• I noticed that when loading a complex epg basically a sum of epg for different satellites, with a db size over 10MB the dreambox (dm7080) loose readiness and has a long loading time at boot. Is it possibile to query the dbase instead of loading it ? It's kind of non-sense (unless there are other reasons unknown to me) to keep a dbase on a hard disk or memory stick and then load everything, instead of just query it from its location.
    Thanks.

  • Keep it at the Default decice insted of moving to a slow one mounted later. And if You dont understand the advantages of a Memory resident database better dont offer advice...

  • is your image up to date? if I remember correctly there were also some improvements regarding speed (quite a few updates ago).

    Gruss
    Dre


    Boxen (im Einsatz): DM920, DM900, DMOne
    Developer Project Merlin - we are OpenSource

  • Thanks to all of you, for your deep interest in my feature request. It would be intersting to know how you guys run this. I used last compiled image and copied a 20 mb db, so I am wondering how with 80MB epg, you do not notice any slow down, not even at boot...quite strange. Then I tested my 7080 and 820 and also friends with others dreambox and with different images (ihad, newnigma, oozoon and even non official..those openthings ) problem is still there. Is it a series of dreamboxes with same problem? could be.
    @gutemine 'memory resident' is not always an advantage, rather is a disadvantage, DB are meant to be queried not to be resident, otherwise you use a data file. As you well know this is always opinable though. Nowdays I don't see any 'slow' remote device, we have very fast ssd-hd, usb3 and triple N wifi, they all offer more then sufficient speed for remote queries, even for a 7 days multi epg.
    May be coders could implement a simple switch 'EPG local or remote' and try out with a REST protocol, it would be something interesting to test. Kind of those app for the cell phones that provide you tv-epg.
    Very best to all

  • The openATV image does not even use an EPG database, so your tests don't seem to have been very thorough :winking_face:


    Btw: My epg.db is located in /etc/enigma2/ and therefore located on the eMMC (as by default). It currently has ~70 MB and my box boots in about 40 seconds :smiling_face: The only thing that slows down with an increasing epg db is the shutdown. But DMM implemented some speedup workarounds for that some time ago.

    so long
    m0rphU

  • Ok, I did a few tests. First I've used the Online recovery and then made a software update. This way I have an up-to-date image without anything that might slow down the box.


    Fresh image, without channel list: ~20 seconds boot time
    With channel list, a litte bit of zapping: ~21 seconds boot time, 12 MB epg.db


    I think what takes the most time during boot are plugins and network mounts.


    EDIT: Ok, I've did another test. I placed the epg.db (72 MB) from my productive image in the basic image that took 20 secs for booting. Now the box boots in ~24 sec.
    So a (according to you) extremely huge epg database slows down the boot by 4 seconds. I don't think thats a problem at all.



    P.S.: My box is connected with DVB-C. So I have less then 500 stations in my whole channel list. My bouquet actually holds around 70 channels and I usually only update EPG for those (through EPGrefresh).
    Of course I am not using any CrossEPG, XMLTV, epgbackup or anything else like that. Maybe they cause your problems but I don't think so as gutemine can't approve the problems :winking_face:

    so long
    m0rphU

    Einmal editiert, zuletzt von m0rphU ()

  • This is in line with my tests when I developed all the various EPGdb Plugins. Every 10MB needs 2 secs more, so even a 100 MB epg.db increases boottime only to around 40-45 secs.


    And yes the shutdown problem is fixed since DMM followed my advice to first write it out temporary to /tmp and then copy the file to the permanent location in one single write.


    BUT people have the glorious idea to switch epg.db location to Harddisk and spin-up time there adds another 10 secs easily even for small epg.db

  • Thank you guys for your tests, this confirm the slowdown every X seconds for X mb of epg. Thanks again to Gutemine, for his hard work.
    Now what is your suggestion to avoid that ? Do we have to live with this or is it something that might be improved ? What are your opinions/suggestions about it ?
    Again my idea is the same of cell phone app that gives me in 2 seconds a full epg (weekly) of more then 80 channel (like the italian sky epg) without having the cell phone storing hundreds of megabytes on its memory.
    Best to all.

  • I still don't get your point.
    First of all, I don't think 4 secs for 70 channels is much worse than 2 secs for 80 channels. So I don't think there is any need to improve anything :face_with_rolling_eyes:


    Then your comparison is a bit unclear. The EPG comes via the DVB transport streams and is then stored in a sqlite database. For both I don't see any possibility for a RESTful interface or any other HTTP transfers. There are no web servers or network connections included at all.
    Furthermore I guess the app you are refering to receives the data from the dreambox and thus proftis from the pre-loading of the epg db to the dreambox's memory?


    Maybe you could elaobarate a bit more on your idea, as I think nobody of the folks in this thread understand your idea. At least I don't understand it at all :confused_face:

    so long
    m0rphU

  • Let's do it this way - you use my EPDdbBackup Plugin and save a copy of the Database on the filesystem.
    Then you re-write the Graphical EPG with sql commands which make it faster then how enigma2 reads it from Memory with his current C Code.


    If you have achived this Goal you may come back and we continue discussion and then probably even Ghost wil listen :smiling_face_with_sunglasses:


    Then maybe I (or DMM) will tell you the trick how easy and even faster you could run your SQL code directly on the Memory resident SQLlite Database.


    After finishing this goal you can continue with a re-write for AutoTimer, so that it does his job with sql running directly on the Database, instead of using good old EPG calls in python.


    And to finish the discussion until THEN from my side:


    I'm NOT willing to trade 10 seconds boot time for waiting 5 seconds more EVERY time I want a reasonable amount of EPG to be shown on the TV screen, so either you can do it faster or we don't have a case.


    PS: Finally your proposed 'solution' is completely missleading - if you just want to boot FAST you can use my EPGdbBackup Plugin to produce an EMPTY epg.db and test how fast this one loads on boot. Then you use 1 line of code in autostart of EPGdbBackup plugin to load whatever EPG database size you have AFTER the box has bootet. If you insist that you still want to have EPG after booting it, is also only 5 sql codelines to delete ALL epg except the start channel (or quickly zap there and fill the empty db with this epg data on shutdown) und do the rest later as described ... so there are much simplier solutions possible - and yes I tried all these already, so I know that they work - but as I already said I can live with 10 secs more ...


    Ciao
    gutemine

    3 Mal editiert, zuletzt von Lost in Translation ()

  • @ morphu
    "The EPG comes via the DVB transport streams"....not true, not all providers provide that. Of course you must be German. try to get Mediaset or Tvsat or Italian sky on dvb stream, same for others providers even non italian, but anyway this is not the point. The example of using an application like the sky epg on ipad or a cell phone is as simple as in 2 seconds you have everything, May be this could be something to consider. That's all.


    @gutemine
    I start from the point that I dont' have to use your plugin, the dreambox should already have this features, from the manufacturer. I am not looking for Ghost attention nor to know any tricks.
    I did not propose any solution, mine was a feature request: have the possibility to read epg externally from dreambox. Not worth ? Would like to ear that from someone here.
    I know you spent a lot of times on epg..... but for instance your modified epgimport does not work on dvb-t channels (at least the one I use) so not all you do is gold.
    With respect and best regards.
    Ciao
    Satrunner
    PS
    Try to be less aggrssive, we are here to make things better, not fight each other (and I never did with you).

  • It's always easy to complain but many fail to proof that it can be done better. And there's a reason for no epg on some channels. It's a proprietary solution by the provider to make sure EPG is only available on their licensed devices.

    Gruss
    Dre


    Boxen (im Einsatz): DM920, DM900, DMOne
    Developer Project Merlin - we are OpenSource

  • Now back on topic: I still don't know, what your real problem is:

    I noticed ... db size over 10MB .... loose readiness and has a long loading time at boot....


    The "long loading time on boot" has been proven to be a 20secs in addition for 100MB of EPG database. And if thats to long for you, then gutemine told you, how to change this to be nearly 0 with things already existing, just have to put it together.
    As "normal" users won't have 100MB of EPG database (you only get it when using external tools to fill the database, I don#t think it is possible to have such a big database just with data from the transport- stream) I disagree with you, that DMM has to take care of that "problem".


    But you did not explain, what "loose readiness" means: Does the box get slow? Do you have spinning gears? What exactly is the problem? And are you sure, it comes from big EPG- Database?



    Now lets look at your other point: I totally disagree with your following statement:

    It's kind of non-sense to keep a dbase on a hard disk or memory stick and then load everything, instead of just query it from its location.


    If I have 1GB of RAM, then OF COURSE I want to keep my database (even if it is 100MB) there and query it from there instead of spinning up the Hard- Disk or accessing a slow USB stick for EVERY SINGLE ZAP (because Now / Next comes from EPG).


    The storage of the epg- db is simply there to survive a boot, as RAM content will NOT survive...
    Please explain, where the "non-sense" is...

  • bschaar already gave you the correct answer that mapping of channel serviceref to the xml based EPG source is not done by the Plugin it has to be done either by (your) hand or you give the doglover a servicereference list and the channel names and then he can include them already as mapped.
    So your DVB-T Channel argument is also broken unless you give enough information to verify.


    Finally please read your original request/post, which performence wise is simply nonsense. That's why I asked you to produce some proof that you can query faster from the epg db file then from Memory before we might even consider to follow your argument.


    Neither myself nor DMM work like a free wishing fountain btw.


    If you think there is a bug in the EPG Plugins that I adapted for DreamOS and the epg.db it uses (and there are many as I'm a confessing poor coder) then there are the kit and support Threads for them where you maybe will get help if you post more then "doesn't work"


    Ciao
    gutemine

  • Dear Gutemine, no need of doglover since his channels work ok. Anyway we preferred to make our own rytec. It does not matter since it's still a webgrab epg.
    So if you download Italian dvb-t channels from epgimport will not work on7080-820. Strange since the same file works on a 7020 with OE2.0
    Now in a collaborative spirit, (we have no secrets or tricks to hide), I will make the example for italian channel, for which I am sure it works (we tested it with DE-epg) it may differs for the dvb-t channel of others countries, but I am sure you will get into the right way.
    So here we go:
    Rytec/webgrab uses HEX, so if in T_service you write DVB_NAMESPACE = EEEE0000 will not work.
    Since this is working in OE2.0 WE were surprised (yes, it's not mine the solution, but from our good staff member).......
    but let's go further:
    Let's say we have a channel (3, 2151, 905, 272, 4008574976, u'2015-10-23 21:28:22') if we look into OE2.2 we find (21, 2151, 905, 272, -286392320, u'2015-10-23 19:35:20'), what? a negative number? yep....
    and then again
    Converting EEEE0000 in dec = 4008574976 ....but we saw it was a negative number!! so now we do need that as a negative number for enigma2
    So let's convert that negative number.... -286392320 to HEX = FFFFFFFFEEEE0000
    At end if in the pid I use this in the conversion all dvb-t channel will work. That has been tested with DE-epg ..... Now you know the trick and I am sure you will make epgimport 2.4 :winking_face:
    Why this is not applicable to dvb-s with 82000 or C00000 we do not know, so may be GHOST can tell us the answer, besides the feature request :smiling_face:
    Ciao
    Satrunner
    PS
    I take the occasion to thank Eidii for the solution

    3 Mal editiert, zuletzt von Satrunner ()

  • please first try to understand the difference between signed and unsigned integers in python and SQLlite... and yes I was lazy, that is why the code contains this (ugly, I know)


    Code
    if self.dvbnamespace > 2147483647:          	
                                        	self.dvbnamespace -= 4294967296


    Second the parser was built to deliver results on the xml sources Rytec delivers. You can put into yours whatever you want (including girlfriends measurements) but then it needs to be adapted to browse them correctly.


    That's why it is OpenSource btw.


    Third, If you don't post the matching serviceref nobody can tell if the EPG entries are right or wrong.


    And I decide on my own if I do new versions - it is OpenSource, you can easily create your own or a better one, so any blame game is useless in this case.


    Ciao
    gutemine