ReconstructApSc - Reconstruct missing or corrupt .ap and .sc files of movies

  • There is a new plugin, ReconstructApSc, which makes it possible to reconstruct missing or corrupt .ap and .sc files of recorded movies. These files are used for precise seeking and winding in the movie, and also for finding the cut positions when using the plugin MovieCut to cut movies. This plugin scans through .ts files, and stores the structural information into the .ap and .sc files.


    Examples of situations when this plugin may be useful include:

    • Movies recorded with Enigma2 from before the spring of 2009 doesn't have .sc files, so rewind/fast forward won't work properly.
    • The previous version of MovieCut did not know about .sc, and therefore produced no such files for the cut movies.
    • If you have cut your movies on a PC, or downloaded movies from the web, you may not get any .ap or .sc files.
    • If a specific movie is impossible to seek correctly in, or just gets black or a frozen picture when trying to fast forward or rewind, then one may suspect that the coresponding .ap and .sc have got corrupt for some reason.
    • After a disk crash it may be possible to rescue the .ts files (because their specific structure) but perhaps not the others.
    • Maybe you have removed the .ap and .sc files because they took up disk space and you didn't know what they were used for.

    Once installed, the plugin shows up in the menu in the movie list, as "Reconstruct AP/SC". When started, it will give you three alternatives:

    • Reconstruct .ap and .sc for the selected movie.
      If you suspect a specific file has got a corrupt .ap or .sc, this is the choice. Reconstruction will be performed in the background, so you can continue using the dreambox. It will tell you when processing is finished.
    • Reconstruct all missing .ap and .sc files in the directory.
      A typical situation for this choice is that you have a directory with many older movies that don't have any .sc files. Note that if several movies in a directory miss their .sc files, processing will take considerable time and disk activity (count some 5 - 10 minutes per Gigabyte of a movie, depending on the hard disk), since every such movie has to be scanned through. If the process should be interupted for some reason, it will however start over again the next time with the file in which it was interrupted.
    • Check the progress of any existing reconstruction process.
      Especially if you do a whole directory, you may wonder how many files it has processed so far. With this choice you can see the output from the reconstruction process so far, to see which file it is currently processing.

    The plugin is available from the Plugins menu (the experimental feed). I hope that it can be of use.


    Best Regards
    Anders Holst

  • Anders,


    First of all: Thanks for the fantastic MovieCut tool. :thumbs_up:


    I also do have .ts files without the corresponing .ap files, so I was happy when I saw this new tool.


    But where exactly can I download it? I can't seem to find the Plugins menu (the experimental feed) :frowning_face:
    Thanks in advance.


    vGnp

  • Zitat

    But where exactly can I download it?

    If you use the "experimental image" (as opposed to the more stable "official release"), then the Plugins menu will automatically give you plugins from the experimental feed. With the Plugins menu I meant:
    Main menu -> Plugins -> Download -> Extensions -> reconstructapsc


    Alternatively you can fetch it directly from http://dreamboxupdate.com/open…/1.5/dm7025/experimental/
    Scroll down to "enigma2-plugin-extensions-reconstructapsc_experimental", select the version of latest date, and install it manually on the box.


    Best Regards
    Anders Holst

  • Alternatively you can fetch it directly from http://dreamboxupdate.com/open…/1.5/dm7025/experimental/
    Scroll down to "enigma2-plugin-extensions-reconstructapsc_experimental", select the version of latest date, and install it manually on the box.

    Anders,


    Thank you very much. I had tried to find it on elitedvb, and Google wasn't really helping also. Again, Thanks.


    [Blockierte Grafik: http://i50.tinypic.com/r20py1.jpg] [Blockierte Grafik: http://i50.tinypic.com/r20py1.jpg] [Blockierte Grafik: http://i50.tinypic.com/r20py1.jpg]


    Best Regards,


    vGnp

  • Hello,
    I'm having problems with this plugin. When I transfer a movie in TS-format from my PC to the DM800 (with[Moderator] Image is in violation of the Boardrules [/Moderator] ) the Movie-Entry displayed by the Mediaplayer contains also a Film-Length (in minutes). This movie can easily be played by the Dreambox.
    Now, when I index this movie with the reconstructapsc-plugin, the Film-Length disapears in the Movie-Entry of the Mediaplayer and the movie can no longer be played. It seems that the play-back function ultimately jumps to the end of the video-file. After a remove of *.ts.ap and *.ts.sc one can play the movie again.
    Does anyone has an idea what the problem could be?
    Thanks, Willi

    Einmal editiert, zuletzt von floh ()

  • I have the same problems as Willihelmi. Sometimes I transcode HD and HQ files from onlinetvrecorder.com (these are MP4/MP3 in an AVI container) to ts. The .ap and .sc files seems to have wrong timecodes so cutedit/moviecut on the box doesn't work (and I have to do it on the PC in the AVI file first - this isn't as convenient as on the box :winking_face: ).
    Anders, if you need a sample file, just send me a "PN".

    DM 800 HD PVR DVB-S2 OoZooN OE1.6-2011-05-09-experimental, Samsung UE40B6000

  • I have tried to look into this, but have not been able to repeat it, mainly because I have no access to a movie of the kind that causes problems. (HelgeBS didn't have any suitable movies for the moment, unfortunately). If anyone can offer me a small piece of a problematic movie, some 20 seconds or so would be sufficient, I could look into it again.


    There were some speculation on another forum that it had to do with a missing or corrupt .meta file, but that seems not to be the case - that should have slightly different symptoms. My guess is that the movie has a format that is not recognized by the current version of ReconstructApSc, and that the .ap file gets too small, maybe empty. My guess, without being able to confirm it, is that removing .ap only, but leaving .sc, will make the movie viewable again.


    Anyway, the behavior is not as designed, so if I can have a look at the format, I could at least make ReconstructApSc recognize it and just skip .ap if it can't produce it correctly.


    Best Regards
    Anders Holst

  • I don't have a .meta file either! That is the problem? I have a movie, but it's pretty big (over 5 GBytes). I can create a torrent and send it to you (my upload speed is 8 Mbps so it should transfer in a decent amount of time)


    And you are right about .ap file being too small. For example:


    I have a .ts movie, size 3.205.252.608 bytes, which have a .ap file of 1.877.568 bytes - this one is OK.


    After processing a 5.274.810.000 .ts file, I get a .ap file of 1.242.944 bytes, which seem to be the problem.

    Einmal editiert, zuletzt von DarthMaul40 ()

  • DartMaul40 wrote:

    Zitat

    I don't have a .meta file either! That is the problem?

    No. A corrupt .meta file could be a problem. A missing .meta should not.


    Anders

  • DartMaul40 wrote:

    No. A corrupt .meta file could be a problem. A missing .meta should not.


    Anders

    If you want, I will cut a few seconds from one of those movies and send it to you. Can you recommend a tool that can cut mpeg2 ts files?

    Einmal editiert, zuletzt von DarthMaul40 ()

  • Zitat

    If you want, I will cut a few seconds from one of those movies and send it to you.

    I tried to send you an email, but maybe it failed.


    Yes please, but if you first email me the full .ap file I can check it to find an interesting section to cut out, and email you an appropriate command to cut it out from the movie.


    Best Regards
    Anders Holst

  • I tried to send you an email, but maybe it failed.


    Yes please, but if you first email me the full .ap file I can check it to find an interesting section to cut out, and email you an appropriate command to cut it out from the movie.


    Best Regards
    Anders Holst

    OK, I'll send you the .ap file first.

  • I have now analyzed two example movies that I got (thanks DarthMaul40 and HelgeBS). There were in total three different problems with them. Below are the technical details for those who care about such things.


    For those who don't, here is the synopsis: Download a new ReconstructApSc with the plugin manager. It should at least make the movies playable. Patches have been sent to DMM for the problems with winding and length calculation.


    Details:


    One of the movies did not play at all with the .ap file constructed by ReconstructApSc, although it could be played without that file. The movie was SD but very loosely packed so it was large as a typical HD. The problem turned out to be that normally the sequence start markers and picture start markers are in the same ts-frame, whereas in this case they were in consecutive ts-frames. ReconstructApSc has up to now stored the position of ts-frames with picture starts, but apparently it has to store it to sequence starts instead to work properly. I have updated ReconstructApSc to reflect this, and it is available for download the normal way.


    But there was another problem too with this movie: After having corrected the .ap-file, it was possible to play it, but not rewind it nor wind faster that x8 forward. For this the .sc-file is used. However, the problem is not with the contents of that file, which is correct, but with the way it is used by enigma2: It jumps to consecutive I-frame picture starts when fast-forwarding or rewinding, but just as above it should jump to sequence starts instead. I have prepared a fix for this and submitted to DMM. It may as usual take some time until it takes effect though.


    Now to the second movie. This was a HD movie (or at least it was in the H264 format) but very highly compressed so it was not larger than a typical SD. It played alright, but the problem was that with the .ap-file from ReconstructApSc the displayed movie length was far too short, and the position gauge jumped back and forth during playback. It turned out that this was because the I-frames was very far apart, up to almost 10 seconds simetimes. Currently enigma2 interprets gaps between I-frames over 5 seconds as a "discontinuity" in the movie, and when an .ap-file exists, it is used to find those gaps and compensate for them in the length calculation and position gauge. Compare a situation when this feature works as intended: A movie with a two hour discontinuity in the middle, which without the .ap-file would seem two hours longer than it is, but with the .ap-file this gap is detected and compensated for. However, if up to 10 seconds can occur normally in the movie, the threshold for when it is considered a discontinuity should be increased to (at least) 10 seconds. I have submitted such a patch too to DMM, and again it may take some time before it is applied.


    Something I have not done is to make ReconstructApSc construct the .meta file as well when it is missing. This might possibly be done some day in the future.


    Best Regards
    Anders Holst

  • Hey aholst!


    since I am using your great tool also in my permanent-timeshift plugin to reconstruct Ap&Sc files when saving a timeshift-record permantly as a movie on HDD, I have one little question/suggestion:


    I am using enigma's Task-Manager to execute your binary which also has a progress-bar. But that progress bar only works if the program outputs a % number to see how many percent of a file was done.


    Do you think it would be possible to output progress in percent when your binary is creating files? On large *.ts files it takes a very long time to create those files, so it would be nice to know how many % of a file are already done. If it's possible and not too much work, it would be nice if you would consider it for a future version of this tool.


    Thanks in advance and great job with that program! :grinning_squinting_face:

    Theorie ist, wenn man alles weiß, aber nichts funktioniert.
    Praxis ist, wenn alles funktioniert aber niemand weiß warum.

  • anders,


    great job! Thank you. :smiling_face:

    DM 8000 HD PVR mit iCVS (OE 1.6)
    DM 7020 HD mit iCVS (OE 1.6)
    DM 7025+ mit iCVS (OE 1.6)
    DMM 7020 HD v2 mit DMM experimental (OE 2.0)