Shell Script for copying .nfi to Flash ???

  • Hello !


    As you might know I spent some time to write the multiboot tool for the DM 7025 and for normal DM 7025 users it is really nice in the meantime.


    One of the few remaining things that annoy me is that I don't have a way to integrate the whole software in a freshly flashed image without running scripts, or installing something on top of it.


    On the other hand I've seen that noggie and others created nice pythin scripts for unpacking nfi files directly to CF cards or on the PC.


    Well, multiboot for DM 7025 avoids all this great stuff which took me quite some time to not even understand half of it :winking_face:


    But I mastered already to copy directly into the flash existing images
    on the Cf card by simply mounting the /dev/mtdblock/3 and /2 device which contain the / and /boot as plain jffs2 and then use the normal tar to fill them with a copy of an CF card image - works fine so far and is already included in teh current BETA of multiboot.


    To come back to my problem - if I would add into the multiboot shell script the possibility to unpack the .nfi to the entire FLASH (e.g. the /dev/mtdblock/0 device), when bootet from one of the Cf Images, then I could easily add the required links and maybe do other things like modifying /etc/fstab when mounting it before rebooting as I do already when I copy the CF Images to Flash.


    I tried around with the dd command using the .nfi file as source and various /dev/mtdblock/X as target, but the result is always a corrupt filesystem giving IO errors. I also tried specifying correct skips and seeks with appropiate blocksizes to not destroy my bootloader, but the outcome is never as needed.


    Can somebody give me a hint what the proper comands would be to copy an .nfi file back to Flash ?


    Actually this is the easiest case (no need to temporary unpack and then move to CF card partition or something like that), and it is strange that nobody implemented this yet (or I didn't find it), maybe the WebIF for
    flashing is too good :winking_face:


    But if I would enable multiboot to flash .nfi images directly to flash, then this would be a really great tool, until others implement something better :winking_face:


    Please help !
    thowi

    2 Mal editiert, zuletzt von thowi ()

  • Zitat

    Original von noggie
    I believe tmbinc's unpacking tools come closest to what you want. See Experimental: NFI2CF


    The file at the pointer always downloads with 0 bytes.


    BUT putting nfi file to CF card is NOT what i want, I simply want to put the nfi file where it belongs (the Flash!) but instead of using Dreamup or the WebIF I want to do it from the multiboot shell script when booted from another CF Image (so that flash is unused and you can copy to it without prolems), so that after putting it to the flash I can mount it and add the addon menu link, change fstab,....


    Multiboot doesn't really need to put nfi to Cf card directly, stealing the image from flash with tar commands is OK in my opinion and much easier to handle.


    In this case I overcome the problem of the scripts I found that not all image have the mtd kernel modules, that you need temporary files for putting the jffs2 filesystem on,....


    It is really strange that nobody posted a way to do this, because in my understanding this would be the simpliest case when starting to play with nfi files :winking_face:


    The point is that nobody used a Dreambox which is ALREADY bootet from a CF card Image to unpack an nfi file to Flash, but this is exactly what I'm looking for.


    Because then only the first time when you install multiboot you have to
    install it into the image (and on CF card), when it then copies the images to CF card it always installs itself - on a PC you could call it a boot virus :winking_face: Exactly this I would like to do also for a flash copy, and for this I would need the correct commands to copy the nfi file to the /dev/mtdblock/X devices.


    please help
    thowi

    4 Mal editiert, zuletzt von thowi ()

  • Hello noggie !


    Just to give you another hint.


    I already tried to steal from your scripts before I started playing with dd.


    Here is my cp2flash that I created from your Unpack_image by kicking out all the unneded stuff:


    #! /usr/bin/python
    import os, sys, getopt, struct, time
    from machine_config import *
    from common_routines import *


    #################################################################
    # This is GPL software.
    # Written by noggie and modified by thowi
    #################################################################
    #
    # This script will unpack a .nfi image for the 7025 on the Flash
    #
    extract_nfi(['/dev/null', '/dev/mtdblock/2', '/dev/mtdblock/3'])
    os.system('/bin/sync')


    The above should simply unpack the NFI File to the 2 Flash devices
    when booted from CF card Image and then called with:


    > cp2flash < myimage.nfi


    but it fails in line 71 of the common routines with IO Error [Errno 5] Input/output error


    Any idea what I'm doing wrong ?


    I think the reason is because the Flash mtdevice has a fixed size and you cann't write beyond, but then this would mean that part_size is not properly calculated ?

    3 Mal editiert, zuletzt von thowi ()

  • As you've found out, this won't work. The whole MTD/JFFS2/NAND-flash implementation stops you from using the standard Linux utilities to copy data directly to the flash. The problem is that NAND-flashes contain so-called OOB (out-of-band) data in addition to the normal data blocks. When you do a straight write to one of the device files (like /dev/mtdblock/3), only the data blocks are copied, while the OOB-data is computed and written by the MTD driver "invisibly" to the user. The unpacked .nfi file consists of complete jffs2 images, OOB-data included, so if you try to just dump this to the device, you will find that
    1) it takes more space than it should on the flash (OOB-data is effectively written twice), and
    2) the result is not a correct jffs2 file system, so it's unusable


    OK, you might say, let's just strip off the OOB data also from the .nfi, write the data blocks to the device, and let the MTD device driver recompute the OOB data. Well, this won't work either, since jffs2 is actually using some of the OOB data for filesystem meta data. So you still won't have a valid jffs2 file system.


    There are some programs in the mtd-utils package that should be able to do the job, but I have not used any of them (except flash-eraseall) myself. After a bad experience I had, when some scripts I made to backup the flash caused bad blocks to appear on a beta-tester's machine, I've stayed away from mucking around too much with NAND-flash. It's just too easy to make mistakes that are next to impossible to recover from.


    And, BTW, as a personal opinion, I think you're going about this the wrong way. What you need is a program (running on the dreambox) that can unpack an .nfi file to the CF, and keep away from the flash altogether. The best starting point for such a program, that I know of, is the one that tmbinc wrote. It will need one semi-major change (i.e. unpacking to files instead of doing everything in memory) and some minor changes to become just what you need to unpack a .nfi directly to the flash (CF, that is).

    Einmal editiert, zuletzt von noggie ()

  • the attached tool can write a NFI file directly to flash.


    there are a number of issues on the DM7025 regarding the new kernel, so if you get corrupted OOB information, you have a problem which i don't know how to fix (i made a kernel hack).


    On non-7020, you need to add support for the NFi1-Header (just skip the first 32 bytes, if you really want).


    please be careful, it can overwrite your OOB information. Don't use it unless you know how to restore it!

  • Hello !


    Thanks both of you for your usefull input.


    The reason that noggie pointed out that it is dangerous to fiddle around directly with the flash without OS Protection was exactly the reason why I implemented everything on the Cf card so far, and used only "properly with the tools from DM flashed images" as source for the copies.


    On the other hand in the latest releases of multiboot it worked nice to simply mount the /dev/mtdblock/2&3 devices while booted from CF and copying with tar there existing CF card images back to flash
    (already a big stept forward).


    In this case all the Flash memory handling is done by the OS and the drivers in this case, meaning I should be save not to fuck up anything.


    As I pointed out already the main reason why I wanted to unpack the nfi file is NOT that I want to create my own flash program, it was because I wanted to have a way to have the flash filesystem on hand afterwards and add the link to my multiboot tool afterwards, so that when then booted all functionality would be available for multibooting without the need for further installation (everything needed is on CF card already).


    Maybe all these problems with the NAND memory are also the reason why musicbob doesn't want to port the flash wizzard to the 7025 or hasn't succeeded so far :winking_face:


    On the other hand if the small bootloader from DM can do the job
    and is fully supported NOT to do any of the mentiod dangerous things there must be a way to do the same thing also when booted from
    the CF card !


    So there are 3 dumb questions:


    1) I think these sources of the boot loader are not public, but wouldn't it be possible for the Developers to steal this small code sequence (which cann't be much different to tmbinc's code) to have a clean standalone nfi2flash tool ?


    2) the DM bootlader when doing the flash is running on the Dreambox as binary executable (or on the frontprocessor - I know dumb question ?) - would it be possible to run it also when booted (from CF card) - my dumb idea would be like mounting the /dev/mdtblock/1 device where I think it is located (?), and simply call it also with the nfi file as input - because it has to run standalone there would be no need
    for kernel hacks, etc... ? It is like running a dos programm under unix, but why not :frowning_face:


    3) if I remember right the 7020 had the capability to flash from CF card by putting the nfi file and such a small standalone utility on it, would a similar approach be possible for 7025 ?


    BTW: I'm just a little old shell script programmer (and not even a good one, I think 30-40% of the multiboot.sh code is redundant because of me being too lazy to use subroutines,... and other nice features of sh)


    Therefore I don't have any programming environment for the DM 7025 at hand (well I have a linux PC) or are intersting in implementing one (sorry for posting here in the OE section), so would it be possible to get the C code of tmbinc compiled and put the binary here ?


    I'll do some further testing, and then decide if any of this functionality will ever show up in multiboot or if I'll stay away from it :winking_face:


    Thanks a lot for your feedback and help so far
    thowi

    Einmal editiert, zuletzt von thowi ()

  • Hi,


    1.) the secondstage sources are public, with the minor exception of some 7025 support code which license is *still* unclear (thanks, ati..). This however doesn't touch anything of the dreambox-specific code, so that code is all there.


    2.) it's a normal mips (or powerpc) program, running on the dreambox.


    It's extremely hard making it run under linux, and generally doesn't make soo much sense. If you compare the "mynandwrite.c", and the bootloader's flash routines, they are almost identical, except for all the http stuff.


    3.) yes, sure. There is such a tool, but i need to fix lcd output there..


  • Thanks a lot - so your code already stole from the standard bootloader :winking_face:


    Which makes me feel a little more safe to use it - could you please compile and link it so that I have a binary for testing (hopefully it doesn't need any strange libraries, because I would need that it runs on any image)


    BTW I yesterday played a little bit with the standard bootloader (by loggin in with telnet to get the menu interface after booting with pressed up button for flashing) - it allows for Flash to speficy alternatives for autoexec.bat, which would mean that with a little bit tweaking it could be used for multiboot too.


    What about catching th down button on the front for choosing the image by pressing it 1,2.3 times then starting again from 0 (just display output of another LCD tick to show if it is 1,2,3) and then using autoexec.bat1-3 as default like my multiboot does on the Cf card :winking_face:


    If you then press the up botton again (which originally halted the image) it should save and reboot with the changed autoexec.bat.


    Would be an easy way of enabling multiboot without the need to write a 'real' bootloader with menus and TV output like the Flashwizzard does. (entering with telnet and manually type the new autoexec is already possible, so there is not much missing)

  • Multiboot 3.0 is now done with support for multibooting from Harddisk (even without CF card), so I would like to continue working on nfi files next week(end), could therefore somebody post the bin file of tmbinc's c program ?

    Einmal editiert, zuletzt von thowi ()

  • Zitat

    Originally posted by thowi
    Maybe all these problems with the NAND memory are also the reason why musicbob doesn't want to port the flash wizzard to the 7025 or hasn't succeeded so far :winking_face:


    7025 is OE, just like 7020, so there are no major problems in porting FlashWizard.
    You wont believe it, but the only reason that actually delay the porting of FlashWizard to 7025 is the boot menu porting.
    And, to be honest, there is not a big need of a multiboot solution for the moment, as there are very few firmwares to install :winking_face: I can count 2 actually :grinning_squinting_face:


    The FlashWizard's boot-menu is library-aware and font-aware, and that's the reason which allowed it to work on all developed firmwares around, from tuxbox 1.07 to 7020's OE nowadays.


    FlashWizard last np beta is currently able to unpack and install images on 7025 (and do backup-restores as well) on CF, HDD, USB devices, just like other dreamboxes.


    As soon as I will port the boot-menu to new fb, then FW 7025 will be released.... but I'm actually involved in some other projects for 7025... :winking_face:
    Cu :smiling_face:

    FlashWizard PRO developer

    4 Mal editiert, zuletzt von musicbob ()


  • Thanks musicbob for your input & feedback (hopefully people will now not bomb you now with can we have it requests).


    Actually I didn't develop multiboot as a competition for FlashWizzard, it was more as a kind of workaround until it finally would show up, and then the interest became bigger and bigger and the features more and more.


    And as the name says Multiboot is NOT focused on the Flashing simplification I just wanted a nice and easy way to copy and boot various images without the limitation of the FW which relies on a current
    image in Flash (how many LZA patch questions were posted in the last months if images didn't work) and further PC interaction when copying between images, etc...


    As long as develoment is going in so many directions with differnet speed, enthusiasm and results it is not such a bad habit to use entire partitions which are completely seperate for booting them individually.
    And adding a nice GUI that works as a plugin with further features like backup, settings copy,... just happened over time.


    For me it is also a kind of hobby to learn about dreambox, brush up my rusty shell scripting experiences and see how far a dumb user can
    get :winking_face:


    So we all apreciate your work and Thanks again for your input.


    ciao
    thowi

    2 Mal editiert, zuletzt von thowi ()