Load code to certain regions of the flash

[Rafael Pires]

In my application I want load some drivers to a region of the flash (specified by a custom partition table), and use those drivers on my app. Later i want be able to build my application and use those drivers without have to flash that section of the flash again. Basically i want to have pre-compiled functions stored on the flash and have those functions available on the IDF to build my own application. How can i load some parts of the code to a specific flash region? And, how can i have those functions available on the composer?

Best Regards,

Rafael Pires

[kolban]

Howdy Rafael,
I don't have any technical answers to your questions but I am interested in the back story. What problem/puzzle/solution does having "preloaded" libraries of function solve/resolve?

My best guess is that you want to modularize an Over The Air (OTA) refresh of an application so that only portions of the application that are likely to be changed need be changed. That is only a guess though and I'd love to hear the design thinking and goal you have in mind.

[ESP_Sprite]

I would also be very interested in the purpose of this... For the 'how', it is possible in multiple ways. The easiest way may be to write your code, link it using a custom linker script (that makes it end up somewhere in the ESP32 address space) and end up with a binary blob with all the functions in it. Then use a script to convert the symbols in it into a linker file (for example, my elf_to_ld.sh hack is still in esp-idf/components/esp32/ld, you could use that.) Then all you would need to do is use the MMU to map the flash segment into the address space at the address specified in your linker script. (Unfortunately, esp-idf doesn't have an easy option to do this at the moment - if needed, please add a github issue for it) and then you can link your application against the linker script containing the symbols.

There are more complex options - something like a semi-dynamic loader comes to mind - bit as far as I know this is the easiest.

[WiFive]

Then all you would need to do is use the MMU to map the flash segment into the address space at the address specified in your linker script.

What about data/bss/IRAM?

[ESP_Sprite]

Ah - fair point, I neglected any static variables. Hmm, that's a hard one, you'd have to modify the esp-idf to steer clear of those regions as well.

[WiFive]

Ah - fair point, I neglected any static variables. Hmm, that's a hard one, you'd have to modify the esp-idf to steer clear of those regions as well.

...and you'd have to modify the bootloader to load those sections too.

[ESP_Sprite]

FWIW, I've been looking at this a little longer. In essence, normal OSses also have this problem because a program in e.g. Linux has to be able to load multiple shared libraries per process, so the shared libraries can't use fixed addresses for their static/global variables. That is solved in one of two ways: either the dynamic linker 'fixes up' the shared library by essentially re-writing the machine code to adjust the variable addresses. The other way is to write position-independent code (PIC). (Assuming 'code in a certain region in flash' is not supposed to be pre-linked to the eventual executable, what you actually want is a sort-of shared object/library in flash.)

Given that we want to execute the code from flash (and not load it into IRAM first), only the second option remains: we don't want to modify and re-flash the shared object every time. The Xtensa compiler seems to support PIC just fine. Unfortunately, the executable generated still needs fixup when we use globals: the addresses to the globals are stored in a few bytes before each function (essentially, they are literals in xtensa-speak) and my guess is that it's up to the dynamic linker to fix these up to point to the actual data segment.

So all in all: I do think you can compile a bunch of code (with -fPIC), eke out the .text region using objcopy or a well-chosen linker script and dump it into flash; then in the ESP32 app memmap the flash pages into IRAM and call the functions you've defined there, optionally using something like a pointer table in the object. Globals and static variables seem impossible without copying the code to RAM, however. It's not impossible to code without these, though: just use the classic 'object-oriented' way of passing a 'this' struct with all globals in it to every function that needs it.

[ESP_igrr]

I think if we can get something similar to ARM's -mno-data-is-text-relative (which makes static data addressable via GOT) into Xtensa target, that would be enough to have dynamic loading with no caveats.
Currently the obvious (but not nice for any non-trivial code) workaround is to *not* use static variables in the loadable module (only global/common). Then you get GOT entries for that data, which can be filled in by the loader.

[ESP_Sprite]

Well, it's going to be hairy at any rate. It's essentially a catch-22: because you don't know where your static variables live, you need a pointer to these in a known location, but because the only known locations can be relative to your code, they have to be in read-only memory. The options I see are:
[*] Go the X86_64 way, that is using a dedicated register for the static variable location. Apart from the fact that this needs a compiler modification, this is pretty hairy because the windowing ABI makes it hard to have a constant register; they rotate every time a subroutine is called.
[*]Go the uboot way and use a global (well, thread-local) variable; maybe use one of the MISC* processor registers for this. Can be done, but needs a compiler change as well.
[*]Only load libraries into IRAM. That way, we can fixup stuff in memory.