Stopping ULP & ULP Lockup

[RalimTEk]

Hi,

I have been running into a strange issue on some of my esp32 units (all rev1).
I use a small ULP task to read two ADC inputs, and also to control some LED indicators.

Sometimes after an esp_reset() call, the ULP keeps running (which in itself might be okay), however when the code starts up and tries to reload the code into the ULP (RTC Slow mem), sometimes the ULP doesn't start running at all even when the timer is started. Also had some cases where the ULP does keep running, but it appears to be running really slow (blinking a light goes at half speed).

The code flow is :

• Move led state by 1 and decode to on/off state

• Take 2 adc measurements

• store adc measurements to ram

• halt

I cannot find any way to turn OFF the timer that runs the ULP, so I cannot stop it before a restart.
Also I have not found any reliable method to recover from a "crashed" ULP / one that is running slow. calling esp_reset repeatedly sometimes unsticks it after a few iterations but sometimes it never unsticks.

What are methods to recover from this situation?
I have tried hacks such as zero'ing out the slow ram before loading the ulp code, but this doesnt seem to improve things.

Power cycling the esp is not an option at this point, so I am open to any software methods to reset the ULP block / unstick this unit.

[boarchuz]

At the end of https://docs.espressif.com/projects/esp ... s/ulp.html :

To disable the timer (effectively preventing the ULP program from running again), clear the RTC_CNTL_ULP_CP_SLP_TIMER_EN bit in the RTC_CNTL_STATE0_REG register. This can be done both from ULP code and from the main program.

Give that a shot. Another option, if your preference was to preserve the ULP running on reset, might be to have the ULP program 'flag' that it's still running. eg. ULP sets global variable to 0 every loop; meanwhile, when main processor boots, this global variable is set to 1, then waits for it to be set back to 0 by ULP if running; if it times out waiting for the flip, assume ULP isn't running->start ULP. (I'd definitely be using the proper way over this hack though.)

[RalimTEk]

At the end of https://docs.espressif.com/projects/esp ... s/ulp.html :

To disable the timer (effectively preventing the ULP program from running again), clear the RTC_CNTL_ULP_CP_SLP_TIMER_EN bit in the RTC_CNTL_STATE0_REG register. This can be done both from ULP code and from the main program.

Give that a shot. Another option, if your preference was to preserve the ULP running on reset, might be to have the ULP program 'flag' that it's still running. eg. ULP sets global variable to 0 every loop; meanwhile, when main processor boots, this global variable is set to 1, then waits for it to be set back to 0 by ULP if running; if it times out waiting for the flip, assume ULP isn't running->start ULP. (I'd definitely be using the proper way over this hack though.)

My problem is that the ULP is still Running but simultaneously not. Disabling and re-enabling the timer does not clear the issue, and even repeatedly doing this with delays in between (100ms), the ULP never starts running again.

I have already been trying this without any improvement to the failure modes. Also forcing RTC domain power down (and then delay and release) also does not unstick this issue.

Edit: I do not want the ULP to keep running over reboots, but it does, and the esp_reset does not appear to stop it.

[colman]

I have the same issue, I want to know how to stop a running ULP.

Colman

[elshnkhll]

I think I will try giving it HALT command as a fist instruction and calling ulp_run() again, when I got a spare time. Let me know if someone have already tried it.

[boarchuz]

I ended up running into this too when I implemented OTA (with the ULP running) and the subsequent esp_reset() left all RTC peripherals (including ULP) in an undefined and possibly unrecoverable state.

I think this is a pretty serious flaw with the soft reset that should at least carry a clear warning. And hopefully all IDF initialisation does not assume registers are in default states on "reset".

After giving up on an elegant fix I decided to add a little check to the bootloader to catch any reset causes that might avoid the RTC domain, and trigger an immediate RTC_WDT to ensure everything is completely reset to defaults.

[dizcza]

After giving up on an elegant fix I decided to add a little check to the bootloader to catch any reset causes that might avoid the RTC domain, and trigger an immediate RTC_WDT to ensure everything is completely reset to defaults.

Could you please share the code or instructions on how to do this? I'm not proficient in tinkering with the bootloader. Thanks.

And I cannot hide my dismay thinking that such a delicate issue has been neither solved nor pinned down in the docs in three years...

[boarchuz]

Could you please share the code or instructions on how to do this? I'm not proficient in tinkering with the bootloader

It will probably be easiest to use the bootloader hooks feature. Take a look at examples/custom_bootloader/bootloader_hooks.

Bootloader RTC reset example:

Code: Select all

#include "soc/rtc.h"
#include "rtc_wdt.h"
#include "hal/wdt_hal.h"
#include "soc/clk_tree_defs.h"

// Change to a short delay if desired
#define BL_RESET_DELAY_MS 0

static void __attribute__((noreturn)) bootloader_rtc_reset(void)
{
    for(;;)  {
        rwdt_ll_write_protect_disable(&RTCCNTL);
        rwdt_ll_feed(&RTCCNTL);
        RTCCNTL.wdt_config0.val = (
            RTC_CNTL_WDT_EN |
            (WDT_STAGE_ACTION_RESET_RTC << RTC_CNTL_WDT_STG0_S) |
            RTC_CNTL_WDT_APPCPU_RESET_EN |
            RTC_CNTL_WDT_PROCPU_RESET_EN |
            (WDT_RESET_SIG_LENGTH_3_2us << RTC_CNTL_WDT_CPU_RESET_LENGTH_S) |
            (WDT_RESET_SIG_LENGTH_3_2us << RTC_CNTL_WDT_SYS_RESET_LENGTH_S)
        );
        RTCCNTL.wdt_config1 = (uint32_t)((uint64_t)BL_RESET_DELAY_MS * SOC_CLK_RC_SLOW_FREQ_APPROX / 1000);
        rwdt_ll_write_protect_enable(&RTCCNTL);
        esp_rom_delay_us((BL_RESET_DELAY_MS + 100) * 1000);
    }
}

You can paste that above your bootloader_before_init and use it in there, eg.

Code: Select all

#include "esp_rom_sys.h"
#include "soc/reset_reasons.h"

void bootloader_before_init(void) {
    switch(esp_rom_get_reset_reason(0))
    {
        case RESET_REASON_CORE_DEEP_SLEEP:
            // Not an RTC reset, but you don't want it catching a deep sleep wakeup here. (Remove if deep sleep will never be used.)
            break;
        case RESET_REASON_CHIP_POWER_ON:
        case RESET_REASON_SYS_BROWN_OUT:
        case RESET_REASON_SYS_RTC_WDT:
            // These are all RTC resets
            break;
        default:
            // For any other reason, stop execution here and force an RTC reset to guarantee everything is reset to default state
            bootloader_rtc_reset();
    }
}

[dizcza]

So I changed the hook.c in the bootloader_hooks example to the following:

Code: Select all

#include "esp_rom_sys.h"
#include "soc/reset_reasons.h"

// Change to a short delay if desired
#define BL_RESET_DELAY_MS 0


/* Function used to tell the linker to include this file
 * with all its symbols.
 */
void bootloader_hooks_include(void){
}


static void __attribute__((noreturn)) bootloader_rtc_reset(void)
{
    for(;;)  {
        rwdt_ll_write_protect_disable(&RTCCNTL);
        rwdt_ll_feed(&RTCCNTL);
        RTCCNTL.wdt_config0.val = (
            RTC_CNTL_WDT_EN |
            (WDT_STAGE_ACTION_RESET_RTC << RTC_CNTL_WDT_STG0_S) |
            RTC_CNTL_WDT_APPCPU_RESET_EN |
            RTC_CNTL_WDT_PROCPU_RESET_EN |
            (WDT_RESET_SIG_LENGTH_3_2us << RTC_CNTL_WDT_CPU_RESET_LENGTH_S) |
            (WDT_RESET_SIG_LENGTH_3_2us << RTC_CNTL_WDT_SYS_RESET_LENGTH_S)
        );
        RTCCNTL.wdt_config1 = (uint32_t)((uint64_t)BL_RESET_DELAY_MS * SOC_CLK_RC_SLOW_FREQ_APPROX / 1000);
        rwdt_ll_write_protect_enable(&RTCCNTL);
        esp_rom_delay_us((BL_RESET_DELAY_MS + 100) * 1000);
    }
}


void bootloader_before_init(void) {
    switch(esp_rom_get_reset_reason(0))
    {
        case RESET_REASON_CORE_DEEP_SLEEP:
            // Not an RTC reset, but you don't want it catching a deep sleep wakeup here. (Remove if deep sleep will never be used.)
            break;
        case RESET_REASON_CHIP_POWER_ON:
        case RESET_REASON_SYS_BROWN_OUT:
        case RESET_REASON_SYS_RTC_WDT:
            // These are all RTC resets
            break;
        default:
            // For any other reason, stop execution here and force an RTC reset to guarantee everything is reset to default state
            bootloader_rtc_reset();
    }
}

But compiling the app gives me:

Code: Select all

/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c: In function 'bootloader_rtc_reset':
/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c:18:9: error: implicit declaration of function 'rwdt_ll_write_protect_disable' [-Werror=implicit-function-declaration]
   18 |         rwdt_ll_write_protect_disable(&RTCCNTL);
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c:18:40: error: 'RTCCNTL' undeclared (first use in this function)
   18 |         rwdt_ll_write_protect_disable(&RTCCNTL);
      |                                        ^~~~~~~
/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c:18:40: note: each undeclared identifier is reported only once for each function it appears in
/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c:19:9: error: implicit declaration of function 'rwdt_ll_feed' [-Werror=implicit-function-declaration]
   19 |         rwdt_ll_feed(&RTCCNTL);
      |         ^~~~~~~~~~~~
/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c:21:13: error: 'RTC_CNTL_WDT_EN' undeclared (first use in this function)
   21 |             RTC_CNTL_WDT_EN |
      |             ^~~~~~~~~~~~~~~
/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c:22:14: error: 'WDT_STAGE_ACTION_RESET_RTC' undeclared (first use in this function)
   22 |             (WDT_STAGE_ACTION_RESET_RTC << RTC_CNTL_WDT_STG0_S) |
      |              ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c:22:44: error: 'RTC_CNTL_WDT_STG0_S' undeclared (first use in this function)
   22 |             (WDT_STAGE_ACTION_RESET_RTC << RTC_CNTL_WDT_STG0_S) |
      |                                            ^~~~~~~~~~~~~~~~~~~
/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c:23:13: error: 'RTC_CNTL_WDT_APPCPU_RESET_EN' undeclared (first use in this function)
   23 |             RTC_CNTL_WDT_APPCPU_RESET_EN |
      |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c:24:13: error: 'RTC_CNTL_WDT_PROCPU_RESET_EN' undeclared (first use in this function)
   24 |             RTC_CNTL_WDT_PROCPU_RESET_EN |
      |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c:25:14: error: 'WDT_RESET_SIG_LENGTH_3_2us' undeclared (first use in this function)
   25 |             (WDT_RESET_SIG_LENGTH_3_2us << RTC_CNTL_WDT_CPU_RESET_LENGTH_S) |
      |              ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c:25:44: error: 'RTC_CNTL_WDT_CPU_RESET_LENGTH_S' undeclared (first use in this function)
   25 |             (WDT_RESET_SIG_LENGTH_3_2us << RTC_CNTL_WDT_CPU_RESET_LENGTH_S) |
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c:26:44: error: 'RTC_CNTL_WDT_SYS_RESET_LENGTH_S' undeclared (first use in this function)
   26 |             (WDT_RESET_SIG_LENGTH_3_2us << RTC_CNTL_WDT_SYS_RESET_LENGTH_S)
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c:28:72: error: 'SOC_CLK_RC_SLOW_FREQ_APPROX' undeclared (first use in this function)
   28 |         RTCCNTL.wdt_config1 = (uint32_t)((uint64_t)BL_RESET_DELAY_MS * SOC_CLK_RC_SLOW_FREQ_APPROX / 1000);
      |                                                                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dizcza/Projects/Embedded/eclipse-workspace/bootloader_hooks/bootloader_components/my_boot_hooks/hooks.c:29:9: error: implicit declaration of function 'rwdt_ll_write_protect_enable' [-Werror=implicit-function-declaration]
   29 |         rwdt_ll_write_protect_enable(&RTCCNTL);

I tried to find the missing headers but then I hit another error: I found "rwdt_ll_feed" in "components/hal/esp32/include/hal/rwdt_ll.h" but I can import neither "hal/rwdt_ll.h" nor "rwdt_ll.h" headers (No such file or directory).

I'm using ESP-IDF v5.0.

[boarchuz]

You're missing a bunch of the #includes from my above comment.
And you will need to add component dependencies to your 'my_boot_hooks' component's CMakeLists.txt:

Code: Select all

idf_component_register(
    SRCS "hooks.c"
    REQUIRES soc hal esp_hw_support
)