GPIO interrupts lost (hardware race condition)

[ybuyankin]

Hi all,

I've found some serios problem with the GPIO interrupts. My hardware is generating 3 independent IRQs (about 200 per second each) on GPIO 34,36 & 39, configured as rising edge. Quite rarely (in a couple of hours or so) one of the IRQ signals is not getting latched in the GPIO IRQ status register and therefore does not generate the interrupt so this channel gets stuck.

Code uses the gpio_intr_service but that does not really matter. I've used the logic analyzer and a sync signal (some LED pin) generated in software to track this condition. I've found that it happens when new interrupt comes exactly (within ~100 ns) after I trigger the LED right before exiting the app IRQ handler, and next thing will be writing the corresponding GPIO bit to GPIO.out1_w1tc to clear the IRQ. Check the gpio_intr_service function in the esp-idf/components/driver/gpio.c:

Code: Select all

    do {
        if (gpio_num < 32) {
            if (gpio_intr_status & BIT(gpio_num)) { //gpio0-gpio31
                if (gpio_isr_func[gpio_num].fn != NULL) {
                    gpio_isr_func[gpio_num].fn(gpio_isr_func[gpio_num].args);
                }
                GPIO.status_w1tc = BIT(gpio_num);
            }
        } else {
            if (gpio_intr_status_h & BIT(gpio_num - 32)) {
                if (gpio_isr_func[gpio_num].fn != NULL) {
                    gpio_isr_func[gpio_num].fn(gpio_isr_func[gpio_num].args); // my handler is called here
                }
                GPIO.status1_w1tc.intr_st = BIT(gpio_num - 32); // that's the problematic write
            }
        }
    } while (++gpio_num < GPIO_PIN_COUNT);

That most likely means that writing into the GPIO.out1_w1tc to clear the previous IRQ causes race condition in hardware with another IRQ coming in at the same moment thus preventing the latter from latching in. This w1tc register definitely helps with software issues of the same kind but the hardware side of this thing is unfortunately still vulnerable.

It took me more than a week to actually figure this thing out (as stuff like this is the last thing you could really expect, one looks for signal integrity issues, own code bugs etc). There could be some workarounds depending on the actual hardware which uses the interrupts but this should be fixed in the silicon I guess.

[WiFive]

There is a new version of that function in master, it might help. Otherwise if your interrupts are going to be that close together you may want to use rmt to sample the line at 12.5ns and you can count the number of edges.

[ybuyankin]

I've checked the master, but there was no related changes for this function, it was just optimized. It's a bit tricky to compensate for hardware bug in software, but I'll probably do that by comparing actual GPIO state before and after clearing the IRQ and rising an IRQ bit on difference.

For your second suggestion, actually interrupts are not that frequent at ~100-200 Hz, and I don't need to count or somehow measure them, but to fetch data from external source which generates the interrupt on data arrival. The problem arises when interrupt signal comes exactly at the same time as another channel interrupt request gets cleared. So it could happen to absolutely anyone having several independent GPIO IRQs regardless of interrupts rate, just on coincidence. That's really dangerous.

[WiFive]

I've checked the master, but there was no related changes for this function, it was just optimized.

Well the w1tc is only written once now rather than once per set bit.

With edge interrupt type if the edge and the clear happen on the same clock cycle you could be correct. Using level interrupt could be a workaround.

[costaud]

Does this only happen on GPIO34/36/39 ?

[ESP_houwenxiang]

Hi, ybuyankin,

There is a similar issue on github https://github.com/espressif/esp-idf/issues/3132, please refer to it and take a test. at the same time, we will try to reproduce this issue. Can you provide the code how you configure the GPIOs ?

thanks !!

[ESP_houwenxiang]

Hi,
Can you share your progress?

thanks !!

[ybuyankin]

First, I have managed to fix the problem. Thanks for your suggestions.

My original code used the basic gpio_install_isr_service / gpio_isr_handler_add combo. IRQs were set up in loop like

Code: Select all

gpio_set_direction(GPIO_Pin, GPIO_MODE_INPUT);    
gpio_set_intr_type(GPIO_Pin, GPIO_PIN_INTR_POSEDGE);
gpio_isr_handler_add(GPIO_Pin, my_gpio_isr_handler, (void*) ch);

I've got the latest gpio.c code modified in a way that could check the 0->1 transition and call the handler if it was not locked in the IRQ register. Also I I'm only using higher pins:

Code: Select all

static void IRAM_ATTR gpio_intr_service(void* arg)
{

    //read status to get interrupt status for GPIO0-31 - don't have any yet, need to add workarond as below if would have
/*    const uint32_t gpio_intr_status = GPIO.status;
    if (gpio_intr_status) {
        gpio_isr_loop(gpio_intr_status, 0);
        GPIO.status_w1tc = gpio_intr_status;
    }*/

    //read status1 to get interrupt status for GPIO32-39
    uint32_t gpio_intr_status_h = GPIO.status1.intr_st;
    if (gpio_intr_status_h) {
        gpio_isr_loop(gpio_intr_status_h, 32);
        uint32_t gpio_state_before = GPIO.in1.val;
        GPIO.status1_w1tc.intr_st = gpio_intr_status_h; // could case IRQ loss
        uint32_t gpio_state_after = GPIO.in1.val;
        uint32_t gpio_state = (gpio_state_before ^ gpio_state_after) & gpio_state_after; // anything new has arrived?
        gpio_intr_status_h = GPIO.status1.intr_st; //read the new IRQ state
        if(!(gpio_state & gpio_intr_status_h)) // was IRQ lost during status clear?
        {
            gpio_isr_loop(gpio_state, 32); // it's tempting to just set IRQ bit and leave but it could cause exactly the same problem again! so just call the handler function
        }
    }
}

I have to add that it does not actually work exactly as expected as I'm getting these software interrupts (I've also added some debug code to count them) much more often than the issue was happening before fix. Maybe because of the propagation delay in the IRQ pipeline or whatever. Anyway it does the job. I'll probably look deeper into that later.

[ESP_houwenxiang]

Hi, ybuyankin

Are you experiencing this problem in this situation and are you using dual core or single core?

[ESP_houwenxiang]

Hi, ybuyankin

We can use level type interrupt to walk around this issue.

Code: Select all

const uint64_t INPUT_PIIN_MASK = (1ULL<<GPIO_NUM_34) | (1ULL<<GPIO_NUM_36) | (1ULL<<GPIO_NUM_39);

typedef struct io_isr_param_ {
    uint32_t io_tagle;
    uint32_t io_number;
    uint32_t isr_cnt;
} io_isr_param_t;

uint32_t isr_mode[2] = {GPIO_INTR_HIGH_LEVEL, GPIO_INTR_LOW_LEVEL};

io_isr_param_t io34_param = {
    .io_tagle = 0x1,
    .io_number = 34,
};

io_isr_param_t io36_param = {
    .io_tagle = 0x1,
    .io_number = 36,
};

io_isr_param_t io39_param = {
    .io_tagle = 0x1,
    .io_number = 39,
};

IRAM_ATTR void gpio_isr(void *param)
{
    io_isr_param_t *io_param = (io_isr_param_t *)param;
    if(io_param->io_tagle) {
        //do some thing here;
    }
    gpio_set_intr_type(io_param->io_number, isr_mode[io_param->io_tagle]);
    io_param->io_tagle ^= 0x1;
}

void io_init()
{
    gpio_config_t io_conf = {
        .intr_type = GPIO_PIN_INTR_DISABLE,
        .mode = GPIO_MODE_INPUT,
        .pin_bit_mask = INPUT_PIIN_MASK,
        .pull_down_en = 0,
        .pull_up_en = 1,
    };
    //configure GPIO with the given settings
    gpio_config(&io_conf);
    if(gpio_install_isr_service( 0 ) != ESP_OK) {
        printf("ISR install fail\n");
        return;
    }
    //hook isr handler for specific gpio pin
    gpio_isr_handler_add(GPIO_NUM_34, gpio_isr, (void*) &io34_param);
    gpio_isr_handler_add(GPIO_NUM_36, gpio_isr, (void*) &io36_param);
    gpio_isr_handler_add(GPIO_NUM_39, gpio_isr, (void*) &io39_param);
    gpio_set_intr_type(GPIO_NUM_34, GPIO_INTR_HIGH_LEVEL);
    gpio_intr_enable(GPIO_NUM_34);
    gpio_set_intr_type(GPIO_NUM_36, GPIO_INTR_HIGH_LEVEL);
    gpio_intr_enable(GPIO_NUM_36);
    gpio_set_intr_type(GPIO_NUM_39, GPIO_INTR_HIGH_LEVEL);
    gpio_intr_enable(GPIO_NUM_39);
}

thanks !!