I2C Busy Timeout and wedged I2C Bus

[0xffff]

Hi, We have a custom board w/ ESP32-WROOM module that uses a I2C to read various signals through a MCP23017 GPIO expansion module. A task reads these pins every minute for various control functions. We have noticed, perhaps since upgrading to IDF 3.0.2, that occasionally the I2C gets wedged and we get a stream of "Busy Timeout!" log messages, which, based upon the code in esp32-has-i2c.c, occurs upon write when the i2c status register indicates that the bus is busy:

Code: Select all

    if (i2c->dev->status_reg.bus_busy == 1)
    {
        log_e( "Busy Timeout! Addr: %x", address >> 1 );
        I2C_MUTEX_UNLOCK();
        return I2C_ERROR_BUSY;
    }

In fact, scoping shows that in this state, the clock and data lines are running continuously - it's wedged.

It is quite difficult to reproduce the problem - I have had some success w/ test software that reads these bits very rapidly while other tasks perform mqtt transmissions and modbus reads.

I have put in some bandaid detection software that resets the line when this happens, so we're limping along, but I'm curious if anyone has experience with this, or any advice on what could be causing it.

Thanks

[0xffff]

Can someone from Espressif indicate what condition / register states can cause the clock and data lines to pulse continuously, and how to reset these? turns out my hack was insufficient.

Thanks

[mikemoy]

Post more of you code. I have been using I2C allot, and never had any issues with it.

[0xffff]

Thanks - yes, I also generally seem to not have any problems until I do a bunch of modbus and mqtt activity although I'm not sure why that is - it's possible, I suppose, that one of those tasks is corrupting memory. The i2c code is as follows - two methods called at the top level are to "check the i2c shield" and to read the gpio registers:

Code: Select all

bool McpHandler::check_i2c_shield()
{
	byte error;
	Wire.beginTransmission( MCP_DEVICE_ADDRESS );
	error = Wire.endTransmission();
    ESP_LOGI(__func__, "Checked I2C shield at address 0x%x - error = %d", MCP_DEVICE_ADDRESS,error);
	return ( error == 0 );
}

uint8_t McpHandler::read_gpio_state() {
    // oxff is a bad result!
    uint8_t result = BAD_GPIO_READ;
    bool bad_read = false;
    // ESP_LOGD( __func__, "reading gpio_states" );
    if( xSemaphoreTake( xMcpSemaphore, WAIT_FOR_SEMAPHORE_MS ) ) {

        uint16_t all_bits = mcp.readGPIOAB();
        // The read bits are actually the high 8 bits so we right-shift by 8
        result = all_bits >> this->gpio_read_offset;
        if ( result == BAD_GPIO_READ )
            bad_read = true;

        if ( !bad_read ) {
            consec_errors = 0;
            this->gpio_state = result;
            time(&this->last_gpio_read);
            ESP_LOGI( __func__, "gpio_states %x", result );
        } else {
            consec_errors++;
            ESP_LOGE(__func__,"BAD gpio read (%d)!",consec_errors);
        }
// this was done as a hack to try to reset the registers and get the system out of its infinite clock mode
        if ( consec_errors >= 2 ) reset();
        xSemaphoreGive(xMcpSemaphore);
    } else {
        ESP_LOGE(__func__, "semaphore not grabbed" );
    }
    return result;
}

mcp referes to the Adafruit_MCP23017. The code is:

Code: Select all

/**
 * Reads all 16 pins (port A and B) into a single 16 bits variable.
 */
uint16_t Adafruit_MCP23017::readGPIOAB() {
	uint16_t ba = 0;
	uint8_t a;

	// read the current GPIO output latches
	Wire.beginTransmission(MCP23017_ADDRESS | i2caddr);
	wiresend(MCP23017_GPIOA);
	Wire.endTransmission();

	Wire.requestFrom(MCP23017_ADDRESS | i2caddr, 2);
	a = wirerecv();
	ba = wirerecv();
	ba <<= 8;
	ba |= a;

	return ba;
}

The two methods "check_i2c_shield()" and "read_gpio_state()" are read in a task with a settable cadence, typically every minute. When I want to invoke the error condition, I run it every 300ms and then force a bunch of other activity.

Again, I'm running the latest "official" release version: 3.0.1 which uses the arduino-esp "polling based" code prior to stickbreaker's updates to make the i2c interaction be interrupt-driven.

I'm hoping someone who understands the hardware well can help me understand what conditions will result in a perpetually running i2c clock/data line, and also how to reset it since the Wire.reset() doesn't seem to do it.

[mikemoy]

Sorry mate, I cant help on this one. When I started with ESP32, I started with arduino-esp32. Things went well until I worked with I2C & SPI. Not sure which one caused the problem, but after letting my device run for several hours, it would reset. I then switched to ESP32-IDF and never had a problem like that since.

[Archibald]

There have been many reports of problems with I²C failing.

There's an alternative Arduino-ESP32 'Wire' library here that is supposed to be better:
https://github.com/stickbreaker/arduino ... aries/Wire

[0xffff]

Yes thanks - am aware of that but the version of arduino that includes this new interrupt-driven i2c code requires an unreleased version of the esp-idf, which introduced other issues in our application.