Unexpected I2C timeout on ESP32

jarnbjo
Posts: 1
Joined: Wed Mar 22, 2023 7:51 am

Unexpected I2C timeout on ESP32

Postby jarnbjo » Wed Mar 22, 2023 8:57 am

I am experiencing a strange I2C timeout in previously functional software, which used to run on an ESP32-S3 chip, after trying to use the same software on an ESP32 (no -S3) chip. The software is rather simple and connects to a u-blox GNSS chip to read UBX navigation messages. The chip's I2C interface exposes registers numbered 0xfd, 0xfe and 0xff. The registers 0xfd and 0xfe contain a 16 bit integer with the number of bytes available to read from the chips internal buffer and multiple reads from register 0xff gives access to the actual UBX message. The chip features a 'register auto-increment' function, meaning that if you read more than one byte after selecting a register, the register pointer will auto-increment and you will get the data in the following registers. If the register pointer has reached 0xff, it stays at 0xff.

Removing all error checking, the code looks more or less like this:

Code: Select all


// initialize the I2C bus with i2c_param_config and i2c_driver_install,
// then in a loop, use the following code to read individual UBX messages

uint8_t *buffer = ...;
uint8_t reg = 0xfd;

// write 0xfd to the chip (select register 0xfd) and read two bytes
i2c_master_write_read_device(I2C_PORT, ZEDF9P_ADDR, &reg, 1, buffer, 2, I2C_TIMEOUT);

// buffer[0] now contains the value in register 0xfd and buffer[1] the value in register 0xfe,
// the number of available bytes is big endian encoded
uint16_t available = (buffer[0] << 8) | buffer[1];

if (available > 6) { // UBX messages are longer than 6 bytes

    // the register pointer is now at 0xff, so we can read actual data without requesting a specific register
    // first read the 6 byte UBX header:
    i2c_master_read_from_device(I2C_PORT, ZEDF9P_ADDR, buffer, 6, I2C_TIMEOUT);
    
    // buffer[0] and buffer[1] will now contain the UBX prefix 0xb562
    // buffer[2] and buffer[3] will contain the message type (class and id)
    // buffer[4] and buffer[5] the (now little endian encoded) length of the actual message data:
    uint16_t length = buffer[4] | (buffer[5] << 8);
    
    // read the remaining part of the UBX message and two additional bytes with checksum data:
    i2c_master_read_from_device(I2C_PORT, ZEDF9P_ADDR, buffer + 6, length + 2, I2C_TIMEOUT);
    
    // the buffer should now contain one complete UBX message with header, payload data and checksum bytes
    
}
 
As pointed out, this code works perfectly when running on an ESP-S3 chip. After trying the code on an ESP32 chip, the read and write methods started to sproadically return ESP_ERR_TIMEOUT, based on the timing without honouring the requested timeout provided in I2C_TIMEOUT. Even the first call to i2c_master_write_read_device (to read the number of available bytes) could fail with ESP_ERR_TIMEOUT, but a typical flow would look like this:

  • the first call to i2c_master_write_read_device would succeed and indicate that 100 bytes of data are available
  • the second call to i2c_master_read_from_device (reading the 6 bytes UBX header) would fail with ESP_ERR_TIMEOUT within milliseconds, although a much higher timeout was specified with I2C_TIMEOUT
  • reading the number of available bytes from register 0xfd and 0xfe again would show that at least some number of bytes or even all 6 bytes have actually been read although the read method returned ESP_ERR_TIMEOUT
Trying to reduce the I2C bus speed solved the problem, but I had to go down to 20 kbps for the communication to work. Already at 30 kbps, the communication often failed. Trying to run at 100 or even 400 kbps (as I had on the ESP-S3) would almost always fail repeatedly. Looking through the forums, I found indications that the ESP32's internal pull up resistors may be too weak for the I2C bus, so I tried to add external pull up resistors and disabling the internal resistors in the config passed to i2c_param_config, but that did not do any difference.

Considering that I may have a glitch, perhaps even somewhere else in the software, I reimplemented the core logic for I2C communication in a tiny project in Arduino IDE using the Wire library abstraction. The library simplifies the code and leaves less room for own mistakes and I could actually confirm, that when implemented with the Wire library, the communication worked as expected also on the ESP32.

Digging through the implementation of the Wire library for ESP32, the only obviously relevant difference I could find is that in the function i2cInit in esp32-hal-i2c.c the function i2c_set_timeout is called like this:

Code: Select all

i2c_set_timeout((i2c_port_t)i2c_num, I2C_LL_MAX_TIMEOUT); 
For the ESP32, I2C_LL_MAX_TIMEOUT translates through multiple defines to the value 0xFFFFF.

Adding this function call to my software solved the problem there as well. I can now run the I2C bus at 400 kbps (the fastest speed supported by the u-blox chip) without running into the problems with the ESP_ERR_TIMEOUT error.

Just to again clarify: Using i2c_set_timeout is only required when the code runs on the ESP32 chip. When the code runs on the ESP32-S3 chip, it works without using i2c_set_timeout.

Since the documentation is very weak at this point, here are two questions:

What is the difference between the timeout set with i2c_set_timeout and the timeout passed as the last argument to the other read and write functions? In the Wire library, the value passed to i2c_set_timeout (0xFFFFF) seems to translate to about 13ms (it is documented as the number of 80 MHz ticks) and the default value of the timeout passed to the read/write functions is set to 50ms. This does not make obvious sense to me.

Why is the usage of i2c_set_timeout only required on the ESP32 chip and not on the ESP32-S3 chip? Looking through the code examples linked in the Espressif documentation, I find no actual usage of the i2c_set_timeout function and I don't get the impression that it is supposed to be required.

ESP_Sprite
Posts: 9746
Joined: Thu Nov 26, 2015 4:08 am

Re: Unexpected I2C timeout on ESP32

Postby ESP_Sprite » Thu Mar 23, 2023 4:46 am

Glad you managed to resolve it, although it's a shame you had to go through that amount of trouble.
jarnbjo wrote:
Wed Mar 22, 2023 8:57 am
What is the difference between the timeout set with i2c_set_timeout and the timeout passed as the last argument to the other read and write functions? In the Wire library, the value passed to i2c_set_timeout (0xFFFFF) seems to translate to about 13ms (it is documented as the number of 80 MHz ticks) and the default value of the timeout passed to the read/write functions is set to 50ms. This does not make obvious sense to me.
The timeout in the last argument is a timeout for the entire transaction. In theory, this timeout could be hit because other threads are using the I2C bus as well, for instance. It's a software thing. The i2c_set_timeout is a bit-wise timeout: the ESP32 can e.g. release the SCLK line but due to capacitance or another device keeping the line low, the physical line won't go up yet. The ESP32 needs to wait for this to happen, and there's a timeout on this.

Why is the usage of i2c_set_timeout only required on the ESP32 chip and not on the ESP32-S3 chip? Looking through the code examples linked in the Espressif documentation, I find no actual usage of the i2c_set_timeout function and I don't get the impression that it is supposed to be required.
It's a bit of a niche thing... I'm not sure, but I think it has to do with clock stretching; a few I2C devices do this when they need a bit more time to process things; if this is correct, that probably is what your GPS device does as well. I can see that the timeout structure has been changed (more types of timeout) in the S3, so presumably this is handled differently making the call superfluous.

HMesp32
Posts: 1
Joined: Tue Jun 16, 2020 4:51 am

Re: Unexpected I2C timeout on ESP32

Postby HMesp32 » Mon Nov 06, 2023 11:28 pm

How exactly you setup this timeout?
I am using ESP32 with Arduino and I think facing similar issue as I upgraded from 1.0.6 to latest Arduino ESP32 board manager now my I2C is really running slow.

Code: Select all

i2c_set_timeout((i2c_port_t)i2c_num, I2C_LL_MAX_TIMEOUT);
How to setup this in Arduino?

andychess
Posts: 2
Joined: Thu Aug 25, 2022 10:17 am

Re: Unexpected I2C timeout on ESP32

Postby andychess » Tue Jul 02, 2024 6:03 pm

I've been struggling with this issue for the last two days, so thank you very much for sharing your experience:-).

My own setup is a ublox M10S connected to an ESP32-C6 soc and I was wondering if the fault was connected with my hardware design.

Annoyingly, I had just ported the code to the new i2c_master paradigm and, at random intervals, the chip would crash with a message regarding the i2c_ll timeout. Clearly, this is not being handled correctly in the new version, as I was able to roll back to the i2c legacy code (at a cost of 2.5 hours!) and, although the timeout still occurs, it is handled correctly and produces the appropriate esp_err_t response. I would like to post the bug on github, but tbh, having lost two days to this, I think I will have to push on and try to get back on schedule.

It's a huge relief to know you're not alone with such problems, especially as until this eveining, I had found almost nothing on the subject.

Andy

newacecorp
Posts: 1
Joined: Mon Oct 14, 2024 2:58 am

Re: Unexpected I2C timeout on ESP32

Postby newacecorp » Mon Oct 14, 2024 3:07 am

The Ublox chipsets employ I2C clock stretching, where the SCL line is held low for extended periods of time (I've observed in excess of 10ms in some cases). This causes I2C bus timeouts unless the I2C driver timeouts are increased. If you cannot make driver timeout changes, you may consider using the SPI bus instead, provided you can spare the additional I/O pins required.

Who is online

Users browsing this forum: sirOwlBeak and 135 guests