I2C SCL frequency 10% less than it should be at 400kHz

[MartyMacGyver]

I've been investigating this for a few days now, in both Arduino and in the IDF, and it's clear that the I2C clock significantly lags its set value. There's also another problem in terms of the duty cycle itself.

I've looked at the ESP32 I2C HAL code and the way the clock is set is pretty straightforward (it's actually over-simplified, as I'll talk about below). The base clock is APB_CLK_FREQ (which should be 80MHz), and the calculation of the total SCL cycle period comes directly from that (which is divided to become scl_high_period + scl_low_period). However, the net result is an actual I2C clock that's (at 400kHz) about 10% slower than it should be.

Below we see the comparison between the I2C clock as seen on a Particle Photon (which uses a typical ARM processor, the STM32F205RG) and the ESP32 Core Board v2. Both are connected to the same type of device (a 128x32 SSD1306-based OLED) with no other I2C loads. Each board is using its USB-derived 3.3V supply (shared grounds on the board and via USB) and each is running the same Arduino code (there's a small difference in how the frequency gets passed to the HAL, as the Photon uses a parameter to Wire.begin() while the ESP32 uses setFrequency() which passes the frequency to the HAL's i2cSetFrequency() method unmodified, and where it is directly used to set the clock periods):

i2c_problem_analog+digital-orig.jpg (305.01 KiB) Viewed 27770 times

The readings were taken first with a 100MHz logic analyzer, and then with a 100MHz scope (leaving the leads connected to both analyzers will otherwise a slight slowing of both clocks due to RC loading). We see the clock speed of the Photon is a solid 400kHz, while the ESP32 is closer to 363kHz (which is about 10% less than it ought to be).

I've crossed checked this by swapping the two displays (originally I was just using one display but it became clear the display wasn't the problem). Clock stretching isn't occurring (nor is this what it'd look like).

Most notably, this I2C display works particularly poorly with the ESP32 - it becomes glitchy within a minute if not seconds, and eventually requires a hard reset. The display has no such problems with the Photon. It's a generic display and isn't specced for either device in particular, it's just supposed to work with I2C up to 400kHz. In fact, I have as much trouble at 100kHz, it just takes a little longer to become evident.

Now, it might be a sensitive device but again, these displays works quite solidly with the Photon. The only glitch I've seen is when SCL was abnormally loaded (which one would expect to be a problem for any I2C device). The fact that it's so erratic with the ESP32 led me to investigate the I2C signal strength, shape, and the code involved. In addition to the frequency being odd, I noticed something else - the duty cycle.

The typical duty cycle for I2C at 100kHz and 400kHz is not normally 1:1 but closer to 1:2. I'm working on making the appropriate updates to the Arduino-ESP32 and ESP-IDF HALs for this, to better match standard I2C devices and properties, but the current respective HALs use the 1:1 duty cycle across the board, which led me to wonder if this is part of the problem.

Preliminary code to correct the duty cycle shows a waveform more in line with expectation (ignore the analog frequencies shown - the respective values are lower due to effect of probing with both DLA and scope):

i2c_problem_analog+digital-mod.jpg (323.49 KiB) Viewed 27770 times

The clock frequency problem is vexing and persistent. I can't tell if it's some internal clock that's incorrectly set or if the lines assigned to I2C on the ESP32 are loaded incorrectly internally (either a hardware problem or a pin config issue). As we see in the mod above, when I loaded the respective devices with both probes the clocks both slowed somewhat. An I2C master will de-rate the clock based on the RC characteristics of the load it's driving, so this isn't unexpected... but it makes me wonder if the ESP32 is overcompensating for some reason in a way the Photon isn't (for a given external load).

I also tried this with the ESP-WROVER-KIT off to the side and saw the same exact issues (the same kinds of glitches and timing discrepancies). I also tested with with some native ESP IDF I2C code - same problem (though not surprisingly, the ESP32 I2C HAL for the IDF and the Arduino are *very* similar when it comes to setting the frequency and other I2C parameters).

tl;dr - The I2C on the ESP32 underclocks significantly versus its set clock speed, and it also has a duty cycle that seems to be out of spec. I'm working to understand both issues, but while I am able to correct the duty cycle and other timing issues relative to the set clock speed in code, the fact that the I2C clock is significantly slower appears to be either a hardware issue or a pin config issue and thus is a different bug of some sort.

Edit: Both analyzers used (Owon SDS7102 scope using the 10x attenuation setting and the Saleae Logic Pro 16) present a load of about 1Mohm at 10pF. With both devices hooked up at once the load should be about 500Kohm and 20pF. While that doesn't appreciably affect the resistive load (the pull-ups are 4.7Kohm on SDA and SCL respectively) it's a potentially significant capacitive load. I did notice the I2C lines between the ESP32 and the display is very sensitive to capacitive effects when it comes to glitchiness. The Photon seems immune to that, which again makes me wonder why the ESP32 appears to be more sensitive to this and whether it's some pin setting or a silicon problem.

[MartyMacGyver]

More info - I'm using GPIO21 for SDA and GPIO22 for SCL, which appear to be the defaults.

Also, while the clock will drop somewhat with increased RC loading, the ESP32 is super-sensitive to it... I have to really work to get the Photon to react to increased capacitance, while the ESP32 readily drops its SCL frequency. Perhaps that's the primary problem here, though why that is, I'm not sure yet.

Edit: Also, if I put an unusually strong pull-up on SCL (1.5Kohm to 3.3V, effectively 1.1K including the 4.7K pullup on the display itself) the clock rises to 380kHz from 363kHz (again, it's set to 400kHz). That's with or without the IO pin's internal pull-up enabled. SDL looks peculiar from a logic analyzer perspective but it works about the same as without (that it, it still gets glitchy).

[ksmith9109]

Excellent diagnostics....Is there any confirmation of the root cause of the problem?

[MartyMacGyver]

I've heard nothing yet. While I do understand that I2C clocking can vary, the fact that it varies this much for what should be a pretty standard I2C configuration is what troubles me, as well as the errors it appears to be causing. It's not pushing 400kHz no matter what I do, even with just a dummy load of pull-up resistors (though I can close the gap somewhat with a dummy load of around 1Kohm or less, that's not normal at all and the glitches persist).

That said, I hope to make more progress on the *relative* timings (such as they are) this week, particularly the duty cycle and other minimum timings that aren't optimal. I'll post links to the Github branches when I do, but if it doesn't help the present problem (and the glitches that appear related to it) then its makin value will be to at least ensure accurate I2C timings per specs.

[Jakobsen]

Hi

I have seen issues when pushing i2c frequency and i see the i2c hardware into random oscillation state that require a hard reset.

I thing it is important to brake down the issues id on the I2C interface - main gold is to get them out the the way one by one.

From what i can tell you are only writing to the OLED display and that should simplify and isolate sw/hw involved.

When writing to a device the ESP is master and drive the by with only exception during the ACK request. And driving bus in I2C terms mean active pull low or release pin to let the pull up resistors on bus charge bus to high state.

To my knowledge the is no compensation involved in such process and the process should be hard driven from the ESP32 I2C master.

Regarding pull up resistor on the bus. The internal pull up in the ESP32 is very week and external bus pull in the order of 3.3k to 8.2k ohm is needed. If external bus pull is lower <2.2k it is important to check that slaves has sufficient drive to pull down bus well below digital low level threshold level.

I2C clock frequency:
Your observation on exact calculation of i2c scl clock frequency should be easy to test out as there is registers for i2c scl high / low period direct in the address map. Here the system base frequency should be known and I guess error seen relates to a diff here.
I have seen what look to be clock stretching on the 9'th bit as seen for the first 3 x 9'th pulses when in 400kbps.


Driver error during I2C write 100 kbps
This is a normal 3 byte write operation but for some reason the driver keeps clocking data and effectively writes random data (0xFE) until my device gets into error state and flags an error on other error pins. In the plot below the i2c clock is 100 kbsp and the clock stretching seems to have a constant period that is a little longer then scl low period.


To be cont...
/jorgen

[MartyMacGyver]

I'll be interested in seeing what actual frequency you measure for SCL set to 400kHz (excluding stretched cycle(s)). If you test into a dummy load, please specify the resistance you're terminating SCL and SDA with when you measure this.

The registers are being set correctly - it's the output that varies and it varies more than I've seen on other similar devices. I'd expect it should be able to drive a full 400kHz into a simple I2C load of some sort, but so far I've yet to see that happen (and I know it can drive a faster too... if I set it to, say, 450kHz, it'll end up measuring around 400kHz... it just doesn't make any sense why it should be off by that much - it's a pretty simple setup, and that's with the default code, flawed as I think it is for duty cycles...)

[Jakobsen]

Hi
I did 800 kbps just to check if how the system scaled. I was ok but with const clock stretch period.
Yes I will measure some exact frequency and report back. /j

[Jakobsen]

Did some i2c scl measurements at 100, 200, 300, 400 and 800 kbps
Looks like (last plots in this post) we have a 190 ns constant unaccounted delay per cycle scl for a 160MHz and 240MHz system clock using the idf i2c driver - so must be related to amba bus frequency. Any way sub the 190ns in the i2c_scl high / low calculation and you will hit spot on.

Espressif will know how may clock cycles and from what clock to make the hardware do the exact data rate and update the drive

Espressif will also need to explain the non constant delays on the ACK bit period that is seen in the following plots




Constant unaccounted i2c_scl delay of ~190ns per period







/j

[MartyMacGyver]

So we're on the same page, can you share the code you used to test this?

As for other timings, I think part of the problem is how the various timings are set in the HAL. That's part of what I'm looking at, and once I create a modified HAL to share I'd like to see how it differs.

[MartyMacGyver]

My work on the Arduino-ESP32 fix is proceeding well. You can find the work in progress at https://github.com/MartyMacGyver/arduin ... 2c-timings

The main changes are in https://github.com/MartyMacGyver/arduin ... -hal-i2c.c (including annotations that track back to the NXP I2C spec (http://www.nxp.com/documents/user_manual/UM10204.pdf) and the recently updated ESP32 tech ref (https://espressif.com/sites/default/fil ... ual_en.pdf).

The open questions:

• SCL lag persists despite the code updates:
  • 100kHz set -> 97kHz actual
  • 400kHz set -> 360kHz actual
  • 600kHz set -> 519kHz actual
  • 800kHz set -> 661kHz actual
• I2C_SDA_SAMPLE_TIME - If SCL is rising, data should already be valid - the tech ref describes this as a delay before sampling SDA, which should be unnecessary. Yet if this value is small or zero, things go wrong. Why?
• I2C_TIME_OUT - what's the basis for the timeout chosen? This differs in the IDF and seems somewhat arbitrary.
• tSU;DAT from the I2C spec - I don't see this specified anywhere in the tech ref, though

The present value of I2C_SDA_HOLD_TIME at 400kHz seems marginal and increasing it without increasing I2C_SDA_SAMPLE_TIME seemed to help but caused other weird things (particularly, data transitions encoding a 1 (e.g. 010) became very very abrupt). The two settings appear to be correlated in a way that leads to this behavior but I haven't fully characterized it yet.

The I2C OLED glitches appear to be slightly less frequent when raising I2C_SDA_HOLD_TIME, but again I2C_SDA_SAMPLE_TIME seems to have an effect as well. I still think this is deeper than the code given the sensitivity of the I2C circuit and the oddly low actual frequencies, but it'd be useful to understand the correlation above better as well.

Edit: I pushed an updated commit (id 3bf78d2) just now. If you just cloned this please re-clone.

Edit2: if you proportionally increase the I2C_SDA_HOLD_TIME and I2C_SDA_SAMPLE_TIME slightly, it appears you get some very repeatable short glitches during SCL low (the pattern seems related to the data being transmitted, but it's not obvious what the specific correlates are). At 400kHz I varied this from the current value (as of this writing) of 25 for both to 28 for both and started seeing these glitches. At 33 the glitches were even more pronounced and at 50 they were quite large (> 250ns, but still fully within the SCL low period). Not sure what to make of that, but depending on the data and device this causes errors.

Edit3: These glitches could be from the ESP32, or some interaction with the I2C devices being used. Hard to tell any more til I get some feedback on the fundamental SCL problem here.