Possible lockup-issue with `i2c_write_blocking` versus `i2c_write_timeout_us` (maybe conflicting with USB serial?)

[ZodiusInfuser]

Hi, I am working on a Pimoroni RP2040 product that involves regular communication with some onboard I2C devices at 400KHz.

During my testing I was encountering frequent lock-ups after a call to i2c_write_blocking had completed. I checked this by setting a pin high before the call and to low afterwards. The pin would go low, suggesting the function had completed, then the lockup would occur before reaching my next line of code.

This was very strange to me, as I had successfully got the product working with a custom build of CircuitPython, which I had written C level drivers for. Checking how they implement I2C, I saw that i2c_write_timeout_us was used, with a 1 second timeout.

Switching to this call in my code the lock-ups went away, without any timeouts actually getting triggered. Checking the pico-sdk, I see that both functions call i2c_write_blocking_internal, with the latter having a callback that results in the below if statement getting executed.

pico-sdk/src/rp2_common/hardware_i2c/i2c.c

Lines 166 to 172 in 6a7db34

	do {
	if (timeout_check) {
	timeout = timeout_check(ts);
	abort |= timeout;
	}
	tight_loop_contents();
	} while (!timeout && !(i2c->hw->raw_intr_stat & I2C_IC_RAW_INTR_STAT_TX_EMPTY_BITS));

Suspecting that the longer loop time the timeout call was having a positive effect, I added a sleep_us(10) as an else condition, and sure enough the lock-ups from i2c_write_blocking_internal go away. Switching to a debug build also made the issue go away, thus ruling out SWD debugging.

Also, sending CTRL-C via the USB terminal would break the board out of the lock-up, though devices like PIO would not resume.

I am completely baffled by this!

For now I can switch my code to using the timeout function, and carry on. I just thought I would raise it in case there is an actual issue that needs resolving.

P.S: It may be inconsequential, but I have observed that my "debug" pin goes low before I2C comms complete, for the i2c_write_blocking code path:

Whereas my "debug" pin goes low after the I2C comms complete, for the i2c_write_timeout_us code path:

[ZodiusInfuser]

Okay, me just switching to i2c_write_timeout_us isn't enough to stop the lock-ups. Seems adding sleep_us(10) into both of those do...while loops works though. Putting a sleep after the loop (or even after the function call) has no effect.

I'm suspecting I2C may be a red herring though, as the lock-ups (without those added sleeps) seem more prevalent when there is more USB serial output, which is related to changes from the I2C devices. Just tested now and I don't get any lock-ups if I turn off the printouts after removing those sleeps.

[peterharperuk]

Does the problem go away if you use a uart instead of usb? Do you have an example that demonstrates the issue?

[DominusFulguris]

I am also seeing frequent lockups on the i2c bus (both instances).

On i2c1, my code is polling four colour sensors, each on their own pair of SDA/SCL pins (as the sensors are fixed with a single i2c address). Disabling the i2c, reconfiguring the bus pins, and reenabling the i2c was often leaving the SDA pin latched low. Adding a sleep_us(100) between bus switches appears to have worked around the issue. Power cycling the colour sensors clears the bus which indicates it was latching the SDA line.

On the i2c0, I am talking to a single MPU6050. In this case again the SDA line is being latched low by the slave.

The logic analyser trace would indicate that under some circumstances the i2c may not be sending the clock cycle required to clock in the slave device ACK, which leaves the slave holding the SDA line low.

Bit-banging a single clock cycle on SDA causes the slave to release the SDA line.

[CoolNamesAllTaken]

Wow! I was encountering I2C lockups, and it took me a full half day to start considering that something might be wrong other than my own hardware or code. Thank you all very much for bringing up this thread, it's been a big help. I'm attempting to work around the issue with the i2c_write_timeout_us technique mentioned by @ZodiusInfuser for now. Hopefully this will be fixed soon.

How does such a fundamental part of the pico-sdk get broken? I'm surprised that there hasn't been a bigger stink about I2C failing in a fatal way like this. Does this bug only impact certain devices under very particular conditions?

In case it helps, I'm using a M24C02 EEPROM at 100kHz, right up next to GPIO 2-3 on the RP2040. I have 2.2kOhm pullup resistors on both SDA and SCL. I have another device (MCP4017 digipot) on the same net, a few millimeters further from the RP2040 than the EEPROM, and haven't had any lockup issues with that device when writing to it over I2C.

[mitchellcairns]

Wow! I was encountering I2C lockups, and it took me a full half day to start considering that something might be wrong other than my own hardware or code. Thank you all very much for bringing up this thread, it's been a big help. I'm attempting to work around the issue with the i2c_write_timeout_us technique mentioned by @ZodiusInfuser for now. Hopefully this will be fixed soon.

How does such a fundamental part of the pico-sdk get broken? I'm surprised that there hasn't been a bigger stink about I2C failing in a fatal way like this. Does this bug only impact certain devices under very particular conditions?

In case it helps, I'm using a M24C02 EEPROM at 100kHz, right up next to GPIO 2-3 on the RP2040. I have 2.2kOhm pullup resistors on both SDA and SCL. I have another device (MCP4017 digipot) on the same net, a few millimeters further from the RP2040 than the EEPROM, and haven't had any lockup issues with that device when writing to it over I2C.

I just updated my project to SDK 2.1.1 and began having these same issues. I implemented a wrapper function so I can use the SDK 1.5.1 version of the read/write I2C functions and the lockup issue went away completely.