Dual Core Synchronization

[chofsass]

I have multiple tasks on the same core accessing the I2C bus using a freeRTOS mutex for synchronization an that is working. But once I try to use the I2C bus from a task on a different core I get a lot of errors.

I am guessing this is a synchronization issue but I'm not sure. I have tried to use a critical sections (portENTER_CRITICAL, portEXIT_CRITICAL) to fix it but that does not seem to work.

[ESP_Sprite]

Okay, what is your question? Also, it may help if you post your source code and are more precise than 'lots of errors'.

[username]

Have you tried using claim_bus() and release_bus() ?

[chofsass]

Okay, what is your question? Also, it may help if you post your source code and are more precise than 'lots of errors'.

Sorry let me be more clear.

So, I am using the I2C bus on one core (CORE_0) across multiple task on the same core. I am using the xSemaphoreCreateMutex() function provided by freeRTOS to create a mutex to synchronize access to the I2C bus across my task on the same core. This works very well I am getting zero errors.

When I attempt to access the I2C bus from a task located on the other core (CORE_1) using the same synchronization strategy I start getting errors at a rate of about 1000 per sec.

I have tried to use a critical sections (portENTER_CRITICAL, portEXIT_CRITICAL) to fix it but that does not seem to work.

Given that I have shown that I can synchronize access to the I2C bus when only accessing it from tasks located on the same core (CORE_0), why does accessing the I2C bus from the other core (CORE_1) cause my error rate to go from 0 to 1000 times per sec?

Could this be a configuration issue?
Do I need to change my synchronization strategy when synchronizing across cores?
Do both cores have access to the I2C bus?
What could be the issue?

Have you tried using claim_bus() and release_bus() ?

I am not familiar with these functions do you have a reference that I can use?

[ESP_Sprite]

Sorry, you're still being as transparent as mud here. Post your code or relevant snippets, instead of talking about vague 'errors' post the relevant output here. We don't have crystal balls, we cannot divine how your code works and use the APIs.

[PeterR]

A MRE would be useful but guess that's why we ask questions on forums (too overworked/lazy to break it down).
I also have 'an unease' about dual processor synchronisation (using IDF 4.1). The odd exception every now and then. I have plenty of applications under my belt and this is something new for me. Maybe something I have not understood. I have clear evidence of Boost signal differences causing exceptions, maybe Boost but Boost.... Too busy, so just program out those scenarios. I should try C11 signals of course.
Would it help to explain/point to test evidence as to how dual core synchronisation has been proven on the ESP?

[chofsass]

Code: Select all

#include "STI2c.h"
#include "SMCommon.h"
#include "Wire.h"

#define TRACKER() printf("[I2C TRACKER] %u\n", __LINE__);

std::map<uint8_t, std::string> I2cErrorNameLookup =
{
    { i2c_err_t::I2C_ERROR_OK,              "I2C_ERROR_OK"            },
    { i2c_err_t::I2C_ERROR_DEV,             "I2C_ERROR_DEV"           },
    { i2c_err_t::I2C_ERROR_ADDR_NACK,       "I2C_ERROR_ADDR_NACK"     },
    { i2c_err_t::I2C_ERROR_DATA_NACK,       "I2C_ERROR_DATA_NACK"     },
    { i2c_err_t::I2C_ERROR_TIMEOUT,         "I2C_ERROR_TIMEOUT"       },
    { i2c_err_t::I2C_ERROR_BUS,             "I2C_ERROR_BUS"           },
    { i2c_err_t::I2C_ERROR_BUSY,            "I2C_ERROR_BUSY"          },
    { i2c_err_t::I2C_ERROR_MEMORY,          "I2C_ERROR_MEMORY"        },
    { i2c_err_t::I2C_ERROR_CONTINUE,        "I2C_ERROR_CONTINUE"      },
    { i2c_err_t::I2C_ERROR_NO_BEGIN,        "I2C_ERROR_NO_BEGIN"      }
};

namespace STI2c
{
    SemaphoreHandle_t GuardedWire::_lock = xSemaphoreCreateMutex();
    std::map<std::string, uint32_t> GuardedWire::_used = {};
    std::map<std::string, uint32_t> GuardedWire::_failures = {};
    std::map<std::string, TickType_t> GuardedWire::_maxTimes = {};
    std::map<std::string, TickType_t> GuardedWire::_minTimes = {};
    TickType_t GuardedWire::_reportStatusTimeout = xTaskGetTickCount() + pdMS_TO_TICKS(60000);
    std::string GuardedWire::_holder = "unknown";
    int GuardedWire::_holderLineNo = 0;
    bool GuardedWire::_initialized = false;

    GuardedWire::GuardedWire(std::string tag, int lineNo):
    _tag(tag),
    _lineNo(lineNo)
    {
        if (_lock == NULL)
        {
            THROW;
        }

        if (xSemaphoreTake(_lock, pdMS_TO_TICKS(2500)) != pdTRUE)
        {
            ERROR_PRINT("[I2C]%s raised exception\n", _tag.c_str());
            ERROR_PRINT("[I2C]%s %d held for to long\n", _holder.c_str(), _holderLineNo);
            THROW;
        }

        _holder = _tag;
        _holderLineNo = _lineNo;

        if (!_initialized)
        {
            _initialized = true;
            Wire.begin(23, 19, 100000);
            Wire.setTimeOut(20);
        }

        Wire.flush();

        if (_maxTimes.find(_tag) == _maxTimes.end())
        {
            _used[_tag] = 0;
            _failures[_tag] = 0;
            _maxTimes[_tag] = 0;
            _minTimes[_tag] = 10000000;
        }

        _start = xTaskGetTickCount();
    }
    
    GuardedWire::~GuardedWire()
    {
        Wire.flush();

        uint32_t durationMs = (xTaskGetTickCount() - _start)/pdMS_TO_TICKS(1);
        _maxTimes[_tag] = std::max(_maxTimes[_tag], durationMs);
        _minTimes[_tag] = std::min(_minTimes[_tag], durationMs);

        if (xTaskGetTickCount() > _reportStatusTimeout)
        {
            _reportStatusTimeout = xTaskGetTickCount() + pdMS_TO_TICKS(60000);

            for (auto & each : _maxTimes)
            {
                std::string id = each.first;
                
                SMCommon::CsvPrint(
                    "i2c_stats.csv",
                    "name, maxTime, minTime, count, fails",
                    "%s, %u, %u, %u, %u",
                    id.c_str(),
                    _maxTimes[id],
                    _minTimes[id],
                    _used[id],
                    _failures[id]
                );
            }
        }

        xSemaphoreGive(_lock);
    }

    uint16_t GuardedWire::GetTimeOut()
    {
        return Wire.getTimeOut();
    }

    void GuardedWire::SetTimeOut(uint16_t timeOutMillis)
    {
        Wire.setTimeOut(timeOutMillis);
    }

    bool GuardedWire::I2CWrite(I2cDetails details)
    {
        uint8_t error;

        for(uint8_t j=0; j<5; j++)
        {
            _used[_tag]++;

            Wire.beginTransmission(details.address);
            Wire.write(details.reg);

            for(uint8_t i=0; i<details.datalen; i++)
            {
                Wire.write(details.data.data[i]);
            }

            error = Wire.endTransmission(true);

            if(!error)
            {
                return true;
            }

            _failures[_tag]++;
            vTaskDelay(pdMS_TO_TICKS(1));
        }

        ERROR_PRINT("[I2C]%s %d write failed: %s\n", _tag.c_str(), _lineNo, SMCommon::safeLookup<uint8_t>(I2cErrorNameLookup, error));
        return false;
    }

    bool GuardedWire::I2CRead(I2cDetails details, I2CData & results)
    {
        uint8_t *data = results.data;

        for (uint8_t i=0; i<5; i++)
        {
            _used[_tag]++;

            Wire.beginTransmission(details.address);
            Wire.write(details.reg);
            Wire.endTransmission(false);

            if (Wire.requestFrom(details.address, details.datalen, true))
            {
                while(Wire.available())
                {
                    *data++ = Wire.read();
                }
                
                return true;
            }

            _failures[_tag]++;
            vTaskDelay(pdMS_TO_TICKS(30));
        }

        ERROR_PRINT("[I2C]%s %d read failed\n", _tag.c_str(), _lineNo);
        return false;
    }
}

[PeterR]

Need the console outputs for clarity
You don't show the code for second thread on the other core.
Also helps to quote library versions.
Main point, not seeing you're semaphore on I2CWrite() etc. Constructor but not methods.

Once we clear up the semaphore issue then maybe you could explain why

Code: Select all

Wire.endTransmission(true);

would return error & especially with respect to I2C errors?

[chofsass]

The class implements scope guard (RAII) pattern so the lock is acquired when the constructor is called and released when the destructor is called.

If you look in the code you can see that I am caching the error in a map

Code: Select all

_failures[_tag]++;

for reporting later.

So, I think I have a pretty good lead on the issue. In my code I call the begin function on the Wire object once (the very first time the constructor is called). This appears to be associated with the problem.

when sharing memory across cores do you have to use the keyword volatile.

[chofsass]

could Arduino's Wire library not be multi core safe for some reason?