Inter Task Commnication Between Two Cores

[parthbhat13]

Hi,
some insight about the stack i am using
1. PopOs 22.04
2. Vs Code
3. ESP-Idf V4.4
4. ESP32 Wroom-32D 16MB

The Problem:
I am trying to share a structure between two tasks. both are running on individual cpu cores. and the esp32 crashes with different errors.

Details About the tasks:
Task 1 running on cpu 0
Stack size is (1024 * 5)
Priority is 19
This task is intended to talk with a machine which is constantly communicating with the esp32 at every 25mS, i cannot put any longer delays in the task else the communication timing with the machine would not be maintained. with that i have had to even forcefully feed the task watchdog so it does not reset the CPU.

the task watchdog code i am using

Code: [Select all] [Expand/Collapse]

1.  TIMERG0.wdt_wprotect = TIMG_WDT_WKEY_VALUE;
2.  TIMERG0.wdt_feed = 1;
3.  TIMERG0.wdt_wprotect = 0;

GeSHi © Codebox Plus Extension

inside of the task there is a parser which parses the messages coming from the uart and sets the state of the flags accordingly in a state machine, Now i wish to read these flags which is basically a structure, in my main loop which is running on the CPU1

What Did I try?
Plan 1.
initial plan was to use the esp_event_loop library and i did the setting according to the documentation and made an individual task for the event loop to run on the CPU1, The reason why i did not try to run the event loop on the CPU 0 is because anything i do on CPU0 with a lower priority than my uart task, it would not go to attain it.

The problem with plan one:
Everytime when i would call the following function

Code: [Select all] [Expand/Collapse]

1. ESP_ERROR_CHECK(esp_event_post_to(mdbEventLoopHandler,
2.                                                   ESP_MDB_EVENT,
3.                                                   MDB_EVENT_ERROR,
4.                                                   &mdbRuntimeFlags,
5.                                                   sizeof(mdbRuntimeFlags_t),
6.                                                  1));

GeSHi © Codebox Plus Extension

i would get different types of error in my core dump starting with
1. Heap memory error
2. spinLockError
3. portTickEnterCritical error

I did try to add delay to the esp event post but still it did not work.

Plan 2:
i tried to use queue to send the structure.

Code: [Select all] [Expand/Collapse]

1. xQueueSend(mdbSenderQueueHandler, &mdbRuntimeFlags, 1 / portTICK_PERIOD_MS);

GeSHi © Codebox Plus Extension

but even that failed.
with again different set of errors none being constant.
below is the latest error which i have received at the time of writing this post.

Code: Select all

assert failed: spinlock_acquire spinlock.h:122 (result == core_id || result == SPINLOCK_FREE)

Plan 3:
The last plan was to go in the 8-Bit controllers style. where i could make a get function and it would simply return the structure when it would
be called by the main loop. more like the following.

Code: [Select all] [Expand/Collapse]

1. mdbRuntimeFlags_t getMdbFlags(void)
2. {
3.     return mdbRunTimeFlags;
4. }

GeSHi © Codebox Plus Extension

now this actually works, but while it is working, i know this is not the right way to do the things in the RTOS. and i would eventually someday end up with a race condition.

so is there some solution to this? where i can read data from my tight loop running on the CPU0 ?


sorry in advance if my core topic for this post was not right. i would be happy to update the topic if there is any better suited suggestion.

thanks a lot in advance.

[MicroController]

communicating with the esp32 at every 25mS

and your communication task occupies the CPU 100%? There's something wrong with the communication task.

The random errors you see may indicate some kind of memory corruption happening.

[parthbhat13]

Hi,
Thanks for the reply.
the communication task does utilize the cpu, i cant get my head around how can i print the runtime stats although i do have the code for it and when i intend to test it out, the esp32 crashes. i am certain that even with printing the runtime stats i did something wrong.

Code: [Select all] [Expand/Collapse]

1. #include <stdio.h>
2. #include <string.h>
3. #include "esp_console.h"
4. #include "freertos/FreeRTOS.h"
5. #include "freertos/task.h"
6. #include "esp_log.h"
7. #include "ctype.h"
8.  
9. extern char debugStr[];
10. static int showRuntimeStats(int argc, char **argv){
11.     vTaskGetRunTimeStats(debugStr);        
12.     printf("***************************************************\n");
13.     printf("Task          Ticks            Load %%\n");
14.     printf("***************************************************\n");
15.     printf(debugStr);
16.     printf("***************************************************\n");
17.     vTaskList(debugStr);
18.     printf("***************************************************\n");
19.     printf("Task          State    Prio    Stack    Num    Core\n");
20.     printf("***************************************************\n");
21.     printf(debugStr);
22.     printf("***************************************************\n");
23.     heap_caps_print_heap_info(MALLOC_CAP_DEFAULT);              
24.     return 0;
25. }

GeSHi © Codebox Plus Extension

nevertheless, i am still trying hard to fix this, i even tried again to go back to the event loop, and made another task for eventloop running on cpu 0, as well increased the stack size of the IPC. nothing is working all i would get is a spinlock error. more like this.

Code: Select all

Guru Meditation Error: Core  0 panic'ed (StoreProhibited). Exception was unhandled.

Core  0 register dump:
PC      : 0x4008cc31  PS      : 0x00060a33  A0      : 0x8008a616  A1      : 0x3ffc6b60  
0x4008cc31: compare_and_set_native at /home/parthasus/esp/esp-idf/components/esp_hw_support/include/soc/compare_set.h:25
 (inlined by) spinlock_acquire at /home/parthasus/esp/esp-idf/components/esp_hw_support/include/soc/spinlock.h:103
 (inlined by) xPortEnterCriticalTimeout at /home/parthasus/esp/esp-idf/components/freertos/port/xtensa/port.c:301

A2      : 0xe553c04c  A3      : 0xffffffff  A4      : 0x3ffc6bc0  A5      : 0x0000000c  
A6      : 0x00000004  A7      : 0x3ffc69e0  A8      : 0x0000cdcd  A9      : 0x0000cdcd  
A10     : 0x00060a20  A11     : 0x0000abab  A12     : 0xb33fffff  A13     : 0x00060a23  
A14     : 0x00000054  A15     : 0x3ffc6a50  SAR     : 0x00000004  EXCCAUSE: 0x0000001d  
EXCVADDR: 0xe553c04c  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xfffffffc  


Backtrace: 0x4008cc2e:0x3ffc6b60 0x4008a613:0x3ffc6b80 0x400f11e1:0x3ffc6bc0 0x400d732a:0x3ffc6c00 0x4008cac9:0x3ffc6c20
0x4008cc2e: compare_and_set_native at /home/parthasus/esp/esp-idf/components/esp_hw_support/include/soc/compare_set.h:25
 (inlined by) spinlock_acquire at /home/parthasus/esp/esp-idf/components/esp_hw_support/include/soc/spinlock.h:103
 (inlined by) xPortEnterCriticalTimeout at /home/parthasus/esp/esp-idf/components/freertos/port/xtensa/port.c:301

0x4008a613: vPortEnterCritical at /home/parthasus/esp/esp-idf/components/freertos/port/xtensa/include/freertos/portmacro.h:578
 (inlined by) xQueueReceive at /home/parthasus/esp/esp-idf/components/freertos/queue.c:1400

0x400f11e1: esp_event_loop_run at /home/parthasus/esp/esp-idf/components/esp_event/esp_event.c:565 (discriminator 15)

0x400d732a: runEventLooper at /home/parthasus/ESP32/esp32Projects/localPopProjects/RAiMECH_Clients/PaymentDevices/CDV6_Files/CD_RaimechV6/components/mdbHandler/mdbHandler.c:84 (discriminator 13)

0x4008cac9: vPortTaskWrapper at /home/parthasus/esp/esp-idf/components/freertos/port/xtensa/port.c:142

now, eventually what i am planning to do is, move my uart code into ISR. the only confusion i have is to set the tx timeout threshold which i will check, as well if i can do a buffer read and checking of the variables the message which i have received if it would be handled in the interrupt only.

and here the uart task and why it is running so fast is because i am running MDB, MultiDropBus, for communication with a vending machine. everything works fine when the task is running and doing its own job, only when i wish to read the flags from the task ,things go horribly wrong and there is never ever one single constant error which i can work on and try to fix.

any more help from my above explanation would be really very helpful thanks in advance.

[MicroController]

As to the multiple issues your code has, from what you described so far I can only advise two things:
1) Use blocking functions instead of polling/busy-waiting, e.g. via the IDF's UART driver, and
2) trying to display the system's runtime stats won't help in resolving the more fundamental existing problems.

[MicroController]

And maybe #3: Check if you may be overflowing debugStr[] somewhere (or any other array for that matter), which may be the cause for memory corruption and random crashes.

[parthbhat13]

Hi,
thanks for your help, i had to complete my code sooner so here is what i ended up doing. trust me its not at all the best practice but so far does the work.
anyone who would face the similar issue here is my take for the solution.

so what i did is, made a copy of my structure called as

Code: Select all

previousStruct

, as well there is already a main struct where the data is being updated, all i did was a simple comparison and then using the mutex i would send the data back to the main loop.

below are the code snippets.

theres a function which basically compares the two structures thats all and returns a true or false. using that

Code: [Select all] [Expand/Collapse]

1. if(compareStruct(prevMdbRuntimeFlags, mdbRuntimeFlags))
2.         {
3.             ESP_LOGW(TAG, "There is change");
4.             prevMdbRuntimeFlags = mdbRuntimeFlags;
5.             if(xSemaphoreTake(semaphoreMdbGive, (TickType_t) 10))
6.             {
7.                 mdbRuntTimeSendFlags = prevMdbRuntimeFlags;
8.                 xSemaphoreGive(semaphoreMdbGive);
9.             }
10.         }

GeSHi © Codebox Plus Extension

and as for to get this data from the other loop this is what i did

Code: [Select all] [Expand/Collapse]

1. bool getMdbFlagData(mdbRuntimeFlags_t *mdbRuntimeFlags)
2. {
3.     if(xSemaphoreTake(semaphoreMdbGive, (TickType_t) 10))
4.     {
5.         memcpy(mdbRuntimeFlags, &mdbRuntTimeSendFlags, sizeof(mdbRuntimeFlags_t));
6.         xSemaphoreGive(semaphoreMdbGive);
7.         return true;
8.     }
9.  
10.     return false;
11. }

GeSHi © Codebox Plus Extension

now i really know this is ideally not the best way, and personally even i am not liking it. but so far it is doing the job for me. lets see how well it works.
Thanks a lot @MicroController for your time and effort. if you can think of any more easier and better ways id be glad.

[djixon]

Try this example from Xtensa ISA architectrure documentation. It is more related in explanation in difference in between L32I and L32AI instructions and S32I and S32AI instructions. But C compiler is smart enough to recognize volatile variables and use proper instruction form. And not only that, it also have to generate such an assembly code which will prevent reordering of reading/writting instructions in such cases where different corres share the same data.

4.3.12.3 Inter-Processor Communication with the L32AI and S32RI Instructions
L32AI and S32RI are 32-bit load and store instructions with acquire and release semantics. These instructions are useful for controlling the ordering of memory references
in multiprocessor systems, where different memory locations may be used for synchronization and data, so that precise ordering between synchronization references must be
maintained. Other load and store instructions may be executed by processor implementations in any order that produces the same uniprocessor result.
The MEMW instruction is somewhat similar in that it enforces load and store ordering, but
is less selective. MEMW is intended for implementing C’s volatile attribute, and not for
high performance synchronization between processors.
L32AI is used to load a synchronization variable. This load will be performed before any
subsequent load, store, acquire, or release is begun. This ensures that subsequent
loads and stores do not see or modify data that is protected by the synchronization variable.
S32RI is used to store to a synchronization variable. This store will not begin until all
previous loads, stores, acquires, or releases are performed. This ensures that any loads
of the synchronization variable that see the new value will also find all protected data
available as well.
Consider the following example:

Code: Select all

volatile uint incount = 0;
volatile uint outcount = 0;
const uint bsize = 8;
data_t buffer[bsize];
void producer (uint n)
{
 for (uint i = 0; i < n; i += 1) {
  data_t d = newdata(); // produce next datum
  while (outcount == i - bsize); // wait for room
  buffer[i % bsize] = d; // put data in buffer
  incount = i+1; // signal data is ready
  }
}
void consumer (uint n)
{
 for (uint i = 0; i < n; i += 1) {
  while (incount == i); // wait for data
  data_t d = buffer[i % bsize]; // read next datum
  outcount = i+1; // signal data read
  usedata (d); // use datum
 }
}

Here, incount and outcount are synchronization variables, and buffer is a shared
data variable. producer’s writes to incount and consumer’s writes to outcount
must use S32RI and producer’s reads of outcount and consumer’s reads of
incount must use L32AI. If producer’s write to incount were done with a simple
S32I, the processor or memory system might reorder the write to buffer after the write
to incount, thereby allowing consumer to see the wrong data. Similarly, if
consumer’s read of incount were done with a simple L32I, the processor or memory
system might reorder the read to buffer before the read of incount, also causing
consumer to see the wrong data.

Hope that helps.

[ESP_Sprite]

Try this example from Xtensa ISA architectrure documentation. It is more related in explanation in difference in between L32I and L32AI instructions and S32I and S32AI instructions.

I'm 99% sure the ESP32 Xtensa cores don't support release/acquire instructions.

[MicroController]

I'm 99% sure the ESP32 Xtensa cores don't support release/acquire instructions.

Hmm. Apparently they do...
(Not that gcc would care, though.)

[djixon]

@MicroController. Yes, those instructions are supported, but gcc doesn't use them unfortunatelly. However it properly takes care on read/write ordering when volatile variables are involved. Do you see those two memw one after another? Well, it is not so efficient as using ai versions of load/store instructions but it seems that it does the job when atomicity is required.