I2S Record to SDCard using FATFs

[jerome]

Hi All
I have to record to a wave file from I2S input coming from an external ADC (es8388 similar to LyraT board).
My code works fine up to stereo recording 16 bits at 44.1 KHz.
With higher resolution (48K or 96K), a lot of glitches appears in the wave file while recording.
I tried using different DMA buffers sizes and count, I also tried to setup SDCard in high speed mode, unfortunately nothing helps.
I'm sure the issue comes from the SDCard performance while writing, since if I disable SD card writing and pass thru I2S ADC to I2S DAC I can see a clean sine wave on the DAC output, also for 48K or 96K sampling rates.

I'm not using ESP-adf with pipelines to record, I'm using my own code as follow :

Code: Select all

// LOOP RECORDING
do
{
	//read data from I2S bus
	esp_err_t res = i2s_read(I2S_NUM, (void*)i2s_read_buff, I2S_DMA_BUFF_LEN_BYTES, &bytes_read, portMAX_DELAY);
	if(res != ESP_OK)
        {
		ESP_LOGE(TAG, "i2s_read error (ESP_FAIL) -> STOP RECORDING");
		break;
	}

	if(bytes_read != I2S_DMA_BUFF_LEN_BYTES)
	{
		ESP_LOGE(TAG, "i2s_read error (bytes_read != I2S_DMA_BUFF_LEN_BYTES)");
	} 

        // write to SD Card
        size_t bytes_written_sd = fwrite(i2s_read_buff, bytes_read, 1, fileRec);
	if(bytes_written_sd != 1)
	{
		ESP_LOGE(TAG, "fwrite error -> STOP RECORDING");
		break;
	}
} while((xEventGroupGetBits(s_status_event_group) & STOP_RECORDING_BIT) == 0)

Any experience on this ? on how to improve performances of fwrite while using FATFs system ?
Also is there something to do with portMAX_DELAY passed in i2s_read() ?
Any help would be greatly appreciated, as I'm currently out of ideas.
Thanks.

[PeterR]

Hi,
Your problem is probably that you are writing from the same task that reads I2C.
You need to decouple the two activities.
fwrite() can block, erase sectors, wear level, go out for a few pints & a curry. Who knows when it will come back to you?
Meanwhile, whilst fwrite() is doing whatever, you will miss I2C data.

The general approach is:

I2CTask -- writes to --> queue --- read from ---> DataLoggerTask

But at high frequencies this also breaks down depending on read frequency and other pre-emptive/interruptive activities.
As it stands you have a non continuous transfer so there may well be gaps between samples especially if you have higher priority tasks and/or interrupts interrupting your reader task.

You should use continuous I2C transfers instead. Not sure how the ESP32 supports this but typically processors use dual DMA buffers or circular DMA buffer with interrupts when half full etc.
You then read from the buffer half just written whilst the hardware DMA continues to update the other buffer half.


So above simplifies to:
I2C DMA -- writes to --> DMA Buffer (this just runs forever)

I2C DMA Interrupt --- signals ---> Disk Logger Task (this just writes the last DMA buffer captured)

Then your only worry is to set DMA buffer sizes appropriate to sample rate (DMA buffer fill period) & the maximum disk write time.
Capture ESP32 tick before & after fwrite() and printf() duration when duration greater than current largest delay for sanity sake i.e. to rule in/out slow disk access when you see glitches after 20hrs soak test.

http://bbs.esp32.com/viewtopic.php?t=8919 seem relevant but is a writer rather than reader.

[jerome]

Thanks for your detailed answer Peter.
I've separated I2S read from Codec and I2S Write to SDCard on two separated ESP cores.
I2S reads continuously the ADC data, store them to a buffer, and once the buffer is full, this core sets a bit in the RTOs event group that triggers writing buffer to the SDcard.

Code: Select all

char *readBuff;
char *writeBuff;

// LOOP RECORDING FROM I2S (Run on core 0)
do
{
        i2s_read(I2S_NUM, (void*) readBuff, I2S_DMA_BUFF_LEN_BYTES, &bytes_read, portMAX_DELAY);
        // copy readBuff to writeBuff
        memcpy(writeBuff, readBuff, I2S_DMA_BUFF_LEN_BYTES);
        // notify the other core it can write to SD card
	xEventGroupSetBits(s_status_event_group, SDCARD_CAN_WRITE_RECORDED_I2S_BIT);
} while((xEventGroupGetBits(s_status_event_group) & STOP_RECORDING_BIT) == 0);

// LOOP RECORDING TO SDCARD (Run on core 1)
do
{
        // wait until there is something to write on the sdcard
	xEventGroupWaitBits(s_status_event_group, SDCARD_CAN_WRITE_RECORDED_I2S_BIT, false, true, portMAX_DELAY);
        // clear flag 
        xEventGroupClearBits(s_status_event_group, SDCARD_CAN_WRITE_RECORDED_I2S_BIT);
        // write buffer to SD card
        fwrite(writeBuff, SDCARD_CAN_WRITE_RECORDED_I2S_BIT, 1, fileRec);
} while((xEventGroupGetBits(s_status_event_group) & STOP_RECORDING_BIT) == 0);

This unfortunately does not help.
i2s_read is already using DMA with multiple buffers, from my point of view, adding more buffer will result in the issue coming later, this will never work as long as the time to write buffer to SDcard is higher than the time to read an I2S buffer.

[fly135]

It's not enough to just create two threads. You need to create a pool of buffers and a queue to pass them from the read thread to the write thread. If the time to write the buffer is longer than the audio period it's writing, then you will hit a brick wall eventually. But you haven't established that with your code yet. Because your task synchronization is only one way, you are potentially copying to the write buffer while writing to the SD card as well.

John A

[jerome]

You are right, I've modified to :

char *readBuff;
char *writeBuff;

Code: Select all

// LOOP RECORDING FROM I2S (Run on core 0)
xEventGroupSetBits(s_status_event_group, SDCARD_RECORDED_I2S_WRITTEN_BIT);
do
{
        i2s_read(I2S_NUM, (void*) readBuff, I2S_DMA_BUFF_LEN_BYTES, &bytes_read, portMAX_DELAY);
        // wait until write buffer is available
        xEventGroupWaitBits(s_status_event_group, SDCARD_RECORDED_I2S_WRITTEN_BIT, false, true, portMAX_DELAY);
        xEventGroupClearBits(s_status_event_group, SDCARD_RECORDED_I2S_WRITTEN_BIT);
        // copy readBuff to writeBuff
        memcpy(writeBuff, readBuff, I2S_DMA_BUFF_LEN_BYTES);
        // notify the other core it can write to SD card
	xEventGroupSetBits(s_status_event_group, SDCARD_CAN_WRITE_RECORDED_I2S_BIT);
} while((xEventGroupGetBits(s_status_event_group) & STOP_RECORDING_BIT) == 0);

// LOOP RECORDING TO SDCARD (Run on core 1)
do
{
        // wait until there is something to write on the sdcard
	xEventGroupWaitBits(s_status_event_group, SDCARD_CAN_WRITE_RECORDED_I2S_BIT, false, true, portMAX_DELAY);
        // clear flag 
        xEventGroupClearBits(s_status_event_group, SDCARD_CAN_WRITE_RECORDED_I2S_BIT);
        // write buffer to SD card
        fwrite(writeBuff, I2S_DMA_BUFF_LEN_BYTES, 1, fileRec);
        // notify other core the writeBuffer can be used
        xEventGroupSetBits(s_status_event_group, SDCARD_RECORDED_I2S_WRITTEN_BIT);
} while((xEventGroupGetBits(s_status_event_group) & STOP_RECORDING_BIT) == 0);

This unfortunately does not solve my issue.

[fly135]

You might need more buffering than provided by DMA.

[PeterR]

Hi,
Scan read your post and you are still:
I2CRead(), WaitForSDCard, Signal, I2CRead()

How long will it take for WaitForSDCard or really Signal + WaitForSDCard?

Like Fly says; you need a queue (a set of buffers). Read up on circular buffers.

So (e.g. & not using the perfectly usable FreeRTOS queues...):
declare Buffer[16]
I2CRead(Buffer[n]), Signal(n), (n = n++&15), I2CRead(Buffer[n])

IOW you loose the WaitForSDCard using the assumption that if you give the SD-CARD enough time it will catch up i.e. that average performance is acceptable but, from time to time, the card may be a bit laggy.

There are still limitations with this approach. Essentially you are adding I2CRead() setup latency when you do not need to.
At 100Kbps that's 10uS per bit or 160uS per 16bits.
I would have thought that I2CRead() (EDIT: setup) would take 20uS or so, so you should be Ok when properly buffered provided no interrupts etc.

Otherwise its circular DMA for you!

PS
You may see that my code saves the memcpy() which also saves some time.

[fly135]

Been googling for info on the ESP32 flash write speed , and not having any luck. Seems like a while back someone posted some tests in a thread here, but can't find it.

John A

[jerome]

I've implemented circular buffers. It helped a lot but there are still some glitches while recording at 24 bits / 96 Khz.
After further analysis, I found the SD card requires about 5ms to write 16KB, and sometimes, it requires up to 1.6sec to write 16KB.
Increasing the number of buffers helps but I'm now using the following and there are still glitches :

-> I2S DMA: 32 x 1024Bytes
-> Circulars buffer : 256 x 16K (limited to 16K each since malloc does not work above 16K)

I'm continuously reading I2S, filling the current circular buffer of 16KB, once the current buffer is full, the second core is triggered to enable writing this buffer on the SDCard, then it uses the next buffer and so on.

I'm using FATFs system file because I want the SDCard that can be read from a computer.
Increasing frequency of SD Card (High speed mode) does not help, so it seems the problem comes from the FATFs that sometimes lags while continuously writing 16KB buffers to SD Card, I'm using standard fwrite() function.

Is there a way to improve this FATFs system or use a different fwrite() that could bypass some checks, sanity or any other task that reduces writing speed ?

Thanks

[ESP_igrr]

sometimes, it requires up to 1.6sec to write 16KB

There shouldn't be anything in FatFS library that would incur such a delay. However some SD cards may take hundreds of milliseconds to do internal wear levelling operations, when asked to write a sector. In other words, writing sectors to SD cards is not deterministic.

If you have a logic analyzer, you can see if the SD interface is active (clock toggling) while you observe this delay. If it is, it means that some operation is in progress. If you then observe the CMD line, you will see whether this is one single command in progress, or if FATFS is sending a continuous stream of commands.