Way to reduce the SPI buffer size.

[Viento]

Hello.
In my project, the parser reads the G-code from the SD card through the SPI interface, using a usual SD.h library. The problem is that regardless of the size of the SD card buffer, the SPI reads 4092 bytes every time, puts them in the DMA buffer, and reads the next 4092 only when the buffer is free. Because of this, the time for reading g-commands is extremely uneven. Most are read in 20µs, but each 146th is read in over 4500µs. (The average length of g-command is 28 bytes, so 4092/28=146.)
4500µs is approximately equal to the transmission of 4KB at 10MHz. This is a terribly unacceptable time that kills all the advantages of the speed of the microcontroller.

Is there a way to reduce SPI buffer size using arduino IDE? I tried to re-initialize SPI with this construction:

Code: Select all

#include "driver/spi_master.h"
#define PIN_NUM_MISO 19
#define PIN_NUM_MOSI 23
#define PIN_NUM_CLK 18

static void spi_init() {
    spi_bus_config_t buscfg;
       memset(&buscfg, 0, sizeof(spi_bus_config_t));
       buscfg.miso_io_num=PIN_NUM_MISO;
       buscfg.mosi_io_num=PIN_NUM_MOSI;
       buscfg.sclk_io_num=PIN_NUM_CLK;
       buscfg.quadwp_io_num=-1;
       buscfg.quadhd_io_num=-1;
       buscfg.max_transfer_sz=64; //default to 4092 if 0
    spi_bus_initialize(VSPI_HOST, &buscfg, 1);    //DMA channel 1
}; 

This probably works (message "E (280387) spi: spi_bus_initialize(756): SPI bus already initialized" in console), but in general, the situation does not change. Each 146th g-command is still read in 4500µs. SPI bus is still working through some huge 4KB buffer. And I have totally no idea how to divide it into small chunks of 64-128 bytes.

[ESP_Sprite]

That change, even if it worked, does nothing. The max transfer size only influences the amount of DMA descriptors that are allocated; it does not affect whatever the SD library uses them for. You always get one DMA descriptor, good for 4092 bytes, even if the number you fill in there is lower.

I'm not sure how you'd do that in Arduino, but in ESP-IDF, I'd separate the problem into two FreeRTOS tasks. Have one task read the file into e.g. a ringbuffer that is large enough to buffer a few sectors. Have another task grab the data from the ringbuffer and parse the data. That way, the extra buffering the ringbuffer provides enough buffer for the G-code parser to do its thing, even if the SD-card is out to lunch for a moment.

[Viento]

The max transfer size only influences the amount of DMA descriptors that are allocated; it does not affect whatever the SD library uses them for.

Any chance through modifying SD library? (Although I did not find anything in the library mechanisms that could limit DMA.)

I'm not sure how you'd do that in Arduino, but in ESP-IDF, I'd separate the problem into two FreeRTOS tasks. Have one task read the file into e.g. a ringbuffer that is large enough to buffer a few sectors. Have another task grab the data from the ringbuffer and parse the data.

Well, this is plan B. I tried to make a construction on timer interrupts, and the analyzer really worked many times while reading a 4kb chunk. Are FreeRTOS tasks the same thing? I mean, does the DMA filling process itself have the same priority as the task that called it (to make it a low priority)? Interrupts have the highest priority anyway.

[ESP_Sprite]

Any chance through modifying SD library? (Although I did not find anything in the library mechanisms that could limit DMA.)

I couldn't tel you that; could be that that is possible, could be that there is something intrinsic to the SD-card protocol that does not allow for reads of less than a sector.

Well, this is plan B. I tried to make a construction on timer interrupts, and the analyzer really worked many times while reading a 4kb chunk. Are FreeRTOS tasks the same thing? I mean, does the DMA filling process itself have the same priority as the task that called it (to make it a low priority)? Interrupts have the highest priority anyway.

The idea with a task is that you can assign it a priority yourself (high or low) but that it'll only run when it actually has work to do (as in: there is space in the ringbuffer to fill)

[Viento]

Ok, I did it using RTOS. (Ring buffer not suitable, need double buffer.)

Next problem: RTOS is too slow. Tasks are called no more than 1 time per millisecond. Even with the setting "#define CONFIG_FREERTOS_HZ 10000".
I used vTaskDelay(1) to have 100µs between tasks, but it is still 1000µs.

Any way to make it faster?

[ESP_Sprite]

If you use a RTOS-aware structure (like a ringbuffer, queue, whatever) then you don't need to delay: the push into the buffer will block for you while there's no room available. (Also, I don't understand your 'double-buffer' comment? The amount of elements in a ringbuffer is limited by its size, so you can double, triple, however-many-le buffer the things by simply increasing the size of the ringbuffer. Also, perhaps good to mention that by 'ringbuffer' I specifically mean the esp-idf ringbuffer construct, as that is RTOS-aware. If you use the bytebuffer mode of that, you should be good.)

[Viento]

Well, looks like i made it with timer interrupt and xTaskResumeFromISR(xHandle). Method seems to work stably. Commands are parsed every 100µs, a 4kb buffer is filling in between. The time gap should be enough with a margin.

Also, I don't understand your 'double-buffer' comment? The amount of elements in a ringbuffer is limited by its size, so you can double, triple, however-many-le buffer the things by simply increasing the size of the ringbuffer.

Unfortunately, it is not possible to fill an array with file.read(buf,len) and read characters from it at the same time. This creates errors. Therefore, needs to use two separate 4kb arrays. While one is filling up, the second can be safely worked with. Then they replace each other. This is like the Gast system weapon, where one barrel fires while the other is reloading.

[ESP_Sprite]

Unfortunately, it is not possible to fill an array with file.read(buf,len) and read characters from it at the same time. This creates errors. Therefore, needs to use two separate 4kb arrays. While one is filling up, the second can be safely worked with. Then they replace each other. This is like the Gast system weapon, where one barrel fires while the other is reloading.

Ringbuffers can do that as well, implicitly. Specifically in our implementation, what happens is that you request *a portion* of the ringbuffer to write to (in your case a 4K portion of a ringbuffer that is e.g. 16K). The rest of the ringbuffer can still be read fine, and acts as a, well, buffer against reads that take too long.