ESP32 Forum

Posted: **Fri Jun 08, 2018 4:37 pm**

I'm working on a project which needs to support receiving commands from AWS MQTT in JSON format. Those commands may be as large as 40kB. That seems to be a pretty significant chunk of my available stack space and doesn't leave much room for anything else.

For starting my MQTT task:

Code: Select all

void Mqtt_Start_Task( void )
{
   ESP_LOGI(TAG, "Initializing MQTT task");

   g_mqtt_update_queue = xQueueCreate( 10, sizeof(mqtt_update_cmd_t) );

   BaseType_t res = xTaskCreate(&Mqtt_Task, "Mqtt_Task", 1024 * 50, NULL, 5, NULL);

   if (res != pdPASS)
   {
      ESP_LOGE(TAG, "Error creating Mqtt task, %d", res);
      abort();
   }

   ESP_LOGI(TAG, "Remaining free space   : %u", heap_caps_get_free_size(MALLOC_CAP_INTERNAL));
   ESP_LOGI(TAG, "Largest available block: %u", heap_caps_get_largest_free_block(MALLOC_CAP_INTERNAL));
}

In menuconfig, I've set the MQTT receive buffer to be 40960. This should let me hold the largest expected message.

After the task starts, the following is printed:

I (650) mqtt: Remaining free space : 117300
I (660) mqtt: Largest available block: 62600

Once I receive that 40kB message, I'll need to move it to flash for long term, but not permanent storage. Another task will need to read that data back out of flash for continual processing. I feel dangerously close to being out of RAM. My first thought was to switch to the WROVER module, but after looking into it, it appears that the external RAM cannot be utilized for task stack space. Could I still declare a 40kB variable in external RAM for the task to use? The data is largely a simple look up table. Processing the data does not need to be very efficient, but does need to occur at least once per second.

I feel like this is a dumb question, but this project is my first experience with an embedded RTOS. I'm just not really sure what the use case is for the external PSRAM. If I could create my heap in it and have each task's space come from that, that would be ideal. However, I do see issues with it. Second best seems like the external RAM would be used for storing idle tasks and the entire internal RAM could be used for active tasks. Third best, which I THINK is the case, is that I access the external RAM pretty much like it is a peripheral (though I think FreeRTOS hides the complexity of it and I just basically map a variable to an address somehow).

Posted: **Fri Jun 08, 2018 4:45 pm**

Howdy,
Think of RAM usage as being made up of two distinct types. These are "stack" and "heap". Typically, the stack is used for local variables. For example, in the following C function:

Code: Select all

void myFunc(int a, char* b, double c) {
   char mytext[100];
   ...
}

A call to myFunc will push a, b and c onto its local stack and the variable "mytext" will also be placed on the stack.

The heap is where data allocated at runtime is placed. For example,

uint8_t* pMyBuffer = (uint8_t*)malloc(100000);

would allocate at 100000 bytes from the heap.

In your example/story, consider storing your received data in a heap allocated chunk of storage such as that returned by malloc(). This will (by definition) not use any stack space such as your local FreeRTOS stack. The heap requests can be satisfied and allocated on pSRAM.

Posted: **Fri Jun 08, 2018 5:54 pm**

Putting 40K on the stack is IMO never a good thing to do. You should as Kolban says... use the heap. With the ext spi ram you can create storage on the heap with the following....

heap_caps_malloc(size, MALLOC_CAP_SPIRAM);

John A

Posted: **Fri Jun 08, 2018 6:07 pm**

Great, think I got it. Would like one clarification though. You suggest

In your example/story, consider storing your received data in a heap allocated chunk of storage such as that returned by malloc(). This will (by definition) not use any stack space such as your local FreeRTOS stack. The heap requests can be satisfied and allocated on pSRAM.

It makes perfect sense that I'd use the external RAM for storing/processing the data data after pulling it out of flash.

Are you further suggesting that my MQTT task can also use heap allocated storage for receiving the data? I'd love to do this...

...

Trying to figure stuff out on my own here. menuconfig creates a define named CONFIG_AWS_IOT_MQTT_RX_BUF_LEN. This is used to set the size of the MQTT Rx buffer.

Code: Select all

typedef struct _ClientData {
	uint16_t nextPacketId;

	uint32_t packetTimeoutMs;
	uint32_t commandTimeoutMs;
	uint16_t keepAliveInterval;
	uint32_t currentReconnectWaitInterval;
	uint32_t counterNetworkDisconnected;

	/* The below values are initialized with the
	 * lengths of the TX/RX buffers and never modified
	 * afterwards */
	size_t writeBufSize;
	size_t readBufSize;

	unsigned char writeBuf[AWS_IOT_MQTT_TX_BUF_LEN];
	unsigned char readBuf[AWS_IOT_MQTT_RX_BUF_LEN];

#ifdef _ENABLE_THREAD_SUPPORT_
	bool isBlockOnThreadLockEnabled;
	IoT_Mutex_t state_change_mutex;
	IoT_Mutex_t tls_read_mutex;
	IoT_Mutex_t tls_write_mutex;
#endif

	IoT_Client_Connect_Params options;

	MessageHandlers messageHandlers[AWS_IOT_MQTT_NUM_SUBSCRIBE_HANDLERS];
	iot_disconnect_handler disconnectHandler;

	void *disconnectHandlerData;
} ClientData;

So it looks like I could change the struct so that readBuf is a uint8 pointer instead of an array (unsigned char *readBuf; instead of unsigned char readBuf[AWS_IOT_MQTT_RX_BUF_LEN]

At the start of my MQTT task, I would then have

Code: Select all

AWS_IoT_Client client;

client.clientData.readBufSize = AWS_IOT_MQTT_RX_BUF_LEN;
client.clientData.readBuf = malloc(client.clientData.readBufSize);

and.... this seems to be working!

Posted: **Fri Jun 08, 2018 6:09 pm**

fly135 wrote:Putting 40K on the stack is IMO never a good thing to do. You should as Kolban says... use the heap. With the ext spi ram you can create storage on the heap with the following....

heap_caps_malloc(size, MALLOC_CAP_SPIRAM);

John A

I'm ordering a WROVER kit today instead of my DevKitC. I'll update my malloc as you suggested when it arrives.

ESP32 Forum

I'm not quite sure I understand the usage of PSRAM on the WROVER

I'm not quite sure I understand the usage of PSRAM on the WROVER

Re: I'm not quite sure I understand the usage of PSRAM on the WROVER

Re: I'm not quite sure I understand the usage of PSRAM on the WROVER

Re: I'm not quite sure I understand the usage of PSRAM on the WROVER

Re: I'm not quite sure I understand the usage of PSRAM on the WROVER