Problems with promiscuous mode (capture network traffic)

[Staubgeborener]

I use the promiscuous mode of the esp32 to capture network traffic around me.
But i don't get all of the packets. First of all here is my script: https://gist.github.com/Staubgeborener/ ... 336029668b

Everytime the esp32 receive a packet the callback function will be triggered and i check if the mac address in the packets match with the mac address in the filter (line 20), but for now i leave this column empty (see gist). So there is no filter and i should receive everything. All of the packets will be saved on a sd card in a wireshark friendly hexdump. But when turning the esp32 on, i'll mostly get only beacon frames. Maybe some (encrypted) data frames, but mostly (98%) beacons.
So i wrote a program which uses scapy. I send randomly some (thousand) packets (arp, icmp, tcp and beacon frames) in the air. And those packets definitly exist - for example, i send an arp to my second computer and opened wireshark. Wireshark recognized this arp (who is xxx, tell zzz).
But after sending around 30.000 packets, the esp32 only received 30 beacon frames. Thats all. No arp, icmp or tcp. And there we're around ~7.000 beacon frames in the air for around 2 hours.

Most of the (beacon) frames i receive, are those from my network (respectively also my neighbors), but only some from my scapy program. Needless to say: no 'real data' packet, even without filtering.

[ESP_igrr]

It may not explain all the issues you are having, but i suggest measuring the time it takes your promiscuous callback to complete, with all the stuff it does (formatted IO, SD card access, filesystem, 20ms delay?!). Chances are, it takes so long to run that the queue of received frames gets overflown, and some frames are lost.

[Staubgeborener]

The 20ms delay is for feeding the watchdog to avoid error messages. I cut it out now and deactivated the watchdog, otherwise i would get after some seconds/minutes a warning message from it, which will be also saved on sd card.

Meassuring the time was in any case the right point.
Without printf + sd card writing: 0ms
Only printf + without sd card writing: 1 - 30ms
Writing sd card (no difference if using printf or not, because each line will be formated anyway): 10 - 30ms

So the problem is obviously, that i format each hex for the hexdump in a wireshark friendly format. My buffer gets an overflow and discards a lot of packets.
( >> Print Offset, print hex, look if end of packet is reached, look if line has 8 hex, print next hex, and so on)

I have to save everything immediately to the sd card, as soon as the paket arrive.
Do you know a good approach to do so? (For example print/save the whole paket at once instead of going threw an array till i reach sig_len, but how)

[Staubgeborener]

Ok, now it gets weird. I edited these lines in my config

CONFIG_ESP32_WIFI_STATIC_RX_BUFFER_NUM=25
CONFIG_ESP32_WIFI_DYNAMIC_RX_BUFFER_NUM=0

because i thought, this could help to increase my buffer (does it?). As soon as i did this and flashed my esp, my computer shuts down and had the next 10 minuted problems to start again (just turned off after the windows logo). Now it's working, but everytime i flash my device i'll get a looooot of error messages like this:

libraries/WiFiClientSecure/src/ssl_client.d:101: Warnung: NULL-Zeichen gelesen; der Rest der Zeile wird ignoriert

This means something like read NULL characters; rest of the line will be ignored

[ESP_Sprite]

Run a 'make clean'; looks like the hangup of your PC corrupted some ephemeral files that are used to track dependencies. Make clean cleans them up; the next make will re-create them.

[Staubgeborener]

make clean truly solved this.

To get back to my first problem: Is there a way to handle a whole packet at once instead of going threw every index as an array (in case this is my issue)? Or any other way, that my buffer don't throws packets away?

Again: Does CONFIG_ESP32_WIFI_STATIC_RX_BUFFER_NUM=25 and CONFIG_ESP32_WIFI_DYNAMIC_RX_BUFFER_NUM=0 helping me?

It would be also ok, if the packet is as raw as it can be. I would "convert" it later with a second programm or at the end of the code (with a pressure button or when X amounts of packets arrived).

[ESP_Sprite]

I'll ask around. Regardless of this, you may want to make your callback as minimal as possible. In an application I have (https://github.com/SHA2017-badge/bpp/bl ... _sniffer.c) I essentially just do some quick filtering, then push it into a ringbuffer. I then have a thread running on the 2nd core that pops the packets from the ringbuffer again and does the more time-intensive stuff with it. Not only does this decouple the two things, you also can calculate statistics by keeping a counter on how many times you had to drop a packet because of a full ring buffer.

[Staubgeborener]

I will give the ringbuffer and outsourcing to the second core a try. For this, i created with xTaskCreatePinnedToCore a Task und Core 1.

Is it right, that "everything" launches on Core 1 (-> Core 0) when its not defined to run on Core 2 (-> Core 1)? Also, is the numbering correct: Core 2 is defined as Core 1:
xTaskCreatePinnedToCore(secondCore, "secondCore", 10000, NULL, 1, NULL, 1);

Also, do you declared anywhere in your code QueueSetHandle_t? I get the following error messages (ringbuffer still in progress)

In file included from D:/msys32/home/Staubgeborener/esp/esp-idf/promiscuous_mode_esp32/main/promiscuous_mode_esp32.cpp:2:0:
D:/msys32/home/Staubgeborener/esp/esp-idf/components/freertos/include/freertos/ringbuf.h:197:66: error: 'QueueSetHandle_t' has not been declared
BaseType_t xRingbufferAddToQueueSetRead(RingbufHandle_t ringbuf, QueueSetHandle_t xQueueSet);
^
D:/msys32/home/Staubgeborener/esp/esp-idf/components/freertos/include/freertos/ringbuf.h:209:67: error: 'QueueSetHandle_t' has not been declared
BaseType_t xRingbufferAddToQueueSetWrite(RingbufHandle_t ringbuf, QueueSetHandle_t xQueueSet);
^
D:/msys32/home/Staubgeborener/esp/esp-idf/components/freertos/include/freertos/ringbuf.h:221:71: error: 'QueueSetHandle_t' has not been declared
BaseType_t xRingbufferRemoveFromQueueSetRead(RingbufHandle_t ringbuf, QueueSetHandle_t xQueueSet);
^
D:/msys32/home/Staubgeborener/esp/esp-idf/components/freertos/include/freertos/ringbuf.h:233:72: error: 'QueueSetHandle_t' has not been declared
BaseType_t xRingbufferRemoveFromQueueSetWrite(RingbufHandle_t ringbuf, QueueSetHandle_t xQueueSet);
^
D:/msys32/home/Staubgeborener/esp/esp-idf/promiscuous_mode_esp32/main/promiscuous_mode_esp32.cpp: In function 'void setup()':
D:/msys32/home/Staubgeborener/esp/esp-idf/promiscuous_mode_esp32/main/promiscuous_mode_esp32.cpp:50:36: error: invalid conversion from 'RingbufHandle_t {aka void*}' to 'void**' [-fpermissive]
packetRingbuf=xRingbufferCreate(12*1024, RINGBUF_TYPE_NOSPLIT);

See updated Gist in startpost.

[ESP_Sprite]

Is it right, that "everything" launches on Core 1 (-> Core 0) when its not defined to run on Core 2 (-> Core 1)?

No. If you don't specify a core by using ..PinedToCore, the task will be floating, as in whatever core has the least to do at any moment in time will pick up the task.

Also, is the numbering correct: Core 2 is defined as Core 1:
xTaskCreatePinnedToCore(secondCore, "secondCore", 10000, NULL, 1, NULL, 1);

We have two cores, in the documentation called core 0 and core 1, or the pro and app core. Not sure where you get 1 and 2 from

Also, do you declared anywhere in your code QueueSetHandle_t?

Ah, that's in <freertos/queue.h>; include that file before including ringbuf.h to fix that issue. (That's actually a bug in the ringbuf include file; I should fix that one of these days.)

[Staubgeborener]

Ok,

Code: Select all

#include "freertos/FreeRTOS.h"
#include <freertos/queue.h>
extern "C" {
#include "freertos/ringbuf.h"
}

is working perfectly. Also outsourcing everything to the second core with the ringbuffer works. It's waaaay faster and wireshark confirms this.

But i think, i have to really keep the callback tiny. At some point (after 1-2 minutes), the packets seems corrupted. For example there is a datapaket (3859 bytes length) with, lets say, 500 bytes 00 (one after another), some "normal looking bytes" and after that again some hundret a5 (also one after another).

A picture says more than a thousand words:

As i said. The first pakets looks really good. This happens after some time. And a short time later, everything just stop (i vsiualized everything with make monitor).

I think the ringbuffer is full= Or maybe another ringbuffer type? But i think RINGBUF_TYPE_BYTEBUF will make it even worse. What about ALLOWSPLIT?
I need as much as possible from a data paket. All 4 Adresses in the Frame, the payload, etc. Or maybe discard immediately other pakets like beacon frames?

Edit #1: Discarding all other pakets (mgmt, etc) except for data pakets seems to work... until now....
I will run this for an hour or two. But no matter how it goes: What would be another workaround without discarding all mgmt pakets?
And is there a difference between

Code: Select all

xRingbufferSend(packetRingbuf, packet, packet->rx_ctrl.sig_len, 0);
xRingbufferSend(packetRingbuf, packet, sizeof(packet), 0);

Edit #2: The "stop" problem occurred the first time after receiving around 50 pakets. Second time around 70 pakets, third time around 90 pakets, fourth time by 147 pakets.

Edit #3: Okay. I countered all of the pakets. I'll get over 200 pakets before i stopped. But my ringbuffer stopped everytime around ~150 pakets. Even with filtering for only data pakets.