Suggestion for Setting up a Mesh Network

[Usama Masood]

Hi All,

We are trying to setup a mesh network of up to 40000 nodes at a time, accessible through a single (or multiple edge) nodes/routers. For PoC we created a BLE mesh (esp-idf/esp_ble_mesh/nimble) and when we reached to around 200 nodes the performance of mesh became really poor, so much that replies from a node can take upto a minute (even with enhancements related to retransmission delays etc)

We are using ESP32C3 and we cannot cache alot of network data as it runs out of memory.

Our use case involves each node to transmit a ping after each minute or more (currently adjusted to 5 min to decrease the network traffic) and should respond ASAP to requests from out side likely happen with a rate of 1 request per 10 second or more

We are currently trying to figure out an out of box solution or some ideas on how we can achieve this effectively

Happy to share more details, if needed

[username]

We are trying to setup a mesh network of up to 40000 nodes at a time

40k, is that a typo?

[Usama Masood]

No, we need to scale the solution to 40K nodes in a network (with each node 1m apart)

[chegewara]

No, we need to scale the solution to 40K nodes in a network (with each node 1m apart)

I think you may have many issues with such designed mesh network.
- it will be very noisy and may be not reliable
- since you can have "only" 32k unicast addresses you will need at least 2 app/net keys and probably new ble mesh feature which can bridge between them (ble 5.3 i think)
- message can be relayed thru 254 or 255 nodes max, so i dont know if its enough for you
- probably other problems i didnt think about which will pop up when you scale up

[ESP_Sprite]

Could you tell us a little bit more about your use case? Ideally we'd like to know what you are designing, but if you cannot, it would be good to know about the data going to be sent/received: How big is it? From where to where does the data flow go? Is latency important? Standard mesh networking indeed isn't really suitable for your use case, but it's possible there's a different solution.

[Usama Masood]

I think you may have many issues with such designed mesh network.

- it will be very noisy and may be not reliable

Yes correct that is the issue we are trying to overcome

- since you can have "only" 32k unicast addresses you will need at least 2 app/net keys and probably new ble mesh feature which can bridge between them (ble 5.3 i think)

Yes, that is why we decided to have multiple edge devices to connect the mesh to the backend, with each edge device to be owner of its own mesh network

message can be relayed thru 254 or 255 nodes max, so i don't know if its enough for you

Not all of our nodes are relays, currently for this network of around 200 nodes we have 16 relays, and even with that we are seeing that mesh is too chatty to handle minimalistic traffic (the TTL is limited to 16)

Our use case involves nodes reporting their status <32bytes/fragmented payload> at a frequency of 1 minute, this is the only traffic we expect to see in normal working.
From time to time backend will send requests to the nodes broadcast <16 to 128bytes/fragmented payload> for which we expect the node(s) to respond in timely fashion e.g. within 10 seconds
Mesh is isolated and we do not care about encryption or security at this stage

We know the limitations of BLE mesh, we are looking for some other solution ESP-NOW, WIFI mesh (or something else)

[chegewara]

I dont know much about esp-now or other wifi mesh, but i believe they are even more limited, to 255 nodes i think?

In ble mesh (esp mesh bluedroid) you can check "Directed Forwarding" feature.
https://docs.espressif.com/projects/esp ... -list.html

Please correct me, but from your description i understand you have "edge nodes" which are connected to backend over wifi, right?
Is there any reason you cant use mqtt for example (wifi coverage maybe?)?

[ESP_Sprite]

ESP-Now seems to be applicable to your situation, given the low bandwidth requirements, but it's not a mesh tech per se, rather a peer-to-peer tech, so you'd have to roll any store-and-forward and routing discovery yourself.

Any idea how large a surface we're talking about here? How many hops would be needed to go from one end of the network to the other?

[Usama Masood]

Please correct me, but from your description i understand you have "edge nodes" which are connected to backend over wifi, right?
Is there any reason you cant use mqtt for example (wifi coverage maybe?)?

We also need to conserve the power as much as we can, that is why we are thinking more in terms of BLE
In current setup backend<->edge communication is over MQTT, and edge node translate the messages between MQTT to mesh

[Usama Masood]

Any idea how large a surface we're talking about here? How many hops would be needed to go from one end of the network to the other?

That we still have to figure out but from the 200 BLE mesh it seems like we need at most 4 hops, with 16 relays in the system.
The nodes seemed quite overwelled with that much background, and response time became quite high