Using LWIP Raw API

[jdwix615]

Hi,

I'm attempting to implementing protocol using TCP and would like to use to the LwIP Raw API. Apart from this http://lwip.wikia.com/wiki/Raw/TCP lwip specific wiki page I can't find any documentation that talks about using the Raw API from within the ESP IDF.

I am wondering what the process is in terms of using the ESP IDF as I'm running into troubles trying to set up and manage connections via the api. The wiki page I linked makes references to calling timer check functions at regular intervals but I can't work out how or if this has already been implemented in the IDF or how I need to go about it myself.

Thanks.

[ESP_Angus]

Hi jdwix615,

The raw API is not supported because we don't provide any way to run LWIP without a dedicated tcpip worker thread (and the raw API doesn't support threading). It is probably possible to hack something together so the raw API works, but we don't recommend it.

The LWIP netconn API should work (this is essentially the raw API, but made threadsafe). Is this a viable alternative?

Angus

[jdwix615]

Hi Angus,

Ahh yes I guess this makes sense. The netconn API is definitely the way to go.

I've found the LWIP implementation in regards to ESP IDF to be sparsely available. Does it mention anywhere about this API not being supported?

[kolban]

@jdwix615,
Purely for my own curiosity ... can you post why you wish to use the LwIP APIs as opposed to posix sockets API?

[ESP_Angus]

I've found the LWIP implementation in regards to ESP IDF to be sparsely available. Does it mention anywhere about this API not being supported?

Not really. For the most part, we encourage/support the standard/POSIX BSD socket APIs that are also exposed by LWIP. These can generally be used without any LWIP-specific considerations.

[jdwix615]

@kolban
I wanted to implement my own protocols that handled forwarding and routing packets from one interface to another (STA -> AP) and vice versa via callbacks from the API, as opposed to waiting for the data to pass through the whole stack and sending it back out another socket.

[ThinkalVB]

@kolban
I wanted to implement my own protocols that handled forwarding and routing packets from one interface to another (STA -> AP) and vice versa via callbacks from the API, as opposed to waiting for the data to pass through the whole stack and sending it back out another socket.

I am also looking for the same. I am trying to create a mesh network using ESP32. So basically all I want to do is route the raw Ip packets based on their IP address (AP to STA and vice verse). Please let me know if you found something on this. If something needs to build in order for this to works - I can pledge some of my time for its development.
Thinkal VB thinkalvb@gmail.com

[malachib]

I have been using Raw API (by accident) for a long time and it seems to work. This is not an endorsement, I think I may have gotten lucky. I would really, really like to use the Raw API because the callback interface is much more appealing than the sequential one.

Espressif, is there an official word on whether we can use this Raw API in the future? I know docs say unsupported. Any plans to support it in the future? What about tcpip_callback?

I only discovered today Raw API wasn't supported, which bums me out I like the nonblocking pcbs and the zero copy life of pbufs

[chrismerck]

As far as I can tell, it is possible to use the raw API from a user thread if LWIP_TCPIP_CORE_LOCKING is enabled by wrapping your calls in LOCK_TCPIP_CORE / UNLOCK_TCPIP_CORE. Callbacks will occur on the tcpip thread.

I'm considering using raw API in an ESP32 application. Let me explain. I currently maintain a commercial IoT platform based on Sockets/ESP32. We have a few problems with sockets API:

1. Essentially requires a task for every socket. We're running out of internal RAM mainly because of needing a fat stack for every socket task.

2. It's hard to make a highly responsive application with blocking sockets. A `connect`, a `recv` can both block. Even the harmless sounding `send` can block in case of TCP back-pressure. This makes for an awkward situation for a full-duplex application. You basically need a dedicated rx task and then send from another task or timer, hoping that `send` doesn't block.

3. Non-blocking sockets is an option, but this is an esoteric side of the BSD Sockets API. I have vague concerns that there may be bugs or at least surprising limitations in this part of lwIP. Supposedly not all POSIXes are consistent here anyways.

Here's the solutions I'm considering in order of increasing difficulty:

1. Minimize what we do in our socket threads, using queues to shuttle data to and from bigger worker threads which print and otherwise call into deeper stacks (like our lua code). -- Saves a modest amount of stack, does nothing for issue #2.

2. Poll non-blocking sockets. -- Maintains separation of component code, saves stack as we can use timers, but causes massive CPU usage-latency trade-off.

3. Feed all or most of our sockets into `select` in a single task. -- Eliminates several stacks, but causes coupling of all components. A bunch of plumbing is required to manage different parts of the application sharing this select-thread.

4. Use raw API -- Eliminates stacks, solves issue #2, but requires a high-level emulation of the API on top of POSIX non-blocking sockets. Has side benefit of making the application portable to bare-metal applications.