ESP32 configuration at runtime.

[PaulVdBergh]

Hi All,

I'm working on a project for model railroad control (DCC - NMRA). This involves tens of ESP32 SOCs and a Linux SBC (aka RPi). The ESP32s have different roles, for example DCC signal generator, RailCom detector, handheld devices interface, discrete I/O, etc. All the ESP32 devices communicates over Wifi with an MQTT broker on the Linux device. The (headless) Linux device governs the complete layout and interacts with the (human) dispatcher over webpages published by a webserver (apache).

To make this all happens, each and every ESP32 module needs to know the wifi SSID & password and the MQTT brokers IP-address. ATM this info is hard-coded in the source-code. Since I'm the author of the sources, I can eventually modify these constant data and do a recompile to make the changes effective. The constant data can even be integrated into the menuconfig options, making it even more easy. However, if I ever want to make the system available to other model railroad hobbyists, I cannot expect them to have the knowledge to install the development environment, obtain the source-code, modify it, recompile the code and flash the devices. IMHO this would generate an enormous amount of support.

So I'm looking for a solution that allows to configure the ESP at runtime. I first evaluated Bluetooth BLE, but unfortunately there is almost no documentation on the Linux side (Bluez), nor example project re this subject. Another option is to use (or write) an Android (or iOS) app. This implies I have to learn jet another platform/language

My second option is ESP-IDF's smart config. I'm evaluating this option atm. I compiled and flashed the example project ito a ESP-WROVER-KIT and intalled the ESP8266 Smart Config app on my tablet, but to no avail... :

Code: Select all

paulvdbergh@TBTIoT-AMD64:~/TBTIoT/smart_config$ make flash monitor
Flashing binaries to serial port /dev/ttyUSB1 (app at offset 0x10000)...
esptool.py v2.3.1
Connecting........___
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 2000000
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Flash params set to 0x0220                                                      
Compressed 20096 bytes to 11824...                                              
Wrote 20096 bytes (11824 compressed) at 0x00001000 in 0.1 seconds (effective 2021.0 kbit/s)...                                                                  
Hash of data verified.                                                          
Compressed 546112 bytes to 338382...                                            
Wrote 546112 bytes (338382 compressed) at 0x00010000 in 5.4 seconds (effective 805.2 kbit/s)...                                                                 
Hash of data verified.                                                          
Compressed 3072 bytes to 103...                                                 
Wrote 3072 bytes (103 compressed) at 0x00008000 in 0.0 seconds (effective 1454.7 kbit/s)...                                                                     
Hash of data verified.                                                          
                                                                                
Leaving...                                                                      
Hard resetting via RTS pin...                                                   
MONITOR                                                                         
--- idf_monitor on /dev/ttyUSB1 115200 ---                                      
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---           
ets Jun  8 2016 00:22:57                                                        
                                                                                
rst:0x1 (POWERON_RESET),boot:0x3e (SPI_FAST_FLASH_BOOT)                         
flash read err, 1000                                                            
Falling back to built-in command interpreter.                                   
OK                                                                              
>ets Jun  8 2016 00:22:57                                                       
                                                                                
rst:0x10 (RTCWDT_RTC_RESET),boot:0x3e (SPI_FAST_FLASH_BOOT)                     
configsip: 0, SPIWP:0xee                                                        
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00         
mode:DIO, clock div:2                                                           
load:0x3fff0018,len:4                                                           
load:0x3fff001c,len:5664                                                        
ho 0 tail 12 room 4                                                             
load:0x40078000,len:0                                                           
load:0x40078000,len:14324                                                       
entry 0x40078620                                                                
I (30) boot: ESP-IDF v3.1-dev-883-g0236592f 2nd stage bootloader                
I (30) boot: compile time 12:01:32                                              
I (40) boot: Enabling RNG early entropy source...                               
I (40) boot: SPI Speed      : 40MHz                                             
I (40) boot: SPI Mode       : DIO                                               
I (44) boot: SPI Flash Size : 4MB                                               
I (48) boot: Partition Table:                                                   
I (52) boot: ## Label            Usage          Type ST Offset   Length         
I (59) boot:  0 nvs              WiFi data        01 02 00009000 00006000       
I (67) boot:  1 phy_init         RF data          01 01 0000f000 00001000       
I (74) boot:  2 factory          factory app      00 00 00010000 00100000
I (82) boot: End of partition table
I (86) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x11174 ( 70004) map
I (119) esp_image: segment 1: paddr=0x0002119c vaddr=0x3ffb0000 size=0x0379c ( 14236) load
I (125) esp_image: segment 2: paddr=0x00024940 vaddr=0x40080000 size=0x00400 (  1024) load
0x40080000: _iram_start at /home/paulvdbergh/esp/esp-idf/components/freertos/xtensa_vectors.S:1685

I (127) esp_image: segment 3: paddr=0x00024d48 vaddr=0x40080400 size=0x0b2c8 ( 45768) load
I (154) esp_image: segment 4: paddr=0x00030018 vaddr=0x400d0018 size=0x61308 (398088) map
0x400d0018: _stext at ??:?

I (294) esp_image: segment 5: paddr=0x00091328 vaddr=0x4008b6c8 size=0x041ec ( 16876) load
0x4008b6c8: vTaskDelete at /home/paulvdbergh/esp/esp-idf/components/freertos/tasks.c:3529

I (301) esp_image: segment 6: paddr=0x0009551c vaddr=0x400c0000 size=0x00000 (     0) load
I (311) boot: Loaded app from partition at offset 0x10000
I (311) boot: Disabling RNG early entropy source...
I (312) cpu_start: Pro cpu up.
I (316) cpu_start: Starting app cpu, entry point is 0x40080f80
0x40080f80: call_start_cpu1 at /home/paulvdbergh/esp/esp-idf/components/esp32/cpu_start.c:224

I (0) cpu_start: App cpu up.
I (326) heap_init: Initializing. RAM available for dynamic allocation:
I (333) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (339) heap_init: At 3FFB9440 len 00026BC0 (154 KiB): DRAM
I (345) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (352) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (358) heap_init: At 4008F8B4 len 0001074C (65 KiB): IRAM
I (364) cpu_start: Pro cpu start user code
I (47) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (190) wifi: wifi firmware version: 2de7507
I (190) wifi: config NVS flash: enabled
I (190) wifi: config nano formating: disabled
I (190) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (200) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (240) wifi: Init dynamic tx buffer num: 32
I (240) wifi: Init data frame dynamic rx buffer num: 32
I (240) wifi: Init management frame dynamic rx buffer num: 32
I (240) wifi: wifi driver task: 3ffc09b0, prio:23, stack:4096
I (250) wifi: Init static rx buffer num: 10
I (250) wifi: Init dynamic rx buffer num: 32
I (330) phy: phy_version: 386.0, 67c798f, Mar 14 2018, 16:34:06, 0, 0
I (330) wifi: mode : sta (24:0a:c4:11:a7:38)
I (380) smartconfig: SC version: V2.7.1
I (5200) wifi: ic_enable_sniffer
I (5200) sc: SC_STATUS_FINDING_CHANNEL

After some time the tablet app confirms Esptouch fail....

What I finally try to accomplish is that the discovery of ESP devices is done by the Linux machine, perhaps using the techniques used by the ESP smart-config component, and setting up the ESP device so that it can work in the new environment whit the least possible human interaction.

Any sugestions are very welcome.

Thanks in advance,

Paul.

[chegewara]

I have two possible solutions you can test. This is from Neil Kolban repo project, when there is no wifi credential info stored in nvs or data are corrupted then esp32 is running as softAP with small website allowing to input wifi credentials.
https://github.com/nkolban/esp32-snippe ... g/bootwifi

I know you said bluetooth le requires android app and this implies learning another programming language, but i would like to propose this solution with example:
https://github.com/chegewara/esp32-ota-with-ble-setup
For android application you can write quick and easy application with http://appinventor.mit.edu/explore/. Its nodered like application that allows to create android app with minimal knowledge required. In my repo you can find ota_app1.aia file, its app inventor application so you can study how to do it.

[kolban]

Howdy Paul,
In addition to mr @Chegewara's awesome comments ... some questions ...

In your solution, what "factors" can we assume? Will all the ESP32s connect as WiFi stations? Will they be Internet connected?

Lets assume that on your ESP32s you can store the address of your MQTT server persistently (eg. NVS storage). The question then becomes "how do we get that data to the ESP32s?". There are many choices ... but the one you choose will be a function of what you want to end up with.

Some off the cuff possibilities (in no particular order)

* ESP32 is an FTP server ... FTP client into the ESP32 and push a configuration file.
* ESP32 is a UDP listener ... application broadcasts a configuration file.
* ESP32 is internet connected ... ESP32 pulls a configuration file via HTTP.
* ESP32 is an MDNS client ... configuration data associated with the MDNS server entry.
* Integrate with Blynk (https://www.blynk.io/)
... many others

[PaulVdBergh]

Hi,

chegewara and Neil, thanks for the suggestions.

I'm sniffing around in Neil's esp32-snippets bootwifi project. This looks promising, although I have the impression that that code is dependent on Neils framework (Is the webserver based on mongoose ?). OTOH, I came across an example from lucadentella. It's no C++ but pure ESP_IDF C.

I'll test the following scenario:

- The ESP boots as an access point with a known SSID, password and IP-address. The ESP also opens a listening socket.
- The RPi continuously scans for WiFi access points. When a known SSID is detected, a socket to the ESP32 is opened and a json string containing the required configuration values is send over the socket.
- The ESP receives the json string and parses it, so it gets to known the correct config parameters. The ESP now becomes a WiFi station and connects to the correct SSID and MQTT broker.

However, since powering up the whole system at once, there will be multiple ESP devices starting up an access point with the same SSID. I'll try to overcome this by making the SSID partially static (for example "IoTT"), and partially dynamic, based on the mac-address of the ESP.

Do you, experts, think this is a feasible solution?

Greetz,

Paul

[kolban]

Did I understand that the Linux server hosting the MQTT is a RPI and the ESP32s connect to the RPI as an access point?

If that's the case, I'm missing what needs configured? Your ESP32 will be a station, it will connect to the RPI access point ... it will then know the gateway address to which it needs to send traffic which WILL be the RPI .... and now you are off to the races. If you settle on the RPI having a well known SSID, that is likely all you need?

[PaulVdBergh]

Hi Neil,

The RPi is connected to a wifi router/access point over eth0 (cable). This is because, AFAIK, the RPi's wlan0, configured as AP, can only accept a theoretical maximum of 8 concurrent connections, which isn't enough. So the RPi's wlan0 will be configured as station, scanning for APs.
OTOH, the ESP's starts up as a station, broadcasting their SSID. This SSID is in the form of, for example, "IoTT_XXXXXX", where the X-s are numerical representations of the ESP32 mac address (leaving out the manufacturer code).
Now, when the RPi, scanning for APs, encounters an access point for which the SSID starts with "IoTT_", the RPi wlan0 connects to the SSID "IoTT_XXXXXX", opens a socket on a predefined ip-address:port, and sends the config data. This config data contains the SSID of the wifi router/access point etc.
When the ESP has received the data, it reconfigures the wifi, so that it becomes a station, and connects to the wifi router/access point and establish a connection to the MQTT broker on the RPI. It is not so that the IP address of the ESP is the same when in AP-mode as in STA-mode, nor is the ip-address of the RPI (first over wlan0, later over eth0).

Or did I missed something here, I admit that network programming isn't my strongest asset. My knowledge as C/C++ programmer reaches up until opening, using and closing sockets

Of course I'm open to alternatives....

Paul.

PS. : any info re linux C/C++ techniques for scanning for SSID is very welcome too....

[kolban]

Thanks for the description. Appreciate you taking the time to describe it. I think I see your algorithm ...

All ESP32s start advertising as access points.
The RPI connects to each ESP32 in turn and tells it the SSID/password of the router.
Each ESP32 stops being an access point and connects to the router (here we assume that the router can host as many ESP32s as needed)
The ESP32 now connects to the RPI (for MQTT) through the router.

My opinion ... it feels a little "convoluted" ... so if I say that, then I have to proffer an alternative. The immediate issue I think you are trying to solve is how to "bootstrap" the ESP32. How do you tell the ESP32 which router to connect to? There are options available for this which are manual connections where you use an external app (maybe a browser on your phone) and connect to each ESP32 in turn and tell it the access information for the router. While this would indeed work, I think you are looking for a more consumable solution and that has taken you to the current recipe (which ... does look like it will work).

One area I might suggest is the "new kid in town". Recently released (at least I consider it recent) is the new ESP32 MESH technology ... see the following:

http://esp-idf.readthedocs.io/en/latest ... /mesh.html

In this story, the ESP32s conspire amongst themselves to build a self healing and dynamic network that facilitates communication from any to any without having to prescribe manual routing information. This also has the benefit that the ESP32s at the "physical far end" don't need radio connectivity to the router or ALL the other ESP32s. While I haven't made the time to study this in depth, it is a contender to be examined ... even if just to discount for identifiable reasons.

[PaulVdBergh]

My opinion ... it feels a little "convoluted" ...

After a short sleep... Indeed, the RPi act as a access point with a known SSID, so the ESP can connect to that SSID and retreive all the nessecary data to reconnect to the SSID of the router...

As Einstein once told : "Keep it simple, as simple as possible, but no simpler"...