we're in the process of developing a hardware GitLab CI runner so we can run integration tests and unit tests incl getting unit test coverage, all on actual hardware.
Our setup is as follows:
The target device is a ESP32 PoE by Olimex.
A Raspberry Pi 4 with Ubuntu Server 20.04 and gitlab-runner installed is used to run the tests.
An ESP Prog is used for actually getting the unit test coverage (the ESP32 talks JTAG to it for that).
We're using the Raspi so we can utilize its GPIO pins, which are needed for JTAG communication with the ESP32 to get the unit test coverage. On this specific ESP32 pin 12 (which is needed for JTAG) is used for powering the ethernet module, so our scripts can selectively pull that pin to HIGH or LOW depending on what we need in the specific situation. This sounds like it could be responsible for the problem we're having (which I'll describe later), but the problem also occurs when pin 12 is not connected at all.
The Raspberry Pi gets prompted by GitLab to start up a container. The container contains Python code that, among other things, runs a bash script that builds the project via ESP IDF and flashes it. After that, it runs `idf.py monitor` to wait for the ESP32 to provision on our mock device provisioning backend.
The problem is, when the whole process runs in the GitLab Runner, the `idf.py monitor` command's output looks like the ESP32 is crashing. This for some reason doesn't happen if we run the scripts manually (outside of a GitLab runner, but still in the Docker container).
We've confirmed the ESP32 is flashed correctly. If I plug it into my PC after the flashing is done in the CI and run `idf.py monitor` there, I get the expected output.
The relevant output from the CI:
Code: Select all
[INTEGR TEST PREP out]: --- idf_monitor on /dev/ttyUSB0 115200 ---
[INTEGR TEST PREP out]: --- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
[INTEGR TEST PREP out]: 0x40080400: _init at ??:?
[INTEGR TEST PREP out]:
[INTEGR TEST PREP out]: 0x40081438: call_start_cpu1 at /opt/esp/idf/components/esp_system/port/cpu_start.c:142
[INTEGR TEST PREP out]:
[INTEGR TEST PREP out]: 0x400d8b1d: app_main at /builds/gr/clients/dbsus/dsa-plus/esp32/main/main.cpp:305 (discriminator 1)
[INTEGR TEST PREP out]:
[INTEGR TEST PREP out]: 0x40091937: _esp_error_check_failed at /opt/esp/idf/components/esp_system/esp_err.c:47
[INTEGR TEST PREP out]:
[INTEGR TEST PREP out]: 0x40081f4a: panic_abort at /opt/esp/idf/components/esp_system/panic.c:423
[INTEGR TEST PREP out]:
[INTEGR TEST PREP out]: 0x40091941: esp_system_abort at /opt/esp/idf/components/esp_system/esp_system.c:153
[INTEGR TEST PREP out]:
[INTEGR TEST PREP out]: 0x400988d6: abort at /opt/esp/idf/components/newlib/abort.c:38
[INTEGR TEST PREP out]:
[INTEGR TEST PREP out]: 0x40091937: _esp_error_check_failed at /opt/esp/idf/components/esp_system/esp_err.c:47
[INTEGR TEST PREP out]:
[INTEGR TEST PREP out]: 0x400d8b1d: app_main at /builds/gr/clients/dbsus/dsa-plus/esp32/main/main.cpp:305 (discriminator 1)
[INTEGR TEST PREP out]:
[INTEGR TEST PREP out]: 0x401999d7: main_task at /opt/esp/idf/components/freertos/FreeRTOS-Kernel/portable/port_common.c:131 (discriminator 2)
[INTEGR TEST PREP out]:
[INTEGR TEST PREP out]: 0x40095055: vPortTaskWrapper at /opt/esp/idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:154
[INTEGR TEST PREP out]:
[INTEGR TEST PREP out]: 0x40080400: _init at ??:?
[INTEGR TEST PREP out]:
[INTEGR TEST PREP out]: 0x40081438: call_start_cpu1 at /opt/esp/idf/components/esp_system/port/cpu_start.c:142
The question would be: what difference between running the container manually vs via GitLab Runner could cause the `monitor` connection to be faulty?
Thx in advance for any insight.