Connecting your device to the internet with Xively and MQTT

JC_Xively
Posts: 3
Joined: Wed Oct 04, 2017 6:27 pm

Connecting your device to the internet with Xively and MQTT

Postby JC_Xively » Fri Oct 06, 2017 10:51 am

Hi all,

I’m JC, a Firmware Developer at Xively.

We recently released an ESP32 port and demo for our MQTT library, and we’d like to share it with you. We hope you can make the most of it, re-using the code for your own projects.

My intention with this post is to help you get an ESP32 connected to the internet using MQTT+TLS quickly and securely.

The Xively C Client is a fully Open Source library (BSD 3-clause), and the TLS library we’re using in this port is WolfSSL. It’s a good example of secure MQTT communication on an ESP32, and a quick introduction to the Xively platform and the Xively Client library.

You should know there’s a very detailed step by step tutorial of this demo in our Developers Centre: https://developer.xively.com/docs/esp32
Since you’re all familiar with the ESP32 platform, I won't go into as much detail in this post.

Create a Free Xively Developer Account
  1. Register a free account here: https://app.xively.com/register
If you already have an account, log in.

Create your Xively Digital Device

You need to create a "Device" in Xively to get the MQTT credentials for your device:
  1. Log into your free Xively Account
  2. Create the default ESP32 device template
  3. Create a device based on the ESP32 Device Template
  4. Copy your newly created Device Credentials. You'll need them later
More detailed instructions here

Get the Xively C Client MQTT library

Code: Select all

git clone https://github.com/xively/xively-client-c.git
(Or download the ZIP file from GitHub)

Flashing the Pre-Built Binaries for this Example

If you only want to test this demo, you don’t even need to build it. We’ve provided pre-built binaries, so you can flash them directly with esptool.py using these commands:

Code: Select all

cd xively-client-c/examples/esp32/xively_demo
python $IDF_PATH/components/esptool_py/esptool/esptool.py \
            --chip esp32 --port /dev/tty.SLAB_USBtoUART --baud 115200 \
            --before default_reset --after hard_reset write_flash -z \
            --flash_mode dio --flash_freq 40m --flash_size detect \
            0x1000 ./build/bootloader/bootloader.bin \
            0x10000 ./build/xively_demo.bin \
            0x8000 ./build/partitions_singleapp.bin
Once you’ve built the application yourself, you’ll be able to use make flash from the xively_demo folder as usual.

Building the Libraries and Example
  1. Build the libxively.a and libwolfssl.a static libraries:

Code: Select all

cd xively-client-c
make PRESET=ESP32
If successful, both static libraries should now be available in xively-client-c/bin/esp32/

2. Build and flash the ESP32 example:

Code: Select all

cd xively-client-c/examples/esp32/xively_demo
make flash
You’re good to go! Connect to the device’s UART to configure your WiFi and Xively credentials. They’ll be saved to Non-Volatile storage so you don’t need to set them every time.

Run the Demo

The demo’s functionality is rather simple to avoid bloating the code. It uses 2 peripherals: An LED (GPIO17-GND) and a button (GPIO0-GND). Control the LED from the internet publishing “1” or “0” to the …/LED topic; when you press the button, the device will publish “1” or “0” to …/Button (toggles on each button press).

The demo also gathers the credentials from the user at runtime (via UART), and saves them to Non-Volatile Storage so you don’t need to re-enter them every time. If you’d like to update them later, reboot the device and press the GPIO0 button while the LED is blinking rapidly.

Relevant Links
Final Thoughts

Even though this tutorial uses Xively, the library can be pointed at other standard MQTTv3.1 brokers. You may have to modify the Root certificates included in the Client, but the basic MQTT functionality adheres to the specification. The Xively-specific features built on top of it can be disabled.

I'll soon start working on adding OTA Firmware Updates to the ESP32 example. Xively's Secure File Transfer protocol allows you to download new firmware images over the same MQTT connection, without losing regular connectivity or responsiveness.

If you'd like to know more about Xively, our MQTT library, key features of this demo (such as OCSP support in the TLS library), etc. I'm happy to help :)

Thanks!
Juan Carlos Jimenez
Authorized Contractor of LogMeIn Inc.
Last edited by JC_Xively on Mon Oct 09, 2017 2:59 pm, edited 1 time in total.

WiFive
Posts: 3529
Joined: Tue Dec 01, 2015 7:35 am

Re: Connecting your device to the internet with Xively and MQTT

Postby WiFive » Fri Oct 06, 2017 8:16 pm

Since you already support mbedtls why not set it up to link against the mbedtls in esp-idf which has hw accel features?

ShinyGreenRobot
Posts: 1
Joined: Sun Oct 08, 2017 1:05 pm

Re: Connecting your device to the internet with Xively and MQTT

Postby ShinyGreenRobot » Sun Oct 08, 2017 1:17 pm

Hi

I tried building the libxively.a and libwolfssl.a static libraries on a computer that runs Ubuntu 16.04 but it failed to build.

Here is part of the output from the failed build:

Code: Select all

...
...
...
WolfSSL Build Complete.
  [CC] /home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/internal.c
  [CC] /home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/io.c
  [CC] /home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/keys.c
  [CC] /home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/ocsp.c
  [CC] /home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/ssl.c
/home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/ssl.c: In function 'wolfSSL_CertManagerFree':
/home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/ssl.c:2367:19: error: 'WOLFSSL_CERT_MANAGER {aka struct WOLFSSL_CERT_MANAGER}' has no member named 'ocsp_stapling'
             if (cm->ocsp_stapling)
                   ^
/home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/ssl.c:2368:28: error: 'WOLFSSL_CERT_MANAGER {aka struct WOLFSSL_CERT_MANAGER}' has no member named 'ocsp_stapling'
                 FreeOCSP(cm->ocsp_stapling, 1);
                            ^
/home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/ssl.c: In function 'wolfSSL_CertManagerEnableOCSPStapling':
/home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/ssl.c:4793:15: error: 'WOLFSSL_CERT_MANAGER {aka struct WOLFSSL_CERT_MANAGER}' has no member named 'ocsp_stapling'
         if (cm->ocsp_stapling == NULL) {
               ^
/home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/ssl.c:4794:15: error: 'WOLFSSL_CERT_MANAGER {aka struct WOLFSSL_CERT_MANAGER}' has no member named 'ocsp_stapling'
             cm->ocsp_stapling = (WOLFSSL_OCSP*)XMALLOC(sizeof(WOLFSSL_OCSP),
               ^
/home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/ssl.c:4796:19: error: 'WOLFSSL_CERT_MANAGER {aka struct WOLFSSL_CERT_MANAGER}' has no member named 'ocsp_stapling'
             if (cm->ocsp_stapling == NULL)
                   ^
/home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/ssl.c:4799:28: error: 'WOLFSSL_CERT_MANAGER {aka struct WOLFSSL_CERT_MANAGER}' has no member named 'ocsp_stapling'
             if (InitOCSP(cm->ocsp_stapling, cm) != 0) {
                            ^
/home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/ssl.c:4801:28: error: 'WOLFSSL_CERT_MANAGER {aka struct WOLFSSL_CERT_MANAGER}' has no member named 'ocsp_stapling'
                 FreeOCSP(cm->ocsp_stapling, 1);
                            ^
/home/henrik/esp/xively/xively-client-c/src/import/tls/wolfssl/src/ssl.c:4802:19: error: 'WOLFSSL_CERT_MANAGER {aka struct WOLFSSL_CERT_MANAGER}' has no member named 'ocsp_stapling'
                 cm->ocsp_stapling = NULL;
                   ^
Makefile:126: recipe for target '/home/henrik/esp/xively/xively-client-c/obj/esp32/wolfssl/src/ssl.o' failed
make[1]: *** [/home/henrik/esp/xively/xively-client-c/obj/esp32/wolfssl/src/ssl.o] Error 1
make[1]: Leaving directory '/home/henrik/esp/xively/xively-client-c/examples/esp32/xively_demo/wolfssl-make'
make/mt-os/mt-esp32.mk:82: recipe for target 'WOLFSSL_STATIC_LIB' failed
make: *** [WOLFSSL_STATIC_LIB] Error 2
Any idea what it is that is going wrong here?

JC_Xively
Posts: 3
Joined: Wed Oct 04, 2017 6:27 pm

Re: Connecting your device to the internet with Xively and MQTT

Postby JC_Xively » Mon Oct 09, 2017 11:36 am

WiFive wrote:Since you already support mbedtls why not set it up to link against the mbedtls in esp-idf which has hw accel features?
That's a very good question. There are a few factors in play:
  1. The WolfSSL library is configured to use OCSP Stapling. That means the Certification Authority doesn't get a direct call from the devices every time they boot, so the TLS handshake is streamlined and the overall privacy of the system increased. My understanding is that mbedTLS supports OCSP, but no stapling.
  2. The mbedTLS port in our library didn't work right away, probably because of a mismatch in the mbedTLS version provided in IDF and the one we used to write the port. It can be fixed, and probably quite easily, but it would take some extra development time
  3. There are plenty of sample mbedTLS implementations for the ESP32 (because it's in IDF), but not so many for WolfSSL, so it has that bit of extra value
  4. We're partners with WolfSSL, and our clients get free WolfSSL licenses for their devices.
All that being said, if the best fit for your product is to use mbedTLS and the ESP32 hardware acceleration, it's 100% OK. As long as the library supports TLSv1.2 and compatible cyphers and handshakes (it most likely does), the backend shouldn't give you any trouble.
ShinyGreenRobot wrote: I tried building the libxively.a and libwolfssl.a static libraries on a computer that runs Ubuntu 16.04 but it failed to build.

Code: Select all

[...]logs[...]
Any idea what it is that is going wrong here?
Thanks a lot for taking the time to report the issue and send the logs! Some machines seem to be having trouble building WolfSSL with the `NO_WOLFSSL_SERVER` option. The problem can be easily fixed by removing the line `-DNO_WOLFSSL_SERVER` from `xively-client-c/examples/esp32/xively_demo/wolfssl-make/Makefile`. Its only downside is a ~1% size increase on the final application binary, so nothing you need to worry about.

We just pushed that fix to the development branch, and we'll be releasing it to master soon.

albertmbt
Posts: 11
Joined: Thu Nov 09, 2017 11:15 am

Re: Connecting your device to the internet with Xively and MQTT

Postby albertmbt » Wed Nov 15, 2017 12:00 pm

Hi, I'm having issues to build the libxively.a and libwolfssl.a static libraries. When i command make PRESET=ESP32 it ends with a final error and does not create both libraries, it just build "libwolfssl.a" library.

Code: Select all

LD build/xively_demo.elf
xtensa-esp32-elf-gcc.exe: error: C:/msys32/home/Versa2/xively/xively-client-c/bin/esp32/libxively.a: No such file or directory
make: *** [C:/msys32/home/Versa2/esp-idf/make/project.mk:374: /home/Versa2/xively/xively-client-c/examples/esp32/xively_demo/build/xivel                           y_demo.elf] Error 1
any idea why this error?

JC_Xively
Posts: 3
Joined: Wed Oct 04, 2017 6:27 pm

Re: Connecting your device to the internet with Xively and MQTT

Postby JC_Xively » Wed Nov 15, 2017 1:07 pm

Hey albertmbt, if your libxively.a can't be found when trying to link the application, it must have failed to build.
Can you please provide the logs for the Xively Client build error? The ones you got when building for PRESET=ESP32.

If you're trying to run the stock Xively Client code, but libxively fails to build, you can try using the same version of the IDF SDK as we did for development. More info here.

Also, note there is a branch called feat/esp32_fw_updates waiting to be reviewed and merged into development. It has the latest and greatest ESP32 code, including OTA Firmware Updates support, but it's only been tested by me so far. You may wanna give it a try.

Let me know if that helps. If you find any issues in our codebase, you can also report them in GitHub issues; it ensures they don't slip through the cracks :)

Who is online

Users browsing this forum: No registered users and 36 guests