app_insights: Nodes with Self Claiming may not be accessible for Insights... and some UX questions about the platform

[brainstorm]

I'm using insights with a Saola-1 ESP32S2 board and found that message on the logs, I thought I'd ask here for an explanation about why it's the way it is? I'm the author of this hobby project and accompanying PR:

https://github.com/brainstorm/yigedianqi-modbus/
https://github.com/espressif/esp-rainmaker/pull/121

I've also noticed that I can claim the device via rainmaker CLI after it has been self-claimed via the iOS rainmaker mobile app but then I lose access to the mobile app... although I do gain access to ESP-Insights and dashboard by doing that then. I'm sure there's a technical explanation (key handling not being integrated w/ both systems?), but I do find this confusing.

So I have a few other UX questions, if I may... I am a fresh/new user, so forgive my ignorance, I'm evaluating this as I go:

1) Why there's only ESP32S2 self claiming mode available on the menuconfig (using esp-idf 4.4.0)? Is there a claiming mode that would allow me to use both web interfaces and mobile apps without losing data feeds from/to neither?

2) Why there's two RainMaker "data portals": https://dashboard.rainmaker.espressif.com and https://dashboard.insights.espressif.com/. From a user/admin perspective they seem to offer overlapping functionality, so will they be eventually fused into one as a unified dashboard?

3) In both portals I see the following message in the Node logs: "We could not find the firmware image to cross-reference this log entry. Please update a firmware image to get cross-referenced information for this log entry". But I couldn't find an easy explanation about how to generate the .zip package for that cross-referencing to work? A simple `idf.py build` does not seem to generate such a .zip under "build/" :/

4) Why aren't ESP insights sent with MQTT by default? In https://github.com/espressif/esp-insigh ... ng-started there seems to be a preference for the HTTP method, but it involves hardcoding an API key... if the device is already claimed/provisioning and functioning over MQTT, wouldn't it be easier to promote "Insights with MQTT" as the preferred/default method (https://github.com/espressif/esp-insigh ... -with-mqtt)?

My node ID is "hrTavUUPywiHHjrMSMdRAy", just in case you want to take a peek at my particular issue(s). Happy to be contacted over brainstorm at nopcode dot org if there's some private stuff to inquire about.

[ESP_Piyush]

To begin with, let me first explain the primary difference between self claiming and assisted/host (cli) claiming.

In self claiming, the node authenticates itself with the claiming server using a unique secret key programmed in the chip's efuse during manufacturing. The mac address is used as its node_id and it gets MQTT credentials against this id. It does not need any user to do this, and an Internet connection is sufficient.

On the other hand, in assisted/cli claiming, the user authenticates with the claiming service and gets a randomly generated unique node_id and set of credentials that gets programmed onto the node. Since the user is now the owner of the credentials, this user gets admin right to the nodes, which gives access to OTA via dashboard and Insights.

Irrespective of who owns the credentials (node/admin user), the end user association is a separate workflow. Moreover, since the association is linked with the node_id, if you overwrite self claimed credentials with CLI based ones, the node_id changes and so you lose the end user association.

You can read more about claiming and user roles in our docs.

1) Why there's only ESP32S2 self claiming mode available on the menuconfig (using esp-idf 4.4.0)? Is there a claiming mode that would allow me to use both web interfaces and mobile apps without losing data feeds from/to neither?

Assisted claiming, which gives end user as well as admin user privileges works only on BLE and so, it is not available on ESP32S2. We have not enabled this on SoftAP interface, because often, phones will lose Internet connectivity during SoftAP provisioning, as the phone switches from the home Wi-Fi to this SoftAP Wi-Fi.

However, this should not prevent you from doing what you are trying to do. If you first claim an ESP32S2 board from CLI and then add it from the phone app, you will get access to the device from the phone app as well dashboard.

2) Why there's two RainMaker "data portals": https://dashboard.rainmaker.espressif.com and https://dashboard.insights.espressif.com/. From a user/admin perspective they seem to offer overlapping functionality, so will they be eventually fused into one as a unified dashboard?

RainMaker and Insights are actually 2 different products. RainMaker is meant primarily for device control and management, whereas Insights is meant for diagnostics and debugging. However, we understand that most users will require both and so, we are working on better integrating the Insights dashboard into RainMaker dashboard. It should be available soon.

3) In both portals I see the following message in the Node logs: "We could not find the firmware image to cross-reference this log entry. Please update a firmware image to get cross-referenced information for this log entry". But I couldn't find an easy explanation about how to generate the .zip package for that cross-referencing to work? A simple `idf.py build` does not seem to generate such a .zip under "build/" :/

Assuming that you have started off from RainMaker examples and not Insights examples, please refer the docs here to understand how to enable Insights in RainMaker examples, so that idf.py build generates the zip mentioned there.

4) Why aren't ESP insights sent with MQTT by default? In https://github.com/espressif/esp-insigh ... ng-started there seems to be a preference for the HTTP method, but it involves hardcoding an API key... if the device is already claimed/provisioning and functioning over MQTT, wouldn't it be easier to promote "Insights with MQTT" as the preferred/default method (https://github.com/espressif/esp-insigh ... -with-mqtt)?

If you use Insights via RainMaker, MQTT gets used by default, irrespective of what is chosen as the transport via sdkconfig. However, as mentioned earlier, Insights is a separate product and can be used with other systems that have their own, unrelated MQTT connections. A single HTTP API key can then be used across multiple nodes, making the setup and workflow much simpler, and so that is the default in standalone insights.

Regards,
Piyush

[brainstorm]

Thanks for the detailed explanation, all doubts cleared, I will experiment further with that guidance

Now, if I may, I have one more question: I saw in the roadmap that timeseries is a planned feature and I've seen some hints on the properties of the API (i.e: PROP_FLAG_TIME_SERIES).

In my (solar power metering) application I resorted to a third party provider for the timeseries and plotting bit (pvoutput.org): if you are allowed to disclose more details of the roadmap, what's the ETA for this functionality and what do you think the timeseries system specs would be in RainMaker?:

1) Max retention period?
2) Max event rate (limiting)?
3) Max event size payload?
4) Will it ship with any plotting facilities in the dashboard and/or mobile apps?

Thanks in advance!