[JavaScript] JSON parsing faster alternative

[KanyeKanye]

My esp32 chip is used for recording some important data into local memory. I need to send these data to the user webbrowser for drawing a chart. It is about 1MB per hour and I need charts with up to 5days of data (up to 120MB of raw data).

At the moment I have written an http server and JSON parser. When user requests data, raw data from memory are read and parsed into JSON for sanding. Parsing raw data into text json result in extending each 1MB (of raw) into about 2MB (of text).

Local browser waits for JSON file and using JS JSON.parse(this.responseText) parses it into object for adding into an javascript array.
JSON.parse tooks lots of time! About 5 sec for each 2MB (1MB of raw).

Code: Select all

function loadChartData(address) {
    var xhttp = new XMLHttpRequest();
    xhttp.onreadystatechange = loadChartDataReady;
    xhttp.open("GET", "http://192.168.1.1/"+address, true);
    xhttp.send();
}
function loadChartDataReady() {
    if (this.readyState != 4) return;
    if (this.status != 200) { return; }
    try {
        var json = JSON.parse(this.responseText);
        for(var i = 1; i < json.length; i++) {
            var obj = json[i];
            series.splice(i - 1, 0, {ts: obj[0], time: obj[1], s: obj[2], m: obj[3], on: obj[4]} );
        }
        updateChart();
    } catch(err) {
        console.log("error: " + err);
    }
}

I am wandering what can I do for speeding up my code both in terms of used memory and execution time.

• Can I parse raw data somehow in javascript?

• Can I change JSON.parse into some another function for speeding up?

• Can I change XMLHttpRequest into something another for start parsing data while they are still being sent?

[tommeyers]

You can send a body in the html that is not json.

See: https://developer.mozilla.org/en-US/doc ... P/Messages

If you do that you will reduce the load on the esp32 (memory and processors), shorten the transmit time; on the browser you will simplify and reduce the cpu load for json processing.

I think you could also split your body across multiple html messages. That would make concurrent (server browser) processing possible. I can't find an example of that with a quick look.

Tom

[PeterR]

You are using Ajax (Ajaj...).
1) You could chunk requests, for example:

Code: Select all

GET /api/v1/data?start=x&end=y

You may then extract the HTTP query start/end parameters using the ESP library & your webserver would simply serialise between ranges.

2) Custom format
As suggested, you do not need to encode as JSON. You could send CSV. Maybe the first line is a header and following lines are the data.
If you split the header line using the comma deliminator into array headerNames then you will have the Javascript property names.
So:

Code: Select all

myObj[headerNames[i]] = value

I doubt that you will gain in total time but as you are looping on an array of lines you may update your chart whenever you like.

3) Raw data
Be careful. Encode data into network endian . Text has certain advantages!

3) Websocket
A Websocket would allow you to send data in smaller chunks and in real-time.
The ESP webserver does not support websockets but the Asynchronous webserver does.

I would be tempted to use Websockets but a combination of 1) and 2) would be simplest to get running.

[KanyeKanye]

I've decided to send raw data. Bytes are now read and send to the browser without any parsing. That way I could use larger buffers and read more from memory to send. Whats more I know exactly how much data will be send.
On browser, javascript side I parse data using DataView, getUint32, getInt8

[tommeyers]

So, you intend to send data that potentially has byte values 0..255?

If you do you will be back here.

Tom

[PaulNi]

I think getting it parsed faster will not help much.

You biggest concern should be the IO bottleneck. Think of streaming the json to some parser before rewriting the parser itself

[PeterR]

If you use RAW bytes then just make sure you encode/decode using the correct endians!