Different results when pulling API data with MicroPython or Arduino IDE

heikki.hietala · Postby **heikki.hietala** » Sun Jun 02, 2024 5:47 am

Hi all,

I'm using an ESP32 to pull in XML flight data for Finnish airports from the Finnish Aviation Authority's API. I have managed to do it with both MicroPython and Arduino's IDE, but the data is clean only with MicroPython.

The data mass is 900,000 characters on the average, so it cannot be read at once. I read the stream and write it in a file called flights.xml, and then I parse that to get the next arriving or departing flights. This is an example of the data:

Code: [Select all] [Expand/Collapse]

<flights xmlns="http://www.finavia.fi/FlightsService.xsd" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<dep>
<header>
<timestamp>2024-05-17T04:06:15Z</timestamp>
<from>Finavia</from>
<description>Departure flight data</description>
</header>
<body>
<flight>
<h_apt>HEL</h_apt>
<fltnr>AY1103</fltnr>
<sdt>2024-05-16T16:30:00Z</sdt>
<sdate>20240516</sdate>
<acreg>OHATF</acreg>
<actype>AT7</actype>
<mfltnr/>
<cflight_1>QR6174</cflight_1>
<cflight_2>JL6873</cflight_2>
<cflight_3/>
<cflight_4/>
<cflight_5/>
<cflight_6/>
<route_1>VNO</route_1>
<route_2/>
<route_3/>
<route_4/>
<route_n_1>Vilnius</route_n_1>
<route_n_2/>
<route_n_3/>
<route_n_4/>
<route_n_fi_1>Vilna</route_n_fi_1>
<route_n_fi_2/>
<route_n_fi_3/>
<route_n_fi_4/>
<chkarea>200</chkarea>
<chkdsk_1>211</chkdsk_1>
<chkdsk_2>211</chkdsk_2>
<calls_1>2024-05-16T16:13:30Z</calls_1>
<calls_2/>
<calls_3>2024-05-16T16:27:40Z</calls_3>
<calls_4/>
<park>6</park>
<park_prv>8</park_prv>
<gate>6</gate>
<gate_prv>8</gate_prv>
<prm>DEP</prm>
<prt>Departed</prt>
<prt_f>Lähtenyt</prt_f>
<prt_s>Avgått</prt_s>
<est_d>2024-05-16T16:40:00Z</est_d>
<pest_d>2024-05-16T16:40:00Z</pest_d>
<act_d>2024-05-16T16:42:00Z</act_d>
<ablk_d>2024-05-16T16:38:00Z</ablk_d>
<callsign>FIN9Q</callsign>
</flight>
</body>
</dep>
</flights>

GeSHi © Codebox Plus Extension

The issue is that the C program introduces random errors into the XML, always between elements, and mostly the errors are of the type "ff9" or "ff0":

<gate>6</gate>
<gate_prv>8</gate_prv>
ff0
<prm>DEP</prm>

and so on. This is the function that gets the data in chunks of 1024 bytes (the chunk soze has no effect on the error):

Code: [Select all] [Expand/Collapse]

// Function to fetch flight data from API and store in SPIFFS
void fetchFlightData() {
  HTTPClient http;
  http.begin(api_url);
  http.addHeader("app_id", api_key);
  http.addHeader("app_key", app_key);
  int httpCode = http.GET();
 
  if (httpCode > 0) {
    if (httpCode == HTTP_CODE_OK) {
      WiFiClient* stream = http.getStreamPtr();
      File file = SPIFFS.open("/flights.xml", FILE_WRITE);
 
      if (!file) {
        Serial.println("Failed to open file for writing");
        return;
      }
 
      uint8_t buffer[1024];
      int bytesRead;
 
      while (http.connected() && (bytesRead = stream->readBytes(buffer, sizeof(buffer))) > 0) {
        file.write(buffer, bytesRead);
      }
 
      file.close();
      Serial.println("File written successfully");
      cleanFileContents(); // Clean the file after writing
    } else {
      Serial.printf("HTTP GET request failed: %s\n", http.errorToString(httpCode).c_str());
    }
  } else {
    Serial.printf("HTTP GET request failed: %s\n", http.errorToString(httpCode).c_str());
  }
 
  http.end();
}

GeSHi © Codebox Plus Extension

And the relevant bit in Python:

Code: [Select all] [Expand/Collapse]

async def fetch_flight_data():
    print("Getting data from API...")
    headers = {
        "Accept": "application/xml",
        "app_id": API_KEY,
        "app_key": APP_KEY
    }
    response = requests.get(API_URL, headers=headers)
    if response.status_code == 200:
        with open(RESPONSE_FILE, "wb") as file:
            while True:
                chunk = response.raw.read(CHUNK_SIZE)
                if not chunk:
                    break
                file.write(chunk)
        print("Data written to file:", RESPONSE_FILE)
    else:
        print("Error fetching data:", response.status_code)

Any assistance would be of great help as I would rather code in C than in Python.

Different results when pulling API data with MicroPython or Arduino IDE

Different results when pulling API data with MicroPython or Arduino IDE

Who is online

About Us

Extra

Information