ESP-IDF precision of double limited to 7 decimal places?

[xwbxwb]

Dear all,

I am working on a project that involves handling latitude and longitude information, and require a precision of at least 8 decimal places.
It seems like doubles are strictly limited to 7 decimal places in ESP-IDF? Is there anyway to bypass this?
In the following code,
1: we take in a latitude value in string format (p) that has been multiplied by 100 (this is the format passed on via UART from the gps module)
->eg: 2219.00753
2: we convert this from a list of characters (string) to double using atof(2219.00753)
->this is converted correctly and there is no loss of precison (output: 2219.00753)
3: I have to move the decimal places 2 spaces forwards (as latitude should be a value from -90 to +90, not -9000 to +9000)
-> expected output: 22.1900753, actual output: 22.190075
I have made sure to log to the terminal using %lf (ie. double) and that all the variables involved here (temp_lat) is defined as double. Is this a limitation of ESP-IDF and if so, is there anyway to override it? I must have at least 7d.p. to have the required resolution of around 1cm.

Code: Select all

temp_lat=atof(p)/100;
ESP_LOGI("gps decode v2", "what is p for lat %s, what is atof(p) %lf, atof(p)/100 %10lf, templat %lf",p,atof(p),atof(p)/100,temp_lat);

Many thanks for your help.
-Clement

[postpiet]

Code: Select all

ESP_LOGI("gps decode v2", "what is p for lat %s, what is atof(p) %.7f, atof(p)/100 %10lf, templat %lf",p,atof(p),atof(p)/100,temp_lat);

%.7f and %.7lf is the same. If the conversion from string to double is good than it won't be necessary to divide the number to a smaller one.

[MicroController]

Check what the actual value of your double variable is. The base-10 string output may be truncated.

[AntonioTesta]

I think the problem is not the decimal places but the total digits involved (integer part plus decimal part) that seams to be limited to 11 or 12 in total. I do not know about IDF but using Arduino IDE I had the following results:


float QI=17.0/3.0;
float RI=999999.999999/3.111;

void setup() {
Serial.begin(115200);
}

void loop() {
Serial.print("PI=");Serial.println(PI,11);
Serial.print("QI=");Serial.println(QI,11);
Serial.print("RI=");Serial.println(RI,11);
while(1){}
}

Result:

PI=3.14159265359
QI=5.66666650772
RI=321440.06250000000

[xwbxwb]

Check what the actual value of your double variable is. The base-10 string output may be truncated.

Any tips on how to do this? I am already printing long float, and I also sent this value in a http REST response to a debugging portal I built and it has the same result of truncated decimal places. Many thanks!

[xwbxwb]

I think the problem is not the decimal places but the total digits involved (integer part plus decimal part) that seams to be limited to 11 or 12 in total. I do not know about IDF but using Arduino IDE I had the following results:


float QI=17.0/3.0;
float RI=999999.999999/3.111;

void setup() {
Serial.begin(115200);
}

void loop() {
Serial.print("PI=");Serial.println(PI,11);
Serial.print("QI=");Serial.println(QI,11);
Serial.print("RI=");Serial.println(RI,11);
while(1){}
}

Result:

PI=3.14159265359
QI=5.66666650772
RI=321440.06250000000

Hmm. Interesting. It was able to display the value when it is 100 times larger though (decimal point shifted to the right by 2 digits so this is unlikely?

[MicroController]

Check what the actual value of your double variable is. The base-10 string output may be truncated.

Any tips on how to do this? I am already printing long float, and I also sent this value in a http REST response to a debugging portal I built and it has the same result of truncated decimal places. Many thanks!

Given that only the decimal places will be truncated, you can check the string output of temp_lat * 1000 for example to confirm that the value is actually still correct.

I also was about to suggest looking into using int32_t and fixed-point math for speed, precision, and control over rounding; sign + 9 base-10 digits fit nicely into an int32_t. But assuming that you probably won't process thousands of GPS inputs per second and that you may want to use trig functions anyway, this is likely not be worth the effort.

For 'full-precision' string output of doubles you may have to resort to fixed-point though, like int32_t intPart = (int32_t)temp_lat; uint32_t fractPart = (uint32_t)(abs((temp_lat - intPart)) * 1000000); and then output intPart + "." + (0-padded fractPart) ("%06i" may work for 0-padding).