memcpy somehow zeroing out bytes

[rohansingh]

Has anyone else run into an issue with memcpy somehow replacing bytes with null characters? I seem to be "losing" bytes during a copy.

For example, I have a zero-initialized buffer with a capacity of 512 bytes, with 390 bytes filled with non-null data. I attempt to copy 32 additional bytes onto the buffer like this:

Code: [Select all] [Expand/Collapse]

1.   // preserve original contents of *out* for debugging
2.   char* preMemcpy = malloc(512);
3.   memcpy(preMemcpy, out, 390);
4.  
5.   // copy new data into *out*
6.   memcpy(out + 390, newData, 32);

GeSHi © Codebox Plus Extension

After that, I have a breakpoint that gets hit if there are any null characters in the out buffer. The breakpoint is hit sometimes, but not always.When the breakpoint is hit, I'm using the debugger to inspect everything.

The last time this happened, out had 388 bytes of data that matched preMemcpy (instead of 390 bytes), followed by two null characters, followed by all 32 bytes from newData. So somehow, two bytes were "deleted".

What's even more alarming is that this is happening in the bytes prior to the block provided to memcpy. Even though I'm specifying out + 390, it's bytes 388 and 389 that are being replaced by null characters.

This is happening in memory that should be byte-addressable, since it's allocated using:

Code: Select all

heap_caps_malloc(byte_count, MALLOC_CAP_SPIRAM|MALLOC_CAP_8BIT)

Has anyone seen anything like this? Any ideas on how to troubleshoot it further?

[ESP_Angus]

Hi Rohan,

Do any of the memory areas in a single memcpy() call overlap?

Could you please let us know the addresses of the 3 pointers preMemcpy, out, newData?


Angus

[rohansingh]

There is definitely no overlap. I also tried replacing memcpy with memmove and hit the same issue.

This actually only occurs intermittently. I'll try to catch it in debug mode again tomorrow and I'll post the affected memory addresses.

[rohansingh]

OK, it looks like I can get this to recur pretty frequently when I put the device under a lot of load. This code is part of a loop that copies incoming MQTT messages into a destination buffer, up to 32 bytes at a time.

This time, the out buffer started with 611 bytes of data, zero-padded to 1024 bytes total. We attempted to copy 32 bytes from newData to out + 131. All 32 bytes were copied, but the 3 bytes of out prior to the copy point were zeroed out.

Could you please let us know the addresses of the 3 pointers preMemcpy, out, newData?

I can do you better, here are the entire contents of each.

Code: [Select all] [Expand/Collapse]

1. // preMemcpy + 1024
2.   Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F  
3. 3ffec9d0:             7B 22 61 70 70 49 44 22 3A 22 47 79       {"appID":"Gy
4. 3ffec9e0: 41 47 34 50 6F 39 4B 4A 7A 42 32 63 39 37 44 7A   AG4Po9KJzB2c97Dz
5. 3ffec9f0: 79 48 4E 59 6B 38 61 58 65 35 55 75 46 68 57 42   yHNYk8aXe5UuFhWB
6. 3ffeca00: 55 65 57 64 61 44 48 54 6F 22 2C 22 73 63 72 65   UeWdaDHTo","scre
7. 3ffeca10: 65 6E 73 22 3A 5B 22 69 56 42 4F 52 77 30 4B 47   ens":["iVBORw0KG
8. 3ffeca20: 67 6F 41 41 41 41 4E 53 55 68 45 55 67 41 41 41   goAAAANSUhEUgAAA
9. 3ffeca30: 45 41 41 41 41 41 67 43 41 49 41 41 41 41 74 2F   EAAAAAgCAIAAAAt/
10. 3ffeca40: 2B 6E 54 41 41 41 41 75 30 6C 45 51 56 52 34 6E   +nTAAAAu0lEQVR4n
11. 3ffeca50: 4F 79 57 77 51 34 44 49 51 68 45 61 39 50 2F 2F   OyWwQ4DIQhEa9P//
12. 3ffeca60: 32 56 36 4D 44 57 45 46 42 77 33 62 6D 5A 4E 35   2V6MDWEFBw3bmZN5
13. 3ffeca70: 70 30 38 44 4F 41 55 73 50 74 2B 48 59 34 4D 73   p08DOAUsPt+HY4Ms
14. 3ffeca80: 4A 45 42 4E 6A 4C 41 52 67 62 59 79 41 43 62 44   JEBNjLARgbYyACbD
15. 3ffeca90: 79 49 79 73 33 35 6F 72 65 47 61 61 64 51 51 44   yIys35oreGaadQQD
16. 3ffecaa0: 49 32 5A 2B 66 43 69 33 41 4B 2B 6A 44 2F 58 6D   I2Z+fCi3AK+jD/Xm
17. 3ffecab0: 76 75 69 41 74 64 48 43 43 78 51 39 79 31 4C 69   vuiAtdHCCxQ9y1Li
18. 3ffecac0: 79 65 66 47 2F 68 62 50 70 54 5A 30 2B 74 66 71   yefG/hbPpTZ0+tfq
19. 3ffecad0: 71 58 68 67 58 61 67 34 2F 4E 6D 5A 63 49 51 44   qXhgXag4/NmZcIQD
20. 3ffecae0: 33 47 52 63 36 72 5A 41 39 4C 54 62 49 4B 52 48   3GRc6rZA9LTbIKRH
21. 3ffecaf0: 63 43 6A 41 74 41 4F 49 44 33 64 39 6D 67 73 4D   cCjAtAOID3d9mgsM
22. 3ffecb00: 6A 65 51 33 53 7A 38 57 70 54 62 4C 2F 38 50 2B   jeQ3Sz8WpTbL/8P+
23. 3ffecb10: 41 63 37 6D 2F 69 75 36 55 76 69 51 78 34 48 2F   Ac7m/iu6UviQx4H/
24. 3ffecb20: 74 4B 4A 69 75 4F 2F 68 57 53 41 6A 51 79 77 6B   tKJiuO/hWSAjQywk
25. 3ffecb30: 51 45 32 4D 73 44 6D 65 41 50 66 41 41 41 41 2F   QE2MsDmeAPfAAAA/
26. 3ffecb40: 2F 39 55 61 49 63 47 46 6D 45 6E 73 41 41 41 41   /9UaIcGFmEnsAAAA
27. 3ffecb50: 41 42 4A 52 55 35 45 72 6B 4A 67 67 67 3D 3D 22   ABJRU5ErkJggg=="
28. 3ffecb60: 2C 22 69 56 42 4F 52 77 30 4B 47 67 6F 41 41 41   ,"iVBORw0KGgoAAA
29. 3ffecb70: 41 4E 53 55 68 45 55 67 41 41 41 45 41 41 41 41   ANSUhEUgAAAEAAAA
30. 3ffecb80: 41 67 43 41 49 41 41 41 41 74 2F 2B 6E 54 41 41   AgCAIAAAAt/+nTAA
31. 3ffecb90: 41 41 70 30 6C 45 51 56 52 34 6E 4F 79 57 77 51   AAp0lEQVR4nOyWwQ
32. 3ffecba0: 37 44 49 41 78 44 59 64 72 2F 2F 33 4A 32 51 45   7DIAxDYdr//3J2QE
33. 3ffecbb0: 4F 49 71 6F 6B 6A 55 64 78 57 66 71 63 65 6E 4D   OIqokjUdxWfqcenM
34. 3ffecbc0: 4B 44 5A 4F 75 6E 50 42 77 4A 73 4A 45 41 47 77   KDZOunPBwJsJEAGw
35. 3ffecbd0: 6D 77 6B 51 41 62 43 62 44 35 49 69 45 7A 61 77   mwkQAbCbD5IiEzaw
36. 3ffecbe0: 2B 31 56 6A 77 54 56 76 56 41 7A 35 6A 5A 57 4F   +1VjwTVvVAz5jZWO
37. 3ffecbf0: 34 73 6C 32 42 63 5A 6E 7A 32 4D 39 64 56 54 65   4sl2BcZnz2M9dVTe
38. 3ffecc00: 78 6F 6F 64 52 42 48 6B 31 38 59 67 46 6B 2B 54   xoodRBHk18YgFk+T
39. 3ffecc10: 56 33 2F 58 39 56 71 6E 6D 67 47 57 67 67 37 35   V3/X9VqnmgGWgg75
40. 3ffecc20: 32 61 75 4F 2F 4A 79 59 65 5A 4E 53 42 33 65 74   2auO/JyYeZNSB3et
41. 3ffecc30: 62 42 79 41 7A 67 56 00 00 00 00 00 00 00 00 00   bByAzgV.........
42. 3ffecc40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
43. 3ffecc50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
44. 3ffecc60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
45. 3ffecc70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
46. 3ffecc80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
47. 3ffecc90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
48. 3ffecca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
49. 3ffeccb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
50. 3ffeccc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
51. 3ffeccd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
52. 3ffecce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
53. 3ffeccf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
54. 3ffecd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
55. 3ffecd10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
56. 3ffecd20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
57. 3ffecd30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
58. 3ffecd40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
59. 3ffecd50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
60. 3ffecd60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
61. 3ffecd70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
62. 3ffecd80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
63. 3ffecd90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
64. 3ffecda0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
65. 3ffecdb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
66. 3ffecdc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
67. 3ffecdd0: 00 00 00 00                                       ....
68.  
69. // out + 1024
70.   Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F  
71. 3f813970:                         7B 22 61 70 70 49 44 22           {"appID"
72. 3f813980: 3A 22 47 79 41 47 34 50 6F 39 4B 4A 7A 42 32 63   :"GyAG4Po9KJzB2c
73. 3f813990: 39 37 44 7A 79 48 4E 59 6B 38 61 58 65 35 55 75   97DzyHNYk8aXe5Uu
74. 3f8139a0: 46 68 57 42 55 65 57 64 61 44 48 54 6F 22 2C 22   FhWBUeWdaDHTo","
75. 3f8139b0: 73 63 72 65 65 6E 73 22 3A 5B 22 69 56 42 4F 52   screens":["iVBOR
76. 3f8139c0: 77 30 4B 47 67 6F 41 41 41 41 4E 53 55 68 45 55   w0KGgoAAAANSUhEU
77. 3f8139d0: 67 41 41 41 45 41 41 41 41 41 67 43 41 49 41 41   gAAAEAAAAAgCAIAA
78. 3f8139e0: 41 41 74 2F 2B 6E 54 41 41 41 41 75 30 6C 45 51   AAt/+nTAAAAu0lEQ
79. 3f8139f0: 56 52 34 6E 4F 79 57 77 51 34 44 49 51 68 45 61   VR4nOyWwQ4DIQhEa
80. 3f813a00: 39 50 2F 2F 32 56 36 4D 44 57 45 46 42 77 33 62   9P//2V6MDWEFBw3b
81. 3f813a10: 6D 5A 4E 35 70 30 38 44 4F 41 55 73 50 74 2B 48   mZN5p08DOAUsPt+H
82. 3f813a20: 59 34 4D 73 4A 45 42 4E 6A 4C 41 52 67 62 59 79   Y4MsJEBNjLARgbYy
83. 3f813a30: 41 43 62 44 79 49 79 73 33 35 6F 72 65 47 61 61   ACbDyIys35oreGaa
84. 3f813a40: 64 51 51 44 49 32 5A 2B 66 43 69 33 41 4B 2B 6A   dQQDI2Z+fCi3AK+j
85. 3f813a50: 44 2F 58 6D 76 75 69 41 74 64 48 43 43 78 51 39   D/XmvuiAtdHCCxQ9
86. 3f813a60: 79 31 4C 69 79 65 66 47 2F 68 62 50 70 54 5A 30   y1LiyefG/hbPpTZ0
87. 3f813a70: 2B 74 66 71 71 58 68 67 58 61 67 34 2F 4E 6D 5A   +tfqqXhgXag4/NmZ
88. 3f813a80: 63 49 51 44 33 47 52 63 36 72 5A 41 39 4C 54 62   cIQD3GRc6rZA9LTb
89. 3f813a90: 49 4B 52 48 63 43 6A 41 74 41 4F 49 44 33 64 39   IKRHcCjAtAOID3d9
90. 3f813aa0: 6D 67 73 4D 6A 65 51 33 53 7A 38 57 70 54 62 4C   mgsMjeQ3Sz8WpTbL
91. 3f813ab0: 2F 38 50 2B 41 63 37 6D 2F 69 75 36 55 76 69 51   /8P+Ac7m/iu6UviQ
92. 3f813ac0: 78 34 48 2F 74 4B 4A 69 75 4F 2F 68 57 53 41 6A   x4H/tKJiuO/hWSAj
93. 3f813ad0: 51 79 77 6B 51 45 32 4D 73 44 6D 65 41 50 66 41   QywkQE2MsDmeAPfA
94. 3f813ae0: 41 41 41 2F 2F 39 55 61 49 63 47 46 6D 45 6E 73   AAA//9UaIcGFmEns
95. 3f813af0: 41 41 41 41 41 42 4A 52 55 35 45 72 6B 4A 67 67   AAAAABJRU5ErkJgg
96. 3f813b00: 67 3D 3D 22 2C 22 69 56 42 4F 52 77 30 4B 47 67   g==","iVBORw0KGg
97. 3f813b10: 6F 41 41 41 41 4E 53 55 68 45 55 67 41 41 41 45   oAAAANSUhEUgAAAE
98. 3f813b20: 41 41 41 41 41 67 43 41 49 41 41 41 41 74 2F 2B   AAAAAgCAIAAAAt/+
99. 3f813b30: 6E 54 41 41 41 41 70 30 6C 45 51 56 52 34 6E 4F   nTAAAAp0lEQVR4nO
100. 3f813b40: 79 57 77 51 37 44 49 41 78 44 59 64 72 2F 2F 33   yWwQ7DIAxDYdr//3
101. 3f813b50: 4A 32 51 45 4F 49 71 6F 6B 6A 55 64 78 57 66 71   J2QEOIqokjUdxWfq
102. 3f813b60: 63 65 6E 4D 4B 44 5A 4F 75 6E 50 42 77 4A 73 4A   cenMKDZOunPBwJsJ
103. 3f813b70: 45 41 47 77 6D 77 6B 51 41 62 43 62 44 35 49 69   EAGwmwkQAbCbD5Ii
104. 3f813b80: 45 7A 61 77 2B 31 56 6A 77 54 56 76 56 41 7A 35   Ezaw+1VjwTVvVAz5
105. 3f813b90: 6A 5A 57 4F 34 73 6C 32 42 63 5A 6E 7A 32 4D 39   jZWO4sl2BcZnz2M9
106. 3f813ba0: 64 56 54 65 78 6F 6F 64 52 42 48 6B 31 38 59 67   dVTexoodRBHk18Yg
107. 3f813bb0: 46 6B 2B 54 56 33 2F 58 39 56 71 6E 6D 67 47 57   Fk+TV3/X9VqnmgGW
108. 3f813bc0: 67 67 37 35 32 61 75 4F 2F 4A 79 59 65 5A 4E 53   gg752auO/JyYeZNS
109. 3f813bd0: 42 33 65 74 62 42 79 41 00 00 00 52 50 51 44 47   B3etbByA...RPQDG
110. 3f813be0: 54 50 66 69 65 78 77 4A 31 33 6E 2F 34 66 77 48   TPfiexwJ13n/4fwH
111. 3f813bf0: 2F 55 32 79 7A 36 4A 61 4B 38 34 00 00 00 00 00   /U2yz6JaK84.....
112. 3f813c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
113. 3f813c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
114. 3f813c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
115. 3f813c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
116. 3f813c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
117. 3f813c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
118. 3f813c60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
119. 3f813c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
120. 3f813c80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
121. 3f813c90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
122. 3f813ca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
123. 3f813cb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
124. 3f813cc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
125. 3f813cd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
126. 3f813ce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
127. 3f813cf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
128. 3f813d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
129. 3f813d10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
130. 3f813d20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
131. 3f813d30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
132. 3f813d40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
133. 3f813d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
134. 3f813d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
135. 3f813d70: 00 00 00 00 00 00 00 00                           ........
136.  
137. // newData + 32
138.   Offset: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F  
139. 3f80dd20:                                     52 50 51 44               RPQD
140. 3f80dd30: 47 54 50 66 69 65 78 77 4A 31 33 6E 2F 34 66 77   GTPfiexwJ13n/4fw
141. 3f80dd40: 48 2F 55 32 79 7A 36 4A 61 4B 38 34               H/U2yz6JaK84

GeSHi © Codebox Plus Extension

The corrupted bytes in out start near 3f813bd0.

[rohansingh]

@ESP_Angus, do you think this is due to the PSRAM cache issue reported here:
https://github.com/espressif/esp-idf/issues/2892

Looking around, I found a similar report of string corruption here:
https://github.com/espressif/esp-idf/issues/3006

I'm also noticing that I can't repro the issue if I use MALLOC_CAP_INTERNAL.

[ESP_Angus]

Yes, I think you're right that this is a variant of those bugs.

Just for one more data point, what ESP-IDF version are you using?

[PeterR]

HI Angus,
The link is a detailed read & I am not sure what the corrolary is (&| is the jury still out?).

I use PSRAM. What should I do to ensure proper operation?
Is there a full problem statement (rather than discussion)?

Thanks

[ESP_Sprite]

The current state is that we arrived at a proper fix for this issue, but it's not in mainline yet. We're working on that. For the time being, you can try the toolchain that is linked in the discussion; that should solve the issue as well.

[rohansingh]

Yes, I think you're right that this is a variant of those bugs.

Just for one more data point, what ESP-IDF version are you using?

I'm using a fork of v3.3. It is just v3.3 with two commits backported onto it from:
https://github.com/espressif/esp-idf/issues/3592

Here is the exact branch that I am using:
https://github.com/tidbyt/esp-idf/commits/master

For what it's worth, I found that the issue is much more frequent without the linker flag for the PSRAM issue workaround. So it does all seem related.