ESP32 CAM direct access to image data

[alanesq]

Hello, I am carrying out a project in which I apply a beam of light over the camera and I would like to take the values ​​in rgb of pixel face, I would just like that, without all those other features of the ESP32 Cam, I am very lay in programming, will it be that could someone help me with this.

Hi,

I have been having a play with this myself recently (i.e. reading RGB data) and I think I have figured out how to access the RGB data of an image without it getting too complicated.

I have included it in a sketch here which may be of interest: https://github.com/alanesq/esp32cam-demo
if you look at the procedure 'readRGBImage()' this captures a live image, converts it to RGB then shows the first few bytes of data via the serial port.

I have not tested it but this should work with all but the highest resolution camera images as the RGB data is stored in the 4mb psram chip on the esp32cam modules (the highest resolution would require more memory than this to store the RGB data).
BTW-You can check how much free space there is with the command: Serial.println(heap_caps_get_free_size( MALLOC_CAP_SPIRAM));

The relevant code:

Code: [Select all] [Expand/Collapse]

1.  
2.   uint32_t resultsToShow = 50;     // how much data to display
3.    
4.   String tReply = "LIVE IMAGE AS RGB DATA: ";      // reply to send to web client and serial port
5.  
6.   // capture live image (jpg)
7.     camera_fb_t * fb = NULL;
8.     fb = esp_camera_fb_get();  
9.     if (!fb) tReply+=" -Error capturing image from camera- ";  
10.     tReply+="(Image resolution=" + String(fb->width) + "x" + String(fb->height) + ")";         // display image resolution        
11.    
12.   // allocate memory to store rgb data in psram
13.     if (!psramFound()) tReply+=" -Error no psram found- ";
14.     void *ptrVal = NULL;
15.     uint32_t ARRAY_LENGTH = fb->width * fb->height * 3;               // number of pixels in the jpg image x 3
16.     if (heap_caps_get_free_size( MALLOC_CAP_SPIRAM) <  ARRAY_LENGTH) tReply+=" -Error not enough free psram to store rgb data- ";      // check free memory in psram
17.     ptrVal = heap_caps_malloc(ARRAY_LENGTH, MALLOC_CAP_SPIRAM);  
18.     uint8_t *rgb = (uint8_t *)ptrVal;
19.  
20.   // convert jpg to rgb (store in an array 'rgb')
21.     bool jpeg_converted = fmt2rgb888(fb->buf, fb->len, PIXFORMAT_JPEG, rgb);    
22.     if (!jpeg_converted) tReply+=" -Error converting image to RGB- ";
23.  
24.   // display some of the resulting data
25.       for (uint32_t i = 0; i < resultsToShow; i++) {
26.         // // x and y coordinate of the pixel
27.         //   uint16_t x = i % fb->width;
28.         //   uint16_t y = floor(i / fb->width);
29.         if (i%3 == 0) tReply+="B";
30.         else if (i%3 == 1) tReply+="G";
31.         else if (i%3 == 2) tReply+="R";
32.         tReply+= String(rgb[i]) + ",";
33.       }
34.      
35.   // free up memory
36.     esp_camera_fb_return(fb);
37.     heap_caps_free(ptrVal);
38.  
39.   Serial.println(tReply);
40.  

GeSHi © Codebox Plus Extension

btw: I got most of my info from: https://github.com/Makerfabs/Project_To ... era_v2.ino

[filolipe]

thank you very much, your post was of great help, thanks to him I now have a good course to follow.

[alanesq]

thank you very much, your post was of great help, thanks to him I now have a good course to follow.

Glad it helped

BTW-I have been experimenting some more and can confirm that it works as I modified my sketch to send the RGB data over the serial port which I then collected and fed in to PROCESSING where I was able to reconstruct the image

Another option if speed is an issue is to run the camera in RGB565 mode. I have not tried it myself but this looks like some good info on how.
https://eloquentarduino.github.io/2020/ ... d-arduino/

[StefOlivier]

Hello everyBody and more thanks for alanesq

(sorry i'm french)

It was very usefull.

Have a nice day

For contribute, I post what I changed:

Code: Select all

// Move this in Init
WiFiClient client;

// And in void readRGBImage() change return of image
    client = server.client(); 
    delay(3);
    client.write(rgb, ARRAY_LENGTH);
    delay(3);
    client.stop();

[alanesq]

Hello everyBody and more thanks for alanesq

Thanks

As it happens I have just finished re-writing my code this morning to use "client = server.client(); "
so your suggestion of "client.write(rgb, ARRAY_LENGTH);" will be very easy to include and I will add this information to my sketch

It would have made my experiments yesterday much easier if I had realised you can send the whole image over this easily...

---------------

UPDATE: Yes, I was right about it being easier - lol
I have put a PROCESSING file to display the raw RGB file in case it is of use/interest here:
https://github.com/alanesq/esp32cam-dem ... layRGB.pde

[StefOlivier]

Very happy to be useful.

[nataly]

I, too, am trying to directly access the captured image and manipulate the pixel data in it, without success.

I've grabbed a frame in PIXFORMAT_RGB888 at FRAMESIZE_UXGA (i.e. 1600x1200) and I can indeed get it and save it to my SD card and then bring it up in an imaging program and view it. So, getting, saving, and displaying the frame is not a problem.

It's when I try to manipulate the pixel data before saving it out to the SD card that there's a problem. The resultant saved image is invalid and cannot be viewed in my imaging software (paint).

Presumably the data, for the RGB888 format, is in the buffer as a sequence of Image Rows (scanlines), top to bottom, each Image Row consisting of a sequence of Pixels, left to right, and each Pixel consisting of 3 sequential bytes, one each for Red, Blue, and Green, in that order.

Here's the code I using for testing purposes. What it's attempting to do is draw a RED line horizontally across the image, every tenth row (scan line), just so I can see if I'm manipulating the pixel data properly.

Code: Select all

     
      // Access the data as a series of uint8_t
      uint8_t *StartOfFrameBufferAsSequenceOfBytes = (uint8_t *)FrameBuffer->buf;

      size_t NumberOfBytesInFrameBuffer = FrameBuffer->len;

      size_t NumberOfImageRows = 1200;
      size_t NumberOfImagePixelsPerRow = 1600;
      size_t NumberOfBytesPerPixel = 3;
      // (width * height * bytes per pixel)
      // In other words it's a sequence of Image Rows (scanlines), top to bottom.
      // Each Image Row consists of a sequence of Pixels, left to right.
      // Each Pixel consists of 3 bytes, one each for Red, Blue, and Green. 
      // TRY writing a RED line of pixels every tenth row.
      uint8_t *StartOfRowInFrameBufferAsSequenceOfBytes;
      uint8_t *StartOfCurrentPixelInRow;
      uint8_t *CurrentPixelInRow;
      for (size_t RowNumber = 0 ; RowNumber < 1 /*NumberOfImageRows */ ; RowNumber += 10)
      {
        StartOfRowInFrameBufferAsSequenceOfBytes = StartOfFrameBufferAsSequenceOfBytes + (RowNumber * NumberOfImagePixelsPerRow * NumberOfBytesPerPixel);
        StartOfCurrentPixelInRow = StartOfRowInFrameBufferAsSequenceOfBytes;
        for (size_t PixelCounter = 0; PixelCounter < NumberOfImagePixelsPerRow ; ++PixelCounter , StartOfCurrentPixelInRow += NumberOfBytesPerPixel)
        {
          CurrentPixelInRow = StartOfCurrentPixelInRow;
          uint8_t ExistingByte = *CurrentPixelInRow;
          *CurrentPixelInRow = 255;
          ++CurrentPixelInRow;  
          *CurrentPixelInRow = 0;  
          ++CurrentPixelInRow;  
          *CurrentPixelInRow = 0;  
        }
      }

The code runs, but when I subsequently save the image to the SD card, move the card to my computer, and try to view it, nothing shows up. In fact, it indicates that the file format is invalid.

Even if all I do is try to zero out JUST the very first byte in the buffer, which presumably is the Red byte of the upper left-hand corner byte of the image, the resultant image file format ends up being invalid.

What am I doing wrong?

Can you give me the all the code that take capture and get the value of pixel ? please Im very need for that !

[Ihor5999]

Hello, Could you please advise viewer for the specific grayscale image format of ESP32CAM. I tried some popular viewers, they do not recognize those images.
Thank you.
----------------------------
config.pixel_format = PIXFORMAT_GRAYSCALE;
config.frame_size = FRAMESIZE_SXGA;
config.jpeg_quality = 10;
config.fb_count = 1;