ESP32 CAM direct access to image data

[XenonXenon]

Anyone?

[HermannSW]

But I cannot find the code for dl_matrix3du_alloc anywhere and whilst the calling context makes clear that parameters 2 & 3 are width & height I do not know what parms 1 & 4 are.

From:
https://github.com/espressif/esp-face/b ... ix3d.h#L68

Code: Select all

...
/*
 * @brief Allocate a 3D matrix with 8-bits items, the access sequence is NHWC
 *
 * @param n     Number of matrix3d, for filters it is out channels, for others it is 1
 * @param w     Width of matrix3d
 * @param h     Height of matrix3d
 * @param c     Channel of matrix3d
 * @return      3d matrix
 */
dl_matrix3du_t *dl_matrix3du_alloc(int n, int w, int h, int c);
...

On your question on direct access to image data, in capture_handler():
https://github.com/espressif/arduino-es ... d.cpp#L218

this is how you get rgb888 encoded frame that you can directly access (CameraWebServer "Get Still" requests JPEG, "Start Stream" requests for MJPEG stream):

Code: Select all

...
    fb = esp_camera_fb_get();
...
    s = fmt2rgb888(fb->buf, fb->len, fb->format, out_buf);
...

From:
https://github.com/espressif/esp32-came ... ers.h#L110

Code: Select all

...
/**
 * @brief Convert image buffer to RGB888 buffer (used for face detection)
 *
 * @param src       Source buffer in JPEG, RGB565, RGB888, YUYV or GRAYSCALE format
 * @param src_len   Length in bytes of the source buffer
 * @param format    Format of the source image
 * @param rgb_buf   Pointer to the output buffer (width * height * 3)
 *
 * @return true on success
 */
bool fmt2rgb888(const uint8_t *src_buf, size_t src_len, pixformat_t format, uint8_t * rgb_buf);
...

[mrstanlez]

Hello,

I have problem with JPEG buffer to RGB data. I cant read red pixels with SVGA / XGA resolutions, if JPEG Format is captured into framebuffer. And another problem is that PSRAM is always lower and after 4th captured picture my module is restarted. Below code don't works. Any help ? Thank you.

Code: Select all

uint8_t *out_buf = (uint8_t*)ps_malloc(out_len);
fmt2rgb888(fb->buf, fb->len, fb->format, out_buf);  
esp_camera_fb_return(fb);
...
free(out_buf);

[RogerInHawaii]

I, too, am trying to directly access the captured image and manipulate the pixel data in it, without success.

I've grabbed a frame in PIXFORMAT_RGB888 at FRAMESIZE_UXGA (i.e. 1600x1200) and I can indeed get it and save it to my SD card and then bring it up in an imaging program and view it. So, getting, saving, and displaying the frame is not a problem.

It's when I try to manipulate the pixel data before saving it out to the SD card that there's a problem. The resultant saved image is invalid and cannot be viewed in my imaging software (paint).

Presumably the data, for the RGB888 format, is in the buffer as a sequence of Image Rows (scanlines), top to bottom, each Image Row consisting of a sequence of Pixels, left to right, and each Pixel consisting of 3 sequential bytes, one each for Red, Blue, and Green, in that order.

Here's the code I using for testing purposes. What it's attempting to do is draw a RED line horizontally across the image, every tenth row (scan line), just so I can see if I'm manipulating the pixel data properly.

Code: Select all

     
      // Access the data as a series of uint8_t
      uint8_t *StartOfFrameBufferAsSequenceOfBytes = (uint8_t *)FrameBuffer->buf;

      size_t NumberOfBytesInFrameBuffer = FrameBuffer->len;

      size_t NumberOfImageRows = 1200;
      size_t NumberOfImagePixelsPerRow = 1600;
      size_t NumberOfBytesPerPixel = 3;
      // (width * height * bytes per pixel)
      // In other words it's a sequence of Image Rows (scanlines), top to bottom.
      // Each Image Row consists of a sequence of Pixels, left to right.
      // Each Pixel consists of 3 bytes, one each for Red, Blue, and Green. 
      // TRY writing a RED line of pixels every tenth row.
      uint8_t *StartOfRowInFrameBufferAsSequenceOfBytes;
      uint8_t *StartOfCurrentPixelInRow;
      uint8_t *CurrentPixelInRow;
      for (size_t RowNumber = 0 ; RowNumber < 1 /*NumberOfImageRows */ ; RowNumber += 10)
      {
        StartOfRowInFrameBufferAsSequenceOfBytes = StartOfFrameBufferAsSequenceOfBytes + (RowNumber * NumberOfImagePixelsPerRow * NumberOfBytesPerPixel);
        StartOfCurrentPixelInRow = StartOfRowInFrameBufferAsSequenceOfBytes;
        for (size_t PixelCounter = 0; PixelCounter < NumberOfImagePixelsPerRow ; ++PixelCounter , StartOfCurrentPixelInRow += NumberOfBytesPerPixel)
        {
          CurrentPixelInRow = StartOfCurrentPixelInRow;
          uint8_t ExistingByte = *CurrentPixelInRow;
          *CurrentPixelInRow = 255;
          ++CurrentPixelInRow;  
          *CurrentPixelInRow = 0;  
          ++CurrentPixelInRow;  
          *CurrentPixelInRow = 0;  
        }
      }

The code runs, but when I subsequently save the image to the SD card, move the card to my computer, and try to view it, nothing shows up. In fact, it indicates that the file format is invalid.

Even if all I do is try to zero out JUST the very first byte in the buffer, which presumably is the Red byte of the upper left-hand corner byte of the image, the resultant image file format ends up being invalid.

What am I doing wrong?

[5290@gmx.de]

This is exactly what I'm looking for.

Can anyone post me a sample to make e.g. from top left the first pixel always RED, the second BLUE, the third YELLOW...

That would help me so much!

Thank you!

[RogerInHawaii]

I've been doing a LOT of testing on this. In my post, far above, I noted that I was able to get an RGB888 formatted image frame at the highest resolution, FRAMESIZE_UXGA. Well, it turns out that that is not actually true. Per the examples that are available I initially do this:

Code: Select all

  config.pixel_format = PIXFORMAT_JPEG;
  if(psramFound())
  {
    config.frame_size = FRAMESIZE_UXGA;
    config.jpeg_quality = 10 ; // RCG initially 10; //0-63 lower number means higher quality
    config.fb_count = 2;
  }
  else
  {
    config.frame_size = FRAMESIZE_UXGA;
    config.jpeg_quality = 12 ; // RCG initially 12;//0-63 lower number means higher quality
    config.fb_count = 1;
  }
  esp_err_t err = esp_camera_init(&config);

and then later, when I actually am trying to acquire the image:

Code: Select all

    s->pixformat = PIXFORMAT_RGB888;
    camera_fb_t * FrameBuffer = esp_camera_fb_get();

Which does indeed return a frame buffer with image data in it. The problem is that it is NOT in PIXFORMAT_RGB888, as I had specified in the

Code: Select all

s->pixformat = PIXFORMAT_RGB888;

Instead it's in PIXFORMAT_JPEG!!! So any attempt at accessing individual pixel data in that frame, expecting it to be in PIXFORMAT_RGB888, will fail. Worse, any attempt at MODIFYING individual pixel data in that frame, expecting it to be in PIXFORMAT_RGB888, will CORRUPT the image data.

It would seem that

Code: Select all

s->pixformat = PIXFORMAT_RGB888;

actually DOES NOTHING! The format of the returned image frame will be whatever you had specified during the initial config of the camera.

My question, now is "WHAT THE HECK IS THE PURPOSE of setting the pixformat field with a

Code: Select all

s->pixformat = PIXFORMAT_RGB888;

????"

I've encountered a number of examples and comments where it says "the recommended way of getting BITMAP data is to retrieve in JPEG format and then using the JPGtoRGB function to convert it". But why? Why can't I just get it in BITMAP format to begin with? One big issue with using the convert approach is that it has to have BOTH the JPEG data AND the BMP data in memory at the same time, and there's generally not enough memory to hold it all. As far as I can tell the best you can hope for is grabbing the very smallest frame (e.g. FRAMESIZE_QVGA) of JPEG image and then converting to BITMAP. Anything larger and you encounter a memory failure.

So my attempts at getting PIXFORMAT_RGB888 data FRAMESIZE_UXGA will always fail. It's just not possible with the ESP32-CAM because there's only 4MBytes of accessible memory. The thing is, the standard ESP32 (not CAM) can be configured for up to EIGHT MBytes of memory, which would indeed be enough able to handle the largest possible bitmap. But someone chose to only put 4MBytes of memory on it. It just boggles the mind!

So it seems that if you want to directly access or manipulate BITMAP data you have to initially config it for BITMAP data and restrict your image size to the very smallest of sizes.

[ESP_Sprite]

What is your 's' pointer? Is it the config struct? If so, how can you possibly imagine the code will pick up that change after you've already told it to configure the sensor?

In general, your ESP-CAM may actually have 8MiB of RAM on it. The issue here is more an architectural one: the ESP32 can only access up to 4MiB of RAM at the same time without dirty tricks (himem).

(Note that even if you were able to save the full 5M image into SPI flash, the process may have been slow enough for the CCD cells to lose their charge during readout, so I don't think memory would have been your major enemy there.)

[RogerInHawaii]

The 's' pointer is this:

Code: Select all

sensor_t * s = esp_camera_sensor_get();

The setting of the pixel format, as in

Code: Select all

s->pixformat = PIXFORMAT_JPEG;

is taken directly from example code provided by the download of the ESP32-CAM software into the Arduino IDE. From that example code it APPEARS that you need to set an initial format and image size in order for it to allocate initial image memory, as in

Code: Select all

config.pixel_format = PIXFORMAT_JPEG;
config.frame_size = InitialFrameSize; // originally FRAMESIZE_UXGA;
esp_err_t err = esp_camera_init(&config);

BUT that you set the ACTUAL desired format and size in preparation for the call to

Code: Select all

sensor_t * s = esp_camera_sensor_get();
s->pixformat = PIXFORMAT_RGB888;
s->set_framesize(s, FRAMESIZE_VGA);
camera_fb_t * FrameBuffer = esp_camera_fb_get();

That's what I conclude from the various examples that are provided. Doing the s->set_framesize call does indeed set the (potentially smaller) framesize that will be grabbed from the camera, but doing the s->pixformat does NOT set the pixel format, instead it always defaults to whatever had been initially set via the config initialization.

My question remains: What is the purpose of the setting of pixformat as in

Code: Select all

s->pixformat = PIXFORMAT_RGB888;

[alanesq]

Hi,

I have experienced similar problems when wanting to switch image formats
The only way I found to do it (which is far from ideal) is to deinit the camera and re configure it

i.e.
esp_camera_deinit();
// change the settings here
esp_camera_init(&config);

[filolipe]

Hello, I am carrying out a project in which I apply a beam of light over the camera and I would like to take the values ​​in rgb of pixel face, I would just like that, without all those other features of the ESP32 Cam, I am very lay in programming, will it be that could someone help me with this.
I even read the comments above, but being a layman at this, I was unable to accomplish anything on top of that.

Thankful.