ESP32 WROVER MP3/AAC/(IP Audio Stream) Decoding, digital Audio Filtering and SPI Colour Display control

[PepeESP32]

Hello guys,

I am aware the ESP32 (Dual Core) can decode mp3 and other streaming audio formats (basically in SW).
Too, I am aware the ESP32 can drive a Colour TFT via SPI (standard 4 lane).

Basically I would like to get your feelings for possibility of parallel operation (MP3 decoding AND display control).
Do you think it is possible to run both, especially as audio needs seemless streaming...
Resolution will be a considered max of 170RGBx320 pix...Content pretty much static, moving text, that's about it.

We intent to use the WROVER module with 8MB RAM and 4MB ROM.

An on-top if both above are possible, we'd like to have some equalizers (bas/mid/treb) and Speaker correction (optional, 5 IIR 2nd order Filters).

Do you think the target to run all on ESP32 is absurd??

Thanks for your opinions!

[ESP_Sprite]

I'd say this should be doable. We do the decoding in our audio platform as well and that seems to work, controlling mostly static imagery on a display isn't that hard, and I think the CPU should have some time for equalizing as well.

[PepeESP32]

Thanks ESP_Sprite.

These indications are a good hint.

Could you let me know what HW (ESP32 EValsystems or Modules) you use for it?

In the meantime I have found the audio development framework section, which seems to answer a lot of questions. But usually these are single functions (like MP3 etc decoding OR 10 band EQ) but it is hard to predict how a System behaves when together with that some higher BW Display content is processed in Parallel

Thanks you!

Christian

[ESP_Sprite]

We use the ADF SDK you already found in combination with our Lyra devboards. If you want to see if this works and experiment with different configurations, it should be pretty easy and cheap to grab an ESP32 devboard and get ADF up and running.

[psychoacoustic]

Should be fine, WROOM32 (no psram) will do stereo MP3 encode and http stream, stereo EQ via 2048 point FFT (so that's FFT and iFFT) and a constant Q spectrum output via websockets to http clients simultaneously. Note though,I believe the ADK expects psram.

Screen Shot 2019-06-09 at 1.38.33 pm.png (218.08 KiB) Viewed 12462 times

So you should be ok, but my only tip is that I decided to go the web client route rather than try to have a realtime colour display because the web client version was better looking, used far less IO and memory and allows the client to control the device.
Using websockets Safari and Chrome push 30fps no problem with 128 bars on the spectrum, wifi esp to wifi laptop via modem/router. Chrome on Windows 10 also fine haven't tested Edge, Firefox is less efficient but works fine.

Screen Shot 2019-06-09 at 1.36.06 pm.png (274.31 KiB) Viewed 12462 times

I still output the spectrum and VU to serial and have a different uC do the display but for my project the display is more annoying than useful and is currently off.

EDIT:
Video example with audio captured from laptop mic to show sync.
https://youtu.be/l2k0qsd11yg?t=195

[PepeESP32]

Hello ESP Sprite and Psycoacoustic,

sounds like good news.
The questions was of that nature to sort out wether or not we have a chance to do so in ESP32.
We will likely contract a 3rd party copayn to evaluate the capabilities as ESP Sprite said in more detail.
Currently there is not enought resources in our company to develop this ourself.

Regarding te Webinterface and Display:
We plan an interface app to control the part via BT.
Potentially we will export some data to another app due to structure to make it available in an IP network.

Main user Interface control will likely be BT, with Display enabling limited stand alone view for people that do not change configs/or other stuff much to give visual feedback 8also for the few buttons we spend)

[psychoacoustic]

Pepe, due to the similarities of our projects it may be beneficial to collaborate, I'm currently working on a BLE based remote control for my device on top of everything else. If you're interested shoot me a PM, I can arrange a demo of my prototype.

Here is a video from months back while testing the constant Q spectrum.
EDIT Changed video to Youtube:
https://youtu.be/l2k0qsd11yg?t=195
Note that my laptop was struggling with the video capture and the web UI performance is better. I had my usual 100+ tabs open i guess.
The audio is NOT recorded directly from the audio processor but from the mic in the laptop to prove the fast realtime response of the web UI.

[o.copleston]

Hi psychoacoustic,

I love the look of the visualiser you created. I'm working on somthing similar but instead outputting the data to LEDs.
I'd really interested to hear how you implimented a Constant-Q transform if it's something you'd be up for sharing.