Trying to fetch buffer from microphone

Post Reply
ayaz.alavi@gmail.com
Posts: 5
Joined: Wed Jul 20, 2022 9:27 am

Trying to fetch buffer from microphone

Postby ayaz.alavi@gmail.com » Wed Jul 20, 2022 9:33 am

Hi,

I am trying to simply get audio buffer after 20 MS and send that buffer to other methods of the project. Examples are too complicated for such simple use case. Can you anyone guide me whether I am doing it right?

Code: Select all

DLOGD("[1.0] Start codec chip");
    audio_board_handle_t board_handle = audio_board_init();
    audio_hal_ctrl_codec(board_handle->audio_hal, AUDIO_HAL_CODEC_MODE_BOTH, AUDIO_HAL_CTRL_START);
    audio_hal_set_volume(board_handle->audio_hal, 100);
   
    DLOGD( "[2.0] Create i2s stream to read audio data from codec chip");
    i2s_stream_cfg_t i2sr_cfg = I2S_STREAM_CFG_DEFAULT();
    i2sr_cfg.type = AUDIO_STREAM_READER;
    obj->i2s_stream_reader = i2s_stream_init(&i2sr_cfg);    

 
    DLOGD( "[3.0] Init Element");
    audio_element_cfg_t cfg = DEFAULT_AUDIO_ELEMENT_CONFIG();   
    cfg.open = el_open;
    cfg.process = el_process;
    cfg.read = micstream;
    cfg.tag = "my_el";
 
    obj->raw_stream_reader = audio_element_init(&cfg);

 
    DLOGD( "[4.0] Create pipeline");
    audio_pipeline_cfg_t pipeline_cfg = DEFAULT_AUDIO_PIPELINE_CONFIG();
    obj->pipeline = audio_pipeline_init(&pipeline_cfg);
    mem_assert(obj->pipeline);
 
  ESP_LOGI(TAG, "[ 5 ] Set up  event listener");
    audio_event_iface_cfg_t evt_cfg = AUDIO_EVENT_IFACE_DEFAULT_CFG();
    obj->evt = audio_event_iface_init(&evt_cfg);

    ESP_LOGI(TAG, "[5.1] Listening event from all elements of pipeline");
    audio_pipeline_set_listener(obj->pipeline, obj->evt);

    audio_pipeline_register(obj->pipeline, obj->raw_stream_reader, "my_el");
    audio_pipeline_register(obj->pipeline, obj->i2s_stream_reader, "i2sr");

 
    const char *link_tag[3] = {"i2sr", "my_el"};
    audio_pipeline_link(obj->pipeline, link_tag, 2);

audio_pipeline_run(obj->pipeline);
In above

Code: Select all

read
method is not getting called where I am supposed to be having raw data buffer which I can share with other parts of project.

Please help me.
Top
tempo.tian
Posts: 40
Joined: Wed Jun 22, 2022 12:10 pm

Re: Trying to fetch buffer from microphone

Postby tempo.tian » Wed Jul 20, 2022 1:21 pm

After i2s is initialized you can directly call i2s_read API to read data.
Following code do initialize oper. After this you can malloc a buffer and call i2s_read to read data directly.
One point you should pay attention is that make sure fetch data is quicker enough than i2s input rate.
i2s_stream_cfg_t i2sr_cfg = I2S_STREAM_CFG_DEFAULT();
i2sr_cfg.type = AUDIO_STREAM_READER;
obj->i2s_stream_reader = i2s_stream_init(&i2sr_cfg);
Top
ayaz.alavi@gmail.com
Posts: 5
Joined: Wed Jul 20, 2022 9:27 am

Re: Trying to fetch buffer from microphone

Postby ayaz.alavi@gmail.com » Wed Jul 20, 2022 2:13 pm

Thanks for the reply.

I am looking for converting microphone stream to opus codec and then read buffer. Can you provide rough code sample how to do this?
I dont understand what do you mean by i2_read?
Top
ayaz.alavi@gmail.com
Posts: 5
Joined: Wed Jul 20, 2022 9:27 am

Re: Trying to fetch buffer from microphone

Postby ayaz.alavi@gmail.com » Wed Jul 20, 2022 4:35 pm

I changed code to:

Code: Select all

DLOGD("[1.0] Start codec chip");
    audio_board_handle_t board_handle = audio_board_init();
    audio_hal_ctrl_codec(board_handle->audio_hal, AUDIO_HAL_CODEC_MODE_BOTH, AUDIO_HAL_CTRL_START);
    audio_hal_set_volume(board_handle->audio_hal, 100);
   
    DLOGD( "[2.0] Create i2s stream to read audio data from codec chip");
    i2s_stream_cfg_t i2sr_cfg = I2S_STREAM_CFG_DEFAULT();
    i2sr_cfg.type = AUDIO_STREAM_READER;
    i2sr_cfg.i2s_config.sample_rate = 48000;
    i2sr_cfg.out_rb_size = 16 * 1024; // Increase buffer to avoid missing data in bad network conditions
    i2sr_cfg.i2s_port = CODEC_ADC_I2S_PORT;
    pFileSrcContext->i2s_stream_reader = i2s_stream_init(&i2sr_cfg);  


    opus_encoder_cfg_t opus_http_cfg = DEFAULT_OPUS_ENCODER_CONFIG();
    opus_http_cfg.out_rb_size = 16 * 1024;
    audio_element_handle_t opus_decoder = encoder_opus_init(&opus_http_cfg);

    rsp_filter_cfg_t rsp_file_cfg = DEFAULT_RESAMPLE_FILTER_CONFIG();
    rsp_file_cfg.src_rate = 48000;
    rsp_file_cfg.src_ch = 2;
    rsp_file_cfg.dest_rate = 16000;
    rsp_file_cfg.dest_ch = 1;
     rsp_file_cfg.out_rb_size = 16 * 1024;
    audio_element_handle_t resample_for_rec = rsp_filter_init(&rsp_file_cfg);
 
    raw_stream_cfg_t cfg = RAW_STREAM_CFG_DEFAULT();   
    cfg.type = AUDIO_STREAM_READER;
    cfg.out_rb_size = 16 * 1024;
    pFileSrcContext->raw_stream_reader = raw_stream_init(&cfg);

    // ringbuf_handle_t ringbuf1, ringbuf2;
    // ringbuf1 = rb_create(RING_BUFFER_SIZE, 1);
    // audio_element_set_output_ringbuf(pFileSrcContext->i2s_stream_reader, ringbuf1);
    // audio_element_set_input_ringbuf(opus_decoder, ringbuf1);

    // ringbuf2 = rb_create(RING_BUFFER_SIZE, 1);
    // audio_element_set_output_ringbuf( opus_decoder, ringbuf2);
    // audio_element_set_input_ringbuf( pFileSrcContext->raw_stream_reader, ringbuf2);
 
    audio_element_set_event_callback(pFileSrcContext->i2s_stream_reader, audio_element_event_handler, NULL);
    audio_element_set_event_callback(opus_decoder, audio_element_event_handler, NULL);
    audio_element_set_event_callback(pFileSrcContext->raw_stream_reader, audio_element_event_handler, NULL);
    audio_element_set_event_callback(resample_for_rec, audio_element_event_handler, NULL);

    DLOGD( "[5.0] Create pipeline");
    audio_pipeline_cfg_t pipeline_cfg = DEFAULT_AUDIO_PIPELINE_CONFIG();
    pFileSrcContext->pipeline = audio_pipeline_init(&pipeline_cfg);
    mem_assert(pFileSrcContext->pipeline);
 
    DLOGD( "[ 5 ] Set up  event listener");
   esp_periph_config_t periph_cfg = DEFAULT_ESP_PERIPH_SET_CONFIG();
   // esp_periph_set_handle_t set = esp_periph_set_init(&periph_cfg);
    audio_event_iface_cfg_t evt_cfg = AUDIO_EVENT_IFACE_DEFAULT_CFG();
    pFileSrcContext->evt = audio_event_iface_init(&evt_cfg);

    ESP_LOGI(TAG, "[4.1] Listening event from all elements of pipeline");
    audio_pipeline_set_listener(pFileSrcContext->pipeline, pFileSrcContext->evt);

    ESP_LOGI(TAG, "[4.2] Listening event from peripherals");
   // audio_event_iface_set_listener(esp_periph_set_get_event_iface(set), pFileSrcContext->evt);

    audio_pipeline_register(pFileSrcContext->pipeline, pFileSrcContext->i2s_stream_reader, "i2sr");
    audio_pipeline_register(pFileSrcContext->pipeline, resample_for_rec,        "resample");
    audio_pipeline_register(pFileSrcContext->pipeline, opus_decoder,        "opus");
    audio_pipeline_register(pFileSrcContext->pipeline, pFileSrcContext->raw_stream_reader, "my_el");
    
   // audio_pipeline_register(pFileSrcContext->pipeline, i2s_stream_writer, "i2sw");
 
    const char *link_tag[4] = {"i2sr", "resample", "opus", "my_el"};
    audio_pipeline_link(pFileSrcContext->pipeline, &link_tag[0], 4);
but I am getting following on the console:

Code: Select all

E (24732) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (24732) task_wdt:  - IDLE (CPU 0)
E (24732) task_wdt: Tasks currently running:
E (24732) task_wdt: CPU 0: opus
E (24732) task_wdt: CPU 1: IDLE
E (24732) task_wdt: Print CPU 0 (current core) backtrace


Backtrace:0x40152EAF:0x3FFB0840 0x400851AD:0x3FFB0860 0x400F0731:0x3F9A7560 0x400F1C68:0x3F9A9500 0x400EB0A2:0x3F9AA7D0 0x400ECE66:0x3F9AAC40 0x400ED205:0x3F9AD570 0x400E81C2:0x3F9AD590 0x400E85B1:0x3F9AD600 0x400E3716:0x3F9AD620 0x400E38AA:0x3F9AD650 0x4009387D:0x3F9AD680 
0x40152eaf: task_wdt_isr at /Users/ayaz/esp/esp-idf/components/esp_common/src/task_wdt.c:189

0x400851ad: _xt_lowint1 at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/xtensa_vectors.S:1105

0x400f0731: run_prefilter at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/celt/celt_encoder.c:1135 (discriminator 1)

0x400f1c68: celt_encode_with_ec at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/celt/celt_encoder.c:1606

0x400eb0a2: opus_encode_native at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/src/opus_encoder.c:2061

0x400ece66: opus_multistream_encode_native at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/src/opus_multistream_encoder.c:1073

0x400ed205: opus_multistream_encode at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/src/opus_multistream_encoder.c:1153

0x400e81c2: opus_encoder_process at /builds/adf/esp-adf-libs-source/esp_processing/esp-wrapper/opus_encoder.c:709

0x400e85b1: _opus_encoder_process at /builds/adf/esp-adf-libs-source/esp_processing/esp-wrapper/opus_encoder.c:858

0x400e3716: audio_element_process_running at /Users/ayaz/.espressif/esp-adf/components/audio_pipeline/audio_element.c:336

0x400e38aa: audio_element_task at /Users/ayaz/.espressif/esp-adf/components/audio_pipeline/audio_element.c:483

0x4009387d: vPortTaskWrapper at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/port.c:168


E (24732) task_wdt: Print CPU 1 backtrace


Backtrace:0x40087719:0x3FFB0E40 0x400851AD:0x3FFB0E60 0x4000BFED:0x3FFC3860 0x40093B3D:0x3FFC3870 0x4015317B:0x3FFC3890 0x40153187:0x3FFC38C0 0x400DC36D:0x3FFC38E0 0x40091325:0x3FFC3900 0x4009387D:0x3FFC3920 
0x40087719: esp_crosscore_isr at /Users/ayaz/esp/esp-idf/components/esp32/crosscore_int.c:77

0x400851ad: _xt_lowint1 at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/xtensa_vectors.S:1105

0x40093b3d: vPortExitCritical at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/port.c:473

0x4015317b: esp_task_wdt_reset at /Users/ayaz/esp/esp-idf/components/esp_common/src/task_wdt.c:336

0x40153187: idle_hook_cb at /Users/ayaz/esp/esp-idf/components/esp_common/src/task_wdt.c:88

0x400dc36d: esp_vApplicationIdleHook at /Users/ayaz/esp/esp-idf/components/esp_common/src/freertos_hooks.c:51 (discriminator 1)

0x40091325: prvIdleTask at /Users/ayaz/esp/esp-idf/components/freertos/tasks.c:3880 (discriminator 1)

0x4009387d: vPortTaskWrapper at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/port.c:168
Top
ayaz.alavi@gmail.com
Posts: 5
Joined: Wed Jul 20, 2022 9:27 am

Re: Trying to fetch buffer from microphone

Postby ayaz.alavi@gmail.com » Thu Jul 21, 2022 7:09 pm

I am trying to encode microphone stream to get raw bytes but I am getting following the console:

Code: Select all

I (21343) OPUS_ENCODER: opus open
Audio event 8 from my_el element
AEL_STATUS_STATE_RUNNING
readign bytes
Audio event 8 from opus element
AEL_STATUS_STATE_RUNNING
E (22593) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (22593) task_wdt:  - IDLE (CPU 0)
E (22593) task_wdt: Tasks currently running:
E (22593) task_wdt: CPU 0: opus
E (22593) task_wdt: CPU 1: IDLE
E (22593) task_wdt: Print CPU 0 (current core) backtrace


Backtrace:0x40151D6B:0x3FFB0840 0x400851AD:0x3FFB0860 0x401C6539:0x3F9A3D30 0x400F31CB:0x3F9A3E00 0x400EFCAD:0x3F9A5C70 0x400F129C:0x3F9A5CE0 0x400E9F5E:0x3F9A7F70 0x400EBD22:0x3F9A83E0 0x400EC0C1:0x3F9AAD10 0x400E707E:0x3F9AAD30 0x400E746D:0x3F9AADA0 0x400E25D2:0x3F9AADC0 0x400E2766:0x3F9AADF0 0x4009387D:0x3F9AAE20 
0x40151d6b: task_wdt_isr at /Users/ayaz/esp/esp-idf/components/esp_common/src/task_wdt.c:189

0x400851ad: _xt_lowint1 at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/xtensa_vectors.S:1105

0x401c6539: dsps_dotprod_s16_ae32_error at dsps_dotprod_s16_ae32.S.obj:?

0x400f31cb: clt_mdct_forward_c at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/celt/mdct.c:215

0x400efcad: compute_mdcts at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/celt/celt_encoder.c:467 (discriminator 3)

0x400f129c: celt_encode_with_ec at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/celt/celt_encoder.c:1657

0x400e9f5e: opus_encode_native at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/src/opus_encoder.c:2061

0x400ebd22: opus_multistream_encode_native at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/src/opus_multistream_encoder.c:1073

0x400ec0c1: opus_multistream_encode at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/src/opus_multistream_encoder.c:1153

0x400e707e: opus_encoder_process at /builds/adf/esp-adf-libs-source/esp_processing/esp-wrapper/opus_encoder.c:709

0x400e746d: _opus_encoder_process at /builds/adf/esp-adf-libs-source/esp_processing/esp-wrapper/opus_encoder.c:858

0x400e25d2: audio_element_process_running at /Users/ayaz/.espressif/esp-adf/components/audio_pipeline/audio_element.c:336

0x400e2766: audio_element_task at /Users/ayaz/.espressif/esp-adf/components/audio_pipeline/audio_element.c:483

0x4009387d: vPortTaskWrapper at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/port.c:168


E (22593) task_wdt: Print CPU 1 backtrace


Backtrace:0x40087719:0x3FFB0E40 0x400851AD:0x3FFB0E60 0x4000BFED:0x3FFC3860 0x40093B3D:0x3FFC3870 0x40152037:0x3FFC3890 0x40152043:0x3FFC38C0 0x400DB13D:0x3FFC38E0 0x40091325:0x3FFC3900 0x4009387D:0x3FFC3920 
0x40087719: esp_crosscore_isr at /Users/ayaz/esp/esp-idf/components/esp32/crosscore_int.c:77

0x400851ad: _xt_lowint1 at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/xtensa_vectors.S:1105

0x40093b3d: vPortExitCritical at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/port.c:473

0x40152037: esp_task_wdt_reset at /Users/ayaz/esp/esp-idf/components/esp_common/src/task_wdt.c:336

0x40152043: idle_hook_cb at /Users/ayaz/esp/esp-idf/components/esp_common/src/task_wdt.c:88

0x400db13d: esp_vApplicationIdleHook at /Users/ayaz/esp/esp-idf/components/esp_common/src/freertos_hooks.c:51 (discriminator 1)

0x40091325: prvIdleTask at /Users/ayaz/esp/esp-idf/components/freertos/tasks.c:3880 (discriminator 1)

0x4009387d: vPortTaskWrapper at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/port.c:168


message received
WEBSOCKET_EVENT_DATA
Received opcode=10
New certificate has been pre-generated and added to the queue
Bytes read 0 16384
Writing frame to buffer 16384
buffer sent
readign bytes
E (27593) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (27593) task_wdt:  - IDLE (CPU 0)
E (27593) task_wdt: Tasks currently running:
E (27593) task_wdt: CPU 0: opus
E (27593) task_wdt: CPU 1: IDLE
E (27593) task_wdt: Print CPU 0 (current core) backtrace


Backtrace:0x40151D6B:0x3FFB0840 0x400851AD:0x3FFB0860 0x400F8ECF:0x3F9A3D30 0x400F31CB:0x3F9A3E00 0x400EFCAD:0x3F9A5C70 0x400F0C8C:0x3F9A5CE0 0x400E9F5E:0x3F9A7F70 0x400EBD22:0x3F9A83E0 0x400EC0C1:0x3F9AAD10 0x400E707E:0x3F9AAD30 0x400E746D:0x3F9AADA0 0x400E25D2:0x3F9AADC0 0x400E2766:0x3F9AADF0 0x4009387D:0x3F9AAE20 
0x40151d6b: task_wdt_isr at /Users/ayaz/esp/esp-idf/components/esp_common/src/task_wdt.c:189

0x400851ad: _xt_lowint1 at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/xtensa_vectors.S:1105

0x400f8ecf: kf_bfly4 at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/celt/kiss_fft.c:153
 (inlined by) opus_fft_impl at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/celt/kiss_fft.c:554

0x400f31cb: clt_mdct_forward_c at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/celt/mdct.c:215

0x400efcad: compute_mdcts at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/celt/celt_encoder.c:467 (discriminator 3)

0x400f0c8c: celt_encode_with_ec at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/celt/celt_encoder.c:1664

0x400e9f5e: opus_encode_native at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/src/opus_encoder.c:2061

0x400ebd22: opus_multistream_encode_native at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/src/opus_multistream_encoder.c:1073

0x400ec0c1: opus_multistream_encode at /builds/adf/esp-adf-libs-source/esp_codec/esp-opus/src/opus_multistream_encoder.c:1153

0x400e707e: opus_encoder_process at /builds/adf/esp-adf-libs-source/esp_processing/esp-wrapper/opus_encoder.c:709

0x400e746d: _opus_encoder_process at /builds/adf/esp-adf-libs-source/esp_processing/esp-wrapper/opus_encoder.c:858

0x400e25d2: audio_element_process_running at /Users/ayaz/.espressif/esp-adf/components/audio_pipeline/audio_element.c:336

0x400e2766: audio_element_task at /Users/ayaz/.espressif/esp-adf/components/audio_pipeline/audio_element.c:483

0x4009387d: vPortTaskWrapper at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/port.c:168


E (27593) task_wdt: Print CPU 1 backtrace


Backtrace:0x40087719:0x3FFB0E40 0x400851AD:0x3FFB0E60 0x4000BFED:0x3FFC3860 0x40093B3D:0x3FFC3870 0x40152037:0x3FFC3890 0x40152043:0x3FFC38C0 0x400DB13D:0x3FFC38E0 0x40091325:0x3FFC3900 0x4009387D:0x3FFC3920 
0x40087719: esp_crosscore_isr at /Users/ayaz/esp/esp-idf/components/esp32/crosscore_int.c:77

0x400851ad: _xt_lowint1 at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/xtensa_vectors.S:1105

0x40093b3d: vPortExitCritical at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/port.c:473

0x40152037: esp_task_wdt_reset at /Users/ayaz/esp/esp-idf/components/esp_common/src/task_wdt.c:336

0x40152043: idle_hook_cb at /Users/ayaz/esp/esp-idf/components/esp_common/src/task_wdt.c:88

0x400db13d: esp_vApplicationIdleHook at /Users/ayaz/esp/esp-idf/components/esp_common/src/freertos_hooks.c:51 (discriminator 1)

0x40091325: prvIdleTask at /Users/ayaz/esp/esp-idf/components/freertos/tasks.c:3880 (discriminator 1)

0x4009387d: vPortTaskWrapper at /Users/ayaz/esp/esp-idf/components/freertos/port/xtensa/port.c:168
Can anyone let me know what is the problem? I am using simple i2s_stream -> opus_encoder -> raw_stream.
Top
felixcollins
Posts: 125
Joined: Fri May 24, 2019 2:02 am

Re: Trying to fetch buffer from microphone

Postby felixcollins » Wed Aug 03, 2022 2:45 am

You got an idle watchdog timeout on core0. The task that was running ("opus" doing fft?) was hogging core 0 so much that the idle task got blocked. You could try disabling the the idle watchdog. You will need to study task scheduling and cpu allocation for rtos.
Top
Post Reply

Return to “ESP-ADF”

Who is online

Users browsing this forum: No registered users and 6 guests