USBHID and MSC drive on internal Flash simultaneously

[hairPullRequest]

I am trying to make a HID USB program for ESP32-S2 and to have an MSC on the internal Flash mounted as device towards PC, First part is super easy using the demo programs from C:\.platformio\packages\framework-arduinoespressif32\libraries\USB\examples. My problem is that there is no example for MSC using Flash, only for a RamDisk and a FirmwareMSC .
I have found a nice example in chegewara's TinyUSB packet, but trying to combine this library with the internal lib of arduino-espressif looks very hard to me, while only simply creating the FlashUSB dev object ruins the demo program.
I am using Platformio arduino espressif for developement and here is the code:

#include <Arduino.h>
#include "FS.h"
#include "SD.h"


#include "USB.h"
#include "USBHIDMouse.h"
#include "USBHIDKeyboard.h"
USBHIDMouse Mouse;
USBHIDKeyboard Keyboard;

/**
* Simple MSC device, use fat partition
* author: chegewara
*/
#include "flashdisk.h"


#if 0
FlashUSB dev;

char *l1 = "ffat";
void setupFlash()
{

if (dev.init("/fat1", "ffat"))
{
if (dev.begin())
{
Serial.println("MSC lun 1 begin");
}
else
log_e("LUN 1 failed");
}
}
#endif

void setup()
{ // initialize the buttons' inputs:

Serial.begin(115200);
delay(100);
// initialize mouse control:
Mouse.begin();
Keyboard.begin();
USB.begin();
}

void loop()
{
// use serial input to control the mouse:
if (Serial.available() > 0)
{
char inChar = Serial.read();
Serial.printf(" > %c ", inChar);
switch (inChar)
{
case 'u':
// move mouse up
Mouse.move(0, -40);
Serial.println(" up ");
break;
case 'd':
// move mouse down
Mouse.move(0, 40);
Serial.println(" down ");
break;
}
}

delay(5);
}

This is the GOOD version and the output with debuglevel 5:
ESP-ROM:esp32s2-rc4-20191025
Build:Oct 25 2019
rst:0x1 (POWERON),boot:0x8 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3ffe6100,len:0x524
load:0x4004c000,len:0xa70
load:0x40050000,len:0x2914
entry 0x4004c18c
[ 497][D][esp32-hal-tinyusb.c:673] tinyusb_enable_interface(): Interface HID enabled
[ 497][D][USBHID.cpp:61] tinyusb_enable_hid_device(): Device[0] len: 79
[ 500][D][USBHID.cpp:61] tinyusb_enable_hid_device(): Device[1] len: 67
E (96) esp_core_dump_flash:[ 620][D][esp32-hal-tinyusb.c:562] tinyusb_load_enabled_interfaces(): Load Done: if_num: 1, descr_len: 41, if_mask: 0x4
[ 1073][V][USBHID.cpp:245] tud_hid_set_idle_cb(): instance: 0, idle_rate:0
[ 1074][V][USBHID.cpp:224] tud_hid_descriptor_report_cb(): instance: 0
[ 1076][D][USBHID.cpp:176] tinyusb_load_enabled_hid_devices(): Loaded HID Desriptor with the following reports:
[ 1085][D][USBHID.cpp:184] tinyusb_load_enabled_hid_devices(): ID: 2, Type: INPUT, Size: 5, Usage: MOUSE
[ 1095][D][USBHID.cpp:184] tinyusb_load_enabled_hid_devices(): ID: 1, Type: INPUT, Size: 8, Usage: KEYBOARD
[ 1106][D][USBHID.cpp:184] tinyusb_load_enabled_hid_devices(): ID: 1, Type: OUTPUT, Size: 1, Usage: KEYBOARD

Simply moving the
FlashUSB dev;
line above the #if 0 - gives output with missing loaded HID descriptors.
ESP-ROM:esp32s2-rc4-20191025
Build:Oct 25 2019
rst:0x1 (POWERON),boot:0x8 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3ffe6100,len:0x524
load:0x4004c000,len:0xa70
load:0x40050000,len:0x2914
entry 0x4004c18c
[ 498][D][esp32-hal-tinyusb.c:673] tinyusb_enable_interface(): Interface HID enabled
[ 498][D][USBHID.cpp:61] tinyusb_enable_hid_device(): Device[0] len: 79
[ 501][D][USBHID.cpp:61] tinyusb_enable_hid_device(): Device[1] len: 67
E (97) esp_core_dump_flash:[ 621][D][esp32-hal-tinyusb.c:562] tinyusb_load_enabled_interfaces(): Load Done: if_num: 1, descr_len: 41, if_mask: 0x4

Unfortunately I am newbie concerning USBs, and descriptors on a level needed to get past this obstacle. The best solution would be probably to have a simple internal Flash demo program with the ESP32-S2 own internal library. I feel that the solution is really easy - but only for someone familiar with USB.

Can someone help me a little bit here?

[vincentpernice]

Hello, I'm bumping this as I'm in the same boat : I'm trying to expose FFAT file system as a mass storage flash drive over USB.
I got the ram disk example working and I have implemented this on another MCU using tinyUSB, using FATFS too.

Here I understand the access to FATFS uses an interface that has the low level diskio implementation already written for a number of media : SD card, mmc, onboard flash / SPI flash so that it's transparent to the user.

The usual system with MSD is that you "only" need the onWrite and Onread commands that point to the actual low level functions to the media.

I modified the FFAT lib to have a getDrive() method that returns the logical drive from which you can call the FATFS disk_read and disk_write but this is leading to crash

here's my implementation. I tried using both disk_read / disk_write and their ff_ variants
any ideas ? thanks

Code: Select all

// FFat getDrive
unsigned char F_Fat::getDrive() {
	BYTE pdrv = 0;
	if(_wl_handle != WL_INVALID_HANDLE){
		pdrv = ff_diskio_get_pdrv_wl(_wl_handle);
	}
	return((unsigned char)pdrv);
} 


setup
[...]
  HWSerial.begin(115200);
  HWSerial.setDebugOutput(true);
  USB.onEvent(usbEventCallback);
  MSC.vendorID("ME");//max 8 chars
  MSC.productID("MSC-DRIVE");//max 16 chars
  MSC.productRevision("1.0");//max 4 chars
  MSC.onStartStop(onStartStop);
  MSC.onRead(onRead);
  MSC.onWrite(onWrite);
  MSC.mediaPresent(true);
  MSC.begin(DISK_SECTOR_COUNT, DISK_SECTOR_SIZE);
  Serial.begin(115200);
  USB.begin();  


[...]
// MSC / FATFS / FFAT callbacks
static int32_t onWrite(uint32_t lba, uint32_t offset, uint8_t* buffer, uint32_t bufsize){
  HWSerial.printf("MSC WRITE: lba: %u, offset: %u, bufsize: %u\n", lba, offset, bufsize);
  int32_t res;
  res = ff_disk_write(FFat.getDrive(),(uint8_t*)buffer, lba, bufsize);
  //memcpy(msc_disk[lba] + offset, buffer, bufsize);
  return bufsize;
}

static int32_t onRead(uint32_t lba, uint32_t offset, void* buffer, uint32_t bufsize){
  HWSerial.printf("MSC READ: lba: %u, offset: %u, bufsize: %u\n", lba, offset, bufsize);
  int32_t res;
  res = ff_disk_read(FFat.getDrive(),(uint8_t*)buffer, lba, bufsize);
  //memcpy(buffer, msc_disk[lba] + offset, bufsize);
  return bufsize;
}

static bool onStartStop(uint8_t power_condition, bool start, bool load_eject){
  HWSerial.printf("MSC START/STOP: power: %u, start: %u, eject: %u\n", power_condition, start, load_eject);
  return true;
}

static void usbEventCallback(void* arg, esp_event_base_t event_base, int32_t event_id, void* event_data){
  if(event_base == ARDUINO_USB_EVENTS){
    arduino_usb_event_data_t * data = (arduino_usb_event_data_t*)event_data;
    switch (event_id){
      case ARDUINO_USB_STARTED_EVENT:
        HWSerial.println("USB PLUGGED");
        break;
      case ARDUINO_USB_STOPPED_EVENT:
        HWSerial.println("USB UNPLUGGED");
        break;
      case ARDUINO_USB_SUSPEND_EVENT:
        HWSerial.printf("USB SUSPENDED: remote_wakeup_en: %u\n", data->suspend.remote_wakeup_en);
        break;
      case ARDUINO_USB_RESUME_EVENT:
        HWSerial.println("USB RESUMED");
        break;
      
      default:
        break;
    }
  }
}

[vincentpernice]

my bad for the double posting (new on this forum).
Update:

I did check in the IDF and the FFat arduino library indeed registers the low level implementations of diskio (disk_read, disk_write etc) in the esp_vfs_fat_spiflash_mount() function and calls ff_diskio_register_wl_partition(). Those are then called via linked pointers of the selected implementation for the currently mounted drive, within FATFS.

This works perfectly when testing FFat file examples, from formatting to accessing files.
However when I try to manually dump a sector of the fat using disk_read, I get a crash and for now I don't know why. If that was due to a mutex somewhere, I would be supposed to get an error or receive an empty buffer.
The test was done without the mass storage. I tried calling disk_read as well as ff_disk_read without success.

here's my code. The getDrive() function has been listed in the previous post.

Code: Select all

// This file should be compiled with 'Partition Scheme' (in Tools menu)
// set to 'TinyUSB compatibility 2MB app / 3.7MB ffat'

#include "FS.h"
#include "FFat.h"
#include "ff.h"
#include "diskio_impl.h"


// App should be compiled with CDC serial on boot but worse case we instantiate CDC in the app itself
#define HWSerial Serial0
#define USBSerial Serial

DRESULT ff_disk_read (BYTE pdrv, BYTE* buff, DWORD sector, UINT count);

static const uint16_t DISK_SECTOR_SIZE = 512;    // Should be 512

void dumpFlashSector(uint32_t lbaSector) {
  uint8_t buf[DISK_SECTOR_SIZE+50];
  
  //DRESULT res = ff_disk_read(FFat.getDrive(),buf, lbaSector, 1);
  DRESULT res = disk_read(FFat.getDrive(), buf, lbaSector, 1);
  Serial.printf("disk_read result = %d\n", res);
  for(int i = 0 ; i < DISK_SECTOR_SIZE ; i++) {
    Serial.printf("%02X ", buf[i]);
    if(i%32)
      Serial.println();
  }  
}

void setup() {
  HWSerial.begin(115200);
  HWSerial.setDebugOutput(true);
  // Starts the file system
  //FFat.format();
  
  // Use format on fail
  if(!FFat.begin(true)){
    HWSerial.println("FFat Mount Failed");
    return;
  }
 
  USBSerial.begin();
  
  // we should wait until USB is ready before doing anything else or start the MSD later like in CFX
  int cnt = 1000;
  while(cnt--) {
    delay(1);
    yield();
  }

  Serial.printf("Total space: %10u\n", FFat.totalBytes());
  Serial.printf("Free space: %10u\n", FFat.freeBytes());
  
  uint64_t chipid = ESP.getEfuseMac(); //The chip ID is essentially its MAC address(length: 6 bytes).
  Serial.printf("ESP32 Chip ID = %04X", (uint16_t)(chipid >> 32)); //print High 2 bytes
  Serial.printf("%08X\n", (uint32_t)chipid); //print Low 4bytes.

  Serial.printf("dumping FAT sector #%d:\n", 0);
  dumpFlashSector(0);
}


void loop() {

}

I get this trace over the debug UART

Code: Select all

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x1 (POWERON),boot:0xb (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fcd0108,len:0x1660
load:0x403b6000,len:0xeac
load:0x403ba000,len:0x32c8
entry 0x403b6328
[0;32mI (25) boot: ESP-IDF v4.4-367-gc29343eb94 2nd stage bootloader[0m
[0;32mI (25) boot: compile time 07:12:00[0m
[0;32mI (25) boot: chip revision: 0[0m
[0;32mI (28) qio_mode: Enabling default flash chip QIO[0m
[0;32mI (33) boot.esp32s3: Boot SPI Speed : 80MHz[0m
[0;32mI (38) boot.esp32s3: SPI Mode       : QIO[0m
[0;32mI (43) boot.esp32s3: SPI Flash Size : 8MB[0m
[0;32mI (47) boot: Enabling RNG early entropy source...[0m
[0;32mI (53) boot: Partition Table:[0m
[0;32mI (56) boot: ## Label            Usage          Type ST Offset   Length[0m
[0;32mI (64) boot:  0 nvs              WiFi data        01 02 00009000 00005000[0m
[0;32mI (71) boot:  1 otadata          OTA data         01 00 0000e000 00002000[0m
[0;32mI (79) boot:  2 ota_0            OTA app          00 10 00010000 00200000[0m
[0;32mI (86) boot:  3 ota_1            OTA app          00 11 00210000 00200000[0m
[0;32mI (93) boot:  4 uf2              factory app      00 00 00410000 00040000[0m
[0;32mI (101) boot:  5 ffat             Unknown data     01 81 00450000 003b0000[0m
[0;32mI (109) boot: End of partition table[0m
[0;32mI (613) esp_image: segment 0: paddr=00010020 vaddr=3c040020 size=0fbf0h ( 64496) map[0m
[0;32mI (628) esp_image: segment 1: paddr=0001fc18 vaddr=3fc92100 size=00400h (  1024) load[0m
[0;32mI (629) esp_image: segment 2: paddr=00020020 vaddr=42000020 size=352b0h (217776) map[0m
[0;32mI (667) esp_image: segment 3: paddr=000552d8 vaddr=3fc92500 size=03bbch ( 15292) load[0m
[0;32mI (670) esp_image: segment 4: paddr=00058e9c vaddr=40374000 size=0e0f4h ( 57588) load[0m
[0;32mI (689) boot: Loaded app from partition at offset 0x10000[0m
[0;32mI (689) boot: Disabling RNG early entropy source...[0m
[   707][D][esp32-hal-tinyusb.c:680] tinyusb_enable_interface(): Interface CDC enabled
E (708) esp_core_dump_flash: No core dump partition found!
E (709) esp_core_dump_flash: No core dump partition found!
[   715][D][esp32-hal-tinyusb.c:569] tinyusb_load_enabled_interfaces(): Load Done: if_num: 2, descr_len: 75, if_mask: 0x10
[  1610][I][esp32-hal-psram.c:96] psramInit(): PSRAM enabled

assert failed: spinlock_acquire spinlock.h:122 (result == core_id || result == SPINLOCK_FREE)


Backtrace: 0x4037711e:0x3fcecdc0 0x4037a67d:0x3fcecde0 0x40380521:0x3fcece00 0x4037d521:0x3fcecf30 0x4037aefd:0x3fcecf70 0x4037841a:0x3fcecfb0 0x403784b9:0x3fcecfe0 0x42011769:0x3fced000 0x42012b08:0x3fced020 0x4201299d:0x3fced040 0x42001f58:0x3fced060 0x42002052:0x3fced2c0 0x42004ffe:0x00000000 |<-CORRUPTED

[chegewara]

Sorry, but those 2 libraries are not compatible.
Both are using tinyUSB, but my library was created before espressif and i chose to build it my own way.

[vincentpernice]

nevermind, I found what my issue was, and in the process I learned how to use the JTAG debugger over USB with gdb and openocd, on this new target.
My sector size was wrong, I assumed some sort of cache existed to realign wl_flash to SD card sector size emulation, but it's not even needed than the MSD is totally ok with larger sizes, which makes the FAT buffering much more efficient too, and aligned with the wl_flash physical structure. All working now, my crash was simple memory fault (as debug attested). Bear with me, I've just started to learn coding on the ESP32-S3 !
cheers

[squix78]

Hi Vincent. Do you mind to share your final code? I'm trying to accomplish the same as you and I hope to learn from your code

Cheers, Dani