Using esptool.py with subprocess.Popen() vs. .run()

[graceyoung-ww]

We are building a provisioning script in python that runs esptool.py using the subprocess functions. We do this because we want to show the output to the operator, and also parse that output to get certain information for later steps in provisioning. I know this isn't specific to the esptool.py itself, but I figured someone here may have experienced this weirdness and may have an answer?

The following code is based on the default esp-idf generated flashing function (run from within VS Code). It maps those default parameters for use in the subprocess functions.

Code: Select all

proc_args = ['esptool.py']
proc_args.extend(['-p', f"{DEVICE_PORT}"])
proc_args.extend(['-b', '460800'])
proc_args.extend(['--before', 'default_reset'])
proc_args.extend(['--after', 'hard_reset'])
proc_args.extend(['--chip', 'esp32'])
proc_args.append('write_flash')
proc_args.extend(['--flash_mode', 'dio'])
proc_args.extend(['--flash_freq', '40m'])
proc_args.extend(['--flash_size', 'detect'])
proc_args.extend(['0x10000', f"{BUILD_DIR}/6mw-firmware.bin"])
proc_args.extend(['0x1000', f"{BUILD_DIR}/bootloader/bootloader.bin"])
proc_args.extend(['0x8000', f"{BUILD_DIR}/partition_table/partition-table.bin"])

print('\n\nARGS\n', proc_args)
p = subprocess.run(proc_args, capture_output=True, text=True)
print('\n\nOUTPUT\n', p.stdout)
print('\n\nERRORS\n', p.stderr)

This works, but has the drawback that the entire operation has to have completed before the output is displayed to the operator.

I have found an alternative approach using subproces.Popen. Using this with the default '--help' argument works as expected. That is, each individual line is received and processed in real time.

But if I pass the proc_args as defined above, it does not run the actual arguments (it just continues to display the help text, as though no arguments have been passed):

Code: Select all

p = subprocess.Popen(proc_args, stdout=subprocess.PIPE, universal_newlines=True, shell=True)
for out in iter(p.stdout.readline, ''): # read rest of output
    print(out, end='')
p.stdout.close()
return_code = p.wait()

I've tried with and without the additional parameters (

Code: Select all

shell=True

and

Code: Select all

universal_newlines=True

) and still it doesn't work.

Is there anything obvious about the difference between run() and Popen() that would cause this to work for one, but not the other?

Alternatively, does anyone else have a working example of successfully calling esptool.py from within python with a "tried and tested" result?

Thanks in advance. Grace

P.S. if there's a more appropriate thread or sub-forum for this post, please let me know