Inconsistent Python versions between ESP-IDF tools and native OS

[mwirth]

The ./install.sh command in section "Step 3. Set up the tools" of the Get Started page for the ESP32-S2 downloads a number of large binary tools, saves the downloads and installs them in an .espressif folder. It then creates a Python 2.7 environment and installs a bunch of Python modules in the same directory. This creates a number of Python-related problems:

• Python 2.7 is at end-of-life. Some Python 3.x version should be used.
• It looks like a Python virtual environment is created. The first couple lines of output from this phase are:

  Code: Select all

  Installing Python environment and packages
  Creating a new Python environment in /Users/mike/.espressif/tools/python_env/idf4.4_py2.7_env
  created virtual environment CPython2.7.16.final.0-64 in 14559ms
  

• This virtual environment is problematic. There are no instructions to activate this Python virtual environment later when the tools are used. Should this be automatic?
• if this environment is activated manually, trouble ensues:

  Code: Select all

  % source /Users/mike/.espressif/tools/python_env/idf4.4_py2.7_env/bin/activate 
  (idf4.4_py2.7_env) mike@Mikes-MacBook-Pro-10-1 ~ % get_idf
  Adding ESP-IDF tools to PATH...
  Not using an unsupported version of tool openocd-esp32 found in PATH: unknown.
  Using Python interpreter in python: aliased to /usr/local/bin/python3
  Checking if Python packages are up to date...
  The following Python requirements are not satisfied:
  click>=7.0
  pyserial>=3.0
  future>=0.15.2
  cryptography>=2.1.4
  pyparsing>=2.0.3,<2.4.0
  pyelftools>=0.22
  gdbgui==0.13.2.0
  pygdbmi<=0.9.0.2
  kconfiglib==13.7.1
  reedsolo>=1.5.3,<=1.5.4
  bitstring>=3.1.6
  ecdsa>=0.16.0
  construct==2.10.54
  To install the missing packages, please run "/Users/mike/esp/esp-idf/install.sh"
  Diagnostic information:
      IDF_PYTHON_ENV_PATH: /Users/mike/.espressif/tools/python_env/idf4.4_py2.7_env
      Python interpreter used: /usr/local/bin/python3
      Warning: python interpreter not running from IDF_PYTHON_ENV_PATH
      PATH: /Users/mike/.espressif/tools/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin:/Users/mike/.espressif/tools/tools/xtensa-esp32s2-elf/esp-2020r3-8.4.0/xtensa-esp32s2-elf/bin:/Users/mike/.espressif/tools/tools/xtensa-esp32s3-elf/esp-2020r3-8.4.0/xtensa-esp32s3-elf/bin:/Users/mike/.espressif/tools/tools/riscv32-esp-elf/1.24.0.123_64eb9ff-8.4.0/riscv32-esp-elf/bin:/Users/mike/.espressif/tools/tools/esp32ulp-elf/2.28.51-esp-20191205/esp32ulp-elf-binutils/bin:/Users/mike/.espressif/tools/tools/esp32s2ulp-elf/2.28.51-esp-20191205/esp32s2ulp-elf-binutils/bin:/Users/mike/.espressif/tools/tools/openocd-esp32/v0.10.0-esp32-20200709/openocd-esp32/bin:/Users/mike/.espressif/tools/python_env/idf4.4_py2.7_env/bin:/usr/local/sbin:/Users/mike/Projects/moddable/build/bin/mac/release:/Library/Frameworks/Python.framework/Versions/3.9/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Applications/Wireshark.app/Contents/MacOS:/Users/mike/esp/esp-idf/tools:/Applications/Visual Studio Code.app/Contents/Resources/app/bin
  (idf4.4_py2.7_env) mike@Mikes-MacBook-Pro-10-1 ~ % 
  

• Similar trouble is encountered if the environment is not invoked directly, but the get_idf command is defined (as shown in a later paragraph) and used.
• In any case, conflicts between Python versions installed in the host environment and ESP-IDF related tooling seem probable.
• I've attempted to search this forum for "python version" and a number of related search terms without success.
• is there a discussion of "proper use" of Python versions in ESP-IDF tooling and development in the docs anywhere?

Thanks in advance for any clarification of this issue.

Mike

[ESP_Roland]

Hi mwirth,

python3 will be used instead of python[2] by the installer if it is available. This is a very recent change and not yet on Github. This change will be published soon.

You are not supposed to activate the Python environment manually. Please use "source $IDF_PATH/export.sh" and it will setup everything including the Python environment installed.

[mwirth]

Roland,

Thanks for the quick reply. FYI, on my Big Sur MacBook Pro, Python 2.7 and 3.8 are installed be default (in R/O system image) and invoked by "python" and "python3" respectively. In an effort to avoid python 2.7, I had aliased "python" to "/usr/local/bin/python...", my local installation of Python 3.9.2. This is what caused my problem later (see attached console output).

Looking forward to your release of the Python 3 version of tool building in ESP-IDF

Mike

Code: Select all

PROBLEMS CAUSED BY ALIASING "PYTHON" TO PYTHON 3
====================================================
mike@Mikes-MacBook-Pro-10-1 ~ % echo $IDF_PATH
/Users/mike/esp/esp-idf
mike@Mikes-MacBook-Pro-10-1 ~ % source $IDF_PATH/export.sh
Adding ESP-IDF tools to PATH...
Not using an unsupported version of tool openocd-esp32 found in PATH: unknown.
Using Python interpreter in python: aliased to /usr/local/bin/python3
Checking if Python packages are up to date...
The following Python requirements are not satisfied:
click>=7.0
pyserial>=3.0
future>=0.15.2
cryptography>=2.1.4
pyparsing>=2.0.3,<2.4.0
pyelftools>=0.22
gdbgui==0.13.2.0
pygdbmi<=0.9.0.2
kconfiglib==13.7.1
reedsolo>=1.5.3,<=1.5.4
bitstring>=3.1.6
ecdsa>=0.16.0
construct==2.10.54
To install the missing packages, please run "/Users/mike/esp/esp-idf/install.sh"
Diagnostic information:
    IDF_PYTHON_ENV_PATH: /Users/mike/.espressif/tools/python_env/idf4.4_py2.7_env
    Python interpreter used: /usr/local/bin/python3
    Warning: python interpreter not running from IDF_PYTHON_ENV_PATH
    PATH: /Users/mike/.espressif/tools/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin:/Users/mike/.espressif/tools/tools/xtensa-esp32s2-elf/esp-2020r3-8.4.0/xtensa-esp32s2-elf/bin:/Users/mike/.espressif/tools/tools/xtensa-esp32s3-elf/esp-2020r3-8.4.0/xtensa-esp32s3-elf/bin:/Users/mike/.espressif/tools/tools/riscv32-esp-elf/1.24.0.123_64eb9ff-8.4.0/riscv32-esp-elf/bin:/Users/mike/.espressif/tools/tools/esp32ulp-elf/2.28.51-esp-20191205/esp32ulp-elf-binutils/bin:/Users/mike/.espressif/tools/tools/esp32s2ulp-elf/2.28.51-esp-20191205/esp32s2ulp-elf-binutils/bin:/Users/mike/.espressif/tools/tools/openocd-esp32/v0.10.0-esp32-20200709/openocd-esp32/bin:/Users/mike/.espressif/tools/python_env/idf4.4_py2.7_env/bin:/usr/local/sbin:/Users/mike/Projects/moddable/build/bin/mac/release:/Library/Frameworks/Python.framework/Versions/3.9/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Applications/Wireshark.app/Contents/MacOS:/Users/mike/esp/esp-idf/tools:/Applications/Visual Studio Code.app/Contents/Resources/app/bin
mike@Mikes-MacBook-Pro-10-1 ~ % 

[ESP_Roland]

Are you able to work around this issue until the permanent solution is published, or are you still blocked?

This is also a workaround https://github.com/espressif/esp-idf/pull/6471/files. That PR will be closed when the mentioned solution arrives to Github. The documentation page will be updated at the same time. I cannot estimate when that will happen. It depends on testing and fixing unrelated issues.

[mwirth]

Roland,

Thanks for checking. Sorry I wasn't more clear. Yes, I now have ESP-IDF working for command line use. Still have a problem with the Eclipse plugin, but that's due to a plugin bug.

Mike

PS: Feel free to close this issue.