https://pantsbuild.org/ logo
Join Slack
Powered by
attempting to upgrade our pants script and we get ...
# general
r
attempting to upgrade our pants script and we get this error in CI:
Copy code
./pants version

Bootstrapping Pants using /usr/bin/python3.8

Downloading the virtualenv zipapp

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed


  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 20 9258k   20 1879k    0     0  5910k      0  0:00:01 --:--:--  0:00:01 5891k
100 9258k  100 9258k    0     0  17.5M      0 --:--:-- --:--:-- --:--:-- 17.5M

SHA256 fingerprint of <https://raw.githubusercontent.com/pypa/get-virtualenv/20.4.7/public/virtualenv.pyz> verified.

Installing pantsbuild.pants==2.5.0 into a virtual environment at /home/ubuntu/workspace/PR-58095/./.cache/pants/setup/bootstrap-Linux-x86_64/2.5.0_py38

Traceback (most recent call last):

  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main

    return _run_code(code, main_globals, None,

  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code

    exec(code, run_globals)

  File "/home/ubuntu/workspace/PR-58095/./.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-20.4.7/virtualenv.pyz/__main__.py", line 168, in <module>

  File "/home/ubuntu/workspace/PR-58095/./.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-20.4.7/virtualenv.pyz/__main__.py", line 164, in run

  File "/home/ubuntu/workspace/PR-58095/./.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-20.4.7/virtualenv.pyz/virtualenv/__main__.py", line 18, in run

  File "/home/ubuntu/workspace/PR-58095/./.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-20.4.7/virtualenv.pyz/virtualenv/run/__init__.py", line 30, in cli_run

  File "/home/ubuntu/workspace/PR-58095/./.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-20.4.7/virtualenv.pyz/virtualenv/run/__init__.py", line 48, in session_via_cli

  File "/home/ubuntu/workspace/PR-58095/./.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-20.4.7/virtualenv.pyz/virtualenv/run/__init__.py", line 75, in build_parser

  File "/home/ubuntu/workspace/PR-58095/./.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-20.4.7/virtualenv.pyz/virtualenv/run/plugin/seeders.py", line 8, in __init__

  File "/home/ubuntu/workspace/PR-58095/./.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-20.4.7/virtualenv.pyz/virtualenv/run/plugin/base.py", line 39, in options

  File "/home/ubuntu/workspace/PR-58095/./.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-20.4.7/virtualenv.pyz/virtualenv/run/plugin/base.py", line 18, in entry_points_for

  File "/home/ubuntu/workspace/PR-58095/./.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-20.4.7/virtualenv.pyz/virtualenv/run/plugin/base.py", line 18, in <genexpr>

  File "/usr/lib/python3.8/importlib/metadata.py", line 77, in load

    module = import_module(match.group('module'))

  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import

  File "<frozen importlib._bootstrap>", line 991, in _find_and_load

  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import

  File "<frozen importlib._bootstrap>", line 991, in _find_and_load

  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked

ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'
We run Ubuntu 20.04 in CI with Python 3.8. This does not happen with our local runs. Any idea how to resolve this error?
w
hm! my first guess would be that the 
virtualenv
version that is hardcoded by the script changed between the two versions you are using, and the new version is … not ideal?
also, what is 
/usr/lib/python3.8/bin/python3.8 --version
?
e
The issue here occurs whenever virtualenv is installed already for the interpreter the setup script uses:
Copy code
$ docker run --rm -it ubuntu:20.04
root@789c91e7e2d3:/# apt update &>/dev/null && apt install -y curl python3 python3-distutils &>/dev/null
root@789c91e7e2d3:/# curl -sSL -O <https://raw.githubusercontent.com/pypa/get-virtualenv/20.4.7/public/virtualenv.pyz>
root@789c91e7e2d3:/# python3 virtualenv.pyz a.venv
created virtual environment CPython3.8.5.final.0-64 in 274ms
  creator CPython3Posix(dest=/a.venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==21.1.2, setuptools==57.0.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
root@789c91e7e2d3:/# apt install -y python3-virtualenv &>/dev/null
root@789c91e7e2d3:/# python3 virtualenv.pyz b.venv
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "virtualenv.pyz/__main__.py", line 168, in <module>
  File "virtualenv.pyz/__main__.py", line 164, in run
  File "virtualenv.pyz/virtualenv/__main__.py", line 18, in run
  File "virtualenv.pyz/virtualenv/run/__init__.py", line 30, in cli_run
  File "virtualenv.pyz/virtualenv/run/__init__.py", line 48, in session_via_cli
  File "virtualenv.pyz/virtualenv/run/__init__.py", line 75, in build_parser
  File "virtualenv.pyz/virtualenv/run/plugin/seeders.py", line 8, in __init__
  File "virtualenv.pyz/virtualenv/run/plugin/base.py", line 39, in options
  File "virtualenv.pyz/virtualenv/run/plugin/base.py", line 18, in entry_points_for
  File "virtualenv.pyz/virtualenv/run/plugin/base.py", line 18, in <genexpr>
  File "/usr/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'
root@789c91e7e2d3:/# python3 -sSE virtualenv.pyz b.venv
created virtual environment CPython3.8.5.final.0-64 in 148ms
  creator CPython3Posix(dest=/b.venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==21.1.2, setuptools==57.0.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
root@789c91e7e2d3:/#
The fix I demonstrate there (passing the 
-sSE
flags to python to "scrub" the sys.path) is probably not generalizable. Some python installations have parts of the stdlib installed in 
-S
😕. So - issue explained, but I'm not sure the best way to solve. PEX does this sort of scrubbing in a robust way. Maybe we need to curl down the Pex PEX and use that to execute virtualenv?
IOW:
Copy code
$ docker run --rm -it ubuntu:20.04
root@be286eba0744:/# apt update &>/dev/null && apt install -y curl python3 python3-distutils python3-virtualenv &>/dev/null
root@be286eba0744:/# curl -sSL -O <https://github.com/pantsbuild/pex/releases/download/v2.1.42/pex>
root@be286eba0744:/# python3 pex virtualenv==20.4.7 -mvirtualenv -- a.venv
/root/.pex/unzipped_pexes/478cc1fa371ca40aa3e7dafee735ca438d4a243f/.deps/pex-2.1.42-py2.py3-none-any.whl/pex/tools/commands/venv.py:141: PEXWarning: Encountered collision building venv at /root/.pex/venvs/short/0fc6a8ab from /root/.pex/pip.pex/46820cb5af0dcf9295a4e7f30184cc0e9fa063dc:
1. /root/.pex/venvs/d8e5e9b222cbb0506ad915e4c2c72a50c5a61701/739d26ded7518b7bd7f0ef93c3b66e0e826d110e.73631d2ff32e428d8f982890bb5dd795/lib/python3.8/site-packages/constraints.txt was provided by:
	/root/.pex/pip.pex/46820cb5af0dcf9295a4e7f30184cc0e9fa063dc/.deps/setuptools/constraints.txt
	/root/.pex/pip.pex/46820cb5af0dcf9295a4e7f30184cc0e9fa063dc/.deps/wheel/constraints.txt
  pex_warnings.warn(message)
created virtual environment CPython3.8.5.final.0-64 in 280ms
  creator CPython3Posix(dest=/a.venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==21.1.2, setuptools==57.0.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
root@be286eba0744:/#
I'll post that as a PR for discussion.
Filed https://github.com/pypa/virtualenv/issues/2133 against virtualenv and offered to switch them from .pyz to .pex - we'll see what they think of that!
❤️ 1
@rapid-bird-79300 this should now be fixed: https://github.com/pantsbuild/setup/pull/99 When you get a chance to re-curl down the pants script, let me know if this solves the CI issue.
👀 1
✅ 1
r
@enough-analyst-54434 looks like that did the trick! Thank you so much for looking into this
❤️ 1
5 Views
Open in Slack
PreviousNext
Powered by