https://pantsbuild.org/ logo
Join Slack
Powered by
How can I control the version of pip/virtualenv us...
# general
e
How can I control the version of pip/virtualenv used by pants? Does it just use system pip?
h
Hello! In what context? When you install Pants the first time, or the version of Pip (and Pex) used when running commands like 
./pants test
? Also, what Pants version are you on? (
./pants --version
)
e
when I run pants test, for example
we're on 1.16.0
👍 1
The issue I'm having is that our Jenkins workers have an old pip and I can't upgrade them at the moment.
So we're running into the setuptools "bug"
because a new version is getting pulled in via dependencies
h
So we’re running into the setuptools “bug”
What is the bug?
I pulled up the 1.16.x branch to try to find the option. Looking.
e
Copy code
Exception caught: (pex.resolver.Untranslateable) (backtrace omitted)
Exception message: Package SourcePackage('file:///mnt/workspace/.pants.d/python-setup/resolved_requirements/CPython-2.7.17/setuptools-47.1.0.zip') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
Copy code
stderr:
pkg_resources/py2_warn.py:15: UserWarning: Setuptools no longer works on Python 2
************************************************************
Encountered a version of Setuptools that no longer supports
this version of Python. Please head to
<https://bit.ly/setuptools-py2-sunset> for support.
************************************************************
the test runs fine on a host with a newer pip installed
h
Hm. I asked another maintainer if they know a better solution, but the best I’ve found for Pants 1.16 is the options 
setuptools_version
in both the 
[python-setup]
and 
[pex-build-wrapper]
scopes.
e
It's pip version I want to control though.
Setuptools is already pinned to <45
h
With more recent versions of Pants, there is an option to control the Pex version. As of Pants 1.26, Pex vendors Pip to do all resolution; so controlling the version of Pex controls the version of Pip. There might a similar option with 1.16, but I can’t find it. Other maintainers might know
cc @witty-crayon-22786 as well
e
ok thanks, we can probably upgrade but I'm going to try another workaround first.
h
Is there anything blocking an upgrade to more recent Pants versions? Is the Python 2 -> Python 3 migration for custom plugins in 1.17 a blocker?
e
no, I'm pretty sure we can upgrade
h
Awesome. If you’re able to upgrade to 1.26, this should no longer be an issue. See https://pants.readme.io/docs/upgrade-tips for some tips on upgrading
(It mentions pants.toml, but you can still use pants.ini)
If you’re able to get to 1.25 or 1.26, one of the major new features is file arguments. You can now run 
./pants test project/app.py
, rather than needing to use a target address
e
@echoing-manchester-70122 pants does not use pip at all except in the setup script (the ./pants script.) The error you paste is long after where pants runs fine, but when executing python tasks hits a problem with pex (which does not use pip in your version). Just wanted to clear up that red herring. Now looking at the conversation a bit more...
1st bit, @echoing-manchester-70122 re:
Setuptools is already pinned to <45
What mechanism are you using to do this?
e
requirements.txt
and BUILD
e
OK, and what goal do you hit this failure for. For example, is it 
test
or 
run
or some other goal?
e
test
e
Ok, and, finally I think, what do you have configured - if anything for interpreter_constraints in pants.ini? IE: do you expect Python 2.7.17?
e
>3.6, <4
is the default,
modules that haven't been migrated add their own compatibility line
>2.7, <3
e
And your failure case includes those modules or not?
e
yes, the modules are run individually
so the interpreter version changes for each test
potentially
e
OK, hopefully. I'll assume 2.7.17 is the correct interpreter for your test goal target set.
e
yes
e
OK .. poking around to see which setuptools knob is appropriate or missing here....
So reading closely:
Copy code
.pants.d/python-setup/resolved_requirements/CPython-2.7.17/setuptools-47.1.0.zip
Says the phase in the test goal is resolution of requirements for the test targets themselves. That implies none of the test targets has a direct dependency on your requirements.txt / BUILD pinning of setuptools under 45.
e
correct, and adding setuptools to the BUILD file for the module that's failing fixes it in one place but it still fails in Jenkins
e
To check this, do something like this:
Copy code
$ ./pants dependencies src/python/pants/base: 2>/dev/null | grep 3rdparty/
3rdparty/python:pathspec
3rdparty/python:setproctitle
3rdparty/python:pystache
3rdparty/python:typing-extensions
3rdparty/python:dataclasses
3rdparty/python/twitter/commons:twitter.common.dirutil
Here substitute the targets you pass to test for 
src/python/pants/base:
and change the grep to match wherever your requirements.txt / BUILD live.
e
you mean 
./pants dependees
right?
e
no, 
dependencies
e
Copy code
./pants dependencies src/test/python/.../cloudwatch_to_tsd::
Unknown goals: dependencies
e
Sorry about that - I though we had that goal going way back. How about:
Copy code
./pants depmap --minimal --internal-only src/python/pants/base: | grep 3rdparty | grep setuptools
h
Hm, we wrote 
dependencies
in 2014. Possibly it’s not activated? You’ll want to add 
pants.backend.project_info
to the 
backend_packages
option in the 
[GLOBAL]
scope. It should be activated by default, but maybe got deactivated?
e
Copy code
./pants dependencies src/test/python/p/cloudwatch_to_tsd::
src/test/python/p/cloudwatch_to_tsd:cloudwatch_to_tsd
3rdparty/python:moto
moto==0.4.27
3rdparty/python:requirements.txt
src/main/python/p/cloudwatch_to_tsd:cloudwatch_to_tsd
src/main/python/p/cloudwatch_to_tsd:_cloudwatch_to_tsd
3rdparty/python:boto3
boto3==1.11.3
3rdparty/python:brood
brood==0.3.2
3rdparty/python:pyknox
pyknox==0.3.4
3rdparty/python:schedule
schedule==0.4.2
3rdparty/python:pinstatsd
pinstatsd==1.4.2
3rdparty/python:metrics-agent
metrics-agent==0.4.67
3rdparty/python:setuptools
setuptools<45
we don't have src/python/pants
e
Hmm ok, so that target set does have a direct dependency on setuptools<45 ... thinking of more questions...
e
yeah something else is pulling in the later version
e
Do you have any configuration in pants.ini for setuptools?
e
no
e
Basically does grep match it?
OK
hrm
OK, the only remaining thing I can think of is a Pex bug. That version of Pants uses Pex 1.6.7. I'll see what got solved in changelogs from 1.6.8 - 1.6.12 - the last Pex before Pex 2.x.
e
I've been told it's due to the old base pip version (9.0.1) on ubuntu
that's what started this. If I add setuptools into the BUILD file and run tests with newer pip, it works
e
You have to do both? The above dependeencies output shows setuptools<45 is already in the dependency set.
e
yes, previously setuptools was not in that BUILD file
e
Aha - ok. So on that version of pants I think we have 2 issues to fix: 1. pin setuptools low without hacking BUILDs. 2. upgrade pex from 1.6.7 to 1.6.12 For 1, we support constraints.txt in newer Pants but you don't have that so I think we can use the 
[pytest] requirements
option instead. If you don't have that in your pants.ini, please add:
Copy code
[pytest]
requirements: ["pytest>=3.0.7,<3.7", "setuptools<45"]
The remove your BUILD addition of a setuptools<45 dependency and re-run the tests. Before the re-run you'll need to clean-all unfortantely. It looks like that option is not marked to invalidate things in Pants 1.16.
👍 1
e
makes sense, ok giving that a try
we do clean-all every run
e
For 2, you should be able to add this to pants.ini
Copy code
[GLOBAL]
plugins: ["pex==1.6.12"]
👍 2
e
Copy code
$ ./pants test src/test/python/.../cloudwatch_to_tsd::
22:07:51 [INFO] Resolving new plugins...:
  pex==1.6.12
  pantsbuild.pants.contrib.python.checks==1.16.0
timestamp: 2020-05-28T22:07:51.701232
Exception caught: (pex.resolver.Unsatisfiable) (backtrace omitted)
Exception message: Could not satisfy all requirements for pex==1.6.12:
    pex==1.6.12, pex==1.6.7(from: pantsbuild.pants.contrib.python.checks==1.16.0)
e
Hrm, right.
e
I'lll update pants version
h
Cool. I’m looking for the first version to use that Pex release
e
It looks like 1st stable is 1.23.0
👍 1
h
* 1.18.0 upgrades to Pex 1.6.8 * 1.20.0 upgrades to Pex 1.6.10 * 1.21.0 upgrades to Pex 1.6.11 * 1.23.0 upgrades to Pex 1.6.12
e
fuck's sake, now I am running into this stupid apt_pkg bug
😢 1
h
John, is this fixed in 1.6.12 or earlier? It’ll be easier to upgrade to older Pants versions.
now I am running into this stupid apt_pkg bug
What’s this one?
e
It looks like we never had a 1.22.0, just a 1.22.0rc0, so yeah 1.23.0:
Copy code
$ git describe b720e6937a7a3cbaa5b28bbbfdb5bb4d2310ab70
release_1.22.0rc0-1-gb720e6937
👍 1
👀 1
Adam - if that's all too much, Pex 1.6.9 contains the likely fix so Pants 1.20.0 might suffice.
e
Copy code
Traceback (most recent call last):
  File "/home/amckenna/.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-16.4.3/virtualenv.py", line 22, in <module>
    import distutils.spawn
ModuleNotFoundError: No module named 'distutils.spawn'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'
this is on 3.7
on ubuntu18
Copy code
$ ./pants
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   122  100   122    0     0  10166      0 --:--:-- --:--:-- --:--:-- 10166
100   281  100   281    0     0   5978      0 --:--:-- --:--:-- --:--:--  5978
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 3626k  100 3626k    0     0  22.9M      0 --:--:-- --:--:-- --:--:-- 49.6M
Traceback (most recent call last):
  File "/home/amckenna/.cache/pants/setup/bootstrap-Linux-x86_64/virtualenv-16.4.3/virtualenv.py", line 22, in <module>
    import distutils.spawn
ModuleNotFoundError: No module named 'distutils.spawn'
./pants: line 170: /home/amckenna/.cache/pants/setup/bootstrap-Linux-x86_64/pants.v1BtqB/install/bin/pip: No such file or directory
./pants: line 171: /home/amckenna/.cache/pants/setup/bootstrap-Linux-x86_64/pants.v1BtqB/install/bin/pip: No such file or directory
New virtual environment successfully created at /home/amckenna/.cache/pants/setup/bootstrap-Linux-x86_64/1.23.0_py38.
./pants: line 193: /home/amckenna/.cache/pants/setup/bootstrap-Linux-x86_64/1.23.0_py38/bin/python: No such file or directory
e
That's part of the stdlib so Python3.7 on Ubuntu 18 appears broken: https://docs.python.org/3.7/library/distutils.html
h
does this fix it? 
sudo apt install python3-distutils -y
(disclosure: first thing I found on google)
e
Copy code
$ python3
Python 3.7.5 (default, Nov  7 2019, 10:50:52)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils.spawn
>>>
oh it's the 3.8 one that's barfing maybe
I really hope debian/ubuntu clean up their python stuff soon
🤞 1
this is getting unmanageable
h
Ah, yep. You can change the 
./pants
script to put Py38 behind Py37. If you use the latest 
./pants
script, we do this already: https://github.com/pantsbuild/setup/blob/gh-pages/pants#L110
e
Copy code
ERROR: Invalid requirement, parse error at "'["pytest'"
if I remove that, I get this error:
Copy code
**** Failed to install configparser-5.0.0 (caused by: NonZeroExit("received exit code 1 during execution of `['/usr/bin/python2.7', '-s', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpy_m190mb']` while trying to execute `['/usr/bin/python2.7', '-s', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpy_m190mb']`")
):
stdout:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
copying src/configparser.py -> build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/backports
copying src/backports/__init__.py -> build/lib.linux-x86_64-2.7/backports
creating build/lib.linux-x86_64-2.7/backports/configparser
copying src/backports/configparser/helpers.py -> build/lib.linux-x86_64-2.7/backports/configparser
copying src/backports/configparser/__init__.py -> build/lib.linux-x86_64-2.7/backports/configparser
running egg_info

stderr:
error: 'egg_base' must be a directory name (got `src`)
e
Adam, where is 
RROR: Invalid requirement, parse error at "'["pytest'"
coming from? Can you paste the exact snippet you removed? Is that from a pants.ini entry addition described above?
e
yes
Copy code
[pytest]
timeout_requirements: pytest-timeout==1.2.0
requirements: ["pytest<3.7,>=3.0.7", "setuptools<45"]
cov_requirements: pytest-cov==2.4.0
e
Ah right - its not a list option 😕
Hrm.
h
Normally we had people override things like 
unittest2
(This all got cleaned up in later Pants version)
e
Yeah - that's the only option - pick one you don't actually use.
e
well I'm using 1.23 now
now getting
Copy code
Exception message: Package SourcePackage('file:///mnt/efs-home/amckenna/code/python-commons/.pants.d/python-setup/resolved_requirements/CPython-2.7.15/configparser-5.0.0.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
h
Something like:
Copy code
[pytest]
pytest_plugins = +["setuptools<45"]
e
Maybe 2 less quotes.
h
Yeah, edited
e
= or :?
h
Doesn’t matter with INI. Either works
I got in the habit of 
=
because that’s what TOML uses
e
Copy code
[pytest]
version: >=3.0.7,<3.7
pytest_plugins: +["setuptools<45"]
seems to be the format it wants
h
Yes, that looks right. And if configparser still gives you trouble, you can pin it there too by adding it to 
pytest_plugins
e
Nope still wrong, it wants:
Copy code
[pytest]
version: pytest>=4.6.6,<4.7
pytest_plugins: +["setuptools<45"]
👍 1
any way to turn off the .whl dump?
h
any way to turn off the .whl dump?
What do you mean?
e
Copy code
Dumping distribution: .../moto-0.4.27-py2.py3-none-any.whl
                     Dumping distribution: .../boto3-1.11.3-py2.py3-none-any.whl
now getting
Copy code
22:57:20 00:22       [run]
                     Failed to execute PEX file, missing linux_x86_64-cp-27-cp27mu compatible dependencies for:
                     importlib-metadata
                     zipp

22:57:23 00:25       [coverage-html]
                     Failed to execute PEX file, missing linux_x86_64-cp-27-cp27mu compatible dependencies for:
                     importlib-metadata
                     zipp

22:57:24 00:26       [coverage-xml]
                     Failed to execute PEX file, missing linux_x86_64-cp-27-cp27mu compatible dependencies for:
                     importlib-metadata
                     zipp
                     22:57:24 [DEBUG] pants.base.project_tree:pid=23020: ProjectTree ignore_patterns: None
pin those as well I guess
👍 1
h
I’m not sure what that Dumping thing. Where is that happening? For example, is it in the 
[test]
block of Pants output or something like 
[gen]
?
e
it shows up when I run with 
-ldebug
👍 1
it's before test target is getting run
in the cache target look like
Copy code
23:02:39 00:01       [cache]
                   No cached artifacts for 8 targets.
                   Invalidated 8 targets.
                     Dumping requirement: PythonRequirement(moto==0.4.27)
                     Dumping requirement: PythonRequirement(boto3==1.11.3)
                     Dumping requirement: PythonRequirement(brood==0.3.2)
                     Dumping requirement: PythonRequirement(pyknox==0.3.4)
                     Dumping requirement: PythonRequirement(schedule==0.4.2)
                     Dumping requirement: PythonRequirement(pinstatsd==1.4.2)
                     Dumping requirement: PythonRequirement(metrics-agent==0.4.67)
                     Dumping requirement: PythonRequirement(setuptools<45)
and only after a clean-all
oh, I guess they're being generated for pex
👍 1
h
That sounds right. Offhand, I don’t know of any way to filter out log messages. But we use standard Python logging, and I think they have a way to silence certain deprecation warnings through things like env vars
e
no I didn't mean filtering out the log messages. I thought it was dumping .whl files in the cache for PEX generation and I didn't understand why that was needed for tests, but then I remembered that tests are also run within a pex
👍 1
finally passing with
Copy code
[pytest]
version: pytest>=4.6.6,<4.7
pytest_plugins: +["setuptools<45; python_version < '3'","configparser==4.0.2; python_version < '3'","zipp<2; python_version < '3'","importlib-metadata<1.6; python_version < '3'"]
💯 1
❤️ 1
Thanks for the help!
h
You’re welcome! Sorry it was so much trouble to get working, but I’m glad we figured it out! FYI Pants 1.24 adds support for Flake8. In a later Pants version, 
python.checks
gets deprecated in favor of Flake8. 
python.checks
was a weird custom version of Flake8 with fewer checks and without support for Flake8 plugins. See https://pants.readme.io/docs/python-linters-and-formatters for how to activate Flake8
e
ok cool thanks
what's the new syntax for --test-pytest-options="-v" ?
h
--pytest-args='-v'
, or in your pants.ini, 
args
option in the 
[pytest]
scope. You can still use 
./pants test blah -- -v
too, contrary to the deprecation warning. See https://pants.readme.io/docs/upgrade-tips#ignore-deprecation-messages-with-ignore_pants_warnings for how to ignore the deprecation warning
--pytest-args
is an improvement in that it will “shlex” the strings for you. You can do this:
Copy code
./pants test project:tests --pytest-args='-vv -k test_demo'
e
nice
looks like you all are putting a lot of work in
h
Into Python in particular. It got a lot better the past few months, especially if you try out the V2 experience. Some key highlights: * lock files * more linters / autoformatters * file arguments (work with V1 too) * better caching * less verbose output thanks to a new dynamic UI * much better parallelism; you can run all your tests in parallel (limited by your cores)
e
V2 meaning pex v2?
h
V2 meaning the V2 engine of Pants. The past 3 or so years, we had been slowly rewriting Pants how we wish we had built it from the start. It used to require a lot of work to get caching and parallelism right, and now those things fall out of the design. Same with extensibility: it’s now much easier to write plugins. The past 6 months, it’s come into fruition to the point where Python is now a much better experience than what we were able to do before. Now, we’re putting some last polish onto the V2 Python experience, and then starting to look into supporting other languages
Speaking of which, if you have a few moments, we would love for you to fill out this short survey about how you use Pants: https://t.co/za6xeOxZ6S It helps us to plan the roadmap for Pants, e.g. which language we next rewrite with the V2 engine.
e
Sure I'll take a look
❤️ 1
Open in Slack
PreviousNext
Powered by