How do I upgrade the `pip` inside of the `venv` fo...
# development
b
How do I upgrade the
pip
inside of the
venv
for a package in my
PEX_ROOT
? I upgraded system
pip
, but the
pip
inside the
venv
doesn't seem to mirror it. ( which aligns with `venv`s) Specifically, installing
cryptography
is failing, and I think a newer pip would bypass the build and download built wheels?
1
e
1st off, Pex is hermetic as its ~1st rule. Your system foo should have no effect on it in any aspect if Pex works right. How did you get a pip in your venv in the PEX_ROOT? For venvs you create manually, there is `--pip`:
Copy code
$ pex-tools /dev/null venv -h
usage: pex-tools PATH venv [-h] [--scope {all,deps,srcs}] [-b {false,prepend,append}] [-f] [--collisions-ok] [-p] [--copies] [--compile] [--prompt PROMPT] [--rm {pex,all}] [--emit-warnings] [--pex-root PEX_ROOT] [--disable-cache] [--cache-dir CACHE_DIR] [--tmpdir TMPDIR]
                           [--rcfile RC_FILE]
                           PATH

Creates a venv from the PEX file.

positional arguments:
  PATH                  The directory to create the virtual environment in.

options:
  -h, --help            show this help message and exit
  --scope {all,deps,srcs}
                        The scope of code contained in the Pex that is installed in the venv. By defaultall code is installed and this is generally what you want. However, in some situations it's beneficial to split the venv installation into deps and srcs steps. This is
                        particularly useful when installing a PEX in a container image. See <https://pex.readthedocs.io/en/latest/recipes.html#pex-app-in-a-container> for more information. (default: all)
  -b {false,prepend,append}, --bin-path {false,prepend,append}
                        Add the venv bin dir to the PATH in the __main__.py script. (default: false)
  -f, --force           If the venv directory already exists, overwrite it. (default: False)
  --collisions-ok       Don't error if population of the venv encounters distributions in the PEX file with colliding files, just emit a warning. (default: False)
  -p, --pip             Add pip to the venv. (default: False)
You should be able to use that pip to upgrade itself to any version you like.
b
How did you get a pip in your venv in the PEX_ROOT?
This is through Pants, so whatever Pants PEX incantations
e
Huh, ... is this from
./pants export
?
We shouldn't include
--pip
otherwise - Pants should be using immutable things.
b
No. Specifically I'm locking my
flake8
resolve, and on CI when trying to build the PEX it's started to build cryptography
ProcessExecutionFailure: Process 'Building flake8.pex from 3rdparty/python/lockfiles/flake8.lock' failed with exit code 1.
/home/buildagent/.cache/pants/named_caches/pex_root/venvs/s/3663269c/venv/bin/python3.8 /home/buildagent/.cache/pants/named_caches/pex_root/venvs/s/3663269c/venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /home/buildagent/.cache/pants/named_caches/pex_root/tmp/pip-build-env-3oef15hl/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i <https://pypi.org/simple/> -- 'poetry>=0.12'
Wonder why its trying to pex up poetry 😐
e
Ok, your terminology threw me off I think. You should find no
pip
in your venvs under the PEX_ROOT. Are you referring to the
pip
pex
uses to build your PEX? That's a vendored
pip
and you cannot upgrade it.
b
This
/home/buildagent/.cache/pants/named_caches/pex_root/venvs/s/3663269c/venv/lib/python3.8/site-packages/pip
doens't look like vendored pip to me 🤔
But also, I have no clue why
poetry
need to get involved 😮
e
Probabaly a pep-517 build of an sdist in the resolve.
b
Ahhhhhhhhh
Yes
e
The `poetry-core`provides a PEP-517 build backend alot of folks use.
b
I've got VCS reqs, I bet one of them uses
poetry
Good thought! coke
e
What does coke mean for you? The history is Eric Ayers from Square in Atlanta - where coke is headquartered - and he is as old as me and refers to "You owe me a coke!" when you say the ~same thing as someone else at the ~same time.
b
IDK I saw Eric do it for "you win a coke, thats it"
Perhaps I'm misuing 😛
e
Ah, ok. The origin was the opposite - you lose a coke!
"You owe me a coke!" is roughly from my grandparents era.
Their marketing is deep.
Back to cryptography, can you get a repro using Pex directly (newest) and
--preserve-pip-download-log
?
b
This is a CI machine, so admittedly a bit hard to run cod eon
e
Just running on your machine is fine. That will give good info to start.
Even if the result is success. I can use the lockfile + command line to learn more.
b
FWIW I can't repo, but I also have a rust compiler so that mightve been hidden from me and I didn't know
Ah I can send lockfile, one sec
Specifically it looks like _git+https://github.com/thejcannon/darglint@deco_attr#egg=darglint_ is to blame
e
I'm confused - that has 0 production requirements if I read that correctly.
SO how is that to blame for cryptography, or is cryptograpy needed by the PEP-517 build there indirectly?
I could switch my branch to use the newer lighter
core
backend I suspect
e
Yeah, that's a good idea to try out.
So the relevant debug command though then will use
poetry>=0.12
as the sole requirement.
Copy code
pex3 lock create --python python3.8 --resolver-version pip-2020-resolver "poetry>=0.12" --indent 2 -o lock.json
And I find the cryptography needed is 37.0.4 which is satisfied by cryptography-37.0.4-cp36-abi3-manylinux_2_24_x86_64.whl in my case. What is the arch of the CI machine?
b
Ubuntu 20 64bit
e
Arm or x86?
b
95% sure x86
py 38
e
Then I'm stumped, I'm also x84 64 bit and my test was with 3.8
There are also wheels for ARM.
b
I just bumped
pyproject.toml
on that branch. lets see what happens
e
The more data you can get from the CI machine the better. Flying blind is flying blind.
The pre-released wheels really have the bases covered: https://pypi.org/project/cryptography/37.0.4/#files It makes no sense to me currently why the CI machine would have to resort to building the sdist.
the cp36-abi3-* should be picked up unless the distro is so old it is not manylinux2014 compatible, but Ubuntu 20.* is.
Actually - there is a manylinux2010 wheel even for x86.
b
Newer
pyproject.toml
worked
I'm packing it up, there's like 3 fires going on I gotta put out 🙈
🧯 2