https://pantsbuild.org/ logo
Join Slack
Powered by
I'm seeing `pex` have issues resolving dependencie...
# general
a
I'm seeing 
pex
have issues resolving dependencies and report conflicts when both 
pip
and 
pipenv
work fine. I've narrowed down the issue to a set containing just these two dependencies: 
"grpcio<1.48.1" "google-api-core[grpc] >= 1.22.1"
I suspect this has to do with how the extras 
[grpc]
is handled. Details in thread.
issue with pex
Copy code
% pex --python=python3.8  "grpcio<1.48.1" "google-api-core[grpc] >= 1.22.1" -o output.pex
Failed to resolve compatible distributions:
1: grpcio-status==1.51.1 requires grpcio>=1.51.1 but grpcio 1.47.2 was resolved
also happens with other 
--pip-version
Copy code
% pex --python=python3.8 --pip-version=22.3 "grpcio<1.48.1" "google-api-core[grpc] >= 1.22.1" -o output.pex
Failed to resolve compatible distributions:
1: grpcio-status==1.51.1 requires grpcio>=1.51.1 but grpcio 1.47.2 was resolved
pipenv works fine
Copy code
% pipenv install "grpcio<1.48.1" "google-api-core[grpc] >= 1.22.1"
Creating a virtualenv for this project...
...
āœ” Installation Succeeded 
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
āœ” Success!
it installs these versions:
Copy code
% pipenv graph
google-api-core==2.11.0
  - google-auth [required: >=2.14.1,<3.0dev, installed: 2.15.0]
    - cachetools [required: >=2.0.0,<6.0, installed: 5.2.0]
    - pyasn1-modules [required: >=0.2.1, installed: 0.2.8]
      - pyasn1 [required: >=0.4.6,<0.5.0, installed: 0.4.8]
    - rsa [required: >=3.1.4,<5, installed: 4.9]
      - pyasn1 [required: >=0.1.3, installed: 0.4.8]
    - six [required: >=1.9.0, installed: 1.16.0]
  - googleapis-common-protos [required: >=1.56.2,<2.0dev, installed: 1.57.0]
    - protobuf [required: >=3.19.5,<5.0.0dev,!=4.21.5,!=4.21.4,!=4.21.3,!=4.21.2,!=4.21.1,!=3.20.1,!=3.20.0, installed: 4.21.11]
  - protobuf [required: >=3.19.5,<5.0.0dev,!=4.21.5,!=4.21.4,!=4.21.3,!=4.21.2,!=4.21.1,!=4.21.0,!=3.20.1,!=3.20.0, installed: 4.21.11]
  - requests [required: >=2.18.0,<3.0.0dev, installed: 2.28.1]
    - certifi [required: >=2017.4.17, installed: 2022.12.7]
    - charset-normalizer [required: >=2,<3, installed: 2.1.1]
    - idna [required: >=2.5,<4, installed: 3.4]
    - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.13]
grpcio-status==1.47.2
  - googleapis-common-protos [required: >=1.5.5, installed: 1.57.0]
    - protobuf [required: >=3.19.5,<5.0.0dev,!=4.21.5,!=4.21.4,!=4.21.3,!=4.21.2,!=4.21.1,!=3.20.1,!=3.20.0, installed: 4.21.11]
  - grpcio [required: >=1.47.2, installed: 1.47.2]
    - six [required: >=1.5.2, installed: 1.16.0]
  - protobuf [required: >=3.12.0, installed: 4.21.11]
just tested and 
google-api-core
doesn't need a pin:
Copy code
% pex --python=python3.8 --pip-version=22.3 "grpcio<1.48.1" "google-api-core[grpc]" -o output.pex 
Failed to resolve compatible distributions:
1: grpcio-status==1.51.1 requires grpcio>=1.51.1 but grpcio 1.47.2 was resolved
e
@abundant-autumn-67998 you also probably want 
--resolver-version pip-2020-resolver
, works for me. And this is another backwards compat thing. Lots to type, but less when Pex 3.x releases next year and these will be defaults.
You may not have been aware Pip still carries its old legacy resolver around inside itself!
Oh Google!:
Copy code
$ pex --python=python3.8 --pip-version=22.3 "grpcio<1.48.1" "google-api-core[grpc]" --resolver-version pip-2020-resolver -- -c 'import grpc; print(grpc.__file__)'
/tmp/tmp5swbyre5/.bootstrap/pex/environment.py:654: PEXWarning: The `pkg_resources` package was loaded from a pex vendored version when declaring namespace packages defined by:

1. google-api-core==2.11.0 namespace packages:
  google

2. googleapis-common-protos==1.57.0 namespace packages:
  google
  google.logging

3. google-auth==2.15.0 namespace packages:
  google

These distributions should fix their `install_requires` to include `setuptools`
  pex_warnings.warn(
/home/jsirois/.pex/installed_wheels/9ca6bdc2ca97419d9154a2858e7190109280143c87f3eda4ad6dada2729dea7e/grpcio-1.47.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/grpc/__init__.py
So you need to help Google out and include setuptools:
Copy code
$ pex --python=python3.8 --pip-version=22.3 setuptools "grpcio<1.48.1" "google-api-core[grpc]" --resolver-version pip-2020-resolver -- -c 'imp
ort grpc; print(grpc.__file__)'
pex --python=python3.8 --pip-version=22.3 "grpcio<1.48.1" "google-api-core[grpc]" -o output.pex/home/jsirois/.pex/installed_wheels/9ca6bdc2ca97419d9154a2858e7190109280143c87f3eda4ad6dada2729dea7e/grpcio-1.47.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/grpc/__init__.py
You don't need to, that's just a warning. You could also tell Pex 
--no-emit-warnings
If there is a way to pass through Pip options via 
pipenv
you can probably get it to fail too:
Copy code
$ pip -V
pip 22.3.1 from /home/jsirois/.pyenv/versions/3.11.0rc2/lib/python3.11/site-packages/pip (python 3.11)

$ pip --use-feature x

Usage:
  pip <command> [options]

option --use-feature: invalid choice: 'x' (choose from 'fast-deps', 'truststore', 'no-binary-enable-wheel-cache')

$ pip --use-deprecated x

Usage:
  pip <command> [options]

option --use-deprecated: invalid choice: 'x' (choose from 'legacy-resolver')
So, Pex's default is to pass 
--use-deprecated legacy-resolver
no matter the Pip version to maintain backwards compat. By passing 
--resolver-version pip-2020-resolver
to Pex, that gets newer versions of `--pip-version`using the 2020 (now default) resolver.
a
Ah ok - adding 
--resolver-version
fixes the test case for me. Thanks!
Today we saw a second build issue with different dependencies that can also be resolved with 
--resolver-version=pip-2020-resolver
. I'm now making this our default build flag, however I'm not sure if adding this will break some users that happen to need the other resolver, so we're keeping the old build flags also as a fallback. I was also thinking of doing a different resolution mechanism based on 
pipenv
1. run 
pipenv lock
2. run 
pipenv requirements --exclude-markers > pipenv-output-requirements.txt
3. run 
pex -r pipenv-output-requirements.txt
I wonder if 
pex
will try to re-resolve the dependencies. Is there a way to tell pex to just install the fully resolved set of dependencies in 3 above?
(also is this a bad idea?)
e
So, how tied are you to pipenv locks? Do you know about 
pex3 lock {create,update,exprt}
?
But 
pex --intransitive ...
against pinned requirements will do the trick.
šŸ‘€ 1
It will still "resolve" and that will be slower than using a 
pex3 lock
file, which includes all the artifact URLs to just do a straight fetch from, but it will give you exactly what's in 
requirements.txt
and no more as long as all those reqs are pinned. If they have hashes, of course those are checked / respected too.
a
We're not tied to pipenv. I'm just thinking of other possible ways to do dependency resolution. I didn't know about 
pex3 lock
- could be useful but I'll wait until it is official.
e
It is official.
Pants has been using it as its production lock for a good while now.
Don't let the 
pex3
name fool you, that has ~nothing to do with the Pex version. There is just no way to introduce subcommands in `pex`reasonably until 3.x
So I carved out a 
pex3
console script to host subcommands.
a
ah interesting ok. i'll take a closer look then.
talking about dependencies, is there a way in to include a package from local source directories? for instance, we call 
python setup.py build
and then point 
pex -D
at the build output directory. this lets us package the source without resolving or building the dependencies. i assume there is no pex native way of doing something like this?
e
just point pex at the directory containing the setup.py: 
pex local_project/ ...
This is how Pex builds itself. See the `pex_requirement = "."`here: https://github.com/pantsbuild/pex/blob/312fa138a66e7c1d08a59cd7cffc6912ebb13f7e/scripts/package.py#L28-L59
The Pex project uses Flit / 
pyproject.toml
and not 
setup.py
any longer, but works for any Python buildable project.
a
ah i just tested with 
pex /path/to/local_project --instransitive -o source.pex
and looks like is what i need. it just builds the source without the dependencies. the dependencies are injected at run time with 
PEX_PATH
so i'll try that next.
e
What drives the split and use of PEX_PATH to re-join?
a
we do two builds: 1. local project(s) without any deps - source.pex 2. dependencies of local project(s), without the local project itself - deps.pex our build process builds two pex files from above. our runtime invokes something like 
PEX_PATH=deps.pex source.pex
we identify the combination with a string like 
files=deps-HASH.pex:source-HASH.pex
. typically we don't need to rebuild deps.pex which is what makes deploys fast.
e
Gotcha.
a
for 2 above, we parse requirements.txt and the output of setup.py to create a derived requirements.txt that we feed to pex.
e
So one thing to note about Pex locks is they can be subsetted.
So if you locka project, you can then consume the lock like 
pex --lock lock.json just-this-req
And you'll only get `just-this-req`'s subgraph.
If you pass no reqs, you get the whole lock.
And, of course, you can pass one or more reqs to get their combined subgraph, etc.
a
very interesting. for 2 above we need a severed graph - the subgraph (deps only) without the root node (local source). though maybe leaving the source package in 
deps.pex
is ok because it will get overwritten with 
source.pex
or maybe we can just get the subgraph of (direct deps of source)
e
That would do it.
The subsetting is pretty fast. If you've resolved a whole lock before on a machine, the subset should be ~500ms.
Really depends on the size of the wheels. Some AI / ML stuff is slow just due to GB wheels.
As you are becoming aware, I could spend a whole week catching up Pex docs.
I will do that in the new year.
a
lol yeah. i keep finding more useful stuff in pex every day.
e
The most fun easter egg lies in PEX_TOOLS=1 against a PEX file that was built with 
--include-tools
or `--venv`(or by using the `pex-tools`console script): 
pex-tools my.pex graph --open
a
I tried using 
--intransitive
and can't run the built pex because it needs the dependencies. Using 
PEX_PATH=deps.pex
with the dependencies doesn't seem to satisfy a pex built with 
--intransitive
. I keep getting this error, trying `PEX_PATH=dagster-deps.pex ./intransitive.pex`:
Copy code
No interpreter compatible with the requested constraints was found:

  Failed to resolve requirements from PEX environment @ /Users/shalabh/.pex/unzipped_pexes/4ccbb52b80202f20d56767fbebf1e97a0319d337.
  Needed cp38-cp38-macosx_12_0_x86_64 compatible dependencies for:
   1: dagster
      Required by:
        my-dagster-project 0.0.0
      But this pex had no ProjectName(raw='dagster', normalized='dagster') distributions.
   2: dagster-aws
      Required by:
        my-dagster-project 0.0.0
      But this pex had no ProjectName(raw='dagster-aws', normalized='dagster-aws') distributions.
I tried multiple ways to unpack this or merge the dependencies pex with the intransitive pex, but keep getting the above error. Even 
pex-tools intransitive.pex venv here
wont install the venv.
Oh using `--intransitive --ignore-errors`seems to be the way to go. I can then merge the deps with 
PEX_PATH
without issues.
e
Ok. You can probably also just not use 
--intransitive
. Since you're resolving against hashed requirements, Pip will fail if it needs to grab anything not in the requirements.
10 Views
Open in Slack
PreviousNext
Powered by