I'm seeing `pex` have issues resolving dependencie...
# general
I'm seeing
have issues resolving dependencies and report conflicts when both
work fine. I've narrowed down the issue to a set containing just these two dependencies:
"grpcio<1.48.1" "google-api-core[grpc] >= 1.22.1"
I suspect this has to do with how the extras
is handled. Details in thread.
issue with pex
Copy code
% pex --python=python3.8  "grpcio<1.48.1" "google-api-core[grpc] >= 1.22.1" -o output.pex
Failed to resolve compatible distributions:
1: grpcio-status==1.51.1 requires grpcio>=1.51.1 but grpcio 1.47.2 was resolved
also happens with other
Copy code
% pex --python=python3.8 --pip-version=22.3 "grpcio<1.48.1" "google-api-core[grpc] >= 1.22.1" -o output.pex
Failed to resolve compatible distributions:
1: grpcio-status==1.51.1 requires grpcio>=1.51.1 but grpcio 1.47.2 was resolved
pipenv works fine
Copy code
% pipenv install "grpcio<1.48.1" "google-api-core[grpc] >= 1.22.1"
Creating a virtualenv for this project...
āœ” Installation Succeeded 
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
āœ” Success!
it installs these versions:
Copy code
% pipenv graph
  - google-auth [required: >=2.14.1,<3.0dev, installed: 2.15.0]
    - cachetools [required: >=2.0.0,<6.0, installed: 5.2.0]
    - pyasn1-modules [required: >=0.2.1, installed: 0.2.8]
      - pyasn1 [required: >=0.4.6,<0.5.0, installed: 0.4.8]
    - rsa [required: >=3.1.4,<5, installed: 4.9]
      - pyasn1 [required: >=0.1.3, installed: 0.4.8]
    - six [required: >=1.9.0, installed: 1.16.0]
  - googleapis-common-protos [required: >=1.56.2,<2.0dev, installed: 1.57.0]
    - protobuf [required: >=3.19.5,<5.0.0dev,!=4.21.5,!=4.21.4,!=4.21.3,!=4.21.2,!=4.21.1,!=3.20.1,!=3.20.0, installed: 4.21.11]
  - protobuf [required: >=3.19.5,<5.0.0dev,!=4.21.5,!=4.21.4,!=4.21.3,!=4.21.2,!=4.21.1,!=4.21.0,!=3.20.1,!=3.20.0, installed: 4.21.11]
  - requests [required: >=2.18.0,<3.0.0dev, installed: 2.28.1]
    - certifi [required: >=2017.4.17, installed: 2022.12.7]
    - charset-normalizer [required: >=2,<3, installed: 2.1.1]
    - idna [required: >=2.5,<4, installed: 3.4]
    - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.13]
  - googleapis-common-protos [required: >=1.5.5, installed: 1.57.0]
    - protobuf [required: >=3.19.5,<5.0.0dev,!=4.21.5,!=4.21.4,!=4.21.3,!=4.21.2,!=4.21.1,!=3.20.1,!=3.20.0, installed: 4.21.11]
  - grpcio [required: >=1.47.2, installed: 1.47.2]
    - six [required: >=1.5.2, installed: 1.16.0]
  - protobuf [required: >=3.12.0, installed: 4.21.11]
just tested and
doesn't need a pin:
Copy code
% pex --python=python3.8 --pip-version=22.3 "grpcio<1.48.1" "google-api-core[grpc]" -o output.pex 
Failed to resolve compatible distributions:
1: grpcio-status==1.51.1 requires grpcio>=1.51.1 but grpcio 1.47.2 was resolved
@abundant-autumn-67998 you also probably want
--resolver-version pip-2020-resolver
, works for me. And this is another backwards compat thing. Lots to type, but less when Pex 3.x releases next year and these will be defaults.
You may not have been aware Pip still carries its old legacy resolver around inside itself!
Oh Google!:
Copy code
$ pex --python=python3.8 --pip-version=22.3 "grpcio<1.48.1" "google-api-core[grpc]" --resolver-version pip-2020-resolver -- -c 'import grpc; print(grpc.__file__)'
/tmp/tmp5swbyre5/.bootstrap/pex/environment.py:654: PEXWarning: The `pkg_resources` package was loaded from a pex vendored version when declaring namespace packages defined by:

1. google-api-core==2.11.0 namespace packages:

2. googleapis-common-protos==1.57.0 namespace packages:

3. google-auth==2.15.0 namespace packages:

These distributions should fix their `install_requires` to include `setuptools`
So you need to help Google out and include setuptools:
Copy code
$ pex --python=python3.8 --pip-version=22.3 setuptools "grpcio<1.48.1" "google-api-core[grpc]" --resolver-version pip-2020-resolver -- -c 'imp
ort grpc; print(grpc.__file__)'
pex --python=python3.8 --pip-version=22.3 "grpcio<1.48.1" "google-api-core[grpc]" -o output.pex/home/jsirois/.pex/installed_wheels/9ca6bdc2ca97419d9154a2858e7190109280143c87f3eda4ad6dada2729dea7e/grpcio-1.47.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/grpc/__init__.py
You don't need to, that's just a warning. You could also tell Pex
If there is a way to pass through Pip options via
you can probably get it to fail too:
Copy code
$ pip -V
pip 22.3.1 from /home/jsirois/.pyenv/versions/3.11.0rc2/lib/python3.11/site-packages/pip (python 3.11)

$ pip --use-feature x

  pip <command> [options]

option --use-feature: invalid choice: 'x' (choose from 'fast-deps', 'truststore', 'no-binary-enable-wheel-cache')

$ pip --use-deprecated x

  pip <command> [options]

option --use-deprecated: invalid choice: 'x' (choose from 'legacy-resolver')
So, Pex's default is to pass
--use-deprecated legacy-resolver
no matter the Pip version to maintain backwards compat. By passing
--resolver-version pip-2020-resolver
to Pex, that gets newer versions of `--pip-version`using the 2020 (now default) resolver.
Ah ok - adding
fixes the test case for me. Thanks!
Today we saw a second build issue with different dependencies that can also be resolved with
. I'm now making this our default build flag, however I'm not sure if adding this will break some users that happen to need the other resolver, so we're keeping the old build flags also as a fallback. I was also thinking of doing a different resolution mechanism based on
1. run
pipenv lock
2. run
pipenv requirements --exclude-markers > pipenv-output-requirements.txt
3. run
pex -r pipenv-output-requirements.txt
I wonder if
will try to re-resolve the dependencies. Is there a way to tell pex to just install the fully resolved set of dependencies in 3 above?
(also is this a bad idea?)
So, how tied are you to pipenv locks? Do you know about
pex3 lock {create,update,exprt}
pex --intransitive ...
against pinned requirements will do the trick.
šŸ‘€ 1
It will still "resolve" and that will be slower than using a
pex3 lock
file, which includes all the artifact URLs to just do a straight fetch from, but it will give you exactly what's in
and no more as long as all those reqs are pinned. If they have hashes, of course those are checked / respected too.
We're not tied to pipenv. I'm just thinking of other possible ways to do dependency resolution. I didn't know about
pex3 lock
- could be useful but I'll wait until it is official.
It is official.
Pants has been using it as its production lock for a good while now.
Don't let the
name fool you, that has ~nothing to do with the Pex version. There is just no way to introduce subcommands in `pex`reasonably until 3.x
So I carved out a
console script to host subcommands.
ah interesting ok. i'll take a closer look then.
talking about dependencies, is there a way in to include a package from local source directories? for instance, we call
python setup.py build
and then point
pex -D
at the build output directory. this lets us package the source without resolving or building the dependencies. i assume there is no pex native way of doing something like this?
just point pex at the directory containing the setup.py:
pex local_project/ ...
This is how Pex builds itself. See the `pex_requirement = "."`here: https://github.com/pantsbuild/pex/blob/312fa138a66e7c1d08a59cd7cffc6912ebb13f7e/scripts/package.py#L28-L59
The Pex project uses Flit /
and not
any longer, but works for any Python buildable project.
ah i just tested with
pex /path/to/local_project --instransitive -o source.pex
and looks like is what i need. it just builds the source without the dependencies. the dependencies are injected at run time with
so i'll try that next.
What drives the split and use of PEX_PATH to re-join?
we do two builds: 1. local project(s) without any deps - source.pex 2. dependencies of local project(s), without the local project itself - deps.pex our build process builds two pex files from above. our runtime invokes something like
PEX_PATH=deps.pex source.pex
we identify the combination with a string like
. typically we don't need to rebuild deps.pex which is what makes deploys fast.
for 2 above, we parse requirements.txt and the output of setup.py to create a derived requirements.txt that we feed to pex.
So one thing to note about Pex locks is they can be subsetted.
So if you locka project, you can then consume the lock like
pex --lock lock.json just-this-req
And you'll only get `just-this-req`'s subgraph.
If you pass no reqs, you get the whole lock.
And, of course, you can pass one or more reqs to get their combined subgraph, etc.
very interesting. for 2 above we need a severed graph - the subgraph (deps only) without the root node (local source). though maybe leaving the source package in
is ok because it will get overwritten with
or maybe we can just get the subgraph of (direct deps of source)
That would do it.
The subsetting is pretty fast. If you've resolved a whole lock before on a machine, the subset should be ~500ms.
Really depends on the size of the wheels. Some AI / ML stuff is slow just due to GB wheels.
As you are becoming aware, I could spend a whole week catching up Pex docs.
I will do that in the new year.
lol yeah. i keep finding more useful stuff in pex every day.
The most fun easter egg lies in PEX_TOOLS=1 against a PEX file that was built with
or `--venv`(or by using the `pex-tools`console script):
pex-tools my.pex graph --open
I tried using
and can't run the built pex because it needs the dependencies. Using
with the dependencies doesn't seem to satisfy a pex built with
. I keep getting this error, trying `PEX_PATH=dagster-deps.pex ./intransitive.pex`:
Copy code
No interpreter compatible with the requested constraints was found:

  Failed to resolve requirements from PEX environment @ /Users/shalabh/.pex/unzipped_pexes/4ccbb52b80202f20d56767fbebf1e97a0319d337.
  Needed cp38-cp38-macosx_12_0_x86_64 compatible dependencies for:
   1: dagster
      Required by:
        my-dagster-project 0.0.0
      But this pex had no ProjectName(raw='dagster', normalized='dagster') distributions.
   2: dagster-aws
      Required by:
        my-dagster-project 0.0.0
      But this pex had no ProjectName(raw='dagster-aws', normalized='dagster-aws') distributions.
I tried multiple ways to unpack this or merge the dependencies pex with the intransitive pex, but keep getting the above error. Even
pex-tools intransitive.pex venv here
wont install the venv.
Oh using `--intransitive --ignore-errors`seems to be the way to go. I can then merge the deps with
without issues.
Ok. You can probably also just not use
. Since you're resolving against hashed requirements, Pip will fail if it needs to grab anything not in the requirements.