https://pantsbuild.org/ logo
#general
Title
# general
a

abundant-autumn-67998

12/12/2022, 7:43 PM
I'm seeing
pex
have issues resolving dependencies and report conflicts when both
pip
and
pipenv
work fine. I've narrowed down the issue to a set containing just these two dependencies:
"grpcio<1.48.1" "google-api-core[grpc] >= 1.22.1"
I suspect this has to do with how the extras
[grpc]
is handled. Details in thread.
issue with pex
Copy code
% pex --python=python3.8  "grpcio<1.48.1" "google-api-core[grpc] >= 1.22.1" -o output.pex
Failed to resolve compatible distributions:
1: grpcio-status==1.51.1 requires grpcio>=1.51.1 but grpcio 1.47.2 was resolved
also happens with other
--pip-version
Copy code
% pex --python=python3.8 --pip-version=22.3 "grpcio<1.48.1" "google-api-core[grpc] >= 1.22.1" -o output.pex
Failed to resolve compatible distributions:
1: grpcio-status==1.51.1 requires grpcio>=1.51.1 but grpcio 1.47.2 was resolved
pipenv works fine
Copy code
% pipenv install "grpcio<1.48.1" "google-api-core[grpc] >= 1.22.1"
Creating a virtualenv for this project...
...
āœ” Installation Succeeded 
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
āœ” Success!
it installs these versions:
Copy code
% pipenv graph
google-api-core==2.11.0
  - google-auth [required: >=2.14.1,<3.0dev, installed: 2.15.0]
    - cachetools [required: >=2.0.0,<6.0, installed: 5.2.0]
    - pyasn1-modules [required: >=0.2.1, installed: 0.2.8]
      - pyasn1 [required: >=0.4.6,<0.5.0, installed: 0.4.8]
    - rsa [required: >=3.1.4,<5, installed: 4.9]
      - pyasn1 [required: >=0.1.3, installed: 0.4.8]
    - six [required: >=1.9.0, installed: 1.16.0]
  - googleapis-common-protos [required: >=1.56.2,<2.0dev, installed: 1.57.0]
    - protobuf [required: >=3.19.5,<5.0.0dev,!=4.21.5,!=4.21.4,!=4.21.3,!=4.21.2,!=4.21.1,!=3.20.1,!=3.20.0, installed: 4.21.11]
  - protobuf [required: >=3.19.5,<5.0.0dev,!=4.21.5,!=4.21.4,!=4.21.3,!=4.21.2,!=4.21.1,!=4.21.0,!=3.20.1,!=3.20.0, installed: 4.21.11]
  - requests [required: >=2.18.0,<3.0.0dev, installed: 2.28.1]
    - certifi [required: >=2017.4.17, installed: 2022.12.7]
    - charset-normalizer [required: >=2,<3, installed: 2.1.1]
    - idna [required: >=2.5,<4, installed: 3.4]
    - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.13]
grpcio-status==1.47.2
  - googleapis-common-protos [required: >=1.5.5, installed: 1.57.0]
    - protobuf [required: >=3.19.5,<5.0.0dev,!=4.21.5,!=4.21.4,!=4.21.3,!=4.21.2,!=4.21.1,!=3.20.1,!=3.20.0, installed: 4.21.11]
  - grpcio [required: >=1.47.2, installed: 1.47.2]
    - six [required: >=1.5.2, installed: 1.16.0]
  - protobuf [required: >=3.12.0, installed: 4.21.11]
just tested and
google-api-core
doesn't need a pin:
Copy code
% pex --python=python3.8 --pip-version=22.3 "grpcio<1.48.1" "google-api-core[grpc]" -o output.pex 
Failed to resolve compatible distributions:
1: grpcio-status==1.51.1 requires grpcio>=1.51.1 but grpcio 1.47.2 was resolved
e

enough-analyst-54434

12/12/2022, 7:49 PM
@abundant-autumn-67998 you also probably want
--resolver-version pip-2020-resolver
, works for me. And this is another backwards compat thing. Lots to type, but less when Pex 3.x releases next year and these will be defaults.
You may not have been aware Pip still carries its old legacy resolver around inside itself!
Oh Google!:
Copy code
$ pex --python=python3.8 --pip-version=22.3 "grpcio<1.48.1" "google-api-core[grpc]" --resolver-version pip-2020-resolver -- -c 'import grpc; print(grpc.__file__)'
/tmp/tmp5swbyre5/.bootstrap/pex/environment.py:654: PEXWarning: The `pkg_resources` package was loaded from a pex vendored version when declaring namespace packages defined by:

1. google-api-core==2.11.0 namespace packages:
  google

2. googleapis-common-protos==1.57.0 namespace packages:
  google
  google.logging

3. google-auth==2.15.0 namespace packages:
  google

These distributions should fix their `install_requires` to include `setuptools`
  pex_warnings.warn(
/home/jsirois/.pex/installed_wheels/9ca6bdc2ca97419d9154a2858e7190109280143c87f3eda4ad6dada2729dea7e/grpcio-1.47.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/grpc/__init__.py
So you need to help Google out and include setuptools:
Copy code
$ pex --python=python3.8 --pip-version=22.3 setuptools "grpcio<1.48.1" "google-api-core[grpc]" --resolver-version pip-2020-resolver -- -c 'imp
ort grpc; print(grpc.__file__)'
pex --python=python3.8 --pip-version=22.3 "grpcio<1.48.1" "google-api-core[grpc]" -o output.pex/home/jsirois/.pex/installed_wheels/9ca6bdc2ca97419d9154a2858e7190109280143c87f3eda4ad6dada2729dea7e/grpcio-1.47.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/grpc/__init__.py
You don't need to, that's just a warning. You could also tell Pex
--no-emit-warnings
If there is a way to pass through Pip options via
pipenv
you can probably get it to fail too:
Copy code
$ pip -V
pip 22.3.1 from /home/jsirois/.pyenv/versions/3.11.0rc2/lib/python3.11/site-packages/pip (python 3.11)

$ pip --use-feature x

Usage:
  pip <command> [options]

option --use-feature: invalid choice: 'x' (choose from 'fast-deps', 'truststore', 'no-binary-enable-wheel-cache')

$ pip --use-deprecated x

Usage:
  pip <command> [options]

option --use-deprecated: invalid choice: 'x' (choose from 'legacy-resolver')
So, Pex's default is to pass
--use-deprecated legacy-resolver
no matter the Pip version to maintain backwards compat. By passing
--resolver-version pip-2020-resolver
to Pex, that gets newer versions of `--pip-version`using the 2020 (now default) resolver.
a

abundant-autumn-67998

12/12/2022, 8:10 PM
Ah ok - adding
--resolver-version
fixes the test case for me. Thanks!
Today we saw a second build issue with different dependencies that can also be resolved with
--resolver-version=pip-2020-resolver
. I'm now making this our default build flag, however I'm not sure if adding this will break some users that happen to need the other resolver, so we're keeping the old build flags also as a fallback. I was also thinking of doing a different resolution mechanism based on
pipenv
1. run
pipenv lock
2. run
pipenv requirements --exclude-markers > pipenv-output-requirements.txt
3. run
pex -r pipenv-output-requirements.txt
I wonder if
pex
will try to re-resolve the dependencies. Is there a way to tell pex to just install the fully resolved set of dependencies in 3 above?
(also is this a bad idea?)
e

enough-analyst-54434

12/13/2022, 1:36 AM
So, how tied are you to pipenv locks? Do you know about
pex3 lock {create,update,exprt}
?
But
pex --intransitive ...
against pinned requirements will do the trick.
šŸ‘€ 1
It will still "resolve" and that will be slower than using a
pex3 lock
file, which includes all the artifact URLs to just do a straight fetch from, but it will give you exactly what's in
requirements.txt
and no more as long as all those reqs are pinned. If they have hashes, of course those are checked / respected too.
a

abundant-autumn-67998

12/13/2022, 5:46 PM
We're not tied to pipenv. I'm just thinking of other possible ways to do dependency resolution. I didn't know about
pex3 lock
- could be useful but I'll wait until it is official.
e

enough-analyst-54434

12/13/2022, 5:51 PM
It is official.
Pants has been using it as its production lock for a good while now.
Don't let the
pex3
name fool you, that has ~nothing to do with the Pex version. There is just no way to introduce subcommands in `pex`reasonably until 3.x
So I carved out a
pex3
console script to host subcommands.
a

abundant-autumn-67998

12/13/2022, 5:56 PM
ah interesting ok. i'll take a closer look then.
talking about dependencies, is there a way in to include a package from local source directories? for instance, we call
python setup.py build
and then point
pex -D
at the build output directory. this lets us package the source without resolving or building the dependencies. i assume there is no pex native way of doing something like this?
e

enough-analyst-54434

12/13/2022, 6:06 PM
just point pex at the directory containing the setup.py:
pex local_project/ ...
This is how Pex builds itself. See the `pex_requirement = "."`here: https://github.com/pantsbuild/pex/blob/312fa138a66e7c1d08a59cd7cffc6912ebb13f7e/scripts/package.py#L28-L59
The Pex project uses Flit /
pyproject.toml
and not
setup.py
any longer, but works for any Python buildable project.
a

abundant-autumn-67998

12/13/2022, 6:08 PM
ah i just tested with
pex /path/to/local_project --instransitive -o source.pex
and looks like is what i need. it just builds the source without the dependencies. the dependencies are injected at run time with
PEX_PATH
so i'll try that next.
e

enough-analyst-54434

12/13/2022, 6:09 PM
What drives the split and use of PEX_PATH to re-join?
a

abundant-autumn-67998

12/13/2022, 6:11 PM
we do two builds: 1. local project(s) without any deps - source.pex 2. dependencies of local project(s), without the local project itself - deps.pex our build process builds two pex files from above. our runtime invokes something like
PEX_PATH=deps.pex source.pex
we identify the combination with a string like
files=deps-HASH.pex:source-HASH.pex
. typically we don't need to rebuild deps.pex which is what makes deploys fast.
e

enough-analyst-54434

12/13/2022, 6:12 PM
Gotcha.
a

abundant-autumn-67998

12/13/2022, 6:12 PM
for 2 above, we parse requirements.txt and the output of setup.py to create a derived requirements.txt that we feed to pex.
e

enough-analyst-54434

12/13/2022, 6:13 PM
So one thing to note about Pex locks is they can be subsetted.
So if you locka project, you can then consume the lock like
pex --lock lock.json just-this-req
And you'll only get `just-this-req`'s subgraph.
If you pass no reqs, you get the whole lock.
And, of course, you can pass one or more reqs to get their combined subgraph, etc.
a

abundant-autumn-67998

12/13/2022, 6:16 PM
very interesting. for 2 above we need a severed graph - the subgraph (deps only) without the root node (local source). though maybe leaving the source package in
deps.pex
is ok because it will get overwritten with
source.pex
or maybe we can just get the subgraph of (direct deps of source)
e

enough-analyst-54434

12/13/2022, 6:18 PM
That would do it.
The subsetting is pretty fast. If you've resolved a whole lock before on a machine, the subset should be ~500ms.
Really depends on the size of the wheels. Some AI / ML stuff is slow just due to GB wheels.
As you are becoming aware, I could spend a whole week catching up Pex docs.
I will do that in the new year.
a

abundant-autumn-67998

12/13/2022, 6:21 PM
lol yeah. i keep finding more useful stuff in pex every day.
e

enough-analyst-54434

12/13/2022, 6:24 PM
The most fun easter egg lies in PEX_TOOLS=1 against a PEX file that was built with
--include-tools
or `--venv`(or by using the `pex-tools`console script):
pex-tools my.pex graph --open
a

abundant-autumn-67998

12/16/2022, 2:11 AM
I tried using
--intransitive
and can't run the built pex because it needs the dependencies. Using
PEX_PATH=deps.pex
with the dependencies doesn't seem to satisfy a pex built with
--intransitive
. I keep getting this error, trying `PEX_PATH=dagster-deps.pex ./intransitive.pex`:
Copy code
No interpreter compatible with the requested constraints was found:

  Failed to resolve requirements from PEX environment @ /Users/shalabh/.pex/unzipped_pexes/4ccbb52b80202f20d56767fbebf1e97a0319d337.
  Needed cp38-cp38-macosx_12_0_x86_64 compatible dependencies for:
   1: dagster
      Required by:
        my-dagster-project 0.0.0
      But this pex had no ProjectName(raw='dagster', normalized='dagster') distributions.
   2: dagster-aws
      Required by:
        my-dagster-project 0.0.0
      But this pex had no ProjectName(raw='dagster-aws', normalized='dagster-aws') distributions.
I tried multiple ways to unpack this or merge the dependencies pex with the intransitive pex, but keep getting the above error. Even
pex-tools intransitive.pex venv here
wont install the venv.
Oh using `--intransitive --ignore-errors`seems to be the way to go. I can then merge the deps with
PEX_PATH
without issues.
e

enough-analyst-54434

12/16/2022, 2:36 AM
Ok. You can probably also just not use
--intransitive
. Since you're resolving against hashed requirements, Pip will fail if it needs to grab anything not in the requirements.
10 Views