Hey guys I am getting a `pex environment ResolveError` in `2 Pants #general

Hey guys, I am getting a `pex.environment.ResolveE...

rhythmic-battery-45198

04/11/2022, 8:01 PM

Hey guys, I am getting a

pex.environment.ResolveError

2.11.0rc2

Seems to be related to extra dependencies. Adding details to🧵

✅ 1

rhythmic-battery-45198

04/11/2022, 8:02 PM

Copy code

pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /home/dave/.cache/pants/named_caches/pex_root/unzipped_pexes/780e9986364e8711fb1c8e29713d51ade52ba5f4.
Needed cp38-cp38-manylinux_2_31_x86_64 compatible dependencies for:
 1: matplotlib>=3.0; extra == "plotting"
    Required by:
      FingerprintedDistribution(distribution=GPy 1.9.9 (/home/dave/.cache/pants/named_caches/pex_root/installed_wheels/8b63348de1556ed17a912bcab858509d4eb977b29413a06be18bb5b0844755be/GPy-1.9.9-py3-none-any.whl), fingerprint='b577ba75477ade4f390be4e7b738268b0944f129e21a78f2902894d695771347')
    But this pex had no 'matplotlib' distributions

rhythmic-battery-45198

04/11/2022, 8:03 PM

requirements.txt

does not include the

[plotting]

extra dependencies - it is just

GPy==1.9.9

. (

pex: Building pex :: Resolving GPy==1.9.9

)

rhythmic-battery-45198

04/11/2022, 8:04 PM

pex seems to be skipping

matplotlib

earlier in the logs.

rhythmic-battery-45198

04/11/2022, 8:04 PM

Copy code

pex: Skipping activation of `matplotlib>=3.0; extra == "plotting"` due to environment marker de-selection

rhythmic-battery-45198

04/11/2022, 8:13 PM

If I change my requirement to

GPy[plotting]==1.9.9

and regenerate the lockfile, the resolve error goes away, as expected.

enough-analyst-54434

04/11/2022, 9:42 PM

Copy code

Needed cp38-cp38-manylinux_2_31_x86_64 compatible dependencies for:
 1: matplotlib>=3.0; extra == "plotting"
    Required by:
      FingerprintedDistribution(distribution=GPy 1.9.9 (/home/dave/.cache/pants/named_caches/pex_root/installed_wheels/8b63348de1556ed17a912bcab858509d4eb977b29413a06be18bb5b0844755be/GPy-1.9.9-py3-none-any.whl), fingerprint='b577ba75477ade4f390be4e7b738268b0944f129e21a78f2902894d695771347')
    But this pex had no 'matplotlib' distributions

That can mean there is a

matplotlib

embedded in the Pex, but not one loadable for Python 3.8. Can you run Pants with

--no-process-cleanup

and then go in to the sandbox and run

PEX_TOOLS=1 ./repository.pex info -i4

and report what the repository PEX has in its manifest?

enough-analyst-54434

04/11/2022, 9:43 PM

@#$! slack - did not mean for that to go to channel.

hundreds-father-404

04/11/2022, 9:44 PM

Iirc there is an option to undo it if you click on the message's settings. Otherwise in the past I have deleted the message and resent 😕

enough-analyst-54434

04/11/2022, 9:57 PM

Ah, yup. From the channel it shows up as an option, but not from here in the thread. Thanks Eric.

❤️ 1

hundreds-father-404

04/11/2022, 9:59 PM

Ah, I did not know that is why it sometimes works and does not work! 🤷

rhythmic-battery-45198

04/11/2022, 9:59 PM

Thanks John - will try this in the morning

rhythmic-battery-45198

04/12/2022, 3:23 PM

Looks like the sandbox doesn't have a

repository.pex

Copy code

process-execution1lKr3q ls
local_dists.pex  pex  pytest.pex  requirements.pex  __run.sh  source_files

I dug into it and found the root cause though. While I have a direct dependency on

GPy

, it turns out that I also have a transitive dependency on

GPy[plotting]

. This results in the

pex.environment.ResolveError

. If I change my direct dependency to

GPy[plotting]

, the error goes away. Not sure if this might be a bug or a gap in my understanding.

enough-analyst-54434

04/12/2022, 3:29 PM

Ah, yes

s/repository.pex/requirements.pex/

. That rings a bell, let me check the archives here before asking you for more data.

enough-analyst-54434

04/12/2022, 3:34 PM

This was what was ringing: https://github.com/pantsbuild/pex/issues/616 That instance though was fixed by modern Pex. Can you provide me either all the data I layed out before (with

s/repository.pex/requirements.pex/

) or else the name / version of the transitive dependency that depends on

GPy[plotting]

? Hopefully this is easy to repro with just Pex and those 2 requirements.

rhythmic-battery-45198

04/12/2022, 3:38 PM

Sure thing,

emukit==0.4.8

depends on

GPy[plotting]>=1.9.9

rhythmic-battery-45198

04/12/2022, 3:39 PM

Thanks for the help - and this is not a blocker on my end since I can bubble the

[plotting]

extra up to my requirements.txt as an easy workaround.

enough-analyst-54434

04/12/2022, 3:44 PM

Great, thanks. For apples, what are the interpreter constraints you're using?

rhythmic-battery-45198

04/12/2022, 3:45 PM

Copy code

[python]
interpreter_constraints = [
  "==3.8.*"
]
lockfile_generator = 'pex'
enable_resolves = true

enough-analyst-54434

04/12/2022, 3:45 PM

Thanks

enough-analyst-54434

04/12/2022, 3:52 PM

Ok, Yeah - easy to repro with:

Copy code

$ pex3 lock create --style universal --resolver-version pip-2020-resolver --interpreter-constraint "==3.8.*" GPy emukit==0.4.8 -o lock.json
$ pex --python python3.8 --lock lock.json -o all_from_lock.pex
$ pex-tools all_from_lock.pex info -i4
{
    "bootstrap_hash": "194811fc66631583971925a2a1c59e3bfb6c6d49",
    "build_properties": {
        "pex_version": "2.1.78"
    },
    "code_hash": "7506e475892b062c4c5f74f795b09a08a9feb157",
    "distributions": {
        "Cython-0.29.28-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl": "300800a1c9828650ffeba8325b1afc978c026b1fe81eb923852afc363909ecb3",
        "GPy-1.10.0-py3-none-any.whl": "7618d519f3b108be265f2f963987cf559a560d0c0b836a8f1013c9c87d573aae",
        "decorator-5.1.1-py3-none-any.whl": "7ed6dc469274d0b89fff2c3b5a28d1ddc0937210510a7b13016a8327fc28bc3b",
        "emcee-3.1.1-py2.py3-none-any.whl": "e404ed544fe2c8678b16388b8c35abea3a404a626068d10d0bbb617da540a044",
        "emukit-0.4.8-py3-none-any.whl": "706dfd068037f73b383d3cf8c65c3c926dc67a499e6ba336adebde2585f6c204",
        "numpy-1.22.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "fb7879661e502744c416238e5779680ce4e6d2cb6c3c70f30bd364d1475befe9",
        "paramz-0.9.5-py3-none-any.whl": "809087cc272c458d865f491fe8a112e6f200a4776fda9d2271bb4f22acc5db9c",
        "scipy-1.8.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "bcf91dc99a0fbf9915203b26a1371b5798eab488bf7cbc696b4f83b6ce53b060",
        "setuptools-62.1.0-py3-none-any.whl": "8f28465858fe3db86603d42b80b8e3d7f306d805bbae398986878e1d593e2d50",
        "six-1.16.0-py2.py3-none-any.whl": "3e1c439c88d2e7681372427bab751b3fc99969891e95a714fed9604bf7710213"
    },
    "emit_warnings": true,
    "ignore_errors": false,
    "includes_tools": false,
    "inherit_path": "false",
    "interpreter_constraints": [],
    "pex_hash": "e4b596cc533c996c1b31f32ba75804be1dd13b79",
    "pex_path": null,
    "requirements": [
        "GPy",
        "emukit==0.4.8"
    ],
    "strip_pex_env": true,
    "venv": false,
    "venv_bin_path": "false",
    "venv_copies": false,
    "venv_site_packages_copies": false,
    "pex_root": "/home/jsirois/.pex"
}

No matplotlib via the transitive extra. I'll file a Pex bug here shortly.

👍 1

enough-analyst-54434

04/12/2022, 4:25 PM

https://github.com/pantsbuild/pex/issues/1717

rhythmic-battery-45198

04/13/2022, 4:04 PM

Confirmed that this issue is resolved with pex v2.1.79 💪

❤️ 1

enough-analyst-54434

04/13/2022, 6:19 PM

Excellent. Thanks for checking Dave.

9 Views

Open in Slack

Previous Next