<@U07QKSQKB7S> I may be hitting a `pex` issue as a...
# pex
b
@brief-scientist-13682 I may be hitting a
pex
issue as a follow-up from https://pantsbuild.slack.com/archives/C046T6T9U/p1764087004690069 Thanks to your fix, I can now generate a lockfile where I have this pants.toml config
Copy code
[python.resolves_to_only_binary]
__default__ = [":all:"]

[python.resolves_to_no_binary]
__default__ = ["pyspark"]
However, when I try to package a pex where
pyspark
is a dependency, I hit this failure:
Copy code
ProcessExecutionFailure: Process 'Building 3 requirements for subprojects.example.src.python.example.app.pyspark_job/job.pex from the subprojects/example/3rdparty/python/lockfile.jsonc resolve: dd-internal-pyspark>=1.0.0, numpy>=2.3.0, pyspark' failed with exit code 1.
stdout:

stderr:
Failed to resolve compatible artifacts from lock subprojects/example/3rdparty/python/lockfile.jsonc for 1 target:
1. /Users/gregory.fast/.pyenv/versions/3.12.10/bin/python3.12:
    Failed to resolve all requirements for cp312-cp312-macosx_15_0_arm64 interpreter at /Users/gregory.fast/.pyenv/versions/3.12.10/bin/python3.12 from subprojects/example/3rdparty/python/lockfile.jsonc:

Configured with:
    build: False
    only_build: pyspark
    use_wheel: True

Dependency on pyspark not satisfied, 1 incompatible candidate found:
1.) pyspark 3.5.2 does not have any compatible artifacts:
    <https://depot-read-api-python.us1.ddbuild.io/magicmirror/magicmirror/@current/files/pyspark/3.5.2/sha256%3Abbb36eba09fa24e86e0923d7e7a986041b90c714e11c6aa976f9791fe9edde5e/pyspark-3.5.2.tar.gz>
    requirers:
    via: pyspark
    via: dd-internal-pyspark>=1.0.0 -> pyspark==3.5.2
The artifact URL (from an internal index) is totally valid, that pyspark-3.5.2.tar.gz is there. With
only_build: pyspark
, I would expect this artifact to be compatible.
here's the pex command I see
more accurate/latest pex command from the sandbox
Copy code
./pex --tmpdir .tmp --jobs 3 --no-emit-warnings --pip-version latest --python-path $'/Users/gregory.fast/.pyenv/versions/3.10.13/bin:/Users/gregory.fast/.pyenv/versions/3.10.15/bin:/Users/gregory.fast/.pyenv/versions/3.11.13/bin:/Users/gregory.fast/.pyenv/versions/3.12.10/bin:/Users/gregory.fast/.pyenv/versions/3.8.12/bin:/Users/gregory.fast/.pyenv/versions/dd-pyspark-check/bin:/Users/gregory.fast/.pyenv/versions/dd-pyspark-test/bin:/Users/gregory.fast/.pyenv/versions/temp_ddpyspark/bin' --output-file subprojects.example.src.python.example.app.pyspark_job/job.pex --emit-warnings $'--check=warn' --include-tools --requirements-pex local_dists.pex --interpreter-constraint $'CPython==3.12.*' --python-path /Users/gregory.fast/.pyenv/versions/3.12.10/bin/python3.12 $'--sources-directory=source_files' $'dd-internal-pyspark>=1.0.0' $'numpy>=2.3.0' pyspark --lock subprojects/example/3rdparty/python/lockfile.jsonc --no-pypi $'--index=<https://depot-read-api-python.us1.ddbuild.io/magicmirror/magicmirror/@current/simple/>' --manylinux manylinux2014 --wheel --no-build $'--only-build=pyspark' --layout zipapp
b
Yeah, definitely a Pex bug. I'll be slower than usual over this holiday.
b
Of course, have a happy thanksgiving! Thanks for confirming
b
That fix is released in https://github.com/pex-tool/pex/releases/tag/v2.72.1 Your case now works with
--lock
and
--pylock
resolvers. The prior fix got the Pip resolver working and
--pex-repository
,
--venv-repository
and
--pre-resolved-dists
resolvers were always unaffected. So this should be fixed for all scenarios now.
b
Thank you so much, I really appreciate it. I’ll try out the fix tomorrow
looks good, John! Can't thank you enough for the fast turnaround, especially during the holidays 🙇