fresh-cat-90827
11/13/2021, 5:54 AM2.6.1
to 2.8.0rc5
, on the packaging step, the Linux CI fails with an error message. It looks like Pex now requires all dependencies, transitively, to be present as wheels (--only-binary :all:
argument to the pex
command). It works fine to produce necessary .pex
packages with 2.6.1
atm even though some transitive dependencies are available only as sdist
on the PyPI. I know Pex wouldn’t get the sdist
for multiplatform packages, but the package in question is only for a single platform (linux_x86_64-cp-38-cp38
). @enough-analyst-54434 is this intended?pants.engine.process.ProcessExecutionFailure: Process 'Building project/project-file.pex with 23 requirements: PyYAML, bokeh==2.3.2, fastparquet==0.7.0, (truncated...)' failed with exit code 1.
stdout:
stderr:
ERROR: Could not find a version that satisfies the requirement thrift>=0.11.0 (from fastparquet)
ERROR: No matching distribution found for thrift>=0.11.0
pid 14440 -> /home/jenkins/.cache/pants/named_caches/pex_root/venvs/8c697325637c51a0499eb9481ecd8612923ca1ef/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/jenkins/.cache/pants/named_caches/pex_root --log /tmp/process-executionHQoVqC/.tmp/tmpq80ca8b0/pip.log download --dest /tmp/process-executionHQoVqC/.tmp/tmpqtdby7ip/linux_x86_64-cp-38-cp38 --platform manylinux2014_x86_64 --platform linux_x86_64 --implementation cp --python-version 38 --abi cp38 --only-binary :all: --constraint constraints.txt PyYAML bokeh==2.3.2 fastparquet==0.7.0 (truncated...) --index-url <https://pypi.org/simple/> --extra-index-url <https://hosted-pypi/url/pypi/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
None
enough-analyst-54434
11/13/2021, 10:04 PMpex_binary
target declares `platforms`: https://www.pantsbuild.org/docs/reference-pex_binary#codeplatformscode
Presumably though, your target definitions remained a constant during the upgrade.pex_binary
target definitions did not change during this Pants upgrade I can dig a bit further.--resolve-local-platforms
Pex option. That option will tell Pex to try to find a local interpreter that corresponds to the platform and use that to build wheels, if needed; otherwise to fall back to only accepting wheels as is the default for --platform
.
Pants has never exposed this though and only uses it for awslambda.
That said, the default is for this to be off both in Pex 2.1.42, which Pants 2.6.1 uses as well as in Pex 2.1.54 which Pants 2.8.0rc5 uses; so this longshot idea doesn't appear to be relevant.hundreds-father-404
11/15/2021, 4:40 PMfresh-cat-90827
11/16/2021, 8:10 PM@fresh-cat-90827 if you can confirm that@enough-analyst-54434 no changes to thetarget definitions did not change during this Pants upgrade I can dig a bit further.pex_binary
BUILD
files, the only changes are in the pants.toml
(renaming some deprecated options).@fresh-cat-90827 maybe if you could try going from 2.6.1 to 2.7 first? Even if you don’t land that bump, it will help to bisect where this brokeThanks @hundreds-father-404 for looking into this! I’ve tried 2.7.0, but unfortunately ran into an issue John has fixed for me. See https://pantsbuild.slack.com/archives/C046T6T9U/p1635797857340500?thread_ts=1635797837.340300&cid=C046T6T9U
hundreds-father-404
11/16/2021, 8:16 PMfresh-cat-90827
11/16/2021, 10:48 PM2.7.2rc1
and`2.7.2rc2` give me the same error, Eric.
stderr:
ERROR: Could not find a version that satisfies the requirement thrift>=0.11.0 (from fastparquet)
ERROR: No matching distribution found for thrift>=0.11.0
sdist
so publishing all of them as wheels would imply that I’d have to build a little CI plan to be able to publish on a hosted PyPI wheels for those projects and re-build new wheels when new versions are released which feels a bit too much.
Do you folks think there is any chance I could provide more information that would help us debug why switching to a later version of Pants doesn’t let build PEX packages asking for wheels (when sdist
only available)?enough-analyst-54434
11/24/2021, 3:54 PM--platform
/ --only-binary :all:
Pip args. Pex only emits those when you specify --platform
and Pants, in turn, only sets that for AWS lambda. Otherwise you must set it with pex_binary.platforms
in your BUILD files. Let's start there and work backwards. Is the issue you're seeing either when building AWS lambda targets or pex_binary
targets with platforms
?package
goal (i.e.: Mac or Linux, what interpreter versions are available on the PATH, etc)?fresh-cat-90827
11/24/2021, 5:19 PMpex_binary
on a Linux Ubuntu (20.04) with Python 3.8 being the default interpreter (and the only interpreter available) — ubuntu:focal
Docker image in CI.
pex_binary(
name='name',
description='Desc',
entry_point='some/path/module.py:main',
platforms=['linux_x86_64-cp-38-cp38'],
zip_safe=False,
)
Removing the platforms
argument still does not remove the --only-binary :all:
Pip args, it seems:
stderr:
ERROR: Could not find a version that satisfies the requirement thrift>=0.11.0 (from fastparquet)
ERROR: No matching distribution found for thrift>=0.11.0
pid 3344 -> /home/user/.cache/pants/named_caches/pex_root/venvs/8c697325637c51a0499eb9481ecd8612923ca1ef/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/user/.cache/pants/named_caches/pex_root --log /tmp/process-executionCdtnky/.tmp/tmp_3_141kp/pip.log download --dest /tmp/process-executionCdtnky/.tmp/tmpnzgy4ln7/linux_x86_64-cp-38-cp38 --platform manylinux2014_x86_64 --platform linux_x86_64 --implementation cp --python-version 38 --abi cp38 --only-binary :all: --constraint constraints.txt PyYAML ....(omitted).... scipy --index-url <https://pypi.org/simple/> --extra-index-url <https://user>:****@hosted-pypi/api/pypi/pypi/simple --retries 5 --timeout 15 exited with 1 and STDERR:
None
enough-analyst-54434
11/24/2021, 5:23 PMRemoving theThat should not be true - completely unexpected. So a great place to dig. But before digging, I want to emphasize that withargument still does not remove theplatforms
Pip args, it seems:--only-binary :all:
platforms=['linux_x86_64-cp-38-cp38'],
in place, it should have always been the case that `-only-binary all`was being used and thus sdists in a resolve would fail the build../pants package ....:fred
of that new target also somehow fails and shows `--only-binary all`in the output?fresh-cat-90827
11/24/2021, 6:55 PMThat should not be true - completely unexpected. So a great place to dig. But before digging, I want to emphasize that withha! I am able to build the package within place, it should have always been the case that `-only-binary all`was being used and thus sdists in a resolve would fail the build.platforms=['linux_x86_64-cp-38-cp38'],
sdist
only on the PyPI on Pants 2.6.1, with the platforms
set to Linux as above 🙂
However, with 2.8.0
after removing the platforms
, the package is built fine! It’s my bad — there were a few very similar names of the packages…
So, can you verify that if you copy-paste that target definition, change the name to some else, say ‘fred’ and leave out platforms, thatIndeed, the packages that have noof that new target also somehow fails and shows `--only-binary all`in the output?./pants package ....:fred
platforms
are built fine! So the CI case is fine (if we don’t use platforms
) — we’ll have Linux packages (since they are built on Linux). But then we won’t be able to produce PEX packages (Linux ones) locally on MacOS devices.enough-analyst-54434
11/24/2021, 6:57 PMha! I am able to build the package withOk - looking at that Pants version to see if we did useonly on the PyPI on Pants 2.6.1, with thesdist
set to Linux as aboveplatforms
--resolve-local-platforms
which is the only way that could have worked...--resolve-local-platforms
option in Pants (or make it the default). Then, you should find your resolves working. BUT, be warned, they will only continue to work as long as a local interpreter that exactly matches the platform can be resolved in order to build sdists with.
Does that all make sense?fresh-cat-90827
11/24/2021, 7:12 PMBUT, be warned, they will only continue to work as long as a local interpreter that exactly matches the platform can be resolved in order to build sdists with.Just to confirm, I think we already have it in the CI, otherwise it wouldn’t have worked with
2.6.1
, would it. So, if I specify 'linux_x86_64-cp-38-cp38'
then Python 3.8 on Linux x86_64 should be found which is a standard thing, so no changes are necessary to any machinery on our side. Provided we can have this magical --resolve-local-platforms
in Pants, right?enough-analyst-54434
11/24/2021, 7:16 PMpackage
on a linux without Python 3.8 or on a Mac will still fail with the new --resolve-local-platforms
since no local Linux Python 3.8 interpreter will be found on those machines. If you have tight control of your machines, dev and CI - you'll be good to go. If you don't, you'll be surprised when this re-breaks, but at least now you'll know why.platforms
you really do need to have your own wheelhouse. Hopefully you can continue to get away with not needing one.fresh-cat-90827
11/24/2021, 7:23 PM'linux_x86_64-cp-38-cp38'
as platforms
I won’t be able to produce a package on a MacOS device because there is no Linuxy Python 3.8 found?In the general case of using platforms you really do need to have your own wheelhouse.I think this is where we are going to… Time to spin up a new Jenkins plan 😄 😄
hundreds-father-404
11/24/2021, 7:27 PM--resolve-local-platforms
. That new option would only help if you are on a Linux machine and there are some sdists
If you are on macOS building for the Linux platform, you must have wheels already preparedenough-analyst-54434
11/24/2021, 7:27 PMGotcha. So if I haveCorrect. You also won't be able to on a Linux machine either unless the Linux machine has Python 3.8 on theas'linux_x86_64-cp-38-cp38'
I won’t be able to produce a package on a MacOS device because there is no Linuxy Python 3.8 found?platforms
$PATH
. If you get an employee who wipes their Mac and installls Arch Linux (god bless 'em), they'll just have Python 3.9 or even Python 3.10 by default since its a rolling release style distro.fresh-cat-90827
11/24/2021, 7:51 PM[python-setup]
interpreter_search_paths = ["/usr/bin"]
interpreter_constraints = ["CPython==3.8.*"]
As a side effect, a few Linux users get the support as well since they have /usr/bin/python3
being Python 3.8 as well (luckily, we control the system versions) 🙂
Thanks again for all the help, can’t be more grateful!enough-analyst-54434
11/27/2021, 2:21 AM--resolve-local-platforms
Pex feature is still off by default, but can now be flipped on by default with [pex-binary-defaults] resolve-local-platforms = True
and over-ridden on individual pex_binary
targets with a new resolve_local_platforms
`bool`field. This will be available in the 2.9.0.dev3
release next week.fresh-cat-90827
11/27/2021, 5:02 PM2.9.0.dev3
and after setting
[pex-binary-defaults]
resolve_local_platforms = true
in the pants.toml
my Linux CI can build packages even though many of the transitive dependencies are available only as sdists. Thanks a lot for making this option available.enough-analyst-54434
12/14/2021, 8:41 PM