https://pantsbuild.org/ logo
Join Slack
Powered by
When trying to upgrade from `2.6.1` to `2.8.0rc5`,...
# general
f
When trying to upgrade from 
2.6.1
to 
2.8.0rc5
, on the packaging step, the Linux CI fails with an error message. It looks like Pex now requires all dependencies, transitively, to be present as wheels (
--only-binary :all:
argument to the 
pex
command). It works fine to produce necessary 
.pex
packages with 
2.6.1
atm even though some transitive dependencies are available only as 
sdist
on the PyPI. I know Pex wouldn’t get the 
sdist
for multiplatform packages, but the package in question is only for a single platform (
linux_x86_64-cp-38-cp38
). @enough-analyst-54434 is this intended?
Copy code
pants.engine.process.ProcessExecutionFailure: Process 'Building project/project-file.pex with 23 requirements: PyYAML, bokeh==2.3.2, fastparquet==0.7.0, (truncated...)' failed with exit code 1.

stdout:

stderr:

ERROR: Could not find a version that satisfies the requirement thrift>=0.11.0 (from fastparquet)
ERROR: No matching distribution found for thrift>=0.11.0

pid 14440 -> /home/jenkins/.cache/pants/named_caches/pex_root/venvs/8c697325637c51a0499eb9481ecd8612923ca1ef/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/jenkins/.cache/pants/named_caches/pex_root --log /tmp/process-executionHQoVqC/.tmp/tmpq80ca8b0/pip.log download --dest /tmp/process-executionHQoVqC/.tmp/tmpqtdby7ip/linux_x86_64-cp-38-cp38 --platform manylinux2014_x86_64 --platform linux_x86_64 --implementation cp --python-version 38 --abi cp38 --only-binary :all: --constraint constraints.txt PyYAML bokeh==2.3.2 fastparquet==0.7.0 (truncated...) --index-url <https://pypi.org/simple/> --extra-index-url <https://hosted-pypi/url/pypi/simple> --retries 5 --timeout 15 exited with 1 and STDERR:

None
e
It is intended if your 
pex_binary
target declares `platforms`: https://www.pantsbuild.org/docs/reference-pex_binary#codeplatformscode Presumably though, your target definitions remained a constant during the upgrade.
@fresh-cat-90827 if you can confirm that 
pex_binary
target definitions did not change during this Pants upgrade I can dig a bit further.
The only long-shot idea I had was the 
--resolve-local-platforms
Pex option. That option will tell Pex to try to find a local interpreter that corresponds to the platform and use that to build wheels, if needed; otherwise to fall back to only accepting wheels as is the default for 
--platform
. Pants has never exposed this though and only uses it for awslambda. That said, the default is for this to be off both in Pex 2.1.42, which Pants 2.6.1 uses as well as in Pex 2.1.54 which Pants 2.8.0rc5 uses; so this longshot idea doesn't appear to be relevant.
👍 1
h
@fresh-cat-90827 maybe if you could try going from 2.6.1 to 2.7 first? Even if you don't land that bump, it will help to bisect where this broke
f
@fresh-cat-90827 if you can confirm that 
pex_binary
 target definitions did not change during this Pants upgrade I can dig a bit further.
@enough-analyst-54434 no changes to the 
BUILD
files, the only changes are in the 
pants.toml
(renaming some deprecated options).
@fresh-cat-90827 maybe if you could try going from 2.6.1 to 2.7 first? Even if you don’t land that bump, it will help to bisect where this broke
Thanks @hundreds-father-404 for looking into this! I’ve tried 2.7.0, but unfortunately ran into an issue John has fixed for me. See https://pantsbuild.slack.com/archives/C046T6T9U/p1635797857340500?thread_ts=1635797837.340300&amp;cid=C046T6T9U
h
Hm what about 2.7.2rc2?
👀 1
f
2.7.2rc1
and`2.7.2rc2` give me the same error, Eric.
Copy code
stderr:

ERROR: Could not find a version that satisfies the requirement thrift>=0.11.0 (from fastparquet)
ERROR: No matching distribution found for thrift>=0.11.0
hey! There are quite a few projects that are published only as 
sdist
so publishing all of them as wheels would imply that I’d have to build a little CI plan to be able to publish on a hosted PyPI wheels for those projects and re-build new wheels when new versions are released which feels a bit too much. Do you folks think there is any chance I could provide more information that would help us debug why switching to a later version of Pants doesn’t let build PEX packages asking for wheels (when 
sdist
only available)?
e
@fresh-cat-90827 the issue is the 
--platform
/ 
--only-binary :all:
Pip args. Pex only emits those when you specify 
--platform
and Pants, in turn, only sets that for AWS lambda. Otherwise you must set it with 
pex_binary.platforms
in your BUILD files. Let's start there and work backwards. Is the issue you're seeing either when building AWS lambda targets or 
pex_binary
targets with 
platforms
?
Ideally, can you provide the full target definition and information about the machine where your running the 
package
goal (i.e.: Mac or Linux, what interpreter versions are available on the PATH, etc)?
👍 1
f
Thanks @enough-analyst-54434! I am seeing the issue when building this 
pex_binary
on a Linux Ubuntu (20.04) with Python 3.8 being the default interpreter (and the only interpreter available) — 
ubuntu:focal
Docker image in CI.
Copy code
pex_binary(
  name='name',
  description='Desc',
  entry_point='some/path/module.py:main',
  platforms=['linux_x86_64-cp-38-cp38'],
  zip_safe=False,
)
Removing the 
platforms
argument still does not remove the 
--only-binary :all:
Pip args, it seems:
Copy code
stderr:

ERROR: Could not find a version that satisfies the requirement thrift>=0.11.0 (from fastparquet)

ERROR: No matching distribution found for thrift>=0.11.0

pid 3344 -> /home/user/.cache/pants/named_caches/pex_root/venvs/8c697325637c51a0499eb9481ecd8612923ca1ef/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/user/.cache/pants/named_caches/pex_root --log /tmp/process-executionCdtnky/.tmp/tmp_3_141kp/pip.log download --dest /tmp/process-executionCdtnky/.tmp/tmpnzgy4ln7/linux_x86_64-cp-38-cp38 --platform manylinux2014_x86_64 --platform linux_x86_64 --implementation cp --python-version 38 --abi cp38 --only-binary :all: --constraint constraints.txt PyYAML ....(omitted).... scipy --index-url <https://pypi.org/simple/> --extra-index-url <https://user>:****@hosted-pypi/api/pypi/pypi/simple --retries 5 --timeout 15 exited with 1 and STDERR:

None
e
Removing the 
platforms
 argument still does not remove the 
--only-binary :all:
 Pip args, it seems:
That should not be true - completely unexpected. So a great place to dig. But before digging, I want to emphasize that with 
platforms=['linux_x86_64-cp-38-cp38'],
in place, it should have always been the case that `-only-binary all`was being used and thus sdists in a resolve would fail the build.
So, can you verify that if you copy-paste that target definition, change the name to some else, say 'fred' and leave out platforms, that 
./pants package ....:fred
of that new target also somehow fails and shows `--only-binary all`in the output?
f
That should not be true - completely unexpected. So a great place to dig. But before digging, I want to emphasize that with 
platforms=['linux_x86_64-cp-38-cp38'],
 in place, it should have always been the case that `-only-binary all`was being used and thus sdists in a resolve would fail the build.
ha! I am able to build the package with 
sdist
only on the PyPI on Pants 2.6.1, with the 
platforms
set to Linux as above 🙂 However, with 
2.8.0
after removing the 
platforms
, the package is built fine! It’s my bad — there were a few very similar names of the packages…
So, can you verify that if you copy-paste that target definition, change the name to some else, say ‘fred’ and leave out platforms, that 
./pants package ....:fred
 of that new target also somehow fails and shows `--only-binary all`in the output?
Indeed, the packages that have no 
platforms
are built fine! So the CI case is fine (if we don’t use 
platforms
) — we’ll have Linux packages (since they are built on Linux). But then we won’t be able to produce PEX packages (Linux ones) locally on MacOS devices.
e
ha! I am able to build the package with 
sdist
 only on the PyPI on Pants 2.6.1, with the 
platforms
 set to Linux as above
Ok - looking at that Pants version to see if we did use 
--resolve-local-platforms
which is the only way that could have worked...
Aha. I think you were on the beneficiary side of this Pip bug: https://github.com/pypa/pip/issues/10050 I worked around that bug in Pex here: + https://github.com/pantsbuild/pex/issues/1366 + https://github.com/pantsbuild/pex/pull/1367 That was released in Pex 2.1.43. Pants upgraded to Pex 2.1.44 here: https://github.com/pantsbuild/pants/commit/d7be43a5a24d6ec067e9fa86b7cc27d7b22ed31d That upgrade is in 2.7.0.dev2 and greater. This explains the issue you're seeing then. So - to bring it all back: You have always been on the beneficiary side of a Pip bug that Pex worked around. You always should have been getting the failures you're getting today. The only way to restore the old behavior is to expose a 
--resolve-local-platforms
option in Pants (or make it the default). Then, you should find your resolves working. BUT, be warned, they will only continue to work as long as a local interpreter that exactly matches the platform can be resolved in order to build sdists with. Does that all make sense?
💯 2
And thanks for bearing with me on this. Took a little too long to figure out.
f
ah that’s amazing, thanks, John!
BUT, be warned, they will only continue to work as long as a local interpreter that exactly matches the platform can be resolved in order to build sdists with.
Just to confirm, I think we already have it in the CI, otherwise it wouldn’t have worked with 
2.6.1
, would it. So, if I specify 
'linux_x86_64-cp-38-cp38'
then Python 3.8 on Linux x86_64 should be found which is a standard thing, so no changes are necessary to any machinery on our side. Provided we can have this magical 
--resolve-local-platforms
in Pants, right?
e
Right. But, running that same 
package
on a linux without Python 3.8 or on a Mac will still fail with the new 
--resolve-local-platforms
since no local Linux Python 3.8 interpreter will be found on those machines. If you have tight control of your machines, dev and CI - you'll be good to go. If you don't, you'll be surprised when this re-breaks, but at least now you'll know why.
In the general case of using 
platforms
you really do need to have your own wheelhouse. Hopefully you can continue to get away with not needing one.
f
Gotcha. So if I have 
'linux_x86_64-cp-38-cp38'
as 
platforms
I won’t be able to produce a package on a MacOS device because there is no Linuxy Python 3.8 found?
In the general case of using platforms you really do need to have your own wheelhouse.
I think this is where we are going to… Time to spin up a new Jenkins plan 😄 😄
h
Yeah, that's regardless of 
--resolve-local-platforms
. That new option would only help if you are on a Linux machine and there are some sdists If you are on macOS building for the Linux platform, you must have wheels already prepared
e
Gotcha. So if I have 
'linux_x86_64-cp-38-cp38'
 as 
platforms
 I won’t be able to produce a package on a MacOS device because there is no Linuxy Python 3.8 found?
Correct. You also won't be able to on a Linux machine either unless the Linux machine has Python 3.8 on the 
$PATH
. If you get an employee who wipes their Mac and installls Arch Linux (god bless 'em), they'll just have Python 3.9 or even Python 3.10 by default since its a rolling release style distro.
f
haha. The way I decided to solve this is to use enforce having XCode 12 on MacOS (primary users) devices which ships with Python 3.8:
Copy code
[python-setup]
interpreter_search_paths = ["/usr/bin"]
interpreter_constraints = ["CPython==3.8.*"]
As a side effect, a few Linux users get the support as well since they have 
/usr/bin/python3
being Python 3.8 as well (luckily, we control the system versions) 🙂 Thanks again for all the help, can’t be more grateful!
e
OK, the 
--resolve-local-platforms
Pex feature is still off by default, but can now be flipped on by default with 
[pex-binary-defaults] resolve-local-platforms = True
and over-ridden on individual 
pex_binary
targets with a new 
resolve_local_platforms
`bool`field. This will be available in the 
2.9.0.dev3
release next week.
f
lovely, thanks, John! I’ll experiment with this feature and in the worst case will need to host some wheels 🙂
hey @enough-analyst-54434! I tried 
2.9.0.dev3
and after setting
Copy code
[pex-binary-defaults]
resolve_local_platforms = true
in the 
pants.toml
my Linux CI can build packages even though many of the transitive dependencies are available only as sdists. Thanks a lot for making this option available.
💯 1
e
Excellent. You're welcome.
5 Views
Open in Slack
PreviousNext
Powered by