Hey all, I've got an interesting issue. I have an ...
# general
Hey all, I've got an interesting issue. I have an executable that depends on stream-chat. As you can see from the PyPi page it's OS independent. If I
./pants run
the executable it runs fine. However, I'm trying to build a cross-platform
Copy code
platforms = [
and I get
Copy code
ERROR: Could not find a version that satisfies the requirement stream-chat
ERROR: No matching distribution found for stream-chat
pid 181794 -> /home/oliver/.pyenv/versions/3.7.10/bin/python3.7 /home/oliver/.cache/pants/named_caches/pex_root/pip.pex/cf4106b4c7898b462f7c172dde686a9747103f1a --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/oliver/.cache/pants/named_caches/pex_root --log /tmp/process-execution11pC41/.tmp/tmpgo8mme15/pip.log download --dest /tmp/process-execution11pC41/.tmp/tmp3aui1xab/macosx_10_14_x86_64-cp-37-cp37m --platform macosx_10_14_x86_64 --implementation cp --python-version 37 --abi cp37m --only-binary :all: --constraint constraints.txt firebase-admin google-cloud-firestore pydantic stream-chat typing_extensions --index-url <https://pypi.org/simple/> --find-links <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/> --retries 5 --timeout 15 exited with 1 and STDERR:
why would it have trouble getting
when it's OS independent? Note that if I remove the
line from
./pants package
Hmm, weird. I would imagine this is a common scenario!
I'll take a closer look on Monday
Thanks @happy-kitchen-89482
Which version of Pants is this?
I can't upgrade to 2.5 yet due to another bug
What's the other bug?
And you're running this on linux, it sounds like?
Also, what is in
If I run the underlying pip command from your error message, on linux, without the
flag, it seems to work
I think this is the same experience I had here? https://pantsbuild.slack.com/archives/C046T6T9U/p1628260681111400?thread_ts=1628260681.111400&amp;cid=C046T6T9U Should we expect pex to be able to build cross platform for non-wheel dependencies? I was flipped in this case. On Mac, building for Linux.
It is similar. But in that case the google grpc library contained native code (I think) whereas this sdist is pure python (I think). But this is where my ignorance comes in - maybe it is not possible to use an sdist in a cross-platform way even if it is pure python. @enough-analyst-54434 would know. I will look a little deeper.
But that said, as I mentioned - when I ran the pip command directly (without constraints and a few other options) it did work
Also, directly and naively running
Copy code
pex --platform linux-x86_64-cp-37-m --platform macosx-10.14-x86_64-cp-37-m -o mypex.pex firebase-admin google-cloud-firestore pydantic stream-chat typing_extensions
So now I'm wondering what's in constraints.txt, where that find-links URL comes from, etc.
Can you run with -ldebug and post the whole thing to a gist (redacting anything that needs redacting)?
i think that Chris is right: you need a wheel for the relevant platform, rather than an sdist
šŸ‘ 1
āž• 1
Chris and Stu are right. The relevant pip flag from the original command is
--only-binary :all:
. That's required when passing the 4 platform bits.
šŸ‘ 1
@happy-kitchen-89482 @witty-crayon-22786 thanks for your help! Do you need me to re-run with the
or have you determined that the issue is the
flag? If it's that flag what's the workaround for libs like this that don't provide a wheel? And if it is that flag why was @happy-kitchen-89482 able to run the command without the
but with all the other flags and have it succeed?
is quite large as there's lots of other binaries and libraries in our repo with additional dependencies. The stuff I think is relevant is:
Copy code
Benjy's results don't make sense. Pex has a mode enabled with
that Pants doesn't use IIRC that can turn one of your platform arguments into a normal local resolve, but the other will still be foreign and so wheels cannot be built. To work around, you need to pre-build the needed wheels by hand for each required platform and serve them up from your own cheese shop or find links repo. You then add that repo via
list options: https://www.pantsbuild.org/docs/reference-python-repos
Right now, Pex performs all your requested resolves (in your case 2 - 1 per platform) in parallel and fails if any one resolve fails. It could instead post process the successful resolves looking at wheel tags and metadata to see if any successful resolve works for the failed cases. For platform / python version independent resolves (like yours?), this would result in success. For platform specific resolves it would still fail.
Ah, @plain-carpet-73994 I just noticed you're already using
so that concept at least should make sense. You just need to add the pre-built wheels to that or else add a second find links repo with the pre-builts.
Ah, I was not passing
--only-binary :all:
Oh wait, that is a pip thing
In any case, running
Copy code
pex --platform linux-x86_64-cp-37-m --platform macosx-10.14-x86_64-cp-37-m -o mypex.pex firebase-admin google-cloud-firestore pydantic stream-chat typing_extensions
seems to do the right thing
And hypothetically I don't immediately see why it should not be possible to resolve multiplatform for sdists that are pure python. Is this just a limitation of pip and/or pex, or of the metadata not allowing pip and/or pex to know at build time whether the sdist build can succeed on the current platform?
This is a limitation of Pip. If that Pex command works for you, that seems like a bug.
So, I reproduce Benjy's results, but the resolved stream-chat is old - 1.7.0. That stream-chat does have a wheel: https://pypi.org/project/stream-chat/1.7.0/#files
The only thing Benjy and I are missing is constraints.txt. That clearly must specify a pin for stream-chat that does not have a wheel available on PyPI. Is that right @plain-carpet-73994?
And @hundreds-father-404, assuming I'm right here, this will be a case poetry lockfiles will fail on. That will need some thought.
yeah, that's very old. Per above my
So to summarize: ā€¢ There is a bug in Pants that's causing this to fail when it should succeed. ā€¢ The only available workaround right now is for me to pre-build a wheel from the sdist and put that in a local cheese shop or to update my
to pin a much, much older version. Do I have that all right @enough-analyst-54434?
Aha, sorry, missed that bit. Yeah, so that's the issue - that pin. Which leads to the current Pip limitation inherited by Pex for foreign platform resolves.
No bug in Pants. Arguably a feature should be added to Pex as described above to work around the limitation in Pip.
Your work around description is correct though.
Bummer! I wonder if it'd be possible (or advisable) to put a workaround in Pants; have it download sdists, build wheels, and create a local cheese shop before calling out to pex. Sounds like more than a small amount of work however. Thanks for all your help @enough-analyst-54434
Yeah, sounds like more of a feature of pip than a bug in Pants, alas.
But your suggestion is interesting @plain-carpet-73994, but probably no small amount of work