https://pantsbuild.org/ logo
p

plain-carpet-73994

08/07/2021, 11:47 PM
Hey all, I've got an interesting issue. I have an executable that depends on stream-chat. As you can see from the PyPi page it's OS independent. If I
./pants run
the executable it runs fine. However, I'm trying to build a cross-platform
.pex
with:
Copy code
platforms = [
      'linux-x86_64-cp-37-m',
      'macosx-10.14-x86_64-cp-37-m'
   ]
and I get
Copy code
stderr:
ERROR: Could not find a version that satisfies the requirement stream-chat
ERROR: No matching distribution found for stream-chat
pid 181794 -> /home/oliver/.pyenv/versions/3.7.10/bin/python3.7 /home/oliver/.cache/pants/named_caches/pex_root/pip.pex/cf4106b4c7898b462f7c172dde686a9747103f1a --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/oliver/.cache/pants/named_caches/pex_root --log /tmp/process-execution11pC41/.tmp/tmpgo8mme15/pip.log download --dest /tmp/process-execution11pC41/.tmp/tmp3aui1xab/macosx_10_14_x86_64-cp-37-cp37m --platform macosx_10_14_x86_64 --implementation cp --python-version 37 --abi cp37m --only-binary :all: --constraint constraints.txt firebase-admin google-cloud-firestore pydantic stream-chat typing_extensions --index-url <https://pypi.org/simple/> --find-links <https://us-central1-python.pkg.dev/infrastructure-314723/python-deps/> --retries 5 --timeout 15 exited with 1 and STDERR:
None
why would it have trouble getting
stream-chat
when it's OS independent? Note that if I remove the
macosx-10.14...
line from
platforms
the
./pants package
succeeds.
h

happy-kitchen-89482

08/08/2021, 5:05 AM
Hmm, weird. I would imagine this is a common scenario!
I'll take a closer look on Monday
p

plain-carpet-73994

08/08/2021, 11:38 PM
Thanks @happy-kitchen-89482
h

happy-kitchen-89482

08/09/2021, 2:01 AM
Which version of Pants is this?
p

plain-carpet-73994

08/09/2021, 2:58 AM
2.4.0
I can't upgrade to 2.5 yet due to another bug
h

hundreds-father-404

08/09/2021, 3:04 AM
What's the other bug?
h

happy-kitchen-89482

08/09/2021, 3:44 AM
And you're running this on linux, it sounds like?
Also, what is in
constraints.txt
?
If I run the underlying pip command from your error message, on linux, without the
--constraints
flag, it seems to work
w

worried-salesclerk-37834

08/09/2021, 1:31 PM
I think this is the same experience I had here? https://pantsbuild.slack.com/archives/C046T6T9U/p1628260681111400?thread_ts=1628260681.111400&amp;cid=C046T6T9U Should we expect pex to be able to build cross platform for non-wheel dependencies? I was flipped in this case. On Mac, building for Linux.
h

happy-kitchen-89482

08/09/2021, 2:37 PM
It is similar. But in that case the google grpc library contained native code (I think) whereas this sdist is pure python (I think). But this is where my ignorance comes in - maybe it is not possible to use an sdist in a cross-platform way even if it is pure python. @enough-analyst-54434 would know. I will look a little deeper.
But that said, as I mentioned - when I ran the pip command directly (without constraints and a few other options) it did work
Also, directly and naively running
Copy code
pex --platform linux-x86_64-cp-37-m --platform macosx-10.14-x86_64-cp-37-m -o mypex.pex firebase-admin google-cloud-firestore pydantic stream-chat typing_extensions
works
So now I'm wondering what's in constraints.txt, where that find-links URL comes from, etc.
Can you run with -ldebug and post the whole thing to a gist (redacting anything that needs redacting)?
w

witty-crayon-22786

08/09/2021, 3:57 PM
i think that Chris is right: you need a wheel for the relevant platform, rather than an sdist
šŸ‘ 1
āž• 1
e

enough-analyst-54434

08/09/2021, 5:17 PM
Chris and Stu are right. The relevant pip flag from the original command is
--only-binary :all:
. That's required when passing the 4 platform bits.
šŸ‘ 1
p

plain-carpet-73994

08/09/2021, 5:23 PM
@happy-kitchen-89482 @witty-crayon-22786 thanks for your help! Do you need me to re-run with the
-ldebug
or have you determined that the issue is the
--only-binary
flag? If it's that flag what's the workaround for libs like this that don't provide a wheel? And if it is that flag why was @happy-kitchen-89482 able to run the command without the
constraint.txt
but with all the other flags and have it succeed?
The
constraints.txt
is quite large as there's lots of other binaries and libraries in our repo with additional dependencies. The stuff I think is relevant is:
Copy code
stream-chat==3.11.0
firebase-admin==5.0.1
google-cloud-firestore==2.2.0
pydantic==1.8.2
typing-extensions==3.10.0.0
e

enough-analyst-54434

08/09/2021, 5:58 PM
Benjy's results don't make sense. Pex has a mode enabled with
--resolve-local-platforms
that Pants doesn't use IIRC that can turn one of your platform arguments into a normal local resolve, but the other will still be foreign and so wheels cannot be built. To work around, you need to pre-build the needed wheels by hand for each required platform and serve them up from your own cheese shop or find links repo. You then add that repo via
[python-repos]
indexes
or
repos
list options: https://www.pantsbuild.org/docs/reference-python-repos
Right now, Pex performs all your requested resolves (in your case 2 - 1 per platform) in parallel and fails if any one resolve fails. It could instead post process the successful resolves looking at wheel tags and metadata to see if any successful resolve works for the failed cases. For platform / python version independent resolves (like yours?), this would result in success. For platform specific resolves it would still fail.
Ah, @plain-carpet-73994 I just noticed you're already using
--find-links
so that concept at least should make sense. You just need to add the pre-built wheels to that or else add a second find links repo with the pre-builts.
h

happy-kitchen-89482

08/09/2021, 6:21 PM
Ah, I was not passing
--only-binary :all:
Oh wait, that is a pip thing
In any case, running
Copy code
pex --platform linux-x86_64-cp-37-m --platform macosx-10.14-x86_64-cp-37-m -o mypex.pex firebase-admin google-cloud-firestore pydantic stream-chat typing_extensions
seems to do the right thing
And hypothetically I don't immediately see why it should not be possible to resolve multiplatform for sdists that are pure python. Is this just a limitation of pip and/or pex, or of the metadata not allowing pip and/or pex to know at build time whether the sdist build can succeed on the current platform?
e

enough-analyst-54434

08/09/2021, 6:37 PM
This is a limitation of Pip. If that Pex command works for you, that seems like a bug.
So, I reproduce Benjy's results, but the resolved stream-chat is old - 1.7.0. That stream-chat does have a wheel: https://pypi.org/project/stream-chat/1.7.0/#files
The only thing Benjy and I are missing is constraints.txt. That clearly must specify a pin for stream-chat that does not have a wheel available on PyPI. Is that right @plain-carpet-73994?
And @hundreds-father-404, assuming I'm right here, this will be a case poetry lockfiles will fail on. That will need some thought.
p

plain-carpet-73994

08/09/2021, 6:55 PM
yeah, that's very old. Per above my
constraints.txt
has
stream-chat==3.11.0
.
So to summarize: ā€¢ There is a bug in Pants that's causing this to fail when it should succeed. ā€¢ The only available workaround right now is for me to pre-build a wheel from the sdist and put that in a local cheese shop or to update my
constraints.txt
to pin a much, much older version. Do I have that all right @enough-analyst-54434?
e

enough-analyst-54434

08/09/2021, 6:56 PM
Aha, sorry, missed that bit. Yeah, so that's the issue - that pin. Which leads to the current Pip limitation inherited by Pex for foreign platform resolves.
No bug in Pants. Arguably a feature should be added to Pex as described above to work around the limitation in Pip.
Your work around description is correct though.
p

plain-carpet-73994

08/09/2021, 7:04 PM
Bummer! I wonder if it'd be possible (or advisable) to put a workaround in Pants; have it download sdists, build wheels, and create a local cheese shop before calling out to pex. Sounds like more than a small amount of work however. Thanks for all your help @enough-analyst-54434
h

happy-kitchen-89482

08/09/2021, 8:28 PM
Yeah, sounds like more of a feature of pip than a bug in Pants, alas.
But your suggestion is interesting @plain-carpet-73994, but probably no small amount of work
2 Views