Hi @enough-analyst-54434! We are debugging an issue with pants release at Twitter, which is also the reason master is red right now. We are seeing the same thing that is started to happening in upstream on

Deploy unstable pants pex

Travis shard at your commit updating pex to 2.0.3. https://travis-ci.org/pantsbuild/pants/builds/622371760 We’ve run the pip command that the pex is running. In the directory of wheels some of them are linux_x86_64 and some of them are manylinux1_x86_64. It looks like we were able to resolve both of them with the same command before. Now the platform flag limits us to one or another. Is there a quick an easy fix? Could you help us, or implement it? If not, can we please revert the pex update until we have the issue solved? Thanks! 🙂

Do we need to do something like

--platform=linux|manylinux

when we invoke pip (don’t know if that exists)? Or to change how we’re publishing those wheels from the build wheels job?

For foreign platforms there is no way to say both, but I think we may be lying about foreign here for the linux case. Looking...

What does lying about foreign mean? Treating things which should be equivalent as not?

Correct. Generally if you say platform=a|b|c its a lie, you cannot in fact deal with some dists resolved for a , some b some c. Consider macosx|linux

Interesting, internally we’re seeing it with Linux, so maybe both are broken and we get a random ordering or something?

Perhaps.

The OSX being broken is much more mysterious to me!

The OSX case is a bug: https://github.com/pantsbuild/pex/issues/823 The linux/manylinux case is less clear to me. The appropriate fix for that case may be to run auditwheel to add the appropriate tags to "linux-x86_64" wheels. Doing so for the wheels pants depends on reveals all are manylinuxXYZ compatible. The OSX case though is enough to revert here. PR coming...

John, thank you! I appreciate it 🙂

I think the crux of this issue is the exact thing I mentioned in this thread: https://pantsbuild.slack.com/archives/C087V4P1T/p1573407828009400

The problem is, for a foreign platform

linux

means nothing. This is why pypi doesn't allow it.

linux

just means it works on the linux it was built on, not much more. So if you want to resolve

linux

wheels you are guessing or else have arranged that all wheels available that are platform tagged

linux

are in fact built for the linux you expect. If you have done that arrangement, then all wheels should be built for the platform you expect - ie: all

linux

, no

manylinux

.

When you build and deploy a wheel with platform linux-x86_64 to an index - how do you know which linuxes it works for?

and manylinux wheels are only useful in the event of publishing to pypi, where your consumer’s linux variants are unknown

OK - then if so, why don't you do this for all the wheels in your index?

this is why manylinux is only relevant in wide publishing models - on the public internet, its vastly unknown; in-house, where linux-x86_64 bins are totally fair game it’s typiclly much more controllable

Understood. I'm asking about Twitter

we definitely have a mix.

You arguably should not have a mix

of linux, manylinux1 and manylinux 2010

The problem with a foreign resolve is you have no info about the host. That info is needed to make the manylinux call.

yes, this is why it was a flag!

Exactly, and so it will need to be listed as an explicit psuedo-platform for the foreign case.

(and most linux’ are compatible - thats the point - so it’s typically on by default)

Need to find a way to request >1 platform from pip here

thats the rub tho. because of the way that multi-platform resolves happen, you’re gonna have a Bad Time if you think of it as a platform.

It is thought of as exactly that in tag expansion code

whats the approximate

pip

invocation pex 2.x uses now for resolve?

The exact is revealed with -v or -vvv, not sure which.

mainly curious about the mode e.g.

pip wheel

pip download

etc

For a foreign platform only, the quadruple of --platform --python-version --implementation --abi

so design crux here seems to be:

pip

mode for manylinux resolve is optimized for infer vs explicit, pex was other way around until fused

This simply needs a hack to pip. Roughly the same hack we previously made to pep425tags. I'll do something there and try to make it presentable for upstream pip instead of maintaining a hacked copy like we have been.