another fun pex issue: i'm trying to run my generated pex inside a docker image, but it's complaining about not having the right kind of wheel:

root@1247ee737d61:/code# ./app.pex --help
Failed to execute PEX file. Needed manylinux2014_x86_64-cp-36-cp36m compatible dependencies for:
 1: PyYAML>=3.10; extra == "yaml"
    Required by:
      apispec==4.0.0; python_version >= "3.6"
    But this pex only contains:

What is your repo’s interpreter constraints set to? The default is

>=3.6

. You can inspect

PEX-INFO

in

app.pex

by using

unzip

to see what compatibility it’s claiming to have, which will be based off of your Pants targets’ compatibility settings. At build time, if you claim compatibility with

>=3.6

, then at runtime, any interpreter meeting that constraint may be used. At build time, Pex will use all discovered valid interpreters to get relevant copies of each distribution; but if you didn’t have a Py36 interpreter available at build time, then your built

.pex

won’t include

cp36

wheels and runtime may fail. https://github.com/pantsbuild/pex/issues/1020 tracks redesigning this.

Probably won’t help you that much but I do have that issue when I build my `.pex`a using my local machine (MacOs) and then trying to run it inside a container (python:3.7-slim for instance). I do have a bunch of dependencies that seems to be tied to host/OS libs. So what I do here is build the pex inside the container, that way all the “correct” and needed wheels are there.

hmm this looks related to a caching issue, like the old pex was built for the wrong version

like the old pex was built for the wrong version

Like

dist/app.pex

was stale? Or you recently ran

./pants package

and Pants built the wrong thing?

dist/app.pex was stale, and i don't think just changing the "compatibility" invalidated the cache

i don’t think just changing the “compatibility” invalidated the cache

It would only invalidate the cache if the final resulting compatibility changed. Compatibility is calculated by looking at the transitive closure; each target gets ANDed. If you had a downstream dep at

CPython==3.8

, or that was your global default and you had left off the

compatibility

field, then the final result would be the same.

Yup, that’s another thing that I kinda did a lit bit too… I build my local wheels…

pip install wheel
poetry export -f requirements.txt --without-hashes --dev | grep -v '^-e' > libs/3rdparty/constraints.txt
pip wheel -r libs/3rdparty/constraints.txt -w .wheels

and I do the same thing using a docker container:

docker run --rm -it \
    -v "$(pwd)":/app \
    python:3.7 \
    /bin/bash -c """
        apt-get update -qq;
        apt-get install -y libsasl2-dev;
        pip wheel -r /app/libs/3rdparty/constraints.txt -w /app/.wheels
    """

my CI runs pants inside of docker, with the cache and source roots mounted, i could do the same thing on dev machines with a bit of tricky-scripty magic

haha I rant about not being able to get myself a work laptop running linux all the time lol, a bunch of magical scripts to overcome mac oddness

btw, warning this video is gross

but it does make sense that mac machines can’t easily natively compile code that can run inside a linux docker container

Yeah, that’s a fundamental issue independent of Pants. For building the

pants.pex

that we release, we have two CI jobs that build all our wheels on old macOS and old Centos. For Py36, Py37, and Py38. Then we use those prebuilt wheels, rather than an index like PyPI.

"build on old centos" is the exact principle behind manylinux

a bunch of magical scripts to overcome mac oddness

We’ve spent probably 2-3 weeks collectively trying to work around macOS weirdness with its system Python the past 3 months 🙃

We’ve spent probably 2-3 weeks collectively trying to work around macOS weirdness with its system Python the past 3 months

Giphy trolled me 😞

i thought mac system stuff was so bad that every dev had already switched to homebrew

ha, only if homebrew worked properly, that thing has moods

Indeed, but if

/usr/bin

is on your

$PATH

, then Pex may still end up using it

package management sucks

I’m very happy new macOS versions won’t have Python pre-installed

fat binaries and docker are just ways we're mortgaging the fact that packaging sucks and no-one has a great solution for it

surprised

actually fat binaries are okay

Have you ever heard of the wonderful

pacman

? I actually use

yay

i'm trying to play around with https://nixos.org/learn.html actually

oh cool, a friend of mine sent me that another day, how is it going?

idk, i havent' cracked it yet

Haskell community loves nix, I haven’t tried it either