https://pantsbuild.org/ logo
#pex
Title
# pex
f

flat-zoo-31952

10/20/2020, 7:27 PM
another fun pex issue: i'm trying to run my generated pex inside a docker image, but it's complaining about not having the right kind of wheel:
Copy code
root@1247ee737d61:/code# ./app.pex --help
Failed to execute PEX file. Needed manylinux2014_x86_64-cp-36-cp36m compatible dependencies for:
 1: PyYAML>=3.10; extra == "yaml"
    Required by:
      apispec==4.0.0; python_version >= "3.6"
    But this pex only contains:
also the cp36m thing is weird, since it's built for python 3.8
h

hundreds-father-404

10/20/2020, 7:34 PM
What is your repo’s interpreter constraints set to? The default is
>=3.6
. You can inspect
PEX-INFO
in
app.pex
by using
unzip
to see what compatibility it’s claiming to have, which will be based off of your Pants targets’ compatibility settings. At build time, if you claim compatibility with
>=3.6
, then at runtime, any interpreter meeting that constraint may be used. At build time, Pex will use all discovered valid interpreters to get relevant copies of each distribution; but if you didn’t have a Py36 interpreter available at build time, then your built
.pex
won’t include
cp36
wheels and runtime may fail. https://github.com/pantsbuild/pex/issues/1020 tracks redesigning this.
w

wooden-thailand-8386

10/20/2020, 7:35 PM
Probably won’t help you that much but I do have that issue when I build my `.pex`a using my local machine (MacOs) and then trying to run it inside a container (python:3.7-slim for instance). I do have a bunch of dependencies that seems to be tied to host/OS libs. So what I do here is build the pex inside the container, that way all the “correct” and needed wheels are there.
f

flat-zoo-31952

10/20/2020, 7:54 PM
hmm this looks related to a caching issue, like the old pex was built for the wrong version
i have the prod binary set to
compatibility="==3.8.*"
👍 1
h

hundreds-father-404

10/20/2020, 7:59 PM
like the old pex was built for the wrong version
Like
dist/app.pex
was stale? Or you recently ran
./pants package
and Pants built the wrong thing?
f

flat-zoo-31952

10/20/2020, 8:03 PM
dist/app.pex was stale, and i don't think just changing the "compatibility" invalidated the cache
i added a comment to the entry point source file and it worked though
the manylinux wheel problem ended up being a non-issue, but it's a good point that @wooden-thailand-8386 brought up that this will not work for people developing on mac
h

hundreds-father-404

10/20/2020, 8:06 PM
i don’t think just changing the “compatibility” invalidated the cache
It would only invalidate the cache if the final resulting compatibility changed. Compatibility is calculated by looking at the transitive closure; each target gets ANDed. If you had a downstream dep at
CPython==3.8
, or that was your global default and you had left off the
compatibility
field, then the final result would be the same.
the manylinux wheel problem ended up being a non-issue, but it’s a good point that @wooden-thailand-8386 brought up that this will not work for people developing on mac
You can use the
platforms
field on
pex_binary
to work around that, but, it requires that every single dep has a bdist already, rather than sdist.
w

wooden-thailand-8386

10/20/2020, 8:10 PM
Yup, that’s another thing that I kinda did a lit bit too… I build my local wheels…
Copy code
pip install wheel
poetry export -f requirements.txt --without-hashes --dev | grep -v '^-e' > libs/3rdparty/constraints.txt
pip wheel -r libs/3rdparty/constraints.txt -w .wheels
and I do the same thing using a docker container:
Copy code
docker run --rm -it \
    -v "$(pwd)":/app \
    python:3.7 \
    /bin/bash -c """
        apt-get update -qq;
        apt-get install -y libsasl2-dev;
        pip wheel -r /app/libs/3rdparty/constraints.txt -w /app/.wheels
    """
f

flat-zoo-31952

10/20/2020, 8:12 PM
my CI runs pants inside of docker, with the cache and source roots mounted, i could do the same thing on dev machines with a bit of tricky-scripty magic
magic 1
but it does make sense that mac machines can't easily natively compile code that can run inside a linux docker container
i already know i'm going to name this script
./d-pants
in homage to this classic gem...

https://www.youtube.com/watch?v=dEwjogp4wIo

[warning: toilet humor]
w

wooden-thailand-8386

10/20/2020, 8:15 PM
haha I rant about not being able to get myself a work laptop running linux all the time lol, a bunch of magical scripts to overcome mac oddness
f

flat-zoo-31952

10/20/2020, 8:15 PM
btw, warning this video is gross
🤣 1
h

hundreds-father-404

10/20/2020, 8:17 PM
but it does make sense that mac machines can’t easily natively compile code that can run inside a linux docker container
Yeah, that’s a fundamental issue independent of Pants. For building the
pants.pex
that we release, we have two CI jobs that build all our wheels on old macOS and old Centos. For Py36, Py37, and Py38. Then we use those prebuilt wheels, rather than an index like PyPI.
f

flat-zoo-31952

10/20/2020, 8:18 PM
"build on old centos" is the exact principle behind manylinux
💯 1
h

hundreds-father-404

10/20/2020, 8:18 PM
a bunch of magical scripts to overcome mac oddness
We’ve spent probably 2-3 weeks collectively trying to work around macOS weirdness with its system Python the past 3 months 🙃
w

wooden-thailand-8386

10/20/2020, 8:19 PM
We’ve spent probably 2-3 weeks collectively trying to work around macOS weirdness with its system Python the past 3 months
Giphy trolled me 😞
😄 1
f

flat-zoo-31952

10/20/2020, 8:19 PM
i thought mac system stuff was so bad that every dev had already switched to homebrew
w

wooden-thailand-8386

10/20/2020, 8:20 PM
ha, only if homebrew worked properly, that thing has moods
h

hundreds-father-404

10/20/2020, 8:20 PM
Indeed, but if
/usr/bin
is on your
$PATH
, then Pex may still end up using it
f

flat-zoo-31952

10/20/2020, 8:21 PM
package management sucks
😨 1
1
h

hundreds-father-404

10/20/2020, 8:21 PM
I’m very happy new macOS versions won’t have Python pre-installed
👌 1
f

flat-zoo-31952

10/20/2020, 8:22 PM
fat binaries and docker are just ways we're mortgaging the fact that packaging sucks and no-one has a great solution for it
w

wooden-thailand-8386

10/20/2020, 8:22 PM
surprised
😂 1
@flat-zoo-31952 don’t say that!
f

flat-zoo-31952

10/20/2020, 8:22 PM
actually fat binaries are okay
w

wooden-thailand-8386

10/20/2020, 8:23 PM
Have you ever heard of the wonderful
pacman
? I actually use
yay
f

flat-zoo-31952

10/20/2020, 8:23 PM
i'm trying to play around with https://nixos.org/learn.html actually
w

wooden-thailand-8386

10/20/2020, 8:24 PM
oh cool, a friend of mine sent me that another day, how is it going?
f

flat-zoo-31952

10/20/2020, 8:24 PM
idk, i havent' cracked it yet
🤯 1
h

hundreds-father-404

10/20/2020, 8:27 PM
Haskell community loves nix, I haven’t tried it either