https://pantsbuild.org/ logo
#general
Title
# general
w

worried-salesclerk-37834

08/06/2021, 2:38 PM
I have a repo that uses both pandas and a google library. I'd like to build a pex file that will run on both my mac machine and a linux container image. Since pandas has some bindings under the hood, it requires the pre-built wheels, but google has a core dependency that doesn't have any pre-built wheels (https://pypi.org/project/grpc-google-iam-v1/#files). So when I tried to build the pex for multiple platforms, it tells me that it can't find a matching distribution for that grpc package. I was doing this by specifying something like:
Copy code
pex_binary(
  name="pex_binary"
  platforms=["linux-x86_64....", "macosx..."]
)
Am I missing some sort of config that would let the google dependency build its own wheel, or does it always have to use a wheel when specifying a platform?
f

fresh-cat-90827

08/06/2021, 2:39 PM
From https://github.com/pantsbuild/pants/issues/12205#issuecomment-861516498
When you specify a platform, all distributions necessary must be available as pre-built wheels. It’s up to you to pre-build them and make them available.
is building a wheel an option for you (either manually or as part of a CI pipeline)?
w

worried-salesclerk-37834

08/06/2021, 2:45 PM
Ah, that answers the question, thanks! The route I currently took is just to have a build container for pants that's the same distro as the one that the pex will run in. I might look into building the wheels, but I feel like that might turn into whack-a-mole. Did you end up having to do it for quite a few packages?
Not that we have the same dependencies, just curious how common this is.
f

fresh-cat-90827

08/06/2021, 2:53 PM
The route I currently took is just to have a build container for pants that’s the same distro as the one that the pex will run in.
Sorry I am struggling to understand what this implies, could you please explain? I’d love to understand your situation better.
Did you end up having to do it for quite a few packages?
We have about 50 external dependencies and I’ve built 5 wheels out of them (hosted on a private PyPI). I do agree that it’s not a sustainable situation to build many wheels (in particular those that are not pure Python — with C or Rust components — because you’ll need a toolchain present to build it for each target architecture/version).
depending on the projects, you may have luck asking the project maintainers to build a wheel as part of their pipeline — it’s possible that haven’t realized that it may be actually important for people (even with pure
sdist
packages)
👍 1
See an example https://github.com/bokeh/bokeh/pull/11213 — in version 2.4.0 they started publishing wheels 🙂
w

worried-salesclerk-37834

08/06/2021, 3:00 PM
I made a docker-compose to handle packaging.
Copy code
services:
  build:
    image: "python:3.7"
    volumes:
      - .:/code
    command: sh -c "code/pants package ::"
It's a hack for now, but the files are in my dist which I can then use to build my image.
f

fresh-cat-90827

08/06/2021, 3:34 PM
ah I see. If you ever decide to look at the wheel building, look at https://github.com/pypa/manylinux#docker-images since you are already in Docker space. Good luck!
2 Views