https://pantsbuild.org/ logo
a

alert-psychiatrist-14102

01/28/2023, 7:02 AM
Hi everyone, it's nice to join the community! I'm exploring using pants as an alternative to bazel for managing our python and js monorepo. Wanted to get your help on making the decision, mainly by getting more understanding of the following questions - 1. Is there support for hermetic python? I would not want to rely on local installation of python (not for dev and not for deployment) 2. We have a use case where we build a python server and embed it inside a js application. How would that work with pants? 3. Cross platform. We work on both macos and linux. Is there support for this with pants? Can we build dockers from macos?
👋 2
b

bored-energy-25252

01/28/2023, 7:25 AM
For 3, Pants works fine on macOS and GNU/Linux.
h

high-yak-85899

01/28/2023, 7:37 AM
For 1, yes. Check out the work in here mentioning scie-pants: https://github.com/pantsbuild/scie-pants
b

bitter-ability-32190

01/28/2023, 12:24 PM
Actually I wouldn't say that scie-pants is support for hermetic python. It certainly is half of the story, but we still don't have anything for running user code, yet.
I was musing adding #1 support, leveraging pyenv though. 🧐
h

high-yak-85899

01/28/2023, 5:54 PM
Oh. The first bullet of the scie docs features say “A hermetic Python interpreter appropriate for the current active Pants version”. But are you saying you still need an interpreter locally for development?
b

bitter-ability-32190

01/28/2023, 6:07 PM
Precisely. When doing development for Python using Pants theres the Python used for Pants itself (now being provided by scie-pants) and then the Python(s) used to run your code. The latter is still using system installations
1
t

thousands-france-27863

01/28/2023, 7:03 PM
For 1, you could also stick it in a docker image, or a Serverless function like AWS Lambdas. I believe you can use pants for the docker part also, but haven’t tried myself. Depends on your use case. On the second point, I don’t think there is support for js/ts land. Its on the roadmap but I dunno the status. For this scenario you can use another monorepo tool, like nx or bazel (fairly complex), to manage that side. I plan to try that at some point, but not sure when. Or contributing as this is an open source project ;) I can also say that it works fine on MacOS and linux. In both your local python version might interfere, but stick it in a Virtual Environment, and you are fine. I am not from pants, just evaluating it for the company. So far so good. Hope helps!
a

alert-psychiatrist-14102

01/28/2023, 9:27 PM
Nice, thanks for your answers! A quick follow up. I'm trying out pants and building a docker image from macos machine. I'm hitting the following error -
Copy code
Failed to find compatible interpreter on path /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin.

Examined the following interpreters:
1.) /usr/local/bin/python3.9 CPython==3.9.16
2.)       /usr/bin/python3.9 CPython==3.9.2

No interpreter compatible with the requested constraints was found:

  A distribution for numpy could not be resolved for /usr/local/bin/python3.9.
  Found 1 distribution for numpy that do not apply:
  1.) The wheel tags for numpy 1.24.1 are cp39-cp39-macosx_10_9_x86_64 which do not match the supported tags of /usr/local/bin/python3.9:
  cp39-cp39-manylinux_2_31_x86_64
  ... 661 more ...

  A distribution for numpy could not be resolved for /usr/bin/python3.9.
  Found 1 distribution for numpy that do not apply:
  1.) The wheel tags for numpy 1.24.1 are cp39-cp39-macosx_10_9_x86_64 which do not match the supported tags of /usr/bin/python3.9:
  cp39-cp39-manylinux_2_31_x86_64
  ... 661 more ...
i.e., the numpy 3rdparth dependency was packaged for mac and copied into the docker, which in turn can't run it. What's the correct way to build python dockers with pants? Thx!
e

enough-analyst-54434

01/29/2023, 2:49 AM
@alert-psychiatrist-14102 have a read here and report back your success or failure: https://pantsbuild.slack.com/archives/C046T6T9U/p1674738346393669
@alert-psychiatrist-14102 when you say hermetic, is something like what @bitter-ability-32190 suggested acceptable - i.e. git cloning pyenv and invoking it to download a Python tarball and build it? When I was last in the blaze world, a hermetic Python interpreter meant having the build tool build the interpreter from source (I.E. the build tool could handle compiling and linking C). Pants cannot do that today - there is experimental very early support, but you certainly couldn't use it to build CPython inline and then use that for your CPython distribution for tests, tools, etc.
@high-yak-85899 its better to consider scie-pants a binary and the fact Pants rules are written in Python as opaque. This totally makes sense if you're a Java + Go user of Pants and only seems jarring if you're a Python user of Pants. The fundamental issue is a pre-built CPython - for Linux - is ~impossible to do. Part of a working Python installation is being able to invoke the C compiler to build native extensions (
pip install some-sdist.tar.gz
). Good luck guessing the paths of compiler, linker, various headers and libraries on a Linux system.
I was putting the hermetic Python used by scie-pants on the PATH previously, for discovery / use by Pants Python Processes, but: https://github.com/pantsbuild/scie-pants/issues/69 - and so its no longer on the PATH to avoid kabooms.
a

alert-psychiatrist-14102

01/29/2023, 7:56 AM
Thanks! I'll give it a go and report back. Re hermetic python - the important part for me is having the python binary independent of the host system, both during dev and deployment (we currently deploy python binary as part of our package and have no system dependencies whatsoever). This is not a showstopper, but i'd like to understand the idiomatic solution using pants, as system dependencies are a big drawback for deployments.. ill explore further the pyenv + tarball solution. What would be the equivalent for linux (we deploy on both - macos and linux)?
e

enough-analyst-54434

01/30/2023, 1:16 PM
For deployment, assuming the Python application does not attempt to compile Python C extensions at runtime (~aka, your app is not a Python build tool), using the scie-pants approach works well; I.E. Use something like Python Build Standalone (https://python-build-standalone.readthedocs.io/en/latest/) as your portable interpreter and maybe use something like scie-jump (https://github.com/a-scie/jump) to package your Python app up into a single file binary that uses PBS. Pants does not support building scies yet, but @wide-midnight-78598 has started integration with some custom rules for his own use. The Pex (https://github.com/pantsbuild/pex) tool itself will incorporate scie-jump to allow picking
--par
(spelling to be determined) as a build option to finally obtain system independence for any PEX file. This has been a goal since inception in 2010, basically to have Google's internal PAR.
👍 1
b

bitter-ability-32190

01/30/2023, 3:44 PM
I've also had success with
pyoxy
(https://pyoxidizer.readthedocs.io/en/stable/pyoxy.html) for running Python without system installations. It has fewer features than standalone, but is all single-file which makes it REALLY easy to use.
👍 1
f

faint-businessperson-86903

01/30/2023, 7:44 PM
For this scenario you can use another monorepo tool, like nx or bazel (fairly complex), to manage that side.
At my company we worked around this by adopting two monorepos, one for our webapps written in typescript using turborepo and one for everything else via pants. It sucks splitting the work in two (we all choose monorepo for a reason!), but we’ve found turborepo is very specialized and effective for what it does, and pants not supporting js/ts made it a no brainer.
We’ve never had to embed a python server in a JS app though. I suppose if we took that on with our setup we would have both CI pipelines push/pull them from an internal data store, in a classic polyrepo-esque way.