Is there a way to override `python_distribution` '...
# general
b
Is there a way to override
python_distribution
's behavior of referring to other internal distributions and just build a hermetic sdist including all the necessary source files?
e
How will this fat sdist be used?
b
I install it inside a docker build script
fat sdist is useful because I only care about having a complete environment installed inside a docker image. not looking to publish the package
e
A PEX will have the fat property you're after - all in one file.
b
reasons why I've been using sdist: • Can install inside docker platform-independent, we have devs using macs • If I pip install
--user
, pyspark comes with a complete spark distribution included. Just one less versioning to worry about
e
I don't understand the second point but on the 1st there are a few ways to build a PEX for the container even when on a Mac. Have you seen or tried Pants support for environments?: https://www.pantsbuild.org/docs/environments
If you truly want to stick to sdists, maybe use 1 archive target that depends on the root sdists you need: https://www.pantsbuild.org/docs/reference-archive Then
pants package
that. You'll get a zip (say) with a bunch of sdists inside. In the container unzip it in a directory and then
pip install sdist-dir/*
Method #18 - use a PEX to create a fat sdist. Here is an example PEX built with
--include-tools
(In Pants that's a `pex_binary(..., include_tools=True)`:
Copy code
$ pex --python python3.7 -D src/python -r 3rdparty/python/requirements.txt -m pants.bin.pants_loader --include-tools -o pants.pex
It works when run as a PEX of course:
Copy code
$ ./pants.pex -V
2.17.0.dev4
But you can also use the pex tools you asked for to extract a fat sdist. You'll also get the wheels extracted, but you can ignore them:
Copy code
$ PEX_TOOLS=1 python3.7 ./pants.pex repository extract --dest-dir dist --sources
...
Writing pants-0.0.0+ea90af96740c36a4101573ccc0bdab8ea99db6e9/setup.cfg
Creating tar archive
removing 'pants-0.0.0+ea90af96740c36a4101573ccc0bdab8ea99db6e9' (and everything under it)
PyYAML 6.0: Repacking wheel as /home/jsirois/dev/pantsbuild/pants/dist/PyYAML-6.0-cp37-cp37m-manylinux1_x86_64.manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_5_x86_64.whl...OK
ansicolors 1.1.8: Repacking wheel as /home/jsirois/dev/pantsbuild/pants/dist/ansicolors-1.1.8-py2.py3-none-any.whl...OK
...
Now you can install Pants somewhere else just using the sdist:
Copy code
$ python3.7 -mvenv pants.venv
jsirois@Gill-Windows:~/dev/pantsbuild/pants (main) $ pants.venv/bin/pip install dist/pants-0.0.0+ea90af96740c36a4101573ccc0bdab8ea99db6e9.tar.gz
Processing ./dist/pants-0.0.0+ea90af96740c36a4101573ccc0bdab8ea99db6e9.tar.gz
  Preparing metadata (setup.py) ... done
Collecting PyYAML<7.0,>=6.0
  Using cached PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
...
Using legacy 'setup.py install' for pants, since package 'wheel' is not installed.
Installing collected packages: types-urllib3, types-toml, types-setuptools, types-PyYAML, types-freezegun, python-gnupg, node-semver, ijson, chevron, ansicolors, zipp, websockets, uvloop, urllib3, ujson, typing-extensions, types-requests, tomli, toml, soupsieve, sniffio, six, setuptools, setproctitle, PyYAML, python-dotenv, pyparsing, pygments, py, psutil, pex, mypy-typing-asserts, iniconfig, idna, httptools, debugpy, charset-normalizer, certifi, backports.cached-property, requests, python-multipart, python-lsp-jsonrpc, python-dateutil, pydantic, packaging, importlib_resources, importlib-metadata, h11, graphql-core, fasteners, beautifulsoup4, asgiref, anyio, watchgod, starlette, pluggy, humbug, freezegun, click, attrs, uvicorn, strawberry-graphql, pytest, fastapi, pants
  Attempting uninstall: setuptools
    Found existing installation: setuptools 47.1.0
    Uninstalling setuptools-47.1.0:
      Successfully uninstalled setuptools-47.1.0
  Running setup.py install for pants ... done
Successfully installed PyYAML-6.0 ansicolors-1.1.8 anyio-3.6.2 asgiref-3.6.0 attrs-23.1.0 backports.cached-property-1.0.2 beautifulsoup4-4.11.1 certifi-2022.12.7 charset-normalizer-3.1.0 chevron-0.14.0 click-8.1.3 debugpy-1.6.0 fastapi-0.78.0 fasteners-0.16.3 freezegun-1.2.1 graphql-core-3.2.3 h11-0.14.0 httptools-0.5.0 humbug-0.2.7 idna-3.4 ijson-3.1.4 importlib-metadata-6.6.0 importlib_resources-5.0.7 iniconfig-2.0.0 mypy-typing-asserts-0.1.1 node-semver-0.9.0 packaging-21.3 pants-0.0.0+ea90af96740c36a4101573ccc0bdab8ea99db6e9 pex-2.1.134 pluggy-1.0.0 psutil-5.9.0 py-1.11.0 pydantic-1.10.7 pygments-2.15.1 pyparsing-3.0.9 pytest-7.0.1 python-dateutil-2.8.2 python-dotenv-0.21.1 python-gnupg-0.4.9 python-lsp-jsonrpc-1.0.0 python-multipart-0.0.5 requests-2.29.0 setproctitle-1.3.2 setuptools-63.4.3 six-1.16.0 sniffio-1.3.0 soupsieve-2.4.1 starlette-0.19.1 strawberry-graphql-0.114.0 toml-0.10.2 tomli-2.0.1 types-PyYAML-6.0.3 types-freezegun-1.1.6 types-requests-2.28.1 types-setuptools-62.6.1 types-toml-0.10.8 types-urllib3-1.26.25.10 typing-extensions-4.3.0 ujson-5.7.0 urllib3-1.26.15 uvicorn-0.17.6 uvloop-0.17.0 watchgod-0.8.2 websockets-11.0.2 zipp-3.15.0
And it works:
Copy code
$ pants.venv/bin/python -m pants.bin.pants_loader -V
08:41:29.74 [INFO] Starting: Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.27.0
08:41:31.96 [INFO] Completed: Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.27.0
2.17.0.dev4