https://pantsbuild.org/ logo
Join Slack
Powered by
I can’t seem to get the <example-python> repo pack...
# general
f
I can’t seem to get the example-python repo packaged as an executable 
.pex
properly without using Pants (using standalone 
pex
utility). The sources directory is not copied itself, only its contents. See the thread.
If I run 
./pants package helloworld:pex_binary
, then the 
.pex
file is produced having the 
helloworld
directory with the sources under it.
Copy code
├── .bootstrap
│   └── pex
├── .deps
│   ├── ansicolors-1.1.8-py2.py3-none-any.whl
│   ├── setuptools-56.2.0-py3-none-any.whl
│   └── types_setuptools-57.4.14-py3-none-any.whl
├── PEX-INFO
├── __main__.py
├── helloworld
│   ├── __init__.py
│   ├── greet
│   ├── main.py
│   └── translator
If I run 
pex
on its own with: 
pex --requirement=requirements.txt --sources-directory=helloworld -m main -o dist/purepex.pex
, then the root 
helloworld
directory is not included:
Copy code
├── .bootstrap
│   └── pex
├── .deps
│   ├── ansicolors-1.1.8-py2.py3-none-any.whl
│   ├── setuptools-56.2.0-py3-none-any.whl
│   └── types_setuptools-57.4.17-py3-none-any.whl
├── BUILD
├── PEX-INFO
├── __init__.py
├── __main__.py
├── greet
├── main.py
└── translator
Running the 
.pex
file won’t work because of the 
import
line (there isn’t 
helloworld
package, because the 
greet
and 
translator
ended up in the “root” of the 
.pex
):
Copy code
from helloworld.greet.greeting import Greeter
ModuleNotFoundError: No module named 'helloworld'
I wonder if it’s possible to do with 
pex
what Pants does and copy the sources directory itself, not just its contents? The only workaround I found is to copy the 
helloworld
directory into another directory:
Copy code
$ mkdir dummy-placeholder; cp -r helloworld dummy-placeholder/helloworld/
$ pex --requirement=requirements.txt --sources-directory=dummy-placeholder -m helloworld.main -o dist/purepex.pex
Is it what I have to do or is there a better way?
@enough-analyst-54434 I am sorry, any chance you are aware of this limitation and what would be the best approach?
e
There is no limitation. Just back up your directory tree by 1 level. The Pex repo itself has a similar layout with the 
pex
package at repo root:
Copy code
$ python -mpex -D . -m pex -o pex-with-lots-of-extra-files-too.pex
jsirois@siroisdesign:~/pex$ ./pex-with-lots-of-extra-files-too.pex -V              2.1.92
$ zipinfo -1 pex-with-* | head                           .bootstrap/
.bootstrap/pex/
.bootstrap/pex/__init__.py
.bootstrap/pex/__main__.py
.bootstrap/pex/argparse.py
.bootstrap/pex/attrs.py
.bootstrap/pex/auth.py
.bootstrap/pex/bootstrap.py
.bootstrap/pex/common.py
.bootstrap/pex/compatibility.py
$ zipinfo -1 pex-with-* | tail
tests/test_variables.py
tests/test_vendor.py
tests/tools/
tests/tools/__init__.py
tests/tools/commands/
tests/tools/commands/__init__.py
tests/tools/commands/test_interpreter_command.py
tests/tools/commands/test_repository.py
tests/tools/commands/test_venv.py
tox.ini
f
Thank you so much, @enough-analyst-54434, the 
-D .
did the trick! Will now try to convert packaging workflows for repos outside of the Pants monorepo to use 
pex
instead!
@enough-analyst-54434 just to confirm, if I do want to be specific and only package certain directories out of my current directory, then I’d have to copy those into a dummy directory like I described earlier? I.e. 
pex
won’t let me choose what directories to copy? E.g. say I don’t want to 
tests
from your example above.
e
Correct. Or, if you use something like setuptools (setup.py) or Poetry, Flit, etc (pyproject.toml), just point Pex at the directory containing that file as a requirement. Again, for Pex itself, which uses Flit:
Copy code
pex pex/git/clone/directory -c pex -o much-tidier.pex
🙏 1
@fresh-cat-90827 would something like 
pex --requirement=requirements.txt --sources-directory=".|helloworld" -m main -o dist/purepex.pex
work for you? That uses new syntax for 
-D / --sources-directory
where you can mention the module or package name to restrict the sources directory addition to after a pipe symbol. The symbol could also probably be 
:
.
Something along these lines is easy to add and works like most project management tools like setuptools / Poetry / Flit. It does not give the full flexibility of excludes, but it does allow what seems to be the most common case of a root source directory with extraneous bits (including `.git`dir, etc) that you don't want in the PEX / aren't part of the runtime code and resources files.
f
thanks for giving it a thought, John, this is very helpful. I think, yes, indeed, to be able to specify what do you want to include into the PEX file would be hugely helpful. However, I am not sure whether adding a new syntax for the 
--sources-directory
argument is worth it — I realized now that most likely I’d want to include a few directories (sources, resources directories, etc), and standalone files (license files etc). So I think I am better off creating a new directory, copying everything I need into it, and then passing it as a sources directory. This of course is wrapped in a shell script 🙂
e
Ok. Sounds good!
Open in Slack
PreviousNext
Powered by