https://pantsbuild.org/ logo
Join Slack
Powered by
I think this is somewhat of an antipattern but I'm...
# general
h
I think this is somewhat of an antipattern but I'm curious to get ideas: Is there any way to bundle up a bunch of python sources neatly into one pex binary-like thing? I think what I'm describing is a distribution now that I type it out. Basically, I have a bunch of modules that individually are executable but semantically should be grouped together. They'll all share roughly the same dependencies so packaging them all up as individual pex binaries would be pretty inefficient. I'd like to distribute them (and any transitive third party dependencies) as a unit. Should I look at Pants's wheel building capabilities for this or is there another direction folks would recommend?
e
You may just want to add the conscript dep to your PEX, see more here: https://pypi.org/project/conscript/
h
oh that looks cool. So then I'd add all my scripts as dependencies and they'd all be callable in the same way but I wouldn't be duplicating the storage of shared dependencies?
b
That is neat!
h
That is indeed neat. The other option is to control the PEX entry point at runtime with the 
PEX_MODULE
env var.
E.g., 
PEX_MODULE=<http://my.app|my.app> mypex.pex
or 
PEX_MODULE=<http://my.app:main|my.app:main> mypex.pex
h
@enough-analyst-54434, do you have an example of how this would be built by pants?
I was able to get something built by doing
Copy code
pex_binary(
    name="scripts",
    script="conscript",
    dependencies=[
        "//path/to/my/custom:pex", "//:reqs#conscript"
    ])
but my console script doesn't show up when running that built pex with a 
--help
flag
e
The 
pex_binary
needs to depend directly or indirectly on each 
python_distribution
of yours that has a console script defined.
h
Yeah, looks like I was able to get it to work by setting up a 
python_distribution
target. Seems like I can't depend on that target in my 
pex_binary
target that uses console script.
It can't find the built 
tar.gz
sine 
python_distribution
throws that file in 
tar.gz
.
Controlling the pex entrypoint at run time is looking like a nicer option
poking around github, it seems like I should be able to have a pex depend on a 
python_distribution
. I get
Copy code
ProcessExecutionFailure: Process 'List contents of artifacts produced by //:scripts' failed with exit code 9.
at the moment. But I can run 
./pants package //:scripts
where 
scripts
is my 
python_distribution
target.
h
Are those python_distributions published?
Because if so, Pex can depend on them via requirements strings
And if not, do they need to be distributions at all?
h
They are not published. That just seemed to be what I had to do to make a pex_binary that could call out to things with 
conscript
But I think a distribution actually makes sense for us for right now so I'm going to go chase that down. It would be cool to see some native pants capability of bundling multiple pex binaries together with something like 
conscript
.
h
Gotcha
So Pants can build "local distributions" in this situation, so IIRC it should work to have the 
pex_binary
depend on the `python_distribution`s
It should build wheels and then embed them in the pex...
h
That's what I tried and it got through building the distribution when I was attempting to package that pex and then got the error about (the failed with exit code 9 one)
b
I'm thinking it might be time for this to become a ticket...?
h
probably close. Since I'm gravitating away from the pex option anyway, I'm not as concerned about it for right now.
h
Hmm, that exit code 9 error is definitely a bug. Could I trouble you for a stacktrace (
--print-stacktrace
) ?
h
Sure, I'll throw what I can in a ticket
h
Thanks!
exit code 9 is that the process got SIGKILLed
All that process is doing is running 
unzip
to get the file list from the wheel.
Are you running this on Linux? Could the OOMKiller be involved?
h
Yep, ubuntu 20.04
if it's a memory issue, theoretically I could make my distribution artificially smaller and maybe not see the error?
h
I'm not sure if it's related to the size of the distribution or just how much memory Pants is allowed to use in general on your system
unzip
shouldn't be taking up much RAM just to whizz through the file list in the .whl
3 Views
Open in Slack
PreviousNext
Powered by