https://pantsbuild.org/ logo
Powered by
h

high-yak-85899

04/01/2022, 5:46 PM
I think this is somewhat of an antipattern but I'm curious to get ideas: Is there any way to bundle up a bunch of python sources neatly into one pex binary-like thing? I think what I'm describing is a distribution now that I type it out. Basically, I have a bunch of modules that individually are executable but semantically should be grouped together. They'll all share roughly the same dependencies so packaging them all up as individual pex binaries would be pretty inefficient. I'd like to distribute them (and any transitive third party dependencies) as a unit. Should I look at Pants's wheel building capabilities for this or is there another direction folks would recommend?
e

enough-analyst-54434

04/01/2022, 5:51 PM
You may just want to add the conscript dep to your PEX, see more here: https://pypi.org/project/conscript/
h

high-yak-85899

04/01/2022, 5:58 PM
oh that looks cool. So then I'd add all my scripts as dependencies and they'd all be callable in the same way but I wouldn't be duplicating the storage of shared dependencies?
b

busy-vase-39202

04/01/2022, 6:00 PM
That is neat!
h

happy-kitchen-89482

04/01/2022, 6:10 PM
That is indeed neat. The other option is to control the PEX entry point at runtime with the 
PEX_MODULE
env var.
E.g., 
PEX_MODULE=<http://my.app|my.app> mypex.pex
or 
PEX_MODULE=<http://my.app:main|my.app:main> mypex.pex
h

high-yak-85899

04/01/2022, 7:06 PM
@enough-analyst-54434, do you have an example of how this would be built by pants?
I was able to get something built by doing
Copy code
pex_binary(
    name="scripts",
    script="conscript",
    dependencies=[
        "//path/to/my/custom:pex", "//:reqs#conscript"
    ])
but my console script doesn't show up when running that built pex with a 
--help
flag
e

enough-analyst-54434

04/01/2022, 7:51 PM
The 
pex_binary
needs to depend directly or indirectly on each 
python_distribution
of yours that has a console script defined.
h

high-yak-85899

04/01/2022, 8:09 PM
Yeah, looks like I was able to get it to work by setting up a 
python_distribution
target. Seems like I can't depend on that target in my 
pex_binary
target that uses console script.
It can't find the built 
tar.gz
sine 
python_distribution
throws that file in 
tar.gz
.
Controlling the pex entrypoint at run time is looking like a nicer option
poking around github, it seems like I should be able to have a pex depend on a 
python_distribution
. I get
Copy code
ProcessExecutionFailure: Process 'List contents of artifacts produced by //:scripts' failed with exit code 9.
at the moment. But I can run 
./pants package //:scripts
where 
scripts
is my 
python_distribution
target.
h

happy-kitchen-89482

04/01/2022, 9:15 PM
Are those python_distributions published?
Because if so, Pex can depend on them via requirements strings
And if not, do they need to be distributions at all?
h

high-yak-85899

04/01/2022, 9:23 PM
They are not published. That just seemed to be what I had to do to make a pex_binary that could call out to things with 
conscript
But I think a distribution actually makes sense for us for right now so I'm going to go chase that down. It would be cool to see some native pants capability of bundling multiple pex binaries together with something like 
conscript
.
h

happy-kitchen-89482

04/01/2022, 9:24 PM
Gotcha
So Pants can build "local distributions" in this situation, so IIRC it should work to have the 
pex_binary
depend on the `python_distribution`s
It should build wheels and then embed them in the pex...
h

high-yak-85899

04/01/2022, 9:26 PM
That's what I tried and it got through building the distribution when I was attempting to package that pex and then got the error about (the failed with exit code 9 one)
b

busy-vase-39202

04/01/2022, 9:43 PM
I'm thinking it might be time for this to become a ticket...?
h

high-yak-85899

04/01/2022, 9:44 PM
probably close. Since I'm gravitating away from the pex option anyway, I'm not as concerned about it for right now.
h

happy-kitchen-89482

04/01/2022, 9:48 PM
Hmm, that exit code 9 error is definitely a bug. Could I trouble you for a stacktrace (
--print-stacktrace
) ?
h

high-yak-85899

04/01/2022, 9:49 PM
Sure, I'll throw what I can in a ticket
h

happy-kitchen-89482

04/01/2022, 9:50 PM
Thanks!
exit code 9 is that the process got SIGKILLed
All that process is doing is running 
unzip
to get the file list from the wheel.
Are you running this on Linux? Could the OOMKiller be involved?
h

high-yak-85899

04/01/2022, 9:57 PM
Yep, ubuntu 20.04
if it's a memory issue, theoretically I could make my distribution artificially smaller and maybe not see the error?
h

happy-kitchen-89482

04/01/2022, 11:24 PM
I'm not sure if it's related to the size of the distribution or just how much memory Pants is allowed to use in general on your system
unzip
shouldn't be taking up much RAM just to whizz through the file list in the .whl
3 Views
Powered by