https://pantsbuild.org/ logo
l

little-easter-51691

04/01/2022, 3:21 PM
Hello, I'd like to add the output (a dir with some files) of a python script (which has its own python dependencies) to a folder of an archive of another project. One possible way is to build a pex binary and call it with the experimental_shell step and use its output, but here the pex needs to be on the path to be found by the experimental_shell command. Is there an idiomatic way to achieve this?
A less beautiful way could be to write a shell script, which installs the python deps and calls the python script and produces the output
h

happy-kitchen-89482

04/01/2022, 4:04 PM
To clarify, is Pants also building the archive of the other project? If so, what is its target type?
l

little-easter-51691

04/01/2022, 4:51 PM
Yes, it is using the
archive
target (zip), files of the other project are also picked up via
files
(here I'd add the generated dir/folders in a subdir)
w

witty-crayon-22786

04/01/2022, 5:01 PM
experimental_shell_command
should be able to depend on a
pex_binary
to pull it in… if not, that would be worth filing an issue for
h

happy-kitchen-89482

04/01/2022, 6:07 PM
But does the
archive
target depending on the
experimental_shell_command
cause it to run when you
package
the
archive target
?
w

witty-crayon-22786

04/01/2022, 6:08 PM
yes, it should.
h

happy-kitchen-89482

04/01/2022, 6:08 PM
Ah, then that could work
l

little-easter-51691

04/01/2022, 7:34 PM
Here is a quick example:
Copy code
python_sources()

pex_binary(
    name="binary",
    entry_point="script.py:main",
)

experimental_shell_command(
    name='generate-output',
    command='sh -c ./src.python.myoutputproject/binary.pex',
    dependencies=[':binary'],
    outputs=['foo/'],
    tools=["sh"]
)

archive(
    name="archived-output",
    format="zip",
    files=[":generate-output"]
)
I debugged a bit:
Copy code
myoutputproject cd /tmp/process-execution5SOGuS
➜  process-execution5SOGuS ls
__run.sh  src  src.python.myoutputproject
process-execution5SOGuS cat __run.sh 
#!/bin/bash
# This command line should execute the same process as pants did internally.
export TOOLS=$'mkdir ln sh' ln=/usr/bin/ln mkdir=/usr/bin/mkdir sh=/usr/bin/sh
cd /tmp/process-execution5SOGuS/src/python/myoutputproject
/usr/bin/bash -c $'$mkdir -p .bin;for tool in $TOOLS; do $ln -sf ${!tool} .bin; done;export PATH="$PWD/.bin";sh -c ./src.python.myoutputproject/outputbin.pex'
Here the
__run.sh
script cds into the project folder, the trick is to cd out again
_command_='../../../src.python.myoutputproject/binary.pex',
(not sure if there is variable one could use to point to the created tmp dir)
w

witty-crayon-22786

04/01/2022, 9:44 PM
oh… hm. yea, that’s interesting. definitely part of the reason that
experimental_shell_command
is still marked experimental. would you mind filing an issue? it does seem odd to me that the
cwd
of the process is nested below its working directory… even if the inputs are nested, it doesn’t seem like the script itself should be. or perhaps, all of the inputs should be nested in the same way.
i.e. CWD statically
${process_sandbox}/shell
with all of the process’s inputs nested below the CWD
l

little-easter-51691

04/02/2022, 1:25 AM
Yes, thanks will file an issue later
4 Views