Hello, I'd like to add the output (a dir with some files) of a python script (which has its own python dependencies) to a folder of an archive of another project. One possible way is to build a pex binary and call it with the experimental_shell step and use its output, but here the pex needs to be on the path to be found by the experimental_shell command. Is there an idiomatic way to achieve this?

To clarify, is Pants also building the archive of the other project? If so, what is its target type?

Yes, it is using the

archive

target (zip), files of the other project are also picked up via

files

(here I'd add the generated dir/folders in a subdir)

experimental_shell_command

should be able to depend on a

pex_binary

to pull it in… if not, that would be worth filing an issue for

But does the

archive

target depending on the

experimental_shell_command

cause it to run when you

package

the

archive target

?

yes, it should.

Ah, then that could work

Here is a quick example:

python_sources()

pex_binary(
    name="binary",
    entry_point="script.py:main",
)

experimental_shell_command(
    name='generate-output',
    command='sh -c ./src.python.myoutputproject/binary.pex',
    dependencies=[':binary'],
    outputs=['foo/'],
    tools=["sh"]
)

archive(
    name="archived-output",
    format="zip",
    files=[":generate-output"]
)

I debugged a bit:

myoutputproject cd /tmp/process-execution5SOGuS
➜  process-execution5SOGuS ls
__run.sh  src  src.python.myoutputproject
process-execution5SOGuS cat __run.sh 
#!/bin/bash
# This command line should execute the same process as pants did internally.
export TOOLS=$'mkdir ln sh' ln=/usr/bin/ln mkdir=/usr/bin/mkdir sh=/usr/bin/sh
cd /tmp/process-execution5SOGuS/src/python/myoutputproject
/usr/bin/bash -c $'$mkdir -p .bin;for tool in $TOOLS; do $ln -sf ${!tool} .bin; done;export PATH="$PWD/.bin";sh -c ./src.python.myoutputproject/outputbin.pex'

Here the

__run.sh

script cds into the project folder, the trick is to cd out again

_command_='../../../src.python.myoutputproject/binary.pex',

(not sure if there is variable one could use to point to the created tmp dir)

oh… hm. yea, that’s interesting. definitely part of the reason that

experimental_shell_command

is still marked experimental. would you mind filing an issue? it does seem odd to me that the

cwd

of the process is nested below its working directory… even if the inputs are nested, it doesn’t seem like the script itself should be. or perhaps, all of the inputs should be nested in the same way.

Yes, thanks will file an issue later