https://pantsbuild.org/ logo
#general
Title
# general
a

adamant-magazine-16751

10/25/2022, 3:03 PM
I'm looking for a way to run a python script before packaging a pex_binary. I'd like the script to create a file, that would then be packaged as a resource into the pex. Is writing it as
experimental_shell_command
that executes
python my_script.py
the way to go? What do I do afterwards to access the file?
We're using HTTPSource for one of our dependencies, but here it's unfortunately not available through http
b

bitter-ability-32190

10/25/2022, 3:09 PM
I would absolutely love to say yes. But you've hit the brick wall of Pants assumptions 😞
experimental_shell_command
does generate files that can be consumed. BUT it literally generates a
files
target type, which cannot be treated like a
resource
and therefore cannot be packaged in a
pex_binary
Your only solution today is to write your own in-repo codegen plugin that is almost 1:1 the same as
experimental_shell_command
, but the
output
classvar in the
GenerateSourcesRequest
is a
resource
. ...in fact I think you can have it be 1 statement long:
Copy code
class GenResourceTarget(ShellCommandTarget):
    alias = "generate_resource`

class GenResourceRequest(GenerateSourcesRequest):
    input = ShellCommandSourcesField
    output = ResourceSourceField

@rule
async def do_the_thing(request: MyRequestType) -> GeneratedSources:
    return await Get(GeneratedSources, GenerateFilesFromShellCommandRequest, request)

def rules():
    return [*collect_rules(), UnionRule(GenerateSourcesRequest, GenResourceRequest),]
(didn't test the above)
I'm hopign to fix this in the future. It's my next "big" feature I plan to investiagte/tackle
❤️ 1
a

adamant-magazine-16751

10/25/2022, 3:30 PM
Oh I see. Well, thank you for your suggestion. I'll see if it works for our use case. One last question. If I understand correctly, if we just built a docker_image or an archive and wrote our code around it, we should be able to pull this off with what I wrote before right? It's not perfect, as while developing we often run without building images, but it could work for now
b

bitter-ability-32190

10/25/2022, 3:31 PM
Ah well if you want the file in the
pex_binary
you're SoL. If you want the file in an
archive
or in a
docker_image
(I think) that's actually doable, since those are
file
-aware
a

adamant-magazine-16751

10/25/2022, 3:31 PM
Ok, thanks 🙂
b

bitter-ability-32190

10/25/2022, 3:32 PM
This line tells me the output of an
experimental_shell_command
WILL be in the Docker build context
Oh I forgot to say. One unfortunate side-effect of Pants here as well: Any
dependencies
for your "generator" target will also be dependencies of your thing-that-depends-on-it 😐
OK once we have https://github.com/pantsbuild/pants/pull/17277 and https://github.com/pantsbuild/pants/discussions/17336 I'll have 2 blog posts to write (one about those + @curved-television-6568’s
pipx
-like tip. Then one about Python GPU support in Pants)
🎉 2
h

happy-kitchen-89482

10/25/2022, 5:12 PM
I’m looking into
experimental_shell_command
, and @bitter-ability-32190 has long had thoughts about the mess of files vs resources
1
Sigh, why did we decide that
files
can’t be embedded in a pex? I get that it’s troublesome in theory, but in practice…
b

bitter-ability-32190

10/25/2022, 5:32 PM
@witty-crayon-22786 would say that
files
are different form resources.
files
are expected to be loaded from disk and not embedded in packages.
pex_binary
being like a compiled JAR or a native EXE (with embedded bytes)
w

witty-crayon-22786

10/25/2022, 5:36 PM
if PEX had support for automatically extracting
files
in the CWD it is running in so that you could use
open
with a relative path, then we could support using
files
with PEX… but it doesn’t right now (and it would be a bit awkward: you’d have to be careful where you ran the PEX not to overwrite stuff in the directory)
meanwhile,
resources
are always in the PYTHONPATH, and are never (assumed to be) relative to the cwd.
to be clear though: how you declare that you want to consume something as
resources
or
files
could/should change over time (i.e. to be consuming-target-side rather than producing-target-side). i’m just fairly sure we do actually need the split.
h

happy-kitchen-89482

10/25/2022, 5:51 PM
Sure, but users have expectations about this working, and it doesn’t. In practice, if your source root is the repo root, there is no difference, and it looks like Pants is being obstreperous…
b

bitter-ability-32190

10/25/2022, 5:52 PM
Yeah I don't disagree with Stu (I don't personally use the delineation, but I can see a universe where it matters 😛 ) but how we have it today is not ideal
w

witty-crayon-22786

10/25/2022, 5:52 PM
In practice, if your source root is the repo root, there is no difference, and it looks like Pants is being obstreperous…
that’s not true once you’ve actually deployed a PEX somewhere. say you run it in your home dir… should it extract the
file
into your home dir automatically? should it overwrite whatever is already there?
h

happy-kitchen-89482

10/25/2022, 5:52 PM
But if the distinction is on the consumer side, that starts to make more sense (e.g., “consume these files as resources relative to this source root”)
If the pex has unzipped itself then the file is potentially available via relative path from a caller’s
__file__
, why would it need to be extracted to your homedir?
w

witty-crayon-22786

10/25/2022, 5:55 PM
that is different from “it’s in your source directory / repo root”, and would be different from how
files
work in test running, with archives, etc.: in those cases, the file is relative to the CWD
☝️ 1
4 Views