https://pantsbuild.org/ logo
Join Slack
Powered by
Hey folks, I'm trying to write a build instruction...
# general
v
Hey folks, I'm trying to write a build instruction to generate a file in the source tree (a kind of lock file for the type checker) and I'm fighting with pants a little. What I want is to run a command that generates a file that I check in - like a lock file for the type checker. (this is the command 
poetry run mypy --strict -p my-package | poetry run mypy-json-report parse --output-file report.json --diff-old-report report.json
) I'm using 
run_shell_command
to do this because nothing else seems to put the file in the source tree, probably because it's in the sandbox. This still isn't working because 
run_shell_command
is running in an environment with none of the python deps installed. I can start writing commands to install python, poetry, etc - that feels like the wrong path. Any guidance you have would be appreciated.
This is the build code I'm using:
Copy code
def mypy(service: str, ratchet: str = "mypy-ratchet.json"):
    run_shell_command(
        name="mypy",
        tags=["mypy"],
        command=f"poetry run mypy --strict -p {service} | poetry run mypy-json-report parse --diff-old-report {ratchet}",
        execution_dependencies=[f"{service}:poetry"],
    )
    blank = "{}"
    run_shell_command(
        name="mypy_approve",
        tags=["mypy_approve"],
        command=f"(cat {ratchet} > /dev/null || echo '{blank}' > {ratchet}); poetry run mypy --strict -p {service} | poetry run mypy-json-report parse --output-file {ratchet} --diff-old-report {ratchet}",
        execution_dependencies=[f"{service}:poetry"],
    )

mypy("my-service")
b
I think the 
adhoc_tool
target and/or 
shell_command
's 
executable_dependencies
might be what you as you'e tried... but using 
mypy
as the executable, potentially packaging 
mypy
into a 
pex_binary
target. You can see what I do in my work repo along these lines at https://github.com/pantsbuild/pants/discussions/18235#discussioncomment-6655594
v
That does look a lot closer to what I want. Thank you for the example.
I'm still struggling to make this work. I am building a 
pex_binary
to run both mypy and my report tool. These pex binaries failed to run inside the sandbox. They complained that 
python3.11
was missing. When I added 
python3.11
to 
tools=[...]
on 
experimental_test_shell_command
I got a python that wouldn't run. I guess it's missing the list of other files that go into making a working python env. I feel like I'm missing something basic but I don't know the precise question I should be asking
current state of play in full:
build_macros.py
b
Oh, I'm sorry, I think I gave you misleading advice. I was meaning 
runnable_dependencies
above, not 
executable_dependencies
(which doesn't exist, and I can see that 
execution_dependencies
is the most sensible correction 🙂 ): https://www.pantsbuild.org/stable/reference/targets/adhoc_tool#runnable_dependencies (similar for 
shell_command
) In particular, 
runnable_dependencies
will do all the Python management. It puts an executable with the target name onto the command's 
PATH
, so something like 
runnable_dependencies=[":mypy_pex"]
should be able to call 
command="...; mypy_pex ...; ..."
v
This is all a good learning experience, thanks 🙂 I'm still struggling to make the pex run in the sandbox. It doesn't have access to python in there. I used 
system_binary(name="py311", binary_name="python3.11")
as a runnable dependency but that's not working. The 
python3.11
binary is linked into the sandbox but without the other files it needs to operate. It just doesn't run.
I feel like I'm missing something key. Seems like there should be a python environment somewhere akin to what 
poetry install
would make me, and that accessing this would allow me to simply run python tools as defined in my 
pyproject.toml
Oh, I'm sorry, I think I gave you misleading advice.
I'm appreciating the advice you're giving. I'm learning plenty of things about pants along the way even with a few rabbit holes 🙂
b
Ah, is that with the pex as a runnable dependency? or is the pex still an executable dependency?
v
That's with the pex as 
runnable
and python as 
runnable_dependency
b
hmmm, using a pex as a runnable should let pants choose the right python all automatically, not sure what's going on there
v
Maybe it's getting confused by my pyenv based installation of python3.11? I'll try some more debugging in the sandbox. If it's set up right should I see stuff like the default site-packages linked into the sandbox?
3 Views
Open in Slack
PreviousNext
Powered by