hi 👋 is there a way in Pants to run code before adding a dependency to the PEX file? similar to setup.py when running pip install

If you mean during a pants run, when building the PEX file, then yes, there is a experimental new target to run arbitrary shell commands with

experimental_shell_command

. Docs are yet TBW. But a small example in the PR: https://github.com/pantsbuild/pants/pull/12878

thanks! 🙏

I’ll have to defer that question to the more senior Pants devs..

experimental_shell_command

would allow for that usecase via

curl

, yea. but it would not be hermetic unless you validated the checksum of the file yourself in the script we should definitely support a

url(s)

argument to the

files

target at some point… or a separate

urls

target.

Thanks. If it's something well defined I'd be interested in contributing 😄

It might be, yea... I can write up a ticket for it. Thanks for the offer!

Thanks 🙂

opened https://github.com/pantsbuild/pants/issues/13589 … will probably need feedback on the suggested implementation, since there are at least two routes

thanks, I had a look at the code and see how files/resources work but I'm not sure where the code to download the file should go. also, am I correct thinking that I should reuse

DownloadFile

in this case?

Yep! I can sketch out a little more how it will look, including an example of code that was just merged today like this But first, what do you think of the proposed syntax of the url field? That needs to be settled first

I like the idea of having the url as a nested dictionary (a dataclass would be more readable but I'm not sure if that's supported) do you think checksum should be mandatory to enforce reproducibility for the build or leave that to the user?

I think it should be mandatory. We want to help people reduce the risk of supply chain attacks, and I'd rather optimize for that especially because setting up the target is a one time thing not done frequently. What do you think?

yes, I think it makes sense. there are some cases where the file might not be versioned properly and the same url can return a new version of the file at any point. in this case, the build would fail and need to update the checksum on every external change. Either there's a way to opt-out from the check or must be clear in the docs that this scenario is not supported and this sort of files need to be committed in the repo i.e. publicsuffix.org provides a file in https://publicsuffix.org/list/public_suffix_list.dat, but this might get updated at any time

Great point about the workaround to download the file locally! I think that's a good thing that we would fail if the file changes overnight on you. We really want to reduce the risk of "works on my machine" type errors, or where your build breaks overnight

Agree that you want to know as soon as such files change, rather than potentially get unpredictable errors later which could be difficult to find the root cause of a changed upstream file.. If they change more frequently than is manageable to keep up with, either host a version of the file or check in with the source could be possible alternatives.