https://pantsbuild.org/ logo
c

calm-alarm-89082

11/09/2021, 11:15 AM
hi πŸ‘‹ is there a way in Pants to run code before adding a dependency to the PEX file? similar to setup.py when running pip install
c

curved-television-6568

11/09/2021, 11:21 AM
If you mean during a pants run, when building the PEX file, then yes, there is a experimental new target to run arbitrary shell commands with
experimental_shell_command
. Docs are yet TBW. But a small example in the PR: https://github.com/pantsbuild/pants/pull/12878
This is in 2.8.x
c

calm-alarm-89082

11/09/2021, 11:24 AM
thanks! πŸ™
in our specific case we need to download a file and include it in the PEX. IIRC there was a proposal to extend `files`/`resources` targets, is this still the plan?
c

curved-television-6568

11/09/2021, 11:42 AM
I’ll have to defer that question to the more senior Pants devs..
w

witty-crayon-22786

11/09/2021, 9:24 PM
experimental_shell_command
would allow for that usecase via
curl
, yea. but it would not be hermetic unless you validated the checksum of the file yourself in the script we should definitely support a
url(s)
argument to the
files
target at some point… or a separate
urls
target.
βž• 1
c

calm-alarm-89082

11/11/2021, 6:57 PM
Thanks. If it's something well defined I'd be interested in contributing πŸ˜„
πŸ™ 1
w

witty-crayon-22786

11/11/2021, 7:04 PM
It might be, yea... I can write up a ticket for it. Thanks for the offer!
c

calm-alarm-89082

11/11/2021, 7:39 PM
Thanks πŸ™‚
w

witty-crayon-22786

11/11/2021, 8:00 PM
opened https://github.com/pantsbuild/pants/issues/13589 … will probably need feedback on the suggested implementation, since there are at least two routes
c

calm-alarm-89082

11/13/2021, 12:21 AM
thanks, I had a look at the code and see how files/resources work but I'm not sure where the code to download the file should go. also, am I correct thinking that I should reuse
DownloadFile
in this case?
h

hundreds-father-404

11/13/2021, 12:26 AM
Yep! I can sketch out a little more how it will look, including an example of code that was just merged today like this But first, what do you think of the proposed syntax of the url field? That needs to be settled first
c

calm-alarm-89082

11/13/2021, 3:00 AM
I like the idea of having the url as a nested dictionary (a dataclass would be more readable but I'm not sure if that's supported) do you think checksum should be mandatory to enforce reproducibility for the build or leave that to the user?
h

hundreds-father-404

11/13/2021, 3:37 AM
I think it should be mandatory. We want to help people reduce the risk of supply chain attacks, and I'd rather optimize for that especially because setting up the target is a one time thing not done frequently. What do you think?
c

calm-alarm-89082

11/14/2021, 7:25 PM
yes, I think it makes sense. there are some cases where the file might not be versioned properly and the same url can return a new version of the file at any point. in this case, the build would fail and need to update the checksum on every external change. Either there's a way to opt-out from the check or must be clear in the docs that this scenario is not supported and this sort of files need to be committed in the repo i.e. publicsuffix.org provides a file in https://publicsuffix.org/list/public_suffix_list.dat, but this might get updated at any time
πŸ‘ 1
h

hundreds-father-404

11/14/2021, 7:35 PM
Great point about the workaround to download the file locally! I think that's a good thing that we would fail if the file changes overnight on you. We really want to reduce the risk of "works on my machine" type errors, or where your build breaks overnight
πŸ‘ 1
c

curved-television-6568

11/14/2021, 7:43 PM
Agree that you want to know as soon as such files change, rather than potentially get unpredictable errors later which could be difficult to find the root cause of a changed upstream file.. If they change more frequently than is manageable to keep up with, either host a version of the file or check in with the source could be possible alternatives.
βž• 1
5 Views