https://pantsbuild.org/ logo
#general
Title
# general
b

bulky-alarm-94027

05/16/2022, 9:47 AM
👋 Hi everyone! what is the recommended way of replacing text in a source file as part of the package goal? e.g. i have a template metadata.json file for my app and i have to update the contents with the output of
git describe
c

curved-television-6568

05/16/2022, 11:02 AM
Hi Pablo 👋 Take a look at
experimental_shell_command
for how to run some arbitrary commands during a build. https://www.pantsbuild.org/docs/run-shell-commands
b

bitter-ability-32190

05/16/2022, 11:33 AM
So this is actually tricky.
experimental_shell_command
or any other Pants-native target will be run in a sandbox outside of your repo (for hermeticity). What that means is you essentially aren't able to run
git
commands.
Pants is making progress on the front though! I know @happy-kitchen-89482 has thought about this and other cases.
If you're stuck without this feature, I can share my own cobbled-together solution, but it involves pants plugins (which I hope to open source with other ones I have at some point)
c

curved-television-6568

05/16/2022, 11:37 AM
Yea, that was nagging a bit at the back of my head… although there’s ways around that particular limitation (using
GIT_DIR
), but not the prettiest. Another option is to run the
git describe
prior to the
./pants …
and inject the value as an env var for the shell command, if it’s not dynamic in nature.
And Josh may have a third option lined up there.. 😛
b

bulky-alarm-94027

05/16/2022, 1:06 PM
hi, thank you for the replies and suggestions. I did look at
experimental_shell_command
but it seemed that i would have to end up using
sed
or
awk
which felt counterintuitive for a build system (e.g.
ant
has a
replace
task, or in gradle by creating a function that does it). I did not know that i would not have access to git which also makes it awkward. I also looked a plugins and this seemed to be the best option, but it also felt like overkill.
b

bitter-ability-32190

05/16/2022, 1:16 PM
FWIW wanting git metadata is a common ask IMO (and it makes sense) so we should probably find a way to support it first-class
b

bulky-alarm-94027

05/16/2022, 1:16 PM
@Joshua Cannon can you share your solution?
b

bitter-ability-32190

05/16/2022, 2:22 PM
SO we have a shell script to output the metadata , so I'm using
experimental_shell_command
to run that. Then it's a matter of: • Getting the git command to work • Ensuring the cache "does the right thing" So here's a gist you can add easily as a plugin to get the absolute path to the repo root. Note it is an abs path so remote caching is right out. And here's a gist for a plugin which will code generate a file containing the current git SHA. When used a dependency it will invalidate the relevant rules when git SHA changes. YMMV it's a hack. In your case, you might be able to just twist the latter code to generate the
metadata.json
for you (it already has the build root, see line 47)
👍 1
Feel free to ask questions if you get stuck
b

bulky-alarm-94027

05/16/2022, 2:23 PM
awesome, thank you 🙂
h

happy-kitchen-89482

05/16/2022, 2:45 PM
I recently merged https://github.com/pantsbuild/pants/pull/15374 which supports running
setuptools_scm
specifically (and that calls out to git). It sounds like you want a more generic version of that?
We probably want
experimental_git_command
or something
b

bitter-ability-32190

05/16/2022, 2:47 PM
In our case I want a collection of git info, so I think we should collect the use cases first. That'll help us decide which way to go.
h

happy-kitchen-89482

05/16/2022, 2:48 PM
BTW can you not use
experimental_shell_command
by setting GIT_DIR and GIT_WORK_TREE ?
b

bitter-ability-32190

05/16/2022, 2:48 PM
You'd have to set it to the path to git dir, which not everyone has cloned to the same path, no?
h

happy-kitchen-89482

05/16/2022, 2:50 PM
The same relative path to the buildroot? We'd have to support interpolating
{buildroot}
the same way we support
{chroot}
Assuming
GIT_DIR={buildroot}/.git
seems reasonable
b

bitter-ability-32190

05/16/2022, 2:51 PM
You could but it really raises more questions than answers w.r.t. caching 😅
h

happy-kitchen-89482

05/16/2022, 2:51 PM
True
We'll have to support git natively, we already have scaffolding for this
b

bitter-ability-32190

05/16/2022, 2:51 PM
(which is why I have that awkward
gitstate
target)
h

happy-kitchen-89482

05/16/2022, 2:51 PM
That isn't cached
But no matter how we implement it, that does violate the sandbox, unless we take great care to exclusively run read-only git commands
Otherwise an
experimental_shell_command
can revert all your work in your real buildroot
sandbox be damned...
b

bitter-ability-32190

05/16/2022, 2:55 PM
The other issue is if you get it working, depending on your dep graph, I found our build metadata deeply rooted (lots of files indirectly depend on the python file which reads the metadata). Therefore doing it the "right way" invalidates my dep graph every commit 😅
🤔 1
c

curved-television-6568

05/16/2022, 3:03 PM
Clone the
.git
dir into the sandbox as a bare clone (no worktree) From there you can do all kinds of git-fu in read only mode.
🤯 2
b

bulky-alarm-94027

05/16/2022, 3:13 PM
@bitter-ability-32190 we currently use git describe to version our python package, i wonder if there is any way of curbing how much of the dep graph gets invalidated
b

bitter-ability-32190

05/16/2022, 3:15 PM
The way we solved this is to have the file which reads the metadata use an
experimental_shell_command
which is hermetic (doesn't get invalidated). Then the dep graph is sound although the metadata is stale. Then for our docker images (or really anything we want to package) I explicitly tell pants to ignore that
experimental_shell_command
and use an identical one which depends on that
gitstate
target I provided earlier. So then it's cached behavior depends on commits, but we arne't building docker images every PR, so it's OK
👍 1