https://pantsbuild.org/ logo
#general
Title
# general
a

ambitious-actor-36781

12/07/2022, 3:33 AM
so we're using the lame "gzip ~/.cache/pants" approach in our CICD But have started getting
tar ./lmdb_store/files/e/data.mdb file change as we read it
errors. What might be triggering this? My suspicion is pantsd is flushing changes while tar is running. Is there a way to force pantsd to cleanly flush and exit?
h

happy-kitchen-89482

12/07/2022, 6:48 AM
Hmm, not familiar with that error. By 'lame “gzip ~/.cache/pants” approach' do you mean that you save that dir to (e.g.,) S3 at the end of a run and restore it at the beginning of a subsequent run? If so, what cache key do you use for this?
(not directly related to your issue, just curious)
There is no clean shutdown for pantsd today
@rapid-bird-79300 You're doing something similar I believe? Have you ever hit this issue?
a

ambitious-actor-36781

12/07/2022, 6:56 AM
the not-lame way: reapi server the lame way: at the end of the build,
tar -czf
up the ~/.cache/pants directory, upload it to s3. at the start of the build, pull it from s3 and untar it
h

happy-kitchen-89482

12/07/2022, 7:22 AM
Got it
I would be remiss in not plugging Toolchain's remote caching (and soon remote execution), but that is not a replacement for figuring this out.
Your flushing hypothesis seems reasonable, I think @witty-crayon-22786 would know more
a

ambitious-actor-36781

12/07/2022, 7:51 AM
I would love to use toolchain. Just not an option at the moment :(
g

gifted-agency-25998

12/07/2022, 3:32 PM
I stumbled across https://v1.pantsbuild.org/invoking.html#the-pants-daemon-pantsd earlier today which talks about how to kill the daemon:
To kill the daemon, you can run either
clean-all
or `kill-pantsd`:
```./pants clean-all
./pants kill-pantsd```
e

enough-analyst-54434

12/07/2022, 4:55 PM
@gifted-agency-25998 are you seeing the new red banner on those docs? That advice does not work for Pants v2.
You should consider Pants v1 and Pants v2 totally separate tools. Almost no Pants v1 advice from those pages will be applicable to Pants v2.
r

rapid-bird-79300

12/07/2022, 5:03 PM
Yes we've had very similar issues in the past. We now use a separate pipeline which generates the cache from a clean state (no pulling the cache at the start) and then uploads it to S3. Then CI builds that run pants can pull the cache as needed. We have not seen that issue since.
h

happy-kitchen-89482

12/07/2022, 5:04 PM
Interesting! And also annoying to have to do.
If someone wants to add a clean shutdown command, that would end up being useful!
Also, silly and non-rigorous hack, but does adding a short sleep to wait out the flush help in practice?
e

enough-analyst-54434

12/07/2022, 5:07 PM
I was thinking a capped loop on the tarring.
g

gifted-agency-25998

12/07/2022, 5:32 PM
my bad! For some reason, I was linked to the daemon anchor which meant that I didn't see red banner was above the fold
g

great-appointment-89710

06/01/2023, 10:36 PM
Just wondering if there is now a better solution than the capped loop when tarring? Apart from remote caching of course @happy-kitchen-89482 @enough-analyst-54434
h

happy-kitchen-89482

06/02/2023, 1:07 AM
Nothing has changed about pantsd shutdown since this thread began, no.
👍 1
4 Views