https://pantsbuild.org/ logo
Join Slack
Powered by
Soliciting feedback for the open-source dbt backen...
# plugins
a
Soliciting feedback for the open-source dbt backend I'm working on - dbt utilizes a crude form of caching by regenerating its manifest every time it is invoked and using a partial compressed version of the manifest from the previous run as a cache. In the version I had at work, we basically hacked around this by using 
cp
to pull this file in and out of sandbox but for an open-source version I'm wondering if anyone has any better ideas.
f
Could you use the "append-only cache" feature of Pants' execution sandboxes?
a/k/a "named cache"
Point dbt into the named cache or otherwise arrange for the cached data to be placed there.
a
Circling back to this - are there docs on the 
append_only_caches
anywhere?
Is there any way to read the contents of a folder designated as an 
append_only_cache
into a digest? The append only cache feature works well for what I'm trying to do except I also need access to the content of the files in there.
If not I suppose I can just use bash and manipulate directory names to create a copy
f
Maybe run a 
Process
with the append-only cache and just capture outputs from the cache directory as you would for any other process?
a
The folder I'm pointing the cache to is not appearing in the 
output_digest
but when I preserve the process execution sandbox it is there (as a link into my 
named_caches
)
Interestingly if I specify the exact file I need from the cache in 
output_files
I get an error
Copy code
native_engine.IntrinsicError: Snapshots must be constructed from unique path stats; got duplicates in [Some(".cache/pex_root"), Some(".python-build-standalone"), Some("target")]
f
weird that it finds those to be duplicates
a
Yeah, I see that mypy already uses a workaround, which I noted in my issue. It would be nice if this was handled by the engine implicitly.
f
The code in 
src/python/pants/init/plugin_resolver.py
runs a 
Process
to leak the cache paths back into rule code. That might be a work-around as well.
that is, follow the append-only cache's symlink and print the path to stdout; then use that path back in the rule code to just directly access the disk
note though, this will break with `remote_environment`and 
docker_environement
a
This was working nicely for me (minus the 
exit 1
at the top, that's for the issue below).
10 Views
Open in Slack
PreviousNext
Powered by