https://pantsbuild.org/ logo
Join Slack
Powered by
How do I debug a digest `Get` not seeming to match...
# plugins
b
How do I debug a digest 
Get
not seeming to match on anything? When I try to get 
DigestContents
, there are no files coming back
The globs going into 
Get(Digest, ...)
are what I'm expecting - e.g. 
['.pants.d/tmpciy3o72m/**/module_name/*.py']
if it makes any difference, this is in a 
goal_rule
re-read the docs and it looks like the answer is I can't get a digest of a file in the the pants workdir
Ok, so I just ended up dropping down to regular python IO operations, which I was hoping to avoid. The TLDR of what I'm trying to do is copy files out of the working directory back to the original sources, if there's a better way I'd love to know
h
Yeah the engine ignores that because of 
pants_ignore
Are you able to share the use case for wanting to copy files out of the working directory?
b
I'm working on trying to get django's manage.py to work via 
run
(or a variant)
I have a functional version of this atm, but it's obviously hacky
specifically for django's migrations generation, django generates python migration files in a module next to the data model sources by convention, hence why it writes out to the working directory
👍 1
h
Ah. cc @happy-kitchen-89482 w/ your Django example repo Taking a step back, there could be a simpler solution for your specific repository, although it doesn't generalize great: are you using codegen? The whole reason 
./pants run
uses a tempdir is to support codegen without mutating the build root If that isn't a concern for you, you could basically copy and paste 
run.py
and remove the tmpdir stuff, use a custom goal like 
django-run
. In your 
register.py
, add 
UnionRule(DjangoRunFieldSet, PexBinaryFieldSet)
(see 
run_pex_binary.py
)
b
Can codegen utilize dependencies from my sources?
I'd sorta made an assumption that it didn't because it doesn't use pants' workdir
h
Can codegen utilize dependencies from my sources?
I'm not following what you mean
b
So the AFAIK django's migration generation at least partially initializes the django app, at least enough to load the data models, which means it needs to be able to resolve whatever imports the code it's loading needs
what I have right now is basically a copy/paste of the 
run
goal plus some file copying code at the end, so it sounds like you're saying I could just drop the tempdir and do it right in the build root?
➕ 1
h
Oh, yeah 
./pants run
loads the PYTHONPATH for you. In 
run_pex_binary.py
, this is via setting the env var 
PEX_EXTRA_SYS_PATH
on line 91 With my 
django-run
custom goal recommendation, Django should still be able to access all your dependencies correctly. The main thing that changes is it doesn't use a tmpdir anymore and any codegen you're using (e.g. Protobuf) will get written to the build root, which is irrelevant if not using Pants's codegen
b
It's very typical of django to have some level of bidirectional code loading and disk writing happen
h
Exactly. And to be clear, my suggestion is a workaround - Pants still needs to get a better story for 
./pants run
b
yeah, i was hoping to figure out a clean way to do it but digging through the sources told me there wasn't a clear hook and i just ended up fighting the sandboxing. What I ended up with was a 
django_app
target with an argument to specify globs that would be synced back to the build root on completion of the run. That way I can support arbitrary writes that applications and libraries may have
that target was just an extension of 
pex_binary
with some extra sugar
Copy code
django_app(
    name="testapp",
    entry_point="testapp/manage.py",
    dependencies=["tests:testapp-lib"],
    mutable_files=[
        "**/migrations/*.py",
    ]
)
h
This is really interesting. My current recommendation is not to use 
./pants run
for most 
manage.py
uses, but of course that is not very satisfactory.
For example, 
runserver
runs out of a tmpdir copy, so its autoreloader won't work
and, as you noted, migrations will get written into the tmpdir
But of course this is not great because, for example, if your Django app relies on some codegenned protobuf, you kinda have to use 
./pants run
So I'd be interested in helping solve this
And ideally upstreaming into a publicly-available plugin, if you're OK with that
h
Benjy what if we did write codegen to build root but (optionally) cleaned it up always?
b
Definitely eager to solve this (collab welcome). I've given the non-tmpdir approach a quick try and it can't find the pex, which might be a my-code thing
h
Nathan can you share that code? I can take a look
b
@hundreds-father-404 here it is in all its hacky glory 😅 https://gist.github.com/rhysyngsun/ec9ec740e0ceb182108eae209577baee
my attempt from earlier this morning tried to use digests plus 
workspace.write_digest
to accomplish the copying
h
b
I get this:
Copy code
λ › ./pants django-run tests:testapp                                                                                   odl/ol-django nl/pants-t2
13:30:47.17 [INFO] initializing scheduler...
13:30:47.43 [INFO] scheduler initialized.
/usr/bin/python3.8: can't open file './testapp.pex': [Errno 2] No such file or directory
h
hm I'll send you a diff, one sec
Maybe in a clean branch, try this diff so that 
./pants run
no longer uses a tmpdir: Does Django now do what you want?
Untitled.diff
Untitled.diff
b
yeah, modifying my 
django-run
goal similarly, it works
so the piece i was missing was the correct 
chroot
👍 1
I also think my previous attempt to remove the tmpdir may have also removed the 
write_digest
for 
RunRequest
, which is probably why it couldn't find the pex
➕ 1
h
So..then I think all we need to figure out is how to clean up the files written to build root, like 
requirements.pex
. We cannot rely on Git because the user might have modified Git state before running 
./pants run
@happy-kitchen-89482 what do you think about giving up on the chroot and running directly in the build root? It removes a lot of warts I'm trying to figure out if we can always cleanup, e.g. behavior if you ctrl-c out of program
There is a weird edge with writing directly to build root: if you 
./pants
in another shell while 
./pants run
is still executing, it will see those new temporary files
Yeah, I opened https://github.com/pantsbuild/pants/issues/12129. I think we need to give up on a chroot, too many fundamental issues with it
2 Views
Open in Slack
PreviousNext
Powered by