Hello. I’m having difficulties working with the Py...
# general
Hello. I’m having difficulties working with the Python protobuf integration, and I’m wondering if anyone can suggest any debugging strategies. If I run the
goal, I see the generated Python files where I expect them under
. But when `run`ing a script that depends on these generated files, I get an import error. I’ve included
files in every directory. Is there a way to manually inspect the directories that Pants links at runtime, especially the directory where it stores generated code?
Sorry for the trouble! A good way to debug such things is to run with
), which will preserve the sandbox dirs that Pants runs processes in, and log their locations.
Then you can cd into the dir of the failing process, and see what Pants thought they layout should be
it sounds like a source roots error, perhaps
Thanks! That’s exactly what I was looking for. I must be doing something stupid, because even this is failing, which I would not expect given the module structure:
Copy code
$ tree <sandbox dir>
<sandbox dir>
├── local_dists.pex
├── project
│   ├── __init__.py
│   ├── proto
│   │   ├── __init__.py
│   │   ├── generated_pb2.py
└── script.pex

$ PYTHONPATH=<sandbox dir> python3 -c "from project.proto import generated_pb2"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name 'generated_pb2' from 'project.proto'
and it presumably fails when you
inside the sandbox? (that is a script that ~replicates the process Pants ran)
I don’t see a
file. This is the log I’m using to get the <sandbox dir> above:
Copy code
16:16:24.99 [INFO] Preserving local process execution dir /private/var/folders/tn/0sw9hrgd0zl_py1s8gtmfznr0000gp/T/pants-sandbox-N1XHGI for interactive process
OK, I at least have an explanation for the odd result above. I originally ran it from the local repository (i.e., where
lives), and it searched first in the current directory (where the generated file wouldn’t exist) before checking PYTHONPATH. When I first changed directories to the sandbox directory, then ran the import test, it worked. I verified the sandbox dir is in
too, so I’m not sure why it’s not getting resolved when run via
. Anyway, I’ll continue debugging. Thanks for your help.
Ah, so you did
./pants run path/to/script.py
and it failed?
What happens if you
./pants run path/to:script_target
with the address of the pex_binary target for the script?
I did the latter, though the target is a
, if that matters.
But I guess it implicitly builds and runs a pex file when you
Not exactly. Are you setting
on your
“Running a pex_binary is equivalent to package-ing the target followed by executing the built PEX from the repo root.”
But that is slower and often overkill
So running a
actually makes sense
In your case
and if
then I think the codegen should have worked
So that’s what I’m trying to get to the bottom of
Hey, sorry for the delayed reply. I tried running with
set explicitly, and had the same issue. However, I think I may have sorted this out. The problem seems to mirror my mistake above. Namely, that python will try to resolve modules in the execution directory before looking to the PYTHONPATH. I had my project set up with the top-level python module at the root of the repository. When I simply moved that top-level module directory under a
directory, updated the
config in
), and the various build targets to add the
prefix, it worked. Is it possible that
is replicating/preserving the semantics of python, looking first in the local directory even though it won’t contain generated files? I can try to put together a minimal reproducible example if that would be helpful.
Hmm, I’m glad it works now, but it’s not great that you had to refactor your code. So a minimal repro would be very helpful, thanks! It may be that it has to work this way, but I’d like to at least get to the bottom of why…
Ok, I think this should be a minimal example to reproduce. It contains two subdirectories, one in which the command fails, one in which it succeeds. There’s a README there with more info and example commands.
🙏 1
Could I trouble you to open a bug report for this at https://github.com/pantsbuild/pants/issues/new/choose ? Slack is so ephemeral, and that will last
Yup, will do
Thanks, post a link here when it’s up?
i’m running into this exact isssue now
i have the exact same structure as david and i can’t ./pants run but i can use pex target correctly
ah updating my pex to 2.1.121 (not a simple task) fixed it