https://pantsbuild.org/ logo
#general
Title
# general
r

refined-gigabyte-6122

11/09/2022, 8:28 PM
Hello. I’m having difficulties working with the Python protobuf integration, and I’m wondering if anyone can suggest any debugging strategies. If I run the
export-codegen
goal, I see the generated Python files where I expect them under
dist/codegen
. But when `run`ing a script that depends on these generated files, I get an import error. I’ve included
__init__.py
files in every directory. Is there a way to manually inspect the directories that Pants links at runtime, especially the directory where it stores generated code?
h

happy-kitchen-89482

11/09/2022, 8:33 PM
Sorry for the trouble! A good way to debug such things is to run with
--keep-sandboxes=on_failure
(or
--keep-sandboxes=always
), which will preserve the sandbox dirs that Pants runs processes in, and log their locations.
Then you can cd into the dir of the failing process, and see what Pants thought they layout should be
it sounds like a source roots error, perhaps
r

refined-gigabyte-6122

11/09/2022, 9:08 PM
Thanks! That’s exactly what I was looking for. I must be doing something stupid, because even this is failing, which I would not expect given the module structure:
Copy code
$ tree <sandbox dir>
<sandbox dir>
├── local_dists.pex
...
├── project
│   ├── __init__.py
│   ├── proto
│   │   ├── __init__.py
│   │   ├── generated_pb2.py
...
└── script.pex
...

$ PYTHONPATH=<sandbox dir> python3 -c "from project.proto import generated_pb2"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name 'generated_pb2' from 'project.proto'
h

happy-kitchen-89482

11/09/2022, 9:14 PM
hmmmmmmm
and it presumably fails when you
__run.sh
inside the sandbox? (that is a script that ~replicates the process Pants ran)
r

refined-gigabyte-6122

11/09/2022, 9:18 PM
I don’t see a
__run.sh
file. This is the log I’m using to get the <sandbox dir> above:
Copy code
16:16:24.99 [INFO] Preserving local process execution dir /private/var/folders/tn/0sw9hrgd0zl_py1s8gtmfznr0000gp/T/pants-sandbox-N1XHGI for interactive process
OK, I at least have an explanation for the odd result above. I originally ran it from the local repository (i.e., where
project
lives), and it searched first in the current directory (where the generated file wouldn’t exist) before checking PYTHONPATH. When I first changed directories to the sandbox directory, then ran the import test, it worked. I verified the sandbox dir is in
sys.path
too, so I’m not sure why it’s not getting resolved when run via
pants
. Anyway, I’ll continue debugging. Thanks for your help.
h

happy-kitchen-89482

11/09/2022, 11:59 PM
Ah, so you did
./pants run path/to/script.py
and it failed?
What happens if you
./pants run path/to:script_target
with the address of the pex_binary target for the script?
r

refined-gigabyte-6122

11/10/2022, 1:30 AM
I did the latter, though the target is a
python_source
, if that matters.
But I guess it implicitly builds and runs a pex file when you
run
a
python_source
?
h

happy-kitchen-89482

11/10/2022, 1:51 AM
Not exactly. Are you setting
run_goal_use_sandbox
on your
python_sources
target?
“Running a pex_binary is equivalent to package-ing the target followed by executing the built PEX from the repo root.”
But that is slower and often overkill
So running a
python_source
actually makes sense
In your case
and if
run_goal_use_sandbox=True
then I think the codegen should have worked
So that’s what I’m trying to get to the bottom of
r

refined-gigabyte-6122

11/10/2022, 2:35 PM
Hey, sorry for the delayed reply. I tried running with
run_goal_use_sandbox=True
set explicitly, and had the same issue. However, I think I may have sorted this out. The problem seems to mirror my mistake above. Namely, that python will try to resolve modules in the execution directory before looking to the PYTHONPATH. I had my project set up with the top-level python module at the root of the repository. When I simply moved that top-level module directory under a
src
directory, updated the
root_patterns
config in
pants.toml
(from
["/"]
to
["/src"]
), and the various build targets to add the
src
prefix, it worked. Is it possible that
pants
is replicating/preserving the semantics of python, looking first in the local directory even though it won’t contain generated files? I can try to put together a minimal reproducible example if that would be helpful.
h

happy-kitchen-89482

11/10/2022, 5:08 PM
Hmm, I’m glad it works now, but it’s not great that you had to refactor your code. So a minimal repro would be very helpful, thanks! It may be that it has to work this way, but I’d like to at least get to the bottom of why…
r

refined-gigabyte-6122

11/10/2022, 7:40 PM
Ok, I think this should be a minimal example to reproduce. It contains two subdirectories, one in which the command fails, one in which it succeeds. There’s a README there with more info and example commands.
🙏 1
h

happy-kitchen-89482

11/11/2022, 12:34 AM
Could I trouble you to open a bug report for this at https://github.com/pantsbuild/pants/issues/new/choose ? Slack is so ephemeral, and that will last
r

refined-gigabyte-6122

11/11/2022, 12:35 AM
Yup, will do
h

happy-kitchen-89482

11/11/2022, 1:58 AM
Thanks, post a link here when it’s up?
r

refined-gigabyte-6122

11/11/2022, 3:53 AM
f

fancy-daybreak-62348

01/19/2023, 5:39 PM
i’m running into this exact isssue now
i have the exact same structure as david and i can’t ./pants run but i can use pex target correctly
ah updating my pex to 2.1.121 (not a simple task) fixed it
5 Views