Hello. I’m having difficulties working with the Python protobuf integration, and I’m wondering if anyone can suggest any debugging strategies. If I run the

export-codegen

goal, I see the generated Python files where I expect them under

dist/codegen

. But when `run`ing a script that depends on these generated files, I get an import error. I’ve included

__init__.py

files in every directory. Is there a way to manually inspect the directories that Pants links at runtime, especially the directory where it stores generated code?

Sorry for the trouble! A good way to debug such things is to run with

--keep-sandboxes=on_failure

(or

--keep-sandboxes=always

), which will preserve the sandbox dirs that Pants runs processes in, and log their locations.

Thanks! That’s exactly what I was looking for. I must be doing something stupid, because even this is failing, which I would not expect given the module structure:

$ tree <sandbox dir>
<sandbox dir>
├── local_dists.pex
...
├── project
│   ├── __init__.py
│   ├── proto
│   │   ├── __init__.py
│   │   ├── generated_pb2.py
...
└── script.pex
...

$ PYTHONPATH=<sandbox dir> python3 -c "from project.proto import generated_pb2"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name 'generated_pb2' from 'project.proto'

hmmmmmmm

I don’t see a

__run.sh

file. This is the log I’m using to get the <sandbox dir> above:

16:16:24.99 [INFO] Preserving local process execution dir /private/var/folders/tn/0sw9hrgd0zl_py1s8gtmfznr0000gp/T/pants-sandbox-N1XHGI for interactive process

Ah, so you did

./pants run path/to/script.py

and it failed?

I did the latter, though the target is a

python_source

, if that matters.

Not exactly. Are you setting

run_goal_use_sandbox

on your

python_sources

target?

Hey, sorry for the delayed reply. I tried running with

run_goal_use_sandbox=True

set explicitly, and had the same issue. However, I think I may have sorted this out. The problem seems to mirror my mistake above. Namely, that python will try to resolve modules in the execution directory before looking to the PYTHONPATH. I had my project set up with the top-level python module at the root of the repository. When I simply moved that top-level module directory under a

src

directory, updated the

root_patterns

config in

pants.toml

(from

["/"]

to

["/src"]

), and the various build targets to add the

src

prefix, it worked. Is it possible that

pants

is replicating/preserving the semantics of python, looking first in the local directory even though it won’t contain generated files? I can try to put together a minimal reproducible example if that would be helpful.

Hmm, I’m glad it works now, but it’s not great that you had to refactor your code. So a minimal repro would be very helpful, thanks! It may be that it has to work this way, but I’d like to at least get to the bottom of why…

Ok, I think this should be a minimal example to reproduce. It contains two subdirectories, one in which the command fails, one in which it succeeds. There’s a README there with more info and example commands.

Could I trouble you to open a bug report for this at https://github.com/pantsbuild/pants/issues/new/choose ? Slack is so ephemeral, and that will last

Yup, will do

Thanks, post a link here when it’s up?

i’m running into this exact isssue now