https://pantsbuild.org/ logo
Join Slack
Powered by
I have a 3rd party dependency issue that I can’t q...
# general
l
I have a 3rd party dependency issue that I can’t quite work out how to resolve from the docs. I’m trying to add 
authzed
to a project and can’t quite get pants to properly resolve its dependencies: When doing a 
./pants run src/python/my_project/script.py
which imports from 
authzed
, I get an error with the protobuf module:
Copy code
from google.protobuf.internal import builder as _builder
ModuleNotFoundError: No module named 'google.protobuf'
Now, I can see in my lockfile that 
authzed
properly declares this dep, so I’m not sure why pants isn’t including it. The module name/import mapping are in the defaults here, and my lockfile contains the following:
Copy code
"project_name": "authzed",
          "requires_dists": [
            "async_generator<2.0,>=1.10",
            "google-api-core<3.0.0,>=2.4.0",
            "google_api<0.2.0,>=0.1.12",
            "grpcio==1.50.0",
            "mock<5.0.0,>=4.0.3",
            "protobuf==3.20.2",
            "protoc-gen-validate<0.5.0,>=0.4.1",
            "typing-extensions<5,>=3.7.4"
          ],
          "requires_python": "<4.0,>=3.7",
          "version": "0.7"
This is purely a transitive dependency, do I have to explicitly specify it via a 
python_requirement
target or something?
I’ve also tried specifying the same version of protobuf in my 
requirements.txt
, then updating my lockfile and adding that requirement to the 
python_sources
for this directory. Now running 
./pants dependencies src/python/project/script.py
shows the following:
Copy code
data/agent_permissions.csv:auth_test_data
data/agent_team_association.csv:auth_test_data
data/openfga_schema.json:auth_test_schemas
data/team_permissions.csv:auth_test_data
src/python/requirements.txt:reqs
src/python:reqs#authzed
src/python:reqs#protobuf
But I’m still getting the same module not found at runtime. If I export a virtualenv and run a shell in it I can import it, so it’s definitely there and pants is missing that it should be included in the sandbox under 
run
.
Here’s the 
python_sources
target now:
Copy code
python_sources(
    dependencies=[
        "data:auth_test_data", 
        "data:auth_test_schemas",
        "src/python:reqs#protobuf",
    ]
)
Okay, digging further. I can see that in the venv that pants is running under at 
~/.cache/pants/named_caches/pex_root/venvs/3853055278bac9c525f53437e9c479bdb59c892b/b1863b9ec7674b21b46621ab3494feb14780fc43/lib/python3.10/site-packages
the 
google
folder that should have my protobuf import is actually a symlink to 
google -> ../../../../../../installed_wheels/ce222e27b0de0d7bc63eb043b956996d6dccab14cc3b690aaea91c9cc99dc16e/google_api_core-2.11.0-py3-none-any.whl/google
which only has the core_api as a subdir, and not protobuf or a bunch of other things I’d expect. This is in contrast to the exported venv which has a whole bunch of google packages in that directory.
h
You shouldn't have to do anything to get transitive third-party deps pulled in. So your very first attempt should have worked (you shouldn't need to put it in your requirements.txt etc. and indeed that didn't help). Thanks to your digging, this sounds like an improperly handled (by google) namespace package.
Can you create a small repo that reproduces the problem, and post a github link here? Then we can take a look easily. Thanks!
e
I can save you the trouble here, the issue is not using a 
pex_binary(execution_mode="venv", venv_site_packages_copies=True, ...)
@late-keyboard-89314 one relevant bit to your current issue is if you have 
[pex] venv_use_symlinks = true
configured. It looks like you do from your OP, but this would be good to confirm or deny.
l
I don’t believe so, unless that’s the default? Here’s the entire contents of my `pants.toml`:
Copy code
[GLOBAL]
pants_version = "2.14.0"
backend_packages = [
  "pants.backend.python",
  "pants.backend.python.lint.black",
  "pants.backend.python.lint.isort",
]

[anonymous-telemetry]
enabled = false

[python]
enable_resolves = true
interpreter_constraints = ["CPython>=3.10,<3.11"]

[repl]
shell = "ipython"

[python.resolves]
python-default = "src/python/requirements.lock"

[subprocess-environment]
env_vars.add = [
  "GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1",
  "GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1"
]
e
Yeah, default is false. It's a space optimization that doesn't work for all distributions. If you can try the explicit 
pex_binary
target, that should definitely not be using symlinks. The longer story is my guess is your self diagnosis is not correct. The symlink venvs use multiple site-packages directories nested in site-packages for namespace package distributions. It this trick which does not work with some distributions, and IIRC google stuff is one of those.
l
I’ll try creating an explicit pex_binary target when I get back to my desk; thanks!
e
@late-keyboard-89314 it sounds like you're not currently suffering from this, but, as an aside, I notice you're suffering from this bug: https://github.com/pantsbuild/pants/issues/17978 The default repo-wide IC you have configured is not being used to generate your 
python-default
lock (check the lock file contents to confirm), you need to repeat yourself and set the IC again here currently: https://www.pantsbuild.org/docs/reference-python#resolves_to_interpreter_constraints
11 Views
Open in Slack
PreviousNext
Powered by