https://pantsbuild.org/ logo
Join Slack
Powered by
Hey all, I'm getting tripped up getting Pants to w...
# general
m
Hey all, I'm getting tripped up getting Pants to work with SQLAlchemy dialect dependencies, and hoping someone can help:
Copy code
# requirements.txt
sqlalchemy==1.3.22
pybigquery==0.5.0

# BUILD
python_requirements(
    name="root",
)

python_tests(
    name="tests0",
    dependencies=[":root#pybigquery"],
)

# test_bq.py

def test_engine_init():
    sqlalchemy.create_engine("<bigquery://test-project>")
I see that the pybigquery dependency was installed in the pex from the logs of running 
test
and I verified via 
pdb -> interact
that 
import pybigquery
is successful. But there seems to be something screwed up with the environment setup because creating the sql engine fails with
Copy code
...
        else:
            for impl in pkg_resources.iter_entry_points(self.group, name):
                self.impls[name] = impl.load
                return impl.load()
    
>       raise exc.NoSuchModuleError(
            "Can't load plugin: %s:%s" % (self.group, name)
        )
E       sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:bigquery
Is there something extra needed for Pants (or is pex the culprit?) to correctly register the pybigquery plugin for sqlalchemy?
w
Does that file import pybigquery? Whether or not that "needs' to happen
m
the file doesn't import pybigquery note that the test runs successfully outside of Pants (i.e. importing is not necessary)
fwiw the test still fails only in Pants after adding the import
w
Can you make a small repro repo? I don't think this is a hard solve, but it's just been a while since I've had to add plugins to tooling, and I don't recall off the top of my head
m
w
I'll take a look this eve
gratitude thank you 1
I tried creating a venv and running main.py, but it still fails with the same error
e.g. outside of pants
m
huh strange, it succeeds for me outside of pants
w
I'm using uv to venv and install reqs, so maybe that's different?
m
yup, I can repro the failure when using 
uv
, but not when I go through vanilla venv e.g.
Copy code
> python -m venv .venv
> source .venv/bin/activate
> pip install -r requirements.txt
> python main.py
w
Can we try something stupid?
I think the issue might actually be related to missing setuptools
Copy code
sqlalchemy==1.3.22
pybigquery==0.5.0
setuptools
Copy code
python_sources(
    name="lib",
    dependencies=[":root#setuptools", ":root#pybigquery"]
)
sqlalchemy does an... interesting ambient plugin import mechanism
You'll likely want to look into 
pex_binary
for later usage, if you want to ship all this together
b
Yeah, @wide-midnight-78598 has this right. FWIW, pretty easy to check with pex directly (
pexec
is the same as 
pex3 run
and is available in modern Pex and works like 
uvx
/ 
uv run
, etc.). Bad:
Copy code
:; pexec <https://github.com/kvudata/pants-sqlalchemy-repro/raw/refs/heads/main/main.py>
Traceback (most recent call last):
  File "/home/jsirois/.cache/pex/venvs/3/657b406d657a614d2d29dc85b3765eeab3a824c3/485b182d3b10432d6a5f1f5be4248d540afe6b7b/main.py", line 14, in <module>
    create_engine("<bigquery://test-project>")
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
  File "/home/jsirois/.cache/pex/venvs/3/657b406d657a614d2d29dc85b3765eeab3a824c3/485b182d3b10432d6a5f1f5be4248d540afe6b7b/lib/python3.14/site-packages/sqlalchemy/util/langhelpers.py", line 277, in load
    raise exc.NoSuchModuleError(
        "Can't load plugin: %s:%s" % (self.group, name)
    )
sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:bigquery
Good:
Copy code
:; pexec --with setuptools <https://github.com/kvudata/pants-sqlalchemy-repro/raw/refs/heads/main/main.py>
/home/jsirois/.cache/pex/venvs/3/cc8e31028b70553c2609a1977e302224591f4c91/485b182d3b10432d6a5f1f5be4248d540afe6b7b/lib/python3.14/site-packages/sqlalchemy/util/langhelpers.py:269: UserWarning: pkg_resources is deprecated as an API. See <https://setuptools.pypa.io/en/latest/pkg_resources.html>. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
Traceback (most recent call last):
  File "/home/jsirois/.cache/pex/venvs/3/cc8e31028b70553c2609a1977e302224591f4c91/485b182d3b10432d6a5f1f5be4248d540afe6b7b/main.py", line 14, in <module>
    create_engine("<bigquery://test-project>")
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
  File "/home/jsirois/.cache/pex/venvs/3/cc8e31028b70553c2609a1977e302224591f4c91/485b182d3b10432d6a5f1f5be4248d540afe6b7b/lib/python3.14/site-packages/google/auth/_default.py", line 739, in default
    raise exceptions.DefaultCredentialsError(_CLOUD_SDK_MISSING_CREDENTIALS)
google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see <https://cloud.google.com/docs/authentication/external/set-up-adc> for more information.
Same with `uv`:
Copy code
:; uv run --with setuptools <https://github.com/kvudata/pants-sqlalchemy-repro/raw/refs/heads/main/main.py>
/home/jsirois/.cache/uv/environments-v2/f43f6639882ceeb3/lib/python3.14/site-packages/sqlalchemy/util/langhelpers.py:269: UserWarning: pkg_resources is deprecated as an API. See <https://setuptools.pypa.io/en/latest/pkg_resources.html>. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
Traceback (most recent call last):
  File "/tmp/mainDXmk2e.py", line 14, in <module>
    create_engine("<bigquery://test-project>")
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
  File "/home/jsirois/.cache/uv/environments-v2/f43f6639882ceeb3/lib/python3.14/site-packages/google/auth/_default.py", line 739, in default
    raise exceptions.DefaultCredentialsError(_CLOUD_SDK_MISSING_CREDENTIALS)
google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see <https://cloud.google.com/docs/authentication/external/set-up-adc> for more information.
FWIW, this is fixed ~3 months after after your ancient version of SQLAlchemy was published. This commit to switch from 
pkg_resources
(setuptools) to 
importlib.metadata
landed in the beta series leading up to the 1.4.0 release on March 15th 2021: + commit: https://github.com/sqlalchemy/sqlalchemy/commit/cd03b8f0cecbf72ecd6c99c4d3a6338c8278b40d + relevant bit: https://github.com/sqlalchemy/sqlalchemy/commit/cd03b8f0cecbf72ecd6c99c4d3a6338c8278b40d#diff-1bee33b662869a281c26c03[…]a454d5590dfa9b8f2d4fe1fa24de Note in the relevant bit that if 
pkg_resources
wasn't present, all plugins were silently not loaded. If this was still the active codebase, you'd send them a PR to add 
setuptools
to the 
install_requires
in their 
setup.py
. Leaving out 
setuptools
as a dependency is a widespread error in packages people have released. They just get lucky because they work in a venv that includes setuptools automatically but they release a fundamentally broken package like this. Projects declaring old-school 
pkg_resources
namespace packages were frequent offenders. This case was more egregious though since they explicitly use 
pkg_resources
in their plugin loading system. Once you step outside Python syntax and extension module support, basically everything is a dumpster fire. It's good to be keenly aware of this and take up the burden of educating yourself if you're going to stick with it as a language you attempt to get things done with.
m
They just get lucky because they work in a venv that includes setuptools automatically
TIL, fun stuff. Yeah I haven't delved much into the details of how plugins and packaging works in Python
after after your ancient version of SQLAlchemy
😅 yeah, working with some real old stuff which I'm also trying to migrate over to Pants as well as push for upgrading various old deps like this
the setuptools change works! thanks all for the help here
and thanks for the insights into what's going on here
b
Excellent.
Yeah I haven't delved much into the details of how plugins and packaging works in Python
That's the main point, they don't. Plugins and packaging are external to Python and handled by the community and the community is too big for this to go well. It would have needed to be settled when the community was O(100) people or so.
🫠 1
22 Views
Open in Slack
PreviousNext
Powered by