https://pantsbuild.org/ logo
Join Slack
Powered by
Anyone had issue with missing dependency running m...
# general
c
Anyone had issue with missing dependency running mypy? In my project if I try to check a file with a specific set of `import`/dependencies, the venv pex will be missing sqlalchemy stubs. For example, if I check this file:
Copy code
import sendgrid
import remote_pdb
import sqlalchemy
I get a mypy error about 
sqlalchemy
missing. However, if I check this file:
Copy code
import sendgrid
import sqlalchemy
Then 
sqlalchemy
stubs are present. I did inspect the sandbox, and it seems that the python executable PEX venv is indeed missing the sqlalchemy2-stubs from the failing sandbox. (This is not the mypy cache poisoning bug)
h
Weird. I would not expect the presence of 
remote_pdb
to cause the stubs to go missing. Is this consistent, or is it more of a heisenbug?
A small github repo that reproduces the problem would be really valuable for debugging this
c
The issue seems to come from PEX
I ran the 
requirement.pex
with 
PEX_VERBOSE=10
and I can see
pex: Selected SQLAlchemy 1.4.52 via sqlalchemy==1.4.52; python_full_version >= "3.11.3" and python_full_version < "3.12.0" and discarded SQLAlchemy 1.4.52 via sqlalchemy[asyncio,mypy]==1.4.52; python_full_version >= "3.11.3" and python_full_version < "3.12.0", SQLAlchemy 1.4.52 via sqlalchemy[mypy]==1.4.52; python_full_version >= "3.11.3" and python_full_version < "3.12.0".
we do have the same dependency defined with different extra in our lockfile
I feel that depending on the dependencies set, the wrong sqlalchemy get discarded
it's annoying tho because we generate a requirements.txt from poetry, so if different project have different extra, this bug is going to happen
c
Is the theory that in some cases the dependency graph lands on the one without the 
mypy
extra?
c
I would bet that there is no smart resolution, just keep the first entry in some list and discard the rest
not sure who's responsability it would be to merge the extras 😕
but right now, all dependency are there until the requirements.pex -> requirement_venv.pex
c
You might be able to duct tape this particular case with something like:
Copy code
overrides={
        "sqlalchemy": {
            "dependencies": [ "//3rdparty/py#sqlalchemy2-stubs"]
        },
c
it's brittle tho, I do have other dependency that are redefined with other extra (since the 
poetry export
does all transitive package)
I might write a script the post-process the exported requirements.txt
for example, I do have
Copy code
google-api-core==2.18.0 ; python_full_version >= "3.11.3" and python_full_version < "3.12.0"
google-api-core[grpc]==2.18.0 ; python_full_version >= "3.11.3" and python_full_version < "3.12.0"
so from my understanding, the pex is rolling dice to know if 
grpc
extra will be installed or not
I guess that it makes sense fixing it at the source (requirements.txt), merging extra doesn't make sense as soon as constraint are different
had to fight some cache (
rm -rf ~/.pex ~/.cache/pants
), but seems that removing duplicates in my 
python_requirements
fixed it!
3 Views
Open in Slack
PreviousNext
Powered by