https://pantsbuild.org/ logo
Join Slack
Powered by
I have a 3rd-party dependency that's pinning `pack...
# plugins
c
I have a 3rd-party dependency that's pinning 
packaging~=20.9
, which conflicts with pants own dependency on 
packaging
(or that's what looks like is happening). Is there a way to override the version of a 3rd-party package in the 
python_requirements
target? I basically want to tell Pants/Pex/Pip to ignore the transitive dependency of one of my packages so it will accept 
packaging>=20.9
(possible it will break, but I'm skeptical)
I'm trying to use the 
requirements
and specifying the 
packaging
version constraint that I need, but I'm not sure if it'll work
e
Although Pex supports constraints files 1st class, Pants unfortunately has a muddled history with this and so doesn't currently allow you to mix use of a constraints file with modern resolve setups. As such you have to do something like you're doing.
c
so what I'm doing will work?
Right now I get silent failures as pex/pip downloads every version of my dependency (taking hours to resolve, and eventually fail) all while giving me no debug output
e
I'm not saying it will work. Just saying it's your only option.
c
That's not re-assuring
e
What are your interpreter constraints?
c
interpreter_constraints = [">=3.8.*,<3.11"]
e
Ok, and do you mean that?
Does it have to work on all those versions?
c
I think just 3.8
but we plan to upgrade to 3.10 soon
e
If you only ever support 1 minor version though, you might try 
==3.8.*
now, and 
==3.10.*
when you upgrade.
That does effect the resolve space and can cut down on resolve times.
c
it doesn't appear to be interpreter-constraints related though
e
ut if it's still slow or chaging ICs is not acceptable, the current state of debugging a long lock download is you have to roll up your sleeves and get very dirty.
c
but yeah, it'll make the resolve faster
that's what it seems, the lack of debug logs from pex is... frustrating
e
I'm just about to head AFK for ~1 month; so you'll need to lean on others here and yourself if troubles persist.
that's what it seems, the lack of debug logs from pex is... frustrating
If you can pitch in, I'm always happy for help hacking on Pex!
w
Pants’ own requirements generally should not mix with yours… there are some cases where tools themselves need to be compatible though. what exactly is failing?
…oh, sorry. just realized that this is the #plugins room. so i should assume that this is in a plugin.
c
oh oops, meant to put this in #general
no, this is pants + lockfiles
w
ok then: question stands about what exactly is failing. because all of the tools that Pants runs can have their requirements configured
c
basically, I have a requiremnts.txt with a lockfile using the pex resolver. When I add 
sqlalchemy-oso
to the requirements.txt and re-generate the lockfile, it sits and spins for... idk how long yet. I've encountered similar situations before where it spins for 2-3 hours (with no output) and then spits out 3000 lines of "attempted downloading package==version, but could not resolve constraints"
the only conflicting package appears to be 
packaging
when I reference my requirements.lock, and it's in `pantsbuild.pants`:
Copy code
"project_name": "pantsbuild-pants",                                                                                                                                                  
          "requires_dists": [                                                                                                                                                                  
            "PyYAML<7.0,>=6.0",                                                                                                                                                                
            "ansicolors==1.1.8",                                                                                                                                                               
            "fasteners==0.16.3",                                                                                                                                                               
            "humbug==0.2.7",                                                                                                                                                                   
            "ijson==3.1.4",                                                                                                                                                                    
            "packaging==21.3",                                                                                                                                                                 
            "pex==2.1.84",                                                                                                                                                                     
            "psutil==5.9.0",                                                                                                                                                                   
            "python-lsp-jsonrpc==1.0.0",                                                                                                                                                       
            "setproctitle==1.2.2",                                                                                                                                                             
            "setuptools<58.0,>=56.0.0",                                                                                                                                                        
            "toml==0.10.2",                                                                                                                                                                    
            "types-PyYAML==6.0.3",                                                                                                                                                             
            "types-setuptools==57.4.7",                                                                                                                                                        
            "types-toml==0.10.3",                                                                                                                                                              
            "typing-extensions==4.0.1"                                                                                                                                                         
          ],                                                                                                                                                                                   
          "requires_python": null,                                                                                                                                                             
          "version": "2.11.1rc0"                                                                                                                                                               
        },
w
which version of Pants is this?
c
2.11
I tried bumping to 1rc0 to see if it helped, but it didn't seem to change anything
w
and you are using 
lockfile_generator = pex
?
c
Yes
w
ok: the next step would be to file a bug report against github.com/pantsbuild/pex with as much of the relevant CLI args for PEX as you are able to share. to get them, re-run the slow command with 
./pants -ldebug …
, and then look for the last/long-running line matching:
Copy code
spawned local process as
…which will include the full arguments to PEX
c
There's no debug info at all when running 
./pants -ldebug generate-lockfiles
. The output is identical whether you specify 
-ldebug
or not
w
do you not see any 
spawned local process as
lines? if not, try 
./pants --no-pantsd -ldebug
c
No. I'll add the 
—no-pantsd
when I get back to my desk
w
to be clear: i’m not asking for the output from pex: just from Pants as it spawns pex. i’m certain that line is rendered, it’s more a question of whether the 
-ldebug
flag is taking effect (if not, it would be a bug)
c
image.png
definitely think I've found a bug
w
did you try with 
--no-pantsd
…?
c
found a pip.log:
Copy code
2022-06-01T09:29:45,657 The conflict is caused by:
2022-06-01T09:29:45,657     sqlalchemy-oso 0.26.0 depends on packaging~=20.9
2022-06-01T09:29:45,657     pantsbuild-pants 2.11.1rc2 depends on packaging==21.3
2022-06-01T09:29:45,657     sqlalchemy-oso 0.26.0 depends on packaging~=20.9
2022-06-01T09:29:45,657     pantsbuild-pants 2.11.1rc1 depends on packaging==21.3
2022-06-01T09:29:45,657     sqlalchemy-oso 0.26.0 depends on packaging~=20.9
2022-06-01T09:29:45,657     pantsbuild-pants 2.11.1rc0 depends on packaging==21.3
2022-06-01T09:29:45,657     sqlalchemy-oso 0.26.0 depends on packaging~=20.9
2022-06-01T09:29:45,657     pantsbuild-pants 2.11.0 depends on packaging==21.3
2022-06-01T09:29:45,657     sqlalchemy-oso 0.26.0 depends on packaging~=20.9
2022-06-01T09:29:45,657     pantsbuild-pants 2.11.0rc6 depends on packaging==21.3
2022-06-01T09:29:45,657     sqlalchemy-oso 0.26.0 depends on packaging~=20.9
2022-06-01T09:29:45,657     pantsbuild-pants 2.11.0rc5 depends on packaging==21.3
2022-06-01T09:29:45,657     sqlalchemy-oso 0.26.0 depends on packaging~=20.9
2022-06-01T09:29:45,657     pantsbuild-pants 2.11.0rc4 depends on packaging==21.3
2022-06-01T09:29:45,657     sqlalchemy-oso 0.26.0 depends on packaging~=20.9
2022-06-01T09:29:45,657     pantsbuild-pants 2.11.0rc3 depends on packaging==21.3
2022-06-01T09:29:45,657     sqlalchemy-oso 0.26.0 depends on packaging~=20.9
2022-06-01T09:29:45,657     pantsbuild-pants 2.11.0rc2 depends on packaging==21.3
2022-06-01T09:29:45,657     sqlalchemy-oso 0.26.0 depends on packaging~=20.9
2022-06-01T09:29:45,657     pantsbuild-pants 2.11.0rc1 depends on packaging==21.3
2022-06-01T09:29:45,657     sqlalchemy-oso 0.26.0 depends on packaging~=20.9
2022-06-01T09:29:45,657     pantsbuild-pants 2.11.0rc0 depends on packaging==21.3
2022-06-01T09:29:45,657 
2022-06-01T09:29:45,657 To fix this you could try to:
2022-06-01T09:29:45,657 1. loosen the range of package versions you've specified
2022-06-01T09:29:45,657 2. remove package versions to allow pip attempt to solve the dependency conflict

2022-06-01T09:29:45,657 ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies>
2022-06-01T09:29:45,658 Exception information:
2022-06-01T09:29:45,658 Traceback (most recent call last):
2022-06-01T09:29:45,658   File "/home/noah/.cache/pants/named_caches/pex_root/venvs/s/93b232a7/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 121, in>
2022-06-01T09:29:45,658     self._result = resolver.resolve(
2022-06-01T09:29:45,658   File "/home/noah/.cache/pants/named_caches/pex_root/venvs/s/93b232a7/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 453, in resolve
2022-06-01T09:29:45,658     state = resolution.resolve(requirements, max_rounds=max_rounds)
2022-06-01T09:29:45,658   File "/home/noah/.cache/pants/named_caches/pex_root/venvs/s/93b232a7/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 357, in resolve
2022-06-01T09:29:45,658     raise ResolutionImpossible(causes)
2022-06-01T09:29:45,658 pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('packaging~=20.9'), parent=LinkCandidate('<https://file>>
2022-06-01T09:29:45,658 
2022-06-01T09:29:45,658 The above exception was the direct cause of the following exception:
w
c
image.png
w
… there was no log output at all…? that’s very surprising. you should see hundreds of lines of output first.
c
there's none
w
what happens if you use 
-ldebug --no-dynamic-ui --no-pantsd
?
c
same thing
w
the dynamic UI (“spinners”) still renders?
c
yup
it doesn't seem to be respecting any flags
w
that’s very, very broken… have you modified the 
pants
script somehow?
c
not that I know of?
w
can you kill any 
pantsd
processes and try again?
or, actually: “are there 
pantsd
processes?”
c
oh hmm, I ran 
curl -L -O <https://static.pantsbuild.org/setup/pants> && chmod +x ./pants
and retried and that seemed to fix it. It may have been from the 2.2 version?
idk
so that fixed the debug output, but I think I'm still getting dep conflicts. I'm re-running the 
generate-lockfiles
and I found the 
pip.log
and am tailing it
w
i still need the line i mentioned earlier
as much of the 
spawned local process as
line from the hanging process as you’re able to share on a PEX ticket
c
<redacted>
I redacted some private registry stuff
fwiw I don't think it's hanging, normally resolving our dependency tree takes 500s~ because it's huge. But when there's a conflict, pip downloads other versions of packages to attempt to resolve ones that match the graph. In this case, I think it's trying to downgrade 
pantsbuild.pants
, though I'll have exact log messages soon
w
are the private repositories required to repro do you think? do any of the artifacts only exist in private?
but even so: sometimes the backtracking cases are bugs, since PEX is choosing what to ask pip for, so worth filing.
i’ll see if i can repro
c
uhh, there's one that's redacted, but it has no deps iirc. The exact conflicting package is 
sqlalchemy-oso
and 
pantsbuild.pants
common dep on differing versions of 
packaging
w
sounds good, thanks!
c
np, thank you for the help. Fwiw everything resolves in 220s~ if I comment out 
sqlalchemy-oso
, so I really just need a way to override/ignore that packages dependencies
or remove pants own deps from resolving in this lockfile
w
@calm-ambulance-65371: why is pants in this resolve/lockfile? for a plugin?
c
probably because we have some in-repo plugins
I guess?
it's not explicitly in the requirements.txt
w
i would put your plugin in a separate resolve
you don’t always need to, but particularly since you’re seeing conflicts, it would be worth it.
c
I'll try that, should take me a few min
considering 
pants_requirements()
is in the module for our plugins, I'm guessing that's where the dep on 
pantsbuild.pants
is coming from
w
correct
and i repro the very long run even on macOS, so i can file that
c
yeah, moving those to a different resolve fixed it
w
great: thanks. i’ll add a recommendation to the plugin docs for that.
opened https://github.com/pantsbuild/pex/issues/1798 for the long resolve, but indicated that you’re unblocked
👍 1
Open in Slack
PreviousNext
Powered by