https://pantsbuild.org/ logo
#general
Title
# general
m

melodic-carpenter-39613

02/08/2023, 9:18 AM
Hi all, I've been having a strange issue with locking my python requirements for the last few days. Essentially I am encountering
ResolutionImpossible
when running
./pants generate-lock-files
due to multiple pinned versions of bcrypt (3.2.2 vs 4.0.1), but only have a single pinned version (3.2.2) in my top-level requirements. How would I use pants to debug the constraints for each package similar to
pipdeptree
so that I can work out where the issue is coming from?
r

refined-addition-53644

02/08/2023, 9:22 AM
Hi, have you tried already some of options like
--ldebug
or
--keep-sandboxes=always
mentioned here https://www.pantsbuild.org/docs/troubleshooting
m

melodic-carpenter-39613

02/08/2023, 10:51 AM
Thanks for the suggestion @refined-addition-53644 - Yep I tried that (exert of output attached) . I can then go into the sandbox and inspect the
pex
and
__run.sh
and could in theory invoke the PEX separately and retry. Interestingly there are no files within the
.tmp
, so for instance I cannot view
/tmp/pants-sandbox-0k4B7q/.tmp/pex-pip-log.mzg5u2ix/pip.log
. How would I go about trying to find the pants dependency or sub-dependency that is causing the issue from here?
r

refined-addition-53644

02/08/2023, 12:21 PM
I don’t really have an answer but how would you do it without using pants? I feel like this is coming from pip itself. I read a bit about pipdeptree and it looks like it figures out such issues when you already have these packages installed in some global/virtual env.
m

melodic-carpenter-39613

02/08/2023, 12:43 PM
My first dumb port of call would be to increase the verbosity of
pip install
with the environment variable
PIP_VERBOSE=3
and inspect the output. This is useful for instance when pip is taking a long time to resolve requirements as I can identify situations where pip is attempting to download lots of different versions of packages and testing to see if they satisfy all the package version constraints (in the dependency tree). With pants it is a black box
-ldebug
and/or setting the environment variable
PEX_VERBOSITY=9
doesn't give me any more insight other than knowing that pants is attempting to generate a specific lockfile and the time taken so far to do so - is there a way to get more detail here that I've missed in the pants docs? On the flip side when pip installs everything (which could theoretically include version conflicts which it then warns about), I would see what the dependency tree is for a given virtual environment using deptree and/or pipdeptree which together I can work out where potential version conflicts are coming from. Any help on how to do this sort of debugging using pants would be most appreciated πŸ™‚ πŸ™
Note I've tried using
./pants dependencies --transitive ::
but this gives me a list of all dependencies, but does not attribute each to a particular pants resolve and/or give any version constraints (in the case of
python_requirements
targets.
I guess I have a few questions regarding pants locking in general: 1. Can I ignore dependency conflicts and/or figure out which pants targets are causing the conflict? If yes to the latter, then how do people do it within the pants community? 2. Why do I not get a pip log file within the retained sandbox? 3. Should pants be able to cache lock files? For instance running
./pants generate-lockfiles
was previously taking me 5+ minutes for a relatively modest set of top-level dependencies and re-running the command appears to then take roughly the same amount of time, which feels like I am not benefiting from using pants vs using vanilla pip and is annoying as alternatives such as pipenv (via Pipfile) allow more control over when to lock. Is there a way to make pants smarter here, particularly if no source files have been changed? 4. How do I increase the verbosity of the pants wrapped pip process so that I can identify what is happening during the lock process (e.g. similar to
pipenv lock --verbose
) rather than wait for a failure to spit out an error?
r

refined-addition-53644

02/08/2023, 1:04 PM
This is all bit beyond my expertise now. I would wait for input from @enough-analyst-54434
e

enough-analyst-54434

02/08/2023, 1:11 PM
@melodic-carpenter-39613 Pex has a
--preserve-pip-download-log
but you won't need that in this case. Your output snippet above shows
... bcrypt==3.2.2; python_version >= "3.6" bcrypt==4.0.1; python_version >= "3.6" ...
in the requirements list Pants invokes Pex with; so the error message is dead on. These 2 conflicting requirements are coming directly from top level user requirements in your repo.
Can you grep your codebase to see where these 2 conflicting requirements are defined? Is it just a typo that the environment markers are identical instead of mutually exclusive?
m

melodic-carpenter-39613

02/08/2023, 1:21 PM
Thanks @refined-addition-53644 for your help πŸ™‚ Hi @enough-analyst-54434, I've tried that and have only one version defined anywhere (
bcrypt==3.2.2
in my
requirements.txt
which feeds into
python-default
resolve). I've came to the conclusion that it is not a top-level dependency that I have specified that is requesting
bcrypt==4.0.1
and causing the conflict and is instead some sub-dependency hence why I am interested in getting more verbose output of the pants locking process.
e

enough-analyst-54434

02/08/2023, 1:22 PM
It is definitely coming from your repo, Pants does not do transitive 3rd party dependency analysis, it uses tools like Pex for that (Coursier for JVM, etc).
Do you use a constraints file?
Do you integrate Poetry?
m

melodic-carpenter-39613

02/08/2023, 1:29 PM
Okay good to know that it is an issue within my repo. My requirements file has pinned versions of all the packages feeding into
python-default
, however I do have a
python_requirements
target with a different resolve in a
BUILD
file in the same directory as a pipenv
Pipfile
, however this is not currently causing me issues. How would I filter all targets in
pants dependencies --transitive
to list only
python_requirements
targets, ideally matching
python-default
?
(Oh there is no poetry or
pyproject.toml
dependencies in my repo currently)
e

enough-analyst-54434

02/08/2023, 1:34 PM
Ok great, before looking at your most recent question, can you confirm that means you now see where the 2 bcrypts are defined in your repo? It sounds like you see the 1 in a requirements.txt and the other in a Pipfile and each is associated with a different resolve?
As an aside, the marker
python_version >= "3.6"
is a potential red flag unless your repo is truly consumed by Python 3.6 and greater. If not, the marker can likely be narrowed and, likewise `interpreter_constraints`narrowed - this all helps to shrink the lock solve space and speed up locks.
m

melodic-carpenter-39613

02/08/2023, 1:57 PM
Nope sorry I didn't explain myself very well, when grepping I only ever found one bcrypt entry in the code base. I only mentioned the other resolve with pipenv
Pipfile
just in case it was relevant, but my main problem is with the default lockfile. Due to the fact I could only find one entry for
bcrypt
in the source code I suspected that the conflict is being pulled in as a subdependency of another one of the top-level python requirements.
The marker
python_version >= "3.6"
is valid as we are using interpreter constaints of
interpreter_constraints = [">=3.7.13,<=3.8.0"]
in
pants.toml
, but I guess this marker was likely added before using pants to manage dependencies. Does it harm to keep it in?
e

enough-analyst-54434

02/08/2023, 2:02 PM
No harm, but its super odd since you don't use 3.6 at all!
m

melodic-carpenter-39613

02/08/2023, 2:04 PM
Ha ha true - I think it must've been put in a long time ago when using python 3.6, but then didn't get updated for python 3.7 (which itself is now incredibly long in the tooth πŸ˜†)
e

enough-analyst-54434

02/08/2023, 2:04 PM
Ok, on the grep, I deny your claim. The 2 bcrypt deps must be coming from your repo's source somewhere. Pants has no smarts to divine transitive 3rdparty deps, only Pex does. Its Pants up in your output calling Pex and passing Pex 2 bcrypt reqs.
So, do you have a successful lock from prior and this is updating a lock where the failure happens? Or is this trying to generate 1st lock ever for the resolve?
m

melodic-carpenter-39613

02/08/2023, 2:09 PM
Gotcha - I'll rerun and see what I get πŸ‘€ The lock did previously work, however a number of changes were made recently, including merging in another repo into the monorepo and I was unsure how to effectively debug the root cause / which pants target(s) were to blame. Before the lock failed it was also taking MUCH longer to lock than previously (>5 mins vs 1.5 mins) too and was wondering if there was a way to retain verbose logging from pip.
e

enough-analyst-54434

02/08/2023, 2:10 PM
Can you provide the full content of your prior lock? ALso, what files is the grep -v in your output leaving out? Just Pants generated .lock files or others too?
m

melodic-carpenter-39613

02/08/2023, 2:40 PM
Sure thing and good shout - of interest is the previously mentioned
Pipfile.lock
. I've included a screengrab of the
BUILD
file in case you can provide some further insight into the behaviour I'll rerun the
./pants --keep-sandboxes=always -ldebug generate-lockfiles --resolve=python-default
now
e

enough-analyst-54434

02/08/2023, 2:44 PM
Ok, so pipenv_requirements reads Pipfile.lock and that is definitely the source of the 2nd bcrypt requirement - I think we agree on that? If so, that means the resolve="radar-routines"
pipenv_requirements
are getting used along with the other requirements where the other bcrypt version is defined to create the lock and it's that that we need to debug.
m

melodic-carpenter-39613

02/08/2023, 2:45 PM
Ah understood, is it ever possible to use multiple resolves for a target without using the
parametrize()
function?
e

enough-analyst-54434

02/08/2023, 2:46 PM
Whoa - you just skipped and jumped and I plod.
So do you agree the 2 reqs are defined in your repo and the issue is mixing of the 2 requirement sets? I think that has to be what's going on here.
IOW, maybe reveal the other requirement set BUILD?
And the corresponding pants.toml config sections?
m

melodic-carpenter-39613

02/08/2023, 2:52 PM
I agree on the 2 reqs as it makes sense, especially as there are quite a few targets across multiple directories (pexes mostly, but maybe one python lib) that were recently merged into the mono-repo and therefore some might be using default lock without meaning to.
e

enough-analyst-54434

02/08/2023, 3:00 PM
Ok, good.
Can you provide those interesting snippets? The other requirements / BUILD and pants.toml bits? Its a bit tough to remote debug flying blind and that data would help.
m

melodic-carpenter-39613

02/08/2023, 3:13 PM
Here is my
pants.toml
with some possibly sensitive stuff removed.
And here is the
BUILD
file that feeds as much of the monorepo as possible under the
python-default
resolve.
e

enough-analyst-54434

02/08/2023, 3:24 PM
Alrighty then, assuming the
3rdparty/python/BUILD
requirements targets do default to a resolve of
python-default
- which the docs claim they do, then those targets and your
src/python/apps/radar_routines/BUILD
`pipenv_requirements`target do name different resolves. Clearly? those 2 different resolve targets are getting mixed into the process that creates the
python-default
lock and that's why you see 2 bcrypt requirements being passed to Pex in the lock failure snippet you posted. That sounds buggy if so, but I'm also pretty "multiple-resolves at the Pants level" ignorant - I did all the Pex support work to implement locking, but nothing at the Pants layer.
@melodic-carpenter-39613 you may be best-served by posting a new thread or using "Also send to #general" to ask for help with Pants apparently mixing requirements from multiple resolves into the lock creation process for 1 resolve. As I said, that sounds buggy, but it may be by design and you have to ensure your target roots (
pex_binary
targets, test targets, `python_distribution`targets) never mix resolves and this may be towards your
paramterize
question.
@melodic-carpenter-39613 the only shot in the dark I can think of is giving the
3rdparty/python/BUILD
targets an explicit
resolve="python-default"
just to rule out defaulting weirdness. I'll truly bow out at this point.
m

melodic-carpenter-39613

02/09/2023, 7:07 AM
I tried a different resolve, but no joy. I'll continue to play around with pants and post again if I can convince myself it is definitely a bug. In any case, thanks @enough-analyst-54434 for all of your help πŸ™‚ - it was really appreciated as I didn't know where to begin to start unravelling the possible threads and am happier that I understand a little more about how pants works.
6 Views