I'm having a lot of recurring issues with 3rd part...
# general
r
I'm having a lot of recurring issues with 3rd party dependencies. So before proceeding with a question for each issue, I want to start by clarifying what the "proper" way to use
python_requirement
and
python_requirements
is, in case there's some misuse going on that's messing things up.
How Things are Currently Setup We have various Python projects in a monorepo. Each one has its own requirements file with inexact specifiers. We also have project groupings that each define a virtual environment covering a bundle of projects. These groupings have requirements files with exact specifiers. How I'm Migrating to Pants For now I'm working on a single project grouping and the transitive deps of a single project. This corresponds to a single lockfile. I'm taking all projects and tailoring a
python_requirements
for each requirements file. I'm setting the resolve of all these to the same resolve, and I've done work to ensure the dependency specifiers are all compatible.
Related Problems I'm Running Into 1. 3rd party deps are usually not being picked up even when they're present in the lockfile. The workaround that gets most tests to pass is to manually include
//project/n:reqs
into the
dependencies
of every Python target in
//project/n/**
via
__defaults__
. I still get warnings and apparently this bloats all the Pex files, but things usually run successfully. 2. I constantly get warnings of the form "The target project/whatever/file.py imports
six.iteritems
, but Pants cannot safely infer a dependency because more than one target owns this module, so it is ambiguous which to use: ['project/1:reqs#six', 'project/1:reqs#types-six', 'project/1:reqs1#six', 'project1:reqs1#types-six', 'project2:reqs#six', 'project_group:reqs#six', 'project_group:reqs#types-six']". I don't understand how to "disambiguate" because (1) I'm not even sure how requirements are "bound" to individual Python targets beyond
resolve
and (2) I'm already manually trying to disambiguate by explicitly adding
project/1:reqs
to
dependencies
everywhere. It seems like Pants is upset that I have multiple
python_requirements
bound to the same
resolve
, but I'm not sure how else you'd expect
python_requirements
to work. 3. I often get warnings that core modules like pyyaml can't be resolved. AFAIK these are false positives but not sure why this is happening.
Main Clarifying Questions 1. Is it wrong to have multiple requirement files using the same resolve and with overlapping deps? If it's not always wrong, when is it considered "ambiguous"? Why does it even matter when the lockfile ultimately decides what gets used? 2. How do I diagnose these kinds of dependency resolution errors? I might have seen similar questions on Slack so I'll read back a bit first. I'm happy to take this offline if that would be more appropriate.
h
Hey Navneeth! Happy to clarify this. Would you mind posting all this as a new thread on GitHub Discussions, and we can take the convo there? https://github.com/pantsbuild/pants/discussions
👀 1
I suggest this because Slack is so ephemeral, and you’re asking some really good questions, that would be great to have persisted (and searchable) for the future
👍 1
h
And could you please post the link to the discussion here? I’d like to follow it and learn, I might be having similar problems 🙂
h