Transitive dependency enthusiasts - in our current pip setup, we have a Flask 2 app that depends on dbt-core 1.0.3. They have conflicting dependencies on Jinja2 (3 for the former, 2 for the latter), which we skate around by forcing usage of pip 20.x. I'm looking at the 2.10 docs that were mentioned yesterday wrt dependency conflicts, but I'm thinking this scenario may be different - i.e. I don't have 2 separate apps in the same monorepo with conflicting dependencies, but rather it's 1 app that has conflicting dependencies. Does Pants offer a way around this?

Pants uses Pex which vendors Pip 20.3.4. That version of Pip offers 2 resolve modes: legacy and 2020. The legacy mode is what you must be using and what Pex still defaults to. Pants used to have this option plumbed but removed it starting here: https://github.com/pantsbuild/pants/commit/6f0ba9180e4d4c804e71ca44adc9a69d0753052e I can't recall why. Generally we try to let users interact with the tools they use through Pants as they would directly if at all possible. @hundreds-father-404 may remember the motivation. That said, your situation is definitely untenable! Do you happen to know that the version conflict is fake? I.E.: Have you checked that you can never possibly execute code paths that would blow up due to API differences?

Am I misunderstanding the problem, or doesn't the lockfile solve this? you'd have to pick some version of jinja (3.x I guess) but pip 20.x was doing that too.

pip 20 prints out several conflicts; our tests have not uncovered any issues so far. Our current state is that we use Flask 1.1.4 and Celery 5.2.3, and those have a Click 7/8 conflict. So I tried bumping Flask 1.x to 2.x, which then resulted in a Jinja2 conflict with dbt-core. I'm still new to the Python ecosystem, but these seem like popular packages that a lot of users would run into the same conflicts with.

Aha, right. I hadn't considered hacking up a resolved lock file by hand outside Pants even though you effectively spelled this out / that we use --no-deps for resolves against lock files.

No luck so far - I have this in my pants.toml file:

[python]
requirement_constraints = "constraints.txt"

I still get an error - e.g.: The conflict is caused by: The user requested jinja2==3.0.3 dbt-core 1.0.3 depends on Jinja2==2.11.3 I from the logging that ./pex is being used, but I don't see "--no-deps" in the list of arg; should it be there since I have requirement_constraints set in the "python" section?

Sorry about this - the current state of the world surrounding constraints vs locks is confusing. In short

requirement_constraints

is like a lock in that versions can be all pinned transitively, but does a full resolve anyhow. To get a real lockfile behavior where there is no resolve and the stated pinned requirements are used as-is flat you need 2.10.x: https://pypi.org/project/pantsbuild.pants/2.10.0rc1/ and https://www.pantsbuild.org/v2.10/docs/reference-python#section-enable-resolves This is alpha and the latter docs call out known issues.

Thanks @enough-analyst-54434 - I'm making some progress, but wanted to confirm first that if I try to generate a lockfile for just my conflicting versions of flask/dbt-core, I should still get a SolverProblemError, correct? My understanding is I need to hack together a lockfile by generating multiple lockfiles - without conflicts - and then manually stitch those together to get around the SolverProblemError. That effectively gets me to where my team is with pip 20 and the legacy-resolver - i.e. we're living with a conflicting dependency and relying on tests to ensure it doesn't bite us

Yup, exactly. You need a shoddy resolver. Pip old does this automatically for you, otherwise you resolve by hand - i.e: stitch locks together.

"shoddy" seems about right! I'll report back later, but not sure stitching locks together is going to work - i.e. the dbt-core entry is looking for a specific fingerprint for Jinja 2.x, and I've got the Jinja 3.x entry in there now, and the resolver (rightfully) is complaining . I'll keep hacking away.

But I will circle back to the fact you really don't want to be in this situation if you don't have to. Better to fork off longer term payoff PRs to the underlying projects with bad deps to fix those or else if they aren't bad but just super old, upgrade. But likely you have a constraint that you must stick to these (old)? versions I'm guessing.

Well, we're using both flask and dbt-core, and "pex flask dbt-core" blows up due to the Jinja conflict. In theory, a dbt-core 1.1 release could upgrade to Jinja 3 instead of 2. Makes me wonder what other people looking to use flask/dbt-core in the same app are doing.

I think the implication is there aren't many people doing this! You have to be determined like you and really want it!

Thanks John - I was going to fiddle with building dbt-core with Jinja 3 this morning to see where that goes, as I don't have any knowledge yet of how changes from Jinja 2 to 3 will affect dbt-core