https://pantsbuild.org/ logo
#general
Title
# general
l

lively-exabyte-12840

02/23/2022, 1:45 PM
Transitive dependency enthusiasts - in our current pip setup, we have a Flask 2 app that depends on dbt-core 1.0.3. They have conflicting dependencies on Jinja2 (3 for the former, 2 for the latter), which we skate around by forcing usage of pip 20.x. I'm looking at the 2.10 docs that were mentioned yesterday wrt dependency conflicts, but I'm thinking this scenario may be different - i.e. I don't have 2 separate apps in the same monorepo with conflicting dependencies, but rather it's 1 app that has conflicting dependencies. Does Pants offer a way around this?
I found that I can make a requirements.in file with flask==2.0.3 and dbt-core==1.0.3 (and also celery==5.2.3, another key package), and then use pip-compile twice against it - once with dbt-core commented out, once with flask commented out - and then merge the two results together, forcing usage of Jinja2 3.x . And I can do "pip install --no-deps" to install everything, thus achieving the same approach of pip 20. Is there an equivalent of "--no-deps" when using Pants? I'm looking at making a constraints.txt file / user lockfile to see if that makes a difference
e

enough-analyst-54434

02/23/2022, 3:49 PM
Pants uses Pex which vendors Pip 20.3.4. That version of Pip offers 2 resolve modes: legacy and 2020. The legacy mode is what you must be using and what Pex still defaults to. Pants used to have this option plumbed but removed it starting here: https://github.com/pantsbuild/pants/commit/6f0ba9180e4d4c804e71ca44adc9a69d0753052e I can't recall why. Generally we try to let users interact with the tools they use through Pants as they would directly if at all possible. @hundreds-father-404 may remember the motivation. That said, your situation is definitely untenable! Do you happen to know that the version conflict is fake? I.E.: Have you checked that you can never possibly execute code paths that would blow up due to API differences?
In short, Pants offers no sane way around this today that I can concoct.
h

happy-kitchen-89482

02/23/2022, 4:11 PM
Am I misunderstanding the problem, or doesn't the lockfile solve this? you'd have to pick some version of jinja (3.x I guess) but pip 20.x was doing that too.
l

lively-exabyte-12840

02/23/2022, 4:23 PM
pip 20 prints out several conflicts; our tests have not uncovered any issues so far. Our current state is that we use Flask 1.1.4 and Celery 5.2.3, and those have a Click 7/8 conflict. So I tried bumping Flask 1.x to 2.x, which then resulted in a Jinja2 conflict with dbt-core. I'm still new to the Python ecosystem, but these seem like popular packages that a lot of users would run into the same conflicts with.
I'll try a lockfile; I'm currently using a requirements.txt file that was produced by merging two runs of pip-compile together, and then I'm using a constraints.txt file to force usage of Jinja2 3.x, but my little test program that tries to print out the version of both Flask and DBT bombs because Pants can't find a Jinja2 2.x to make DBT happy
e

enough-analyst-54434

02/23/2022, 4:35 PM
Aha, right. I hadn't considered hacking up a resolved lock file by hand outside Pants even though you effectively spelled this out / that we use --no-deps for resolves against lock files.
l

lively-exabyte-12840

02/23/2022, 5:36 PM
No luck so far - I have this in my pants.toml file:
Copy code
[python]
requirement_constraints = "constraints.txt"
I still get an error - e.g.: The conflict is caused by: The user requested jinja2==3.0.3 dbt-core 1.0.3 depends on Jinja2==2.11.3 I from the logging that ./pex is being used, but I don't see "--no-deps" in the list of arg; should it be there since I have requirement_constraints set in the "python" section?
e

enough-analyst-54434

02/23/2022, 5:41 PM
Sorry about this - the current state of the world surrounding constraints vs locks is confusing. In short
requirement_constraints
is like a lock in that versions can be all pinned transitively, but does a full resolve anyhow. To get a real lockfile behavior where there is no resolve and the stated pinned requirements are used as-is flat you need 2.10.x: https://pypi.org/project/pantsbuild.pants/2.10.0rc1/ and https://www.pantsbuild.org/v2.10/docs/reference-python#section-enable-resolves This is alpha and the latter docs call out known issues.
l

lively-exabyte-12840

02/23/2022, 7:23 PM
Thanks @enough-analyst-54434 - I'm making some progress, but wanted to confirm first that if I try to generate a lockfile for just my conflicting versions of flask/dbt-core, I should still get a SolverProblemError, correct? My understanding is I need to hack together a lockfile by generating multiple lockfiles - without conflicts - and then manually stitch those together to get around the SolverProblemError. That effectively gets me to where my team is with pip 20 and the legacy-resolver - i.e. we're living with a conflicting dependency and relying on tests to ensure it doesn't bite us
e

enough-analyst-54434

02/23/2022, 7:25 PM
Yup, exactly. You need a shoddy resolver. Pip old does this automatically for you, otherwise you resolve by hand - i.e: stitch locks together.
l

lively-exabyte-12840

02/23/2022, 7:29 PM
"shoddy" seems about right! I'll report back later, but not sure stitching locks together is going to work - i.e. the dbt-core entry is looking for a specific fingerprint for Jinja 2.x, and I've got the Jinja 3.x entry in there now, and the resolver (rightfully) is complaining . I'll keep hacking away.
e

enough-analyst-54434

02/23/2022, 7:54 PM
But I will circle back to the fact you really don't want to be in this situation if you don't have to. Better to fork off longer term payoff PRs to the underlying projects with bad deps to fix those or else if they aren't bad but just super old, upgrade. But likely you have a constraint that you must stick to these (old)? versions I'm guessing.
l

lively-exabyte-12840

02/23/2022, 8:42 PM
Well, we're using both flask and dbt-core, and "pex flask dbt-core" blows up due to the Jinja conflict. In theory, a dbt-core 1.1 release could upgrade to Jinja 3 instead of 2. Makes me wonder what other people looking to use flask/dbt-core in the same app are doing.
e

enough-analyst-54434

02/24/2022, 2:50 AM
I think the implication is there aren't many people doing this! You have to be determined like you and really want it!
Ok, I learned about dbt and dbt-core this morning. Jinja is going to continue to be a sticky wicket afaict. Since dbt exposes custom functions in Jinja templates and these templates are core to the ~"dbt programming language" they built on top of SQL, it seems you can expect dbt-core to be tightly tied to Jinja versions going forward.
l

lively-exabyte-12840

02/24/2022, 3:59 PM
Thanks John - I was going to fiddle with building dbt-core with Jinja 3 this morning to see where that goes, as I don't have any knowledge yet of how changes from Jinja 2 to 3 will affect dbt-core