Hi, I am facing an issue with resolves and constraints file. So I have a constraints file, and a requirements.txt file. link to constraints file If I specify: 1.

apache-airflow==2.4.3

in my requirements, the pants will correctly resolve to version 2.4.3 2.

apache-airflow[cncf.kubernetes]==2.4.3

, pants will incorrectly resolve to version 2.5.1 3. When I add

apache-airflow==2.4.3

to the constraints file, then

apache-airflow[cncf.kubernetes]==2.4.3

correctly resolve to 2.4.3 I was expecting pants will combine the constraints from the requirements.txt and the constraints specified via

resolves_to_constraints_file

. which doesn’t seem to be happening. Any ideas? On pants version 2.14.1

When you say "correctly resolve", what are you referring to? An indirect result or the contents of the lock file? Can you share the full contents of the lock file?

sure. Here you go. at line 1470, you can see it somehow it pinned 2.5.1

Ok, great - thanks. At 1st glance this does look reasonable; I'm not sure how Pex / Pip are managing to lock a different version from your top-level pin since the resolver is pip-2020-resolver (pip-legacy-resolver happily ignores constraints and installs other things). I'll try to set up a repro here and I'll file an issue to track this.

Just a separate observation: from the lock file, it looks like you’re specifying exclusively exact versions in your requirements file. That’s a dangerous game since it doesn’t give you any freedom if one of the packages transitively needs a different version range of something listed in your requirements file. I always recommend internally to our developers to pin bounded ranges wherever possible to give some room for lock resolution.

@enough-analyst-54434 Happy to see you were able to reproduce it! @high-yak-85899 I agree with the general principle. Unfortunately, in airflow’s case, we can’t control what is installed in the environment as MWAA (AWS-airflow) doesn’t allow shipping docker containers. We can only ship python files. Therefore, we have to match what is installed in there to ensure our code will work. It is a big issue with airflow, and data-engineering in general actually.

CPython was ready for the data revolution with its C-extensions / numpy / etc as a foundation but Python Packaging was caught totally unprepared.

So despite having a lock file, you still have to lock dependencies in the requirements file?

@high-yak-85899 if the requirements say

x

the lockfile will lock

x

latest but MWAA may have latest-10.

@enough-analyst-54434 I am on pants 2.14.1

Ah, ok. Yeah only in 2.16.x: https://github.com/pantsbuild/pants/pull/17555 That went live in 2.16.0.dev0 and we're now up to https://pypi.org/project/pantsbuild.pants/2.16.0.dev6/

Ah, so the platform doesn’t even allow lock files? That’s a shame since pinning exactly at the requirements still wouldn’t guarantee reproducibility.

@enough-analyst-54434 is it okay to use dev builds? I am trying to put together a data engineering monorepo for our Org, so was sticking to released versions.

That is a personal choice. But it should be OK. If you have CI set up you can run through CI and if all green, then that's a good sign.

I can give 2.16 a shot. I am really trying to reduce the number of pants issues I run into. 🙏

OT: MWAA is such a half baked attempt from AWS to try to provide airflow support. That’s one reason we aren’t using it at all.

Ok, thanks @crooked-country-1937. It does help to have someone test dev/alpha/beta or else stable releases mean nothing (they weren't actually stabilized by real use!); so thanks for giving it a shot. There isn't really a good option besides that at any rate. I'm going to close the issue you filed over in Pex as answered.

Re MWAA: https://github.com/pantsbuild/pants/discussions/17858 (not really any answers yet)