https://pantsbuild.org/ logo
c

crooked-country-1937

02/08/2023, 5:24 AM
Hi, I am facing an issue with resolves and constraints file. So I have a constraints file, and a requirements.txt file. link to constraints file If I specify: 1.
apache-airflow==2.4.3
in my requirements, the pants will correctly resolve to version 2.4.3 2.
apache-airflow[cncf.kubernetes]==2.4.3
, pants will incorrectly resolve to version 2.5.1 3. When I add
apache-airflow==2.4.3
to the constraints file, then
apache-airflow[cncf.kubernetes]==2.4.3
correctly resolve to 2.4.3 I was expecting pants will combine the constraints from the requirements.txt and the constraints specified via
resolves_to_constraints_file
. which doesn’t seem to be happening. Any ideas? On pants version 2.14.1
e

enough-analyst-54434

02/08/2023, 5:30 AM
When you say "correctly resolve", what are you referring to? An indirect result or the contents of the lock file? Can you share the full contents of the lock file?
c

crooked-country-1937

02/08/2023, 5:49 AM
sure. Here you go. at line 1470, you can see it somehow it pinned 2.5.1
e

enough-analyst-54434

02/08/2023, 1:59 PM
Ok, great - thanks. At 1st glance this does look reasonable; I'm not sure how Pex / Pip are managing to lock a different version from your top-level pin since the resolver is pip-2020-resolver (pip-legacy-resolver happily ignores constraints and installs other things). I'll try to set up a repro here and I'll file an issue to track this.
Ok, yes - easy repro. Thanks @crooked-country-1937. Tracking here: https://github.com/pantsbuild/pex/issues/2048
h

high-yak-85899

02/08/2023, 3:08 PM
Just a separate observation: from the lock file, it looks like you’re specifying exclusively exact versions in your requirements file. That’s a dangerous game since it doesn’t give you any freedom if one of the packages transitively needs a different version range of something listed in your requirements file. I always recommend internally to our developers to pin bounded ranges wherever possible to give some room for lock resolution.
👍 1
c

crooked-country-1937

02/08/2023, 4:14 PM
@enough-analyst-54434 Happy to see you were able to reproduce it! @high-yak-85899 I agree with the general principle. Unfortunately, in airflow’s case, we can’t control what is installed in the environment as MWAA (AWS-airflow) doesn’t allow shipping docker containers. We can only ship python files. Therefore, we have to match what is installed in there to ensure our code will work. It is a big issue with airflow, and data-engineering in general actually.
e

enough-analyst-54434

02/08/2023, 4:16 PM
CPython was ready for the data revolution with its C-extensions / numpy / etc as a foundation but Python Packaging was caught totally unprepared.
@crooked-country-1937 what version of Pants are you on? This is all fixed by just telling Pex
--pip-version 22.3
which newer Pants versions allow you to do. It looks like a bug in the default old Pip 20.3.4 resolver used by Pex to support the LCD of Python 2.7.
h

high-yak-85899

02/08/2023, 4:20 PM
So despite having a lock file, you still have to lock dependencies in the requirements file?
e

enough-analyst-54434

02/08/2023, 4:22 PM
@high-yak-85899 if the requirements say
x
the lockfile will lock
x
latest but MWAA may have latest-10.
Basically, IIUC @crooked-country-1937 has a pre-existing external lock (environment instead of file) in the form of MWAA he must comply with.
c

crooked-country-1937

02/08/2023, 4:25 PM
@enough-analyst-54434 I am on pants 2.14.1
e

enough-analyst-54434

02/08/2023, 4:26 PM
Ah, ok. Yeah only in 2.16.x: https://github.com/pantsbuild/pants/pull/17555 That went live in 2.16.0.dev0 and we're now up to https://pypi.org/project/pantsbuild.pants/2.16.0.dev6/
h

high-yak-85899

02/08/2023, 4:26 PM
Ah, so the platform doesn’t even allow lock files? That’s a shame since pinning exactly at the requirements still wouldn’t guarantee reproducibility.
c

crooked-country-1937

02/08/2023, 4:28 PM
@enough-analyst-54434 is it okay to use dev builds? I am trying to put together a data engineering monorepo for our Org, so was sticking to released versions.
e

enough-analyst-54434

02/08/2023, 4:29 PM
That is a personal choice. But it should be OK. If you have CI set up you can run through CI and if all green, then that's a good sign.
Another alternative is to generate the lockfile yourself and tell pants to back off. I can find the config for that and point you at it if you'd rather bridge the gap until 2.16.x stable in that manner.
That would mean though that you have to be careful about regenerating at the right times, aligning inputs, etc. All the brittle bits you might expect by managing the lock file completely separately from Pants knowledge or config.
c

crooked-country-1937

02/08/2023, 4:35 PM
I can give 2.16 a shot. I am really trying to reduce the number of pants issues I run into. 🙏
r

refined-addition-53644

02/08/2023, 4:38 PM
OT: MWAA is such a half baked attempt from AWS to try to provide airflow support. That’s one reason we aren’t using it at all.
💯 1
e

enough-analyst-54434

02/08/2023, 4:50 PM
Ok, thanks @crooked-country-1937. It does help to have someone test dev/alpha/beta or else stable releases mean nothing (they weren't actually stabilized by real use!); so thanks for giving it a shot. There isn't really a good option besides that at any rate. I'm going to close the issue you filed over in Pex as answered.
c

curved-manchester-66006

02/08/2023, 5:44 PM
Re MWAA: https://github.com/pantsbuild/pants/discussions/17858 (not really any answers yet)
6 Views