https://pantsbuild.org/ logo
Join Slack
Powered by
Hi friends! been having a hell of a time with the ...
# general
w
Hi friends! been having a hell of a time with the pex resolver in 1.29.0 and an apache-airflow dependency. This might be a pex bug so let me know if i should take it elsewhere
Untitled
Untitled
This happens frequently
have tried adding the werkzeug==0.16.0 to our requirements.txt file being used in our 
python_requirements
target
have installed werkzeug==0.16.0 in our environment before running pants
it sometimes passes but i cant seem to understand what the determining factor is
👀 1
happy to open an issue
h
I suspect the inconsistency is in resolution order. Under the hood, Pex uses Pip. Pip resolves first it encounters. What is first might be changing, so it sometimes works Are you using a constraints file? https://www.pantsbuild.org/v1.29/docs/python-third-party-dependencies#using-a-lockfile-recommended
That will only work if you’re using the v2 backend for Python. Otherwise, I think there’s a less ergonomic workaround similar to what you’re trying
w
not yet on v2 for this phase of the pipeline unfortunately
👍 1
what's the 'less ergonomic' workaround?
h
Okay, so with your workaround, adding it to 
requirements.txt
is the first step. But it doesn’t actually result in anything being used; it just creates a new target for you. The next step is explicitly depending on it by adding 
3rdparty/python:Werkzeug
to the 
dependencies
field of the impacted target. Where it’s unergonomic is that you may need to add this to multiple targets
Also, taking a step back, Werkzeug isn’t an explicit dep for you, right? It’s only a transitive dependency of other things you’re using, like 
apache-airflow
?
w
correct
👍 1
@damp-quill-59187 FYI
ok so, suppose for an instance that in this particular build we are packaging all our python targets.
that would stand to reason that adding that to any target would work
h
Hmm although you may still get the inconsistency..even with an explicit req, if the problematic req is hit first, that will try installing the wrong version of Werkzeug. Then you’ll get to your explicit req, and I believe it will fail. I’m not totally certain if Pip handles things differently if it was explicitly specified.
that would stand to reason that adding that to any target would work
Correct, if the above message doesn’t end up being an issue.
w
having touble understanding this then
the wrong version isn't specified anywhere
until very recently
other than as a transitive dependency
which is pinned in the Ariflow dependency list
so where is this 'bad' version coming from?
h
The wrong version almost certainly is specified, only not by you all. We know Airflow is using it. We can also infer that at least one other req is using it, which is what causes 
1.0.1
to be resolved. Y’all are using Pipenv, right? Pipenv isn’t complaining about a conflict?
w
this is correct
but we're also maintaining a requirements.txt file (still haven't gotten to work on a 
pipenv_requirements
concpet in pants)
possible theres an impedence mismatch there
so i'll start there i suppose
h
I’m searching if pipenv has a command similar to 
dependees
, that is find who is using 
werkzeug
in your project. I suspect that you have a req that says something loose like 
Werkzeug>0.8
. Then, Pex/Pip will use the latest that’s valid, e.g. 1.0.1. Later, when encountering apache-airflow, Pip complains that it already has resolved 1.0.1 so can’t meet the new constraint
w
it has a pipenv graph
👍 1
h
(This is why Pip is working on a proper resolver - Pip dependency resolution is a mess, and Pex inherits that. Constraint files are a semi-hacky way to fix that)
w
doesn't work well for 
extras
so that's annoying
dep management is so far my least favorite part of python so that tracks
h
Ditto. We’re considering some changes for Pants to generate a constraints file for you and I’m recommending that Pants not try to enter that space, but instead integrate better with things like Pipenv and Poetry. It’s a rotten foundation
Hm, so 
pipenv graph
isn’t showing anything helpful?
w
checking now
h
Otherwise, a hacky way to do this is to set an explicit dep on something ludicrous like 
Werkzeug<0.1
. See what errors when you do that, other than 
apache-airflow
.
w
ok it looks like 
moto
has a 
Werkzeug=*
dependency
💯 1
h
That sounds right. Did y’all recently change the moto version?
w
we recently added the airflow dependency
👍 1
so pipenv resolves this by realizing 
<1
and 
*
are compatible and applies the most restrictive
you're saying theres no way around this for pex?
aside from moving to v2?
w
https://github.com/pantsbuild/pex/issues/943 covers improving the situation here… the 
Requires
lines in the metadata are enough to render the conflict
👍 1
h
Precisely. Pip makes zero attempt to resolve and goes with the first it finds 😞 Some possible fixes, none ideal: 1) Play around with Moto versions to see if there’s one that behaves how you want. 2) See if there’s an 
apache-airflow
version that works with >1.0 3) Upgrade to 2.0 and use a constraints file to force the version you want to always be used.
w
2 is definitely not possible
👍 1
3 is a substantial lift
h
@polite-garden-50641 do you know of any other solutions? We’ve had similar conflicts with Toolchain before we added constraint files
w
was already on my way to 1
pray for me
🙏 1
h
Praying for all Python devs who have to deal with this
I’m really hoping this new Pip resolver works well
w
i mean, ive had my issues but this is quite something
1 is a no go
😢 1
looking to see if we can excise the one build target that uses airflow
👍 1
and keep it deployed as is
h
That’s a good idea
p
As long as the conflict is resolvable, i.e. the two conflicting dependencies don't need to be in the same target, this is solvable. In our repo we do it by having two requirements files and (with the appropriate python_requirements() in the BUILD file) and then we reference the requirement from the matching build file.
👍 1
w
interesting
h
True. If you run 
./pants dependencies --transitive --type=3rdparty path/to:target-with-airflow
, does Moto show up?
p
BTW - this is no longer needed since we now use dependency inferencing and pants seems to be able to do the right thing.
h
(
dependencies2
would be faster)
this is no longer needed since we now use dependency inferencing and pants seems to be able to do the right thing.
True if using Pants 2.0. They’re still on 1.29 and using v1 for this part of the pipeline. But good point for the near future 🙂
w
moto does not show up
p
it used to look something like this:
h
Okay awesome. So what is the goal you’re running? Something like 
./pants binary ::
?
w
./pants setup-py ::
we're all wheels baby
👍 1
p
this is for a vendored version of ES curator (which is using old version of some libraries)
w
our deployment surface cant deal with PEXs
gotcha
h
Okay, and then does 
./pants setup-py ::
include a target that includes both the airflow code and the moto code? You can run 
./pants list --provides ::
to see what has 
setup_py
defined
w
oh yes
for sure
oh wait
you mean one that has both dependencies in a single target?
no
h
A single 
setup_py
that includes both 
moto
and 
apache-airflow
in its transitive dependencies; not necessarily it’s direct dependencies
w
no there is only a single target with apache-airflow
and that wouldbn't be a transitive dep anywhere
h
Okay I’m a little confused. Is that single target with apache-airflow included in any of the targets that run with 
setup-py
? You can run 
./pants dependees --transitive path/to:target
for that
w
yes
there is one target with 
apache-airflow
that runs setup-py
there are many that include moto
if i were to build them all with the 
::
selector would that be a problem?
h
there is one target with apache-airflow that runs setup-py
For this target in particular, 
./pants dependencies --transitive --type=thirdparty
does not include 
moto
, right?
w
correct
h
Okay, great. So then now this comes down to how correct/hermitic v1 
setup-py
is..which I don’t know anything about. @happy-kitchen-89482 with v1 setup-py, are each of the distinct targets treated as completely separate and hermetic? (fwit, v2 definitely would do the right thing, but I know that’s not a great answer rn)
I think he may be in a meeting. In the meantime, maybe try this? Add a tag to the airflow target’s 
tags
field, like 
tags=["dont_build"]
. Then, instead of 
./pants setup-py ::
, use 
./pants --tags='-dont_build' setup-py ::
, followed by 
./pants setup-py path/to:airflow
as a separate command
w
that is another decent workaround
trying the separate reqs one first
👍 1
h
I'll go eyeball the v1 setup-py code
I don't remember offhand what it does
What happens if you build the offending target individually (rather than with 
::
)? I have a feeling that in v1 
::
might be causing one big resolve with conflicts.
👍 1
w
oh the individual target has always worked
h
[PS my "meeting" was lunch...]
a
the new pip resolver is awesome btw
w
so is the requirements thing not going to work?
h
Hmm, so looking at the code, I'm not actually sure why the 
resolve_requirements
task is running at all... Nothing in the setup-py task appears to need resolved requirements (and this makes sense - we're not actually running your code, just bundling it up with some metadata). Going to dig a little deeper.
👍 1
OK, so it runs because v1 runs all tasks in a goal even if you depend on just some of them. setup-py depends on interpreter selection, which is in the "pyprep" goal. Also in that goal is "resolve_requirements"...
Grrrr
So it's dying on work it doesn't even need to be doing
@wonderful-iron-54019 Is the workaround working around?
I'd rather not resort to hacking on 1.29.x if I can avoid it
h
Which is a main reason that we created the v2 engine: so that you can precisely model every step of your build process
w
hey sorry. i ended up commenting out the. entire offendinng 
BUILD
file. Touching our requirements.txt results in a 20min build and didn't want to potentially have to. do that multiple times EOD
this isn't likelly to be a high. touch project for. the time being so hopefully we can get on 2.0 before needing to make modifications
👍 2
Open in Slack
PreviousNext
Powered by