https://pantsbuild.org/ logo
#general
Title
# general
w

wonderful-iron-54019

07/21/2020, 7:44 PM
Hi friends! been having a hell of a time with the pex resolver in 1.29.0 and an apache-airflow dependency. This might be a pex bug so let me know if i should take it elsewhere
Untitled
This happens frequently
have tried adding the werkzeug==0.16.0 to our requirements.txt file being used in our
python_requirements
target
have installed werkzeug==0.16.0 in our environment before running pants
it sometimes passes but i cant seem to understand what the determining factor is
👀 1
happy to open an issue
h

hundreds-father-404

07/21/2020, 7:47 PM
I suspect the inconsistency is in resolution order. Under the hood, Pex uses Pip. Pip resolves first it encounters. What is first might be changing, so it sometimes works Are you using a constraints file? https://www.pantsbuild.org/v1.29/docs/python-third-party-dependencies#using-a-lockfile-recommended
That will only work if you’re using the v2 backend for Python. Otherwise, I think there’s a less ergonomic workaround similar to what you’re trying
w

wonderful-iron-54019

07/21/2020, 7:49 PM
not yet on v2 for this phase of the pipeline unfortunately
👍 1
what's the 'less ergonomic' workaround?
h

hundreds-father-404

07/21/2020, 7:51 PM
Okay, so with your workaround, adding it to
requirements.txt
is the first step. But it doesn’t actually result in anything being used; it just creates a new target for you. The next step is explicitly depending on it by adding
3rdparty/python:Werkzeug
to the
dependencies
field of the impacted target. Where it’s unergonomic is that you may need to add this to multiple targets
Also, taking a step back, Werkzeug isn’t an explicit dep for you, right? It’s only a transitive dependency of other things you’re using, like
apache-airflow
?
w

wonderful-iron-54019

07/21/2020, 7:51 PM
correct
👍 1
@damp-quill-59187 FYI
ok so, suppose for an instance that in this particular build we are packaging all our python targets.
that would stand to reason that adding that to any target would work
h

hundreds-father-404

07/21/2020, 7:53 PM
Hmm although you may still get the inconsistency..even with an explicit req, if the problematic req is hit first, that will try installing the wrong version of Werkzeug. Then you’ll get to your explicit req, and I believe it will fail. I’m not totally certain if Pip handles things differently if it was explicitly specified.
that would stand to reason that adding that to any target would work
Correct, if the above message doesn’t end up being an issue.
w

wonderful-iron-54019

07/21/2020, 7:53 PM
having touble understanding this then
the wrong version isn't specified anywhere
until very recently
other than as a transitive dependency
which is pinned in the Ariflow dependency list
so where is this 'bad' version coming from?
h

hundreds-father-404

07/21/2020, 7:56 PM
The wrong version almost certainly is specified, only not by you all. We know Airflow is using it. We can also infer that at least one other req is using it, which is what causes
1.0.1
to be resolved. Y’all are using Pipenv, right? Pipenv isn’t complaining about a conflict?
w

wonderful-iron-54019

07/21/2020, 7:56 PM
this is correct
but we're also maintaining a requirements.txt file (still haven't gotten to work on a
pipenv_requirements
concpet in pants)
possible theres an impedence mismatch there
so i'll start there i suppose
h

hundreds-father-404

07/21/2020, 7:59 PM
I’m searching if pipenv has a command similar to
dependees
, that is find who is using
werkzeug
in your project. I suspect that you have a req that says something loose like
Werkzeug>0.8
. Then, Pex/Pip will use the latest that’s valid, e.g. 1.0.1. Later, when encountering apache-airflow, Pip complains that it already has resolved 1.0.1 so can’t meet the new constraint
w

wonderful-iron-54019

07/21/2020, 7:59 PM
it has a pipenv graph
👍 1
h

hundreds-father-404

07/21/2020, 7:59 PM
(This is why Pip is working on a proper resolver - Pip dependency resolution is a mess, and Pex inherits that. Constraint files are a semi-hacky way to fix that)
w

wonderful-iron-54019

07/21/2020, 7:59 PM
doesn't work well for
extras
so that's annoying
dep management is so far my least favorite part of python so that tracks
h

hundreds-father-404

07/21/2020, 8:01 PM
Ditto. We’re considering some changes for Pants to generate a constraints file for you and I’m recommending that Pants not try to enter that space, but instead integrate better with things like Pipenv and Poetry. It’s a rotten foundation
Hm, so
pipenv graph
isn’t showing anything helpful?
w

wonderful-iron-54019

07/21/2020, 8:02 PM
checking now
h

hundreds-father-404

07/21/2020, 8:02 PM
Otherwise, a hacky way to do this is to set an explicit dep on something ludicrous like
Werkzeug<0.1
. See what errors when you do that, other than
apache-airflow
.
w

wonderful-iron-54019

07/21/2020, 8:03 PM
ok it looks like
moto
has a
Werkzeug=*
dependency
💯 1
h

hundreds-father-404

07/21/2020, 8:04 PM
That sounds right. Did y’all recently change the moto version?
w

wonderful-iron-54019

07/21/2020, 8:04 PM
we recently added the airflow dependency
👍 1
so pipenv resolves this by realizing
<1
and
*
are compatible and applies the most restrictive
you're saying theres no way around this for pex?
aside from moving to v2?
w

witty-crayon-22786

07/21/2020, 8:07 PM
https://github.com/pantsbuild/pex/issues/943 covers improving the situation here… the
Requires
lines in the metadata are enough to render the conflict
👍 1
h

hundreds-father-404

07/21/2020, 8:07 PM
Precisely. Pip makes zero attempt to resolve and goes with the first it finds 😞 Some possible fixes, none ideal: 1) Play around with Moto versions to see if there’s one that behaves how you want. 2) See if there’s an
apache-airflow
version that works with >1.0 3) Upgrade to 2.0 and use a constraints file to force the version you want to always be used.
w

wonderful-iron-54019

07/21/2020, 8:07 PM
2 is definitely not possible
👍 1
3 is a substantial lift
h

hundreds-father-404

07/21/2020, 8:07 PM
@polite-garden-50641 do you know of any other solutions? We’ve had similar conflicts with Toolchain before we added constraint files
w

wonderful-iron-54019

07/21/2020, 8:08 PM
was already on my way to 1
pray for me
🙏 1
h

hundreds-father-404

07/21/2020, 8:08 PM
Praying for all Python devs who have to deal with this
I’m really hoping this new Pip resolver works well
w

wonderful-iron-54019

07/21/2020, 8:11 PM
i mean, ive had my issues but this is quite something
1 is a no go
😢 1
looking to see if we can excise the one build target that uses airflow
👍 1
and keep it deployed as is
h

hundreds-father-404

07/21/2020, 8:11 PM
That’s a good idea
p

polite-garden-50641

07/21/2020, 8:13 PM
As long as the conflict is resolvable, i.e. the two conflicting dependencies don't need to be in the same target, this is solvable. In our repo we do it by having two requirements files and (with the appropriate python_requirements() in the BUILD file) and then we reference the requirement from the matching build file.
👍 1
w

wonderful-iron-54019

07/21/2020, 8:14 PM
interesting
h

hundreds-father-404

07/21/2020, 8:14 PM
True. If you run
./pants dependencies --transitive --type=3rdparty path/to:target-with-airflow
, does Moto show up?
p

polite-garden-50641

07/21/2020, 8:15 PM
BTW - this is no longer needed since we now use dependency inferencing and pants seems to be able to do the right thing.
h

hundreds-father-404

07/21/2020, 8:15 PM
(
dependencies2
would be faster)
this is no longer needed since we now use dependency inferencing and pants seems to be able to do the right thing.
True if using Pants 2.0. They’re still on 1.29 and using v1 for this part of the pipeline. But good point for the near future 🙂
w

wonderful-iron-54019

07/21/2020, 8:16 PM
moto does not show up
p

polite-garden-50641

07/21/2020, 8:16 PM
it used to look something like this:
h

hundreds-father-404

07/21/2020, 8:16 PM
Okay awesome. So what is the goal you’re running? Something like
./pants binary ::
?
w

wonderful-iron-54019

07/21/2020, 8:16 PM
./pants setup-py ::
we're all wheels baby
👍 1
p

polite-garden-50641

07/21/2020, 8:16 PM
this is for a vendored version of ES curator (which is using old version of some libraries)
w

wonderful-iron-54019

07/21/2020, 8:16 PM
our deployment surface cant deal with PEXs
gotcha
h

hundreds-father-404

07/21/2020, 8:18 PM
Okay, and then does
./pants setup-py ::
include a target that includes both the airflow code and the moto code? You can run
./pants list --provides ::
to see what has
setup_py
defined
w

wonderful-iron-54019

07/21/2020, 8:19 PM
oh yes
for sure
oh wait
you mean one that has both dependencies in a single target?
no
h

hundreds-father-404

07/21/2020, 8:20 PM
A single
setup_py
that includes both
moto
and
apache-airflow
in its transitive dependencies; not necessarily it’s direct dependencies
w

wonderful-iron-54019

07/21/2020, 8:20 PM
no there is only a single target with apache-airflow
and that wouldbn't be a transitive dep anywhere
h

hundreds-father-404

07/21/2020, 8:22 PM
Okay I’m a little confused. Is that single target with apache-airflow included in any of the targets that run with
setup-py
? You can run
./pants dependees --transitive path/to:target
for that
w

wonderful-iron-54019

07/21/2020, 8:22 PM
yes
there is one target with
apache-airflow
that runs setup-py
there are many that include moto
if i were to build them all with the
::
selector would that be a problem?
h

hundreds-father-404

07/21/2020, 8:24 PM
there is one target with apache-airflow that runs setup-py
For this target in particular,
./pants dependencies --transitive --type=thirdparty
does not include
moto
, right?
w

wonderful-iron-54019

07/21/2020, 8:24 PM
correct
h

hundreds-father-404

07/21/2020, 8:26 PM
Okay, great. So then now this comes down to how correct/hermitic v1
setup-py
is..which I don’t know anything about. @happy-kitchen-89482 with v1 setup-py, are each of the distinct targets treated as completely separate and hermetic? (fwit, v2 definitely would do the right thing, but I know that’s not a great answer rn)
I think he may be in a meeting. In the meantime, maybe try this? Add a tag to the airflow target’s
tags
field, like
tags=["dont_build"]
. Then, instead of
./pants setup-py ::
, use
./pants --tags='-dont_build' setup-py ::
, followed by
./pants setup-py path/to:airflow
as a separate command
w

wonderful-iron-54019

07/21/2020, 8:29 PM
that is another decent workaround
trying the separate reqs one first
👍 1
h

happy-kitchen-89482

07/21/2020, 8:31 PM
I'll go eyeball the v1 setup-py code
I don't remember offhand what it does
What happens if you build the offending target individually (rather than with
::
)? I have a feeling that in v1
::
might be causing one big resolve with conflicts.
👍 1
w

wonderful-iron-54019

07/21/2020, 8:33 PM
oh the individual target has always worked
h

happy-kitchen-89482

07/21/2020, 8:33 PM
[PS my "meeting" was lunch...]
a

aloof-angle-91616

07/21/2020, 8:33 PM
the new pip resolver is awesome btw
w

wonderful-iron-54019

07/21/2020, 8:34 PM
so is the requirements thing not going to work?
h

happy-kitchen-89482

07/21/2020, 11:15 PM
Hmm, so looking at the code, I'm not actually sure why the
resolve_requirements
task is running at all... Nothing in the setup-py task appears to need resolved requirements (and this makes sense - we're not actually running your code, just bundling it up with some metadata). Going to dig a little deeper.
👍 1
OK, so it runs because v1 runs all tasks in a goal even if you depend on just some of them. setup-py depends on interpreter selection, which is in the "pyprep" goal. Also in that goal is "resolve_requirements"...
Grrrr
So it's dying on work it doesn't even need to be doing
@wonderful-iron-54019 Is the workaround working around?
I'd rather not resort to hacking on 1.29.x if I can avoid it
h

hundreds-father-404

07/21/2020, 11:41 PM
Which is a main reason that we created the v2 engine: so that you can precisely model every step of your build process
w

wonderful-iron-54019

07/22/2020, 11:57 AM
hey sorry. i ended up commenting out the. entire offendinng
BUILD
file. Touching our requirements.txt results in a 20min build and didn't want to potentially have to. do that multiple times EOD
this isn't likelly to be a high. touch project for. the time being so hopefully we can get on 2.0 before needing to make modifications
👍 2