https://pantsbuild.org/ logo
#general
Title
# general
c

curved-television-6568

09/12/2022, 3:10 PM
3rdparty dependency resolution Q 🧵
1
Here’s my background: We have a dependency on
tensorflow
which works fine for linux and Mac as long as it is x86_64. Now enter Mac M1 which as arm64, there are no prebuilt wheels for this platform, however there is a
tensorflow-macos
that does have wheels for macos arm64. Is there any way to support both of these in the same Pants resolve, or do I need to work around this juggling with Pants resolves to make it work? (running experiments now, posting in case someone already know the answer to this 🙂 )
Currently trying with this in my requirements.txt file, see how it goes..:
Copy code
tensorflow ; sys_platform != "darwin" or platform_machine == "x86_64"
tensorflow-macos ; sys_platform == "darwin" and platform_machine == "arm64"
hmm… that’s taking a fair amount of time to create a lock for….
Copy code
⠂ 2064.65s Generate lockfile for python-default
.. still going..
Ah, just found the
requirements
field of the
python_requirement
target.. think that does what I want 🙂
Copy code
requirements
    type: Iterable[str]
    required

    A pip-style requirement string, e.g. `["Django==3.2.8"]`.

    You can specify multiple requirements for the same project in order to use environment markers, such as
    `["foo>=1.2,<1.3 ; python_version>'3.6'", "foo==0.9 ; python_version<'3'"]`.
Hmm… not sure what’s going on, but using either one of these requirements produces a lockfile in seconds, while when I have both, it doesn’t seem to resolve at all…
Copy code
python_requirement(
    name="test",
    resolve="dummy",
    requirements=[
        "tensorflow ; sys_platform != 'darwin' or platform_machine == 'x86_64'",
        "tensorflow-macos ; sys_platform == 'darwin' and platform_machine == 'arm64'",
    ],
)
(this is on pants 2.13.0 btw)
It’s the pip resolver that’s having issues, but I’m not sure how to help it along…
Copy code
2022-09-12T14:01:32,317 INFO: pip is looking at multiple versions of pyparsing to determine which version is compatible with other requirements. This could take a while.
My attempts at pinning down pyparsing doesn’t seem to do anything..
ping @enough-analyst-54434 if you have a minute sometime to lend me an experts eye to spot the (hopefully) obvious I don’t would be much appreciated, thank you 🙂
Eh, ok the pinning down issue was my bad. Edited requirements in the wrong file. However pinning down one transient dependency leads to the next and next and so on.. doesn’t feel like a viable solution to have to pin all the transient deps will make the requirements file look like a lock file-ish.
Taking a step back, I’m splitting the tensorflow variants into two different resolves for now to unblock myself..
e

enough-analyst-54434

09/12/2022, 7:40 PM
You might run with -ldebug and then use the command line to formulate the same command line, but adding
--pip-version 22.2.2
using Pex 2.1 104. Pants has not been upgraded yet, but that's the latest Pip and it solves some - definitely not all - resolve perf issues.
👍 1
Which is just to say that that would rule in / out a Pants upgrade of Pex as helpful to your case
@curved-television-6568 fwiw, the original requirement.txt variant is the same as the
python_requirement
variant. On both, it might be a bit better for sanity sake to make the markers exactly inverse so you know it's an xor, so:
Copy code
tensorflow ; sys_platform != "darwin" or platform_machine != "arm64"
tensorflow-macos ; sys_platform == "darwin" and platform_machine == "arm64"
None of which will have anything to do with the resolve speed here - just scanning back through your posts.
🙏 1
So, do you actually mean any tensorflow* could you not bound that low with something? Like >= last stable currently?
You're leaving Pip alot of space to cover that you may not actually mean.
c

curved-television-6568

09/12/2022, 8:02 PM
Oh.. could that be the issue? didn’t occur to me, as it was iterating versions for a lot of other packages, without really indicating to me why it needed to do that.
indeed, I can try to use >= for tensorflow to a recent version
e

enough-analyst-54434

09/12/2022, 8:03 PM
Well, if you think about what a resolver needs to do. If it ever needs to backtrack and you've said - all time - that could take a while!
Esp if any transitives have many releases and those transitives are sdist only. That means they may ~all need to be built!
c

curved-television-6568

09/12/2022, 8:04 PM
Ouch. Would be nice if it was made clear why it was backtracking though… maybe it did say, but I missed it gleaning the pip logs…
e

enough-analyst-54434

09/12/2022, 8:04 PM
No, Pip is not clear about this. You need to actually read the Pip verbose logs and try to understand them if you want that info.
c

curved-television-6568

09/12/2022, 8:05 PM
I guess it’s the understanding them part that is a bit lacking 😛
e

enough-analyst-54434

09/12/2022, 8:05 PM
Pex has a --preserve-pip-download-log but Pants does not expose this.
c

curved-television-6568

09/12/2022, 8:06 PM
I found it handy to run
./pants --keep-sandbox=always ..
then
tail -f /sandbox/.tmp/pex…/pip.log
e

enough-analyst-54434

09/12/2022, 8:06 PM
So, the other major thing is making sure your ICs aren't outrageous, at least no more than they need to be.
c

curved-television-6568

09/12/2022, 8:06 PM
the ic is 3.8.*
e

enough-analyst-54434

09/12/2022, 8:06 PM
Ok, that's reasonable.
What version of Pants is this?
c

curved-television-6568

09/12/2022, 8:07 PM
2.13.0
e

enough-analyst-54434

09/12/2022, 8:07 PM
Ok, that has windows culling which make resolves a bit faster, but clearly not helpful here.
c

curved-television-6568

09/12/2022, 8:08 PM
OK, I’ll do some more digging around with your provided pointers see what I can turn up, hopefully understand why it’s backtracking as I was thinking I could avoid that altogether, hopefully.
I’ve broken out the tensorflow stuff into it’s own resolve, so it’s rather smallish in terms of number of dists in it at least…
Oh! Thanks for the hints John, the simple stuff that gives so much. Adding the constraints on tensorflow surfaced the issue in plain text, didn’t even have to read the pip log for this info! 😄
Copy code
The conflict is caused by:
     The user requested keras
     tensorflow 2.10.0 depends on keras<2.11 and >=2.10.0
     tensorflow-macos 2.9.2 depends on keras<2.10.0 and >=2.9.0rc0
Now it only remains to see if I can resolve it too.. heh 😛
Ah, right use the same
>=
for both of them… works 🙌
e

enough-analyst-54434

09/12/2022, 8:22 PM
Ok, just remember that universal locks are at heart a bad idea; so it's best to be distrustful of them generally and make sure you understand what you're asking to be done / what is (or isn't!) being done. I have an overdue writeup on this over at Pex read the docs.
c

curved-television-6568

09/12/2022, 8:33 PM
I’m sure I don’t know exactly why, but believe it does what I want in this case at least. So looking forward to reading it once posted 😉
thanks for nudging me in the right direction ❤️