Here’s my background: We have a dependency on

tensorflow

which works fine for linux and Mac as long as it is x86_64. Now enter Mac M1 which as arm64, there are no prebuilt wheels for this platform, however there is a

tensorflow-macos

that does have wheels for macos arm64. Is there any way to support both of these in the same Pants resolve, or do I need to work around this juggling with Pants resolves to make it work? (running experiments now, posting in case someone already know the answer to this 🙂 )

Currently trying with this in my requirements.txt file, see how it goes..:

tensorflow ; sys_platform != "darwin" or platform_machine == "x86_64"
tensorflow-macos ; sys_platform == "darwin" and platform_machine == "arm64"

hmm… that’s taking a fair amount of time to create a lock for….

⠂ 2064.65s Generate lockfile for python-default

.. still going..

Ah, just found the

requirements

field of the

python_requirement

target.. think that does what I want 🙂

requirements
    type: Iterable[str]
    required

    A pip-style requirement string, e.g. `["Django==3.2.8"]`.

    You can specify multiple requirements for the same project in order to use environment markers, such as
    `["foo>=1.2,<1.3 ; python_version>'3.6'", "foo==0.9 ; python_version<'3'"]`.

Hmm… not sure what’s going on, but using either one of these requirements produces a lockfile in seconds, while when I have both, it doesn’t seem to resolve at all…

python_requirement(
    name="test",
    resolve="dummy",
    requirements=[
        "tensorflow ; sys_platform != 'darwin' or platform_machine == 'x86_64'",
        "tensorflow-macos ; sys_platform == 'darwin' and platform_machine == 'arm64'",
    ],
)

(this is on pants 2.13.0 btw)

It’s the pip resolver that’s having issues, but I’m not sure how to help it along…

2022-09-12T14:01:32,317 INFO: pip is looking at multiple versions of pyparsing to determine which version is compatible with other requirements. This could take a while.

My attempts at pinning down pyparsing doesn’t seem to do anything..

ping @enough-analyst-54434 if you have a minute sometime to lend me an experts eye to spot the (hopefully) obvious I don’t would be much appreciated, thank you 🙂

Eh, ok the pinning down issue was my bad. Edited requirements in the wrong file. However pinning down one transient dependency leads to the next and next and so on.. doesn’t feel like a viable solution to have to pin all the transient deps will make the requirements file look like a lock file-ish.

Taking a step back, I’m splitting the tensorflow variants into two different resolves for now to unblock myself..

Which is just to say that that would rule in / out a Pants upgrade of Pex as helpful to your case

See: https://pantsbuild.slack.com/archives/C18RRR4JK/p1661713160748249?thread_ts=1661713160.748249&cid=C18RRR4JK

@curved-television-6568 fwiw, the original requirement.txt variant is the same as the

python_requirement

variant. On both, it might be a bit better for sanity sake to make the markers exactly inverse so you know it's an xor, so:

tensorflow ; sys_platform != "darwin" or platform_machine != "arm64"
tensorflow-macos ; sys_platform == "darwin" and platform_machine == "arm64"

None of which will have anything to do with the resolve speed here - just scanning back through your posts.

So, do you actually mean any tensorflow* could you not bound that low with something? Like >= last stable currently?

You're leaving Pip alot of space to cover that you may not actually mean.

indeed, I can try to use >= for tensorflow to a recent version

Esp if any transitives have many releases and those transitives are sdist only. That means they may ~all need to be built!

What version of Pants is this?

I’ve broken out the tensorflow stuff into it’s own resolve, so it’s rather smallish in terms of number of dists in it at least…

Oh! Thanks for the hints John, the simple stuff that gives so much. Adding the constraints on tensorflow surfaced the issue in plain text, didn’t even have to read the pip log for this info! 😄

The conflict is caused by:
     The user requested keras
     tensorflow 2.10.0 depends on keras<2.11 and >=2.10.0
     tensorflow-macos 2.9.2 depends on keras<2.10.0 and >=2.9.0rc0

Now it only remains to see if I can resolve it too.. heh 😛

Ah, right use the same

>=

for both of them… works 🙌

thanks for nudging me in the right direction ❤️