https://pantsbuild.org/ logo
Join Slack
Powered by
a little open-ended question here, but has anyone ...
# general
h
a little open-ended question here, but has anyone successfully been able to get pants working with tensorflow on M1? it looks like neither pants nor pip are able to install tensorflow normally:
Copy code
martim@Argon allpdl % ./pants run src/app:analysis-cli -- standardize 3577                                      
19:08:09.21 [INFO] Completed: Building requirements.pex with 27 requirements: Unidecode<2.0.0,>=1.2.0, arn<0.2.0,>=0.1.5, boltons<21.0.0,>=20.2.1, boto3<2.0.0,>=1.15, botocore<2.0.0,>=1.21.60, click<8.0.0,>=7.1.2, contractions<0.0... (540 characters truncated)
19:08:09.21 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
[...]
    raise ExecutionError(

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building requirements.pex with 27 requirements: Unidecode<2.0.0,>=1.2.0, arn<0.2.0,>=0.1.5, boltons<21.0.0,>=20.2.1, boto3<2.0.0,>=1.15, botocore<2.0.0,>=1.21.60, click<8.0.0,>=7.1.2, contractions<0.0.59,>=0.0.58, dataclasses<0.9.0,>=0.8; python_version < "3.7", django-taggit<2.0.0,>=1.1, django<4.0.0,>=3.1.1, elasticsearch==7.12.0, emoji<2.0.0,>=1.6, jsonschema<4.0.0,>=3.0, marshmallow-dataclass<9.0.0,>=8.2.0, nameparser<2.0.0,>=1.0.6, nltk<4.0.0,>=3.6.5, orjson<4.0.0,>=3.3.1, phonenumbers<9.0.0,>=8.12, psycopg2-binary<3.0.0,>=2, redis-py-cluster<3.0.0,>=2.0.0, requests<3.0.0,>=2.24, sentry-sdk[flask]<2.0.0,>=1.5.0, slackclient<3.0.0,>=2.9, tensorflow==2.5.1, titlecase<2.0.0,>=1.1.1, toposort<2.0.0,>=1.6, usaddress<0.6.0,>=0.5.10' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement tensorflow==2.5.1
ERROR: No matching distribution found for tensorflow==2.5.1
pip:
Copy code
martim@Argon allpdl % pip install tensorflow
ERROR: Could not find a version that satisfies the requirement tensorflow (from versions: none)
it seems that the fix requires installing miniforge (see here or here), but: 1. it appears to require using conda to create virtualenvs instead of pyenv+pants, 2. it requires pip-installing 
tensorflow-macos
 and 
tensorflow-metal
 instead of 
tensorflow
, and i’m not sure it’s possible to create that kind of conditional 
pyproject.toml
, even assuming it’s possible to get this working
e
On point 2, the Python packaging ecosystem generally supports environment markers for this purpose (https://www.python.org/dev/peps/pep-0508/#environment-markers). You can say something like this (using requirements.txt example, but generally true of any way you write down requirements:
Copy code
# M1 reqs:
tensorflow-macos; sys_platform == "darwin" and platform_machine == "arm64"
tensorflow-metal; sys_platform == "darwin" and platform_machine == "arm64"

# Everyone else reqs:
tensorflow; sys_platform != "darwin" or platform_machine != "arm64"
🙏 1
q
I have encountered this as well. I gave up using M1 mac for tensorflow and start using intel mac and/or EC2 because we would like to share code among our team for both developing and deploying purpose.
h
thanks @able-match-78954, that’s gotten me much of the way to a solution! what i’ve done:
Copy code
pyenv install miniforge3
pyenv local miniforge3  # uses python 3.9
brew install hdf5
export HDF5_DIR="$(brew --prefix hdf5)"
using this, I’m able to 
pip install tensorflow-macos
I’m also able to define my requirements in `pyproject.toml`:
Copy code
[tool.poetry.dependencies]
tensorflow = { version = "2.5.1", markers = "sys_platform != 'darwin' or platform_machine != 'arm64'" }
tensorflow-macos = { version = "^2.5", markers = "sys_platform == 'darwin' and platform_machine == 'arm64'" }
tensorflow-metal = { version = "0.3.0", markers = "sys_platform == 'darwin' and platform_machine == 'arm64'" }
however, when running commands using pants, I still have two issues: 1. setting 
export HDF5_DIR="$(brew --prefix hdf5)"
doesn’t seem to pass on to 
pants
, I still get an error when installing h5py (a dependency for tensorflow-macos) because it can’t find hdf5 by homebrew (works fine using 
pip
directly though)
Copy code
error: Unable to load dependency HDF5, make sure HDF5 is installed properly
  error: dlopen(libhdf5.dylib, 0x0006): tried: 'libhdf5.dylib' (no such file), '/usr/local/lib/libhdf5.dylib' (no such file), '/usr/lib/libhdf5.dylib' (no such file), '/private/var/folders/4s/0qxzthl16y3_cjxt_ygbw__m0000gn/T/process-executionWusDQS/.tmp/pip-req-build-75rsom89/libhdf5.dylib' (no such file), '/usr/local/lib/libhdf5.dylib' (no such file), '/usr/lib/libhdf5.dylib' (no such file)
2. i’m not sure how to resolve the module mapping: I can add 
poetry_requirements(module_mapping={"tensorflow-macos": ["tensorflow"]})
the 
BUILD
file, but then i’ll get a conflicting dependency between 
tensorflow-macos
and 
tensorflow
, and I don’t know how to resolve it based on 
sys_platform
h
i’m not sure how to resolve the module mapping: I can add poetry_requirements(module_mapping={"tensorflow-macos": ["tensorflow"]}) the BUILD file, but then i’ll get a conflicting dependency between tensorflow-macos and tensorflow, and I don’t know how to resolve it based on sys_platform
Interesting. Ideally what would happen is we have a single 
python_requirement
target that includes both 
tensorflow-macos
and 
tensorflow
. Your environment markers are used to make sure that only one actually gets consumed by Pex/pip. I'm trying to think how we would model that with the 
poetry_requirements
target generator...how do we know that you want 
tensorflow-macos
and 
tensorflow
to be the same thing? But you don't want 
tensorflow-utils
to be bundled with 
tensorflow
, for example, I don't think we can infer anything - you'd need to explicitly set it. Maybe something like this? Wdyt?
Copy code
poetry_requirements(
  group_together=[["tensorflow", "tensorflow-macos", "tensorflow-metal"]],
)
In the meantime, the workaround is really bleh. You'd remove both tensorflow entries from pyproject.toml and create an explicit target like this:
Copy code
python_requirement
  name="tensorflow",
  requirements=[
     "tensorflow==2.5.1; sys_platform != 'darwin' or platform_machine != 'arm64'",
     "tensorflow-macos==2.5.1; sys_platform == 'darwin' and platform_machine == 'arm64'",
     "tensorflow-metal==2.5.1; sys_platform == 'darwin' and platform_machine == 'arm64'",
  ],
)
e
For the 1st: Pants tries really hard to make all actions hermetic. If you want to leak your local environment into an action you almost always need to tell Pants to do that explicitly. In this case you want to configure the following in `pants.toml`:
Copy code
[subprocess-environment]
env_vars.add = ["HDF5_DIR"]
That will leak your local environment variable into processes Pants executes. See: https://www.pantsbuild.org/docs/reference-subprocess-environment
❤️ 1
👍 1
h
@hundreds-father-404 yeah, it’s weird to me why they chose to create a separate package for 
tensorflow-macos
instead of extending the tensorflow package… I think the 
group_together
argument would make sense, and i agree that these cases would have to be explicitly stated (fortunately this doesn’t seem like something very common at all) the only other downside with that workaround is that we’re not adding those packages to a lockfile. just to double check though, other than that, there should be no difference between the workaround and adding tensorflow to pyproject.toml, right? i want to make sure I’m not inadvertently breaking something else…
for future reference, my solution to get tensorflow working on an M1 Mac: • install miniforge3 (
brew install miniforge
might work as well, but we use 
pyenv
to specify our python version) ◦ 
pyenv install miniforge3
• set miniforge3 to the local (or global) python version (uses python 3.9) ◦ 
pyenv local miniforge3
in your monorepo • [i don’t think hdf5 or h5py was needed after all, it was no longer a problem when i updated my poetry.lock file. if not, then 
brew install hdf5
should help] • remove tensorflow from 
pyproject.toml
and update the poetry lock file • [i don’t need to update the module mapping for poetry requirements] • add this to the root 
BUILD
file (use 
python_requirement
instead of 
python_requirement_library
for pants 2.8+):
Copy code
python_requirement_library(
    name="tensorflow",
    requirements=[
        "tensorflow==2.7.0; sys_platform != 'darwin' or platform_machine != 'arm64'",
        "tensorflow-macos==2.7.0; sys_platform == 'darwin' and platform_machine == 'arm64'",
        "tensorflow-metal==0.3.0; sys_platform == 'darwin' and platform_machine == 'arm64'",
    ],
)
• i needed to use tensorflow 2.7.0, any older version caused errors with h5py for some reason i think that covers it. I’d need to re-run these steps on a clean machine to see if i missed anything, but i’ve at least managed to get tensorflow making predictions on my machine now 🙂
unfortunately this doesn’t seem to be working non-locally anymore — I guess we’re not using pants and we might be installing only packages in poetry.lock instead? which means it can’t find the 
tensorflow
package, and unfortunately if i add it back into 
poetry.lock
then i’ll get a 
DuplicateNameError: A target already exists at 'PANTS.root' with name 'tensorflow' and target type 'python_requirement_library'. The 'python_requirement_library' target cannot use the same name.
i wonder if something along these lines might work? https://www.pantsbuild.org/docs/python-third-party-dependencies#warning-multiple-versions-of-the-same-dependency this seems similar to having different versions of the same package
e
unfortunately this doesn’t seem to be working non-locally anymore
Can you explain that more? By non-locally do you mean you check in all your configuration changes after getting Pants to work locally and then when someone else on a different machine pulls in those configuration changes you comitted, Pants does not work on their machine? Or is the non-local local Pants vs local poetry having differing success?
h
by non-locally i mean on AWS: we have jobs that run on EMR and it was failing to add the tensorflow package. I’m actually not 100% sure why that is — it might be that we’re not using pants there and so it only installs packages that are explicitly listed in 
poetry.lock
. my workaround ended up being to add back only tensorflow to 
pyproject.toml
(
tensorflow = { version = "2.5.1", markers = "sys_platform != 'darwin' or platform_machine != 'arm64'" }
) and then update our BUILD files for parts of the code that use tensorflow:
Copy code
python_requirement_library(
    name="tensorflow-macos",
    requirements=[
        "tensorflow-macos==2.7.0; sys_platform == 'darwin' and platform_machine == 'arm64'",
        "tensorflow-metal==0.3.0; sys_platform == 'darwin' and platform_machine == 'arm64'",
    ],
)

python_library(
    name="local-tensorflow",
    sources=["**/*.py"],
    dependencies=[":tensorflow-macos"],
)
the downside here is obviously that we have to explicitly add this to every place where tensorflow is used instead of being able to let pants infer when to use 
tensorflow-macos
h
@high-energy-55500 how viable would it be for you all to upgrade to Pants 2.9 or 2.10? I was going to try to add that new 
group_together
field today to pants 2.10 to help you. The benefit being that you don't need to modify 
pyproject.toml
at all
🎉 2
h
we’re planning on upgrading to 2.9 very soon, i think we only have a couple kinks to work out but a lot of the work in migrating to 2.8 has already been done so i think jumping to 2.9 should be pretty straightforward. having something like this in 2.9 would make us push up the migration even sooner
also, can i just say that the quality of support we get from pants is absolutely phenomenal! i don’t think i’ve ever worked with a tool that has such incredible support and development speed :)
❤️ 4
h
Awesome, glad to hear you're close to 2.8! To clarify, we wouldn't cherry-pick this feature to 2.9 because 2.9 is using this old deprecated macro mechanism for 
poetry_requirements
, which makes it harder to add this feature. I'd be adding it to 2.10, although that release series won't be stabilized till February probably
can i just say that the quality of support we get from pants is absolutely phenomenal! i don’t think i’ve ever worked with a tool that has such incredible support and development speed :)
Ah yay! Thanks 🙂 Would you be comfortable with us tweeting that and adding to https://www.pantsbuild.org/docs/testimonials? It helps a lot to get out the word
h
of course, please do!
❤️ 1
h
Ohhhh @high-energy-55500 I started implementing 
group_together
and realized there's a much simpler solution!! The 
overrides
field and `dependencies`:
Copy code
poetry_requirements(
  overrides={
     "tensorflow": {"dependencies": [":tensorflow-macos", ":tensorflow-metal"]},
  },
)
What this means is that you still have three distinct 
python_requirement
targets. But whenever something depends on 
tensorflow
, you'll end up also pulling in those two other dependencies. This is made possible by the 
overrides
feature added to Pants 2.8+. See https://www.pantsbuild.org/docs/python-third-party-dependencies#requirements-with-undeclared-dependencies
🎉 1
b
Thanks for that sentiment @high-energy-55500 and for allowing us to share it. It really keeps everyone on the team excited about what we're doing when we hear wonderful feedback like that. Much appreciation as well to the community that is so thoughtful about sharing so many constructive, innovative ideas for making Pants even better.
💯 2
@high-energy-55500 for the testimonials page, is "Martim Lobao, Data Scientist, People Data Labs" the correct attribution to use?
h
What this means is that you still have three distinct 
python_requirement
 targets. But whenever something depends on 
tensorflow
, you’ll end up also pulling in those two other dependencies. This is made possible by the 
overrides
 feature added to Pants 2.8+. See https://www.pantsbuild.org/docs/python-third-party-dependencies#requirements-with-undeclared-dependencies
@hundreds-father-404 that’s great! I’ll use 
overrides
as soon as we upgrade from 2.7.0! I believe our biggest blocker in upgrading has been the relative instability of the API for plugins, I think we have several internal plugins we depend on and they can apparently break in even minor versions (like in 2.7.1) @busy-vase-39202 yup, that sounds perfect! 🙂
h
Hm feel free to ask for help in the #plugins channel. Definitely the APIs do change between dev releases, but we try to not change them during stable patch releases like 2.7.0 and 2.7.1. Regardless of that, we're happy to help folks with issues upgrading
🙏 1
Open in Slack
PreviousNext
Powered by