https://pantsbuild.org/ logo
Join Slack
Powered by
`pynput` dependency causes a problem on macos. If ...
# general
a
pynput
dependency causes a problem on macos. If i add it to my 
requrements.txt
and run 
./pant generate-lockfiles
it fails. Looks like it is sdist and depenends on 
evdev
, that can only be built on linux (looks for linux kernel headers). What would be the correct way to solve this?
./pants generate-lockfiles
Copy code
14:37:47.19 [INFO] Completed: Generate lockfile for python-default
14:37:47.20 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
pid 28517 -> /Users/vova/.cache/pants/named_caches/pex_root/venvs/cac1718c056bb509f51fcdcc0c376b33deaaa8ec/c5cc082d0e1f388d750df8e9135b5e6451330091/bin/python -sE /Users/vova/.cache/pants/named_caches/pex_root/venvs/cac1718c056bb509f51fcdcc0c376b33deaaa8ec/c5cc082d0e1f388d750df8e9135b5e6451330091/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /Users/vova/.cache/pants/named_caches/pex_root/pip_cache wheel --no-deps --wheel-dir /Users/vova/.cache/pants/named_caches/pex_root/built_wheels/sdists/evdev-1.6.1.tar.gz/299db8628cc73b237fc1cc57d3c2948faa0756e2a58b6194b5bf81dc2081f1e3/cp39-cp39-macosx_13_0_x86_64.dab8db1956564fb2a2fee235c31335d3 /private/var/folders/1r/vwvwzp312ngc1kfys_7g6x_c0000gp/T/pants-sandbox-YsEKqU/.tmp/tmpzvie6l69/Users.vova..pyenv.versions.3.9.7.bin.python3.9/evdev-1.6.1.tar.gz --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
  ERROR: Command errored out with exit status 1:
   command: /Users/vova/.cache/pants/named_caches/pex_root/venvs/cac1718c056bb509f51fcdcc0c376b33deaaa8ec/c5cc082d0e1f388d750df8e9135b5e6451330091/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/vova/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-hddmd5sj/setup.py'"'"'; __file__='"'"'/Users/vova/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-hddmd5sj/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /Users/vova/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-wheel-p4vaip09
       cwd: /Users/vova/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-hddmd5sj/
  Complete output (37 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-12.1-x86_64-3.9
  creating build/lib.macosx-12.1-x86_64-3.9/evdev
  copying evdev/ff.py -> build/lib.macosx-12.1-x86_64-3.9/evdev
  copying evdev/evtest.py -> build/lib.macosx-12.1-x86_64-3.9/evdev
  copying evdev/device.py -> build/lib.macosx-12.1-x86_64-3.9/evdev
  copying evdev/util.py -> build/lib.macosx-12.1-x86_64-3.9/evdev
  copying evdev/events.py -> build/lib.macosx-12.1-x86_64-3.9/evdev
  copying evdev/ecodes.py -> build/lib.macosx-12.1-x86_64-3.9/evdev
  copying evdev/__init__.py -> build/lib.macosx-12.1-x86_64-3.9/evdev
  copying evdev/uinput.py -> build/lib.macosx-12.1-x86_64-3.9/evdev
  copying evdev/genecodes.py -> build/lib.macosx-12.1-x86_64-3.9/evdev
  copying evdev/eventio.py -> build/lib.macosx-12.1-x86_64-3.9/evdev
  copying evdev/eventio_async.py -> build/lib.macosx-12.1-x86_64-3.9/evdev
  running build_ext
  running build_ecodes
  The 'linux/input.h' and 'linux/input-event-codes.h' include files
  are missing. You will have to install the kernel header files in
  order to continue:

      yum install kernel-headers-$(uname -r)
      apt-get install linux-headers-$(uname -r)
      emerge sys-kernel/linux-headers
      pacman -S kernel-headers

  In case they are installed in a non-standard location, you may use
  the '--evdev-headers' option to specify one or more colon-separated
  paths. For example:

      python setup.py \
        build \
        build_ecodes --evdev-headers path/input.h:path/input-event-codes.h \
        build_ext --include-dirs  path/ \
        install
  ----------------------------------------
  ERROR: Failed building wheel for evdev
ERROR: Failed to build one or more wheels




Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
e
The concept you're missing is not Pants specific. You'll need to use an environment marker in your requirement string to only activate the requirement for linux: https://peps.python.org/pep-0508/#environment-markers
a
hmm, but i want this dependency for macos as well. But seems that 
pynput
itself, has different dependencies based on platform. 
pip install pynput
, for example, works
e
But... yeah, the sdist thing. The lock process still needs metadata and Python is broken here and requires partially building the sdist to get metadata.
So you must generate locks on Linux or provide a pre-built wheel for the sdist.
Yeah, I can think of no way around this besides generating the lock on Linux.
The wheel pre-build will be tricky.
a
got it, thx. I'm wary of the opposite situation. What'll happen if i'll have another dependency that i can only build on macos? I'll be in a deadlock
e
It's true.
a
perhaps pre-build wheels after all? Is there an example somewhere that i can copy from?
e
It's not a cookie cutter thing and so there aren't. Conceptually 2 things are needed, but the 0th thing is dependency metadata for the sdist. To get that without Pip trying to build the sdist on Mac: 1. Build a wheel in a Linux container. Roughly 
pythonX.Y --mvenv venv && venv/bin/pip wheel --no-deps<requirement>
2. Host the wheel somewhere and use 
[python-repos] *
to add that self hosted distribution repo to the search list: https://www.pantsbuild.org/docs/reference-python-repos (For 1 X.Y should be an interpreter meeting you Linux production interpreter constraints. If there are multiple of those - you use both 3.9 & 3.10 in production, you should do this once per each but can get away with not doing that if your Linux CI? environments successfully build the sdist). The difficulty here will be getting Mac Pip to pick the Linux wheel 1st as the metadata source and not the sdist. That may just work out but it may require further options too.
a
thank you so much! I'll give it a go
e
It may be the case Pex could be improved here. This is all from my phone, but I notice the failing command is not 
pip download
(which Pex uses to form the lock resolve) but 
pip wheel
, which, I think, Pex uses to get post-processing extra metadata? If so, that could be narrowed to the PEP'd equivalent of ~`python setup.py egg_info` which does a ~partial eval of setup.py to generate dependency metadata. I'll have a look at that the 2nd half of this coming week when I get back to a keyboard if you're still having trouble here and file an issue.
a
sounds great. I appreciate your help🙏 I'll report here with my findings / solutions
e
Ok, so I just circled back here and tested out evdev / re-examined Pex source code. It is the case that Pex builds a wheel from every sdist-only project it resolves when building the lock in order to get access to a METADATA file with dependency metadata needed to complete the lock. That is what errors in your OP here. And there is a way to gather this METADATA without fully building a wheel, by using https://peps.python.org/pep-0517/#prepare-metadata-for-build-wheel . I tried this with evdev 1.6.1 and it generated a proper METADATA file for evdev without trying to build its C-extensions; so I think this is what Pex should be doing in general for sdists - at least for the 
--style universal
locks Pants uses. I'll file a Pex issue shortly. Once I ship that I'll update you here with Pants configuration you need to try out the change / new feature - not sure which approach I'll use yet.
Alright @alert-psychiatrist-14102 a fix for this 
evdev
case is now released in Pex 2.1.122 and will be available in the next Pants 2.16.x release. You can try it out early with this 
pants.toml
config:
Copy code
[pex-cli]
version = "v2.1.122"
known_versions = [
  "v2.1.122|macos_arm64|2646c982e908cb3a2b0e4149878a4658539e8e2e6dde2534aeb573be1b0ebdf9|4075440",
  "v2.1.122|macos_x86_64|2646c982e908cb3a2b0e4149878a4658539e8e2e6dde2534aeb573be1b0ebdf9|4075440",
  "v2.1.122|linux_x86_64|2646c982e908cb3a2b0e4149878a4658539e8e2e6dde2534aeb573be1b0ebdf9|4075440",
  "v2.1.122|linux_arm64|2646c982e908cb3a2b0e4149878a4658539e8e2e6dde2534aeb573be1b0ebdf9|4075440"
]
a
Wow, what a quick turnaround! Thank you so much @enough-analyst-54434, it works for me:)
Open in Slack
PreviousNext
Powered by