https://pantsbuild.org/ logo
Join Slack
Powered by
Hit a weird one today...while trying to generate l...
# general
f
Hit a weird one today...while trying to generate lockfiles that include the doit package at a specific version (0.33.1), I'm getting a dependency compilation error
Copy code
_fsevents.c:2:10: fatal error: CoreFoundation/CoreFoundation.h: No such file or directory
      2 | #include <CoreFoundation/CoreFoundation.h>
        |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for MacFSEvents
Thing is, I'm on Linux, so I have no idea why it's trying to install or compile 
MacFSEvents
. Looks like this is in a platform-conditional extras_require bit in the setup for doit, which I didn't even know was a thing. But it appears Pants doesn't know how to handle this. I should probably report this as a bug, but I'm posting here for feedback if we've run into this kinda thing before?
✅ 1
I'm not really sure how to work around this
So it appears to be what Pants is passing to pex ...
Copy code
/usr/bin/python3.9 ./pex lock create --tmpdir .tmp --python-path /usr/bin $'--output=lock.json' --no-emit-warnings $'--style=universal' --resolver-version pip-2020-resolver --target-system linux --target-system mac $'--indent=2' --no-pypi $'--index=<https://pypi.org/simple/>' --manylinux manylinux2014 --interpreter-constraint $'CPython==3.10.*' --interpreter-constraint $'CPython==3.11.*' $'doit~=0.33.1' $'fastapi~=0.84.0' $'filelock~=3.7.1' $'httpx~=0.22.0' $'more-itertools~=8.12.0' $'packaging~=21.3' $'paramiko~=2.12.0' $'pydantic~=1.10.2' $'uvicorn~=0.15.0'
h
I guess the "at some point" is now...
the lockfile does have to be compatible with all platforms you intend to use it on
f
Found it in the code. Rabbit hole takes me to... • https://github.com/pantsbuild/pex/issues/1821 • https://github.com/pantsbuild/pants/issues/17276 I can look at tackling this maybe
h
so the assumption is that you're not planning to use it on macos
f
I mean I might want to use it on MacOS, but I would need some kinda 
resolves_to_platform
mapping, which probably necessitates a different lockfile for different platforms, which also means changes to 
[python.resolves]
But IIUC, "the lockfile does have to be compatible with all platforms you intend to use it on" means that "lockfiles don't work with platform-specific deps"... unless you can cook up some environment that has deps needed for all platforms installed, which may not always be possible, especially for proprietary mac libs or kernel-specific linux libs
h
They do if you pre-build the necessary wheels
f
Ah okay, and then configure another index or wheelhouse to grab those wheels. Seems like a pain in my case, but it is a workaround. Turns out in my case that there's a version that doesn't need this macfsevents dep, so that's my workaround. But I wonder how valuable this would be in a general case. I'll take the discussion to those issues I linked I guess
e
So, only PEX 2.1.122+ are smart enough to do less for sdists when locking: https://github.com/pantsbuild/pex/releases/tag/v2.1.122 What version of Pants are you on?
Pex 2.1.121:
Copy code
$ pex pex==2.1.121 -cpex3 -- lock create --style universal --target-system linux --target-system mac --resolver-version pip-2020-resolver doit==0.33.1 -o lock.json --indent 2
pid 668481 -> /home/jsirois/.pex/venvs/d934e3610a9d9ea1b88684f2de3ec2c4b3b706ef/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python -sE /home/jsirois/.pex/venvs/d934e3610a9d9ea1b88684f2de3ec2c4b3b706ef/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/jsirois/.pex/pip_cache wheel --no-deps --wheel-dir /home/jsirois/.pex/built_wheels/sdists/MacFSEvents-0.8.4.tar.gz/bf7283f1d517764ccdc8195b21631dbbac1c506b920bf9a8ea2956b3127651cb/cp310-cp310-manylinux_2_35_x86_64.fa0ed2d81bd647588d4077d0fc01b4f3.work /tmp/tmppzg7jvll/usr.bin.python3.10/MacFSEvents-0.8.4.tar.gz --index-url <https://pypi.org/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
  ERROR: Command errored out with exit status 1:
   command: /home/jsirois/.pex/venvs/d934e3610a9d9ea1b88684f2de3ec2c4b3b706ef/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/jsirois/.pex/pip_cache/.tmp/pip-req-build-q_2r7bc6/setup.py'"'"'; __file__='"'"'/home/jsirois/.pex/pip_cache/.tmp/pip-req-build-q_2r7bc6/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /home/jsirois/.pex/pip_cache/.tmp/pip-wheel-am_g8go5
       cwd: /home/jsirois/.pex/pip_cache/.tmp/pip-req-build-q_2r7bc6/
  Complete output (15 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.10
  copying fsevents.py -> build/lib.linux-x86_64-3.10
  running build_ext
  building '_fsevents' extension
  creating build/temp.linux-x86_64-3.10
  x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/jsirois/.pex/venvs/d934e3610a9d9ea1b88684f2de3ec2c4b3b706ef/5985ed09b49a653d6596b0e14d134c5456cf1a9f/include -I/usr/include/python3.10 -c _fsevents.c -o build/temp.linux-x86_64-3.10/_fsevents.o
  _fsevents.c:2:10: fatal error: CoreFoundation/CoreFoundation.h: No such file or directory
      2 | #include <CoreFoundation/CoreFoundation.h>
        |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for MacFSEvents
ERROR: Failed to build one or more wheels
Pex 2.1.122:
Copy code
$ pex pex==2.1.122 -cpex3 -- lock create --style universal --target-system linux --target-system mac --resolver-version pip-2020-resolver doit==0.33
.1 -o lock.json --indent 2
running dist_info
running dist_info
creating /tmp/tmpog4xcem7/build/pyinotify.egg-info
writing /tmp/tmpog4xcem7/build/pyinotify.egg-info/PKG-INFO
writing dependency_links to /tmp/tmpog4xcem7/build/pyinotify.egg-info/dependency_links.txt
writing top-level names to /tmp/tmpog4xcem7/build/pyinotify.egg-info/top_level.txt
writing manifest file '/tmp/tmpog4xcem7/build/pyinotify.egg-info/SOURCES.txt'
creating /tmp/tmphsbi4_l9/build/MacFSEvents.egg-info
writing /tmp/tmphsbi4_l9/build/MacFSEvents.egg-info/PKG-INFO
writing dependency_links to /tmp/tmphsbi4_l9/build/MacFSEvents.egg-info/dependency_links.txt
writing top-level names to /tmp/tmphsbi4_l9/build/MacFSEvents.egg-info/top_level.txt
writing manifest file '/tmp/tmphsbi4_l9/build/MacFSEvents.egg-info/SOURCES.txt'
reading manifest file '/tmp/tmpog4xcem7/build/pyinotify.egg-info/SOURCES.txt'
reading manifest template '<http://MANIFEST.in|MANIFEST.in>'
writing manifest file '/tmp/tmpog4xcem7/build/pyinotify.egg-info/SOURCES.txt'
creating '/tmp/tmpog4xcem7/build/pyinotify.dist-info'
reading manifest file '/tmp/tmphsbi4_l9/build/MacFSEvents.egg-info/SOURCES.txt'
writing manifest file '/tmp/tmphsbi4_l9/build/MacFSEvents.egg-info/SOURCES.txt'
creating '/tmp/tmphsbi4_l9/build/MacFSEvents.dist-info'
adding license file "COPYING" (matched pattern "COPYING*")
adding license file "LICENSE.txt" (matched pattern "LICEN[CS]E*")

$ jq . lock.json
{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "61f594d1f4c295fa5cd9014ceb3a1fc4a70b0de1164b94fbc2d854ccba056f9f",
              "url": "<https://files.pythonhosted.org/packages/15/80/44286939ca215e88fa827b2aeb6fa3fd2b4a7af322485c7170d6f9fd96e0/cloudpickle-2.2.1-py3-none-any.whl>"
            },
            {
              "algorithm": "sha256",
              "hash": "d89684b8de9e34a2a43b3460fbca07d09d6e25ce858df4d5a44240403b6178f5",
              "url": "<https://files.pythonhosted.org/packages/5f/51/913ecca3970a2227cf4d5e8937df52cc28f465ac442216110b8e3323262d/cloudpickle-2.2.1.tar.gz>"
            }
          ],
          "project_name": "cloudpickle",
          "requires_dists": [],
          "requires_python": ">=3.6",
          "version": "2.2.1"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "211fc0de3fd9ee31e5c4ccb36bc1a4054b5c4a4a44f915ca413896155b684bfa",
              "url": "<https://files.pythonhosted.org/packages/b9/af/3540a68099c4b8c3a9b80b3e1fdfeed7a920d97d9cb2038bd35278de06ed/doit-0.33.1-py3-none-any.whl>"
            },
            {
              "algorithm": "sha256",
              "hash": "37c3b35c2151647b968b2af24481112b2f813c30f695366db0639d529190a143",
              "url": "<https://files.pythonhosted.org/packages/0a/b3/fcbb2e07f36e6a1c32b897c4bdc2db0e6da4a85a4fec820a5a01fd7fc2cd/doit-0.33.1.tar.gz>"
            }
          ],
          "project_name": "doit",
          "requires_dists": [
            "cloudpickle",
            "macfsevents; sys_platform == \"darwin\"",
            "pyinotify; sys_platform == \"linux\""
          ],
          "requires_python": ">=3.5",
          "version": "0.33.1"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "bf7283f1d517764ccdc8195b21631dbbac1c506b920bf9a8ea2956b3127651cb",
              "url": "<https://files.pythonhosted.org/packages/f3/80/0d780e46286819558305c3d698f0c700cba2af0b04e86840f528cc302802/MacFSEvents-0.8.4.tar.gz>"
            }
          ],
          "project_name": "macfsevents",
          "requires_dists": [],
          "requires_python": null,
          "version": "0.8.4"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "9c998a5d7606ca835065cdabc013ae6c66eb9ea76a00a1e3bc6e0cfe2b4f71f4",
              "url": "<https://files.pythonhosted.org/packages/e3/c0/fd5b18dde17c1249658521f69598f3252f11d9d7a980c5be8619970646e1/pyinotify-0.9.6.tar.gz>"
            }
          ],
          "project_name": "pyinotify",
          "requires_dists": [],
          "requires_python": null,
          "version": "0.9.6"
        }
      ],
      "platform_tag": null
    }
  ],
  "path_mappings": {},
  "pex_version": "2.1.122",
  "pip_version": "20.3.4-patched",
  "prefer_older_binary": false,
  "requirements": [
    "doit==0.33.1"
  ],
  "requires_python": [],
  "resolver_version": "pip-2020-resolver",
  "style": "universal",
  "target_systems": [
    "linux",
    "mac"
  ],
  "transitive": true,
  "use_pep517": null
}
Although 
pex
often needs pre-built wheels, `pex3 lock create`should almost never. The setup.py would have to be pretty nasty / broken to trip it up. There are some of those out there in the wild, but not many afaict.
@flat-zoo-31952 the bug you found is fixed and added 
--target-system
. The bug you were looking for was https://github.com/pantsbuild/pex/issues/2050 - but also fixed as of 2.1.122.
👀 1
f
I’m on pants 2.15.1 there. I suppose I could pull a newer version of pants to build the lockfile and then jump back to our current version
e
You can upgrade to whatever Pex in any Pants
Just a sec...
Here is the setup for current 2.17.x (https://www.pantsbuild.org/v2.17/docs/reference-pex-cli#known_versions):
Copy code
[pex-cli]
version = "v2.1.134"
known_versions = [
  "v2.1.134|macos_arm64|82b24645769c19483c1306c1ba7a888471a5e1df3a2b538788bc7e0d1b20dbf0|4085867",
  "v2.1.134|macos_x86_64|82b24645769c19483c1306c1ba7a888471a5e1df3a2b538788bc7e0d1b20dbf0|4085867",
  "v2.1.134|linux_x86_64|82b24645769c19483c1306c1ba7a888471a5e1df3a2b538788bc7e0d1b20dbf0|4085867",
  "v2.1.134|linux_arm64|82b24645769c19483c1306c1ba7a888471a5e1df3a2b538788bc7e0d1b20dbf0|4085867"
]
Pex is only up to 2.1.137, so that's pretty fresh there to try out.
Just add that to 
pants.toml
2.1.137 (latest) has a hash of faad51a6a108fba9d40b2a10e82a2646fccbaf8c3d9be47818f4bffae02d94b8 and a size of 4098329: https://github.com/pantsbuild/pex/actions/runs/4968020531/jobs/8890381267#step:4:121
If you want to try that.
f
finally got around to trying this, it works with a newer pex, thanks!
3 Views
Open in Slack
PreviousNext
Powered by