plain-carpet-73994
10/14/2021, 6:50 PMopencv-python
but there isn't an ARM build for that on PyPi. You can apt-get install python3-opencv
but pants won't pick that up. I do understand why: it breaks the hermetic build idea as you're using an outside dependency that could be updated via apt
at any time. You can't can't correctly generate a constraints file because whatever tool you're using (e.g. pip freeze
or pip-compile
) wouldn't be aware of one of the dependencies and it's requirements/version. But, getting it to build on ARM looks like it's going to be a big time suck. Is there an "escape hatch" that allows one to use a globally installed Python library?
("no" is a perfectly reasonable answer here I think, was just hoping...)hundreds-father-404
10/14/2021, 6:53 PM.whl
fileplain-carpet-73994
10/14/2021, 7:00 PMapt
it just installs an .so
file:
# dpkg -L python3-opencv
/.
/usr
/usr/lib
/usr/lib/python3
/usr/lib/python3/dist-packages
/usr/lib/python3/dist-packages/cv2.cpython-39-aarch64-linux-gnu.so
/usr/share
/usr/share/doc
/usr/share/doc/python3-opencv
/usr/share/doc/python3-opencv/changelog.Debian.gz
/usr/share/doc/python3-opencv/copyright
but I don't get any of the metadata (e.g. it's 3rd party dependencies, etc.) that I'd need to build a wheel. To build it into a wheel I think the quickest path to success would be to checkout the OpenCV source and try to setup.py bdist_wheel
but I suspect that's going to end up taking a bunch of time (e.g. ARM was far less popular when the version of OpenCV we need was released) so I was hoping for a "morally wrong but expedient" way to tell Pants to just use the dependency as installed by apt
. Does that make sense?hundreds-father-404
10/14/2021, 7:02 PM.so
I think you'd need to build the wheelflat-zoo-31952
10/14/2021, 7:03 PMhappy-kitchen-89482
10/14/2021, 8:10 PMpython3-opencv
what does that actually do?flat-zoo-31952
10/14/2021, 8:45 PMapt-get install python3-opencv
will install the required files in one of those configured dirs (probably something like /usr/lib/python3.x/site-packages
)plain-carpet-73994
10/14/2021, 8:46 PMHow does the system python find and consume that requirement?
# dpkg -L python3-opencv
/.
/usr
/usr/lib
/usr/lib/python3
/usr/lib/python3/dist-packages
/usr/lib/python3/dist-packages/cv2.cpython-39-aarch64-linux-gnu.so
/usr/share
/usr/share/doc
/usr/share/doc/python3-opencv
/usr/share/doc/python3-opencv/changelog.Debian.gz
/usr/share/doc/python3-opencv/copyright
It puts in /usr/lib/python3/dist-packages
which is on $PYTHONPATH
.flat-zoo-31952
10/14/2021, 8:47 PMYes, using globally installed Python things is wrong.It's not wrong, it just a choice that has its benefits and costs. For a lot of people these days sticking with system-installed packages has far more costs than benefits, so that often gets distilled into the idea that it's wrong. I think it's a reasonable thing for Pants to want to support some day.
python_system_library(
name = "opencv-system",
python_binaries = ["/usr/bin/python3.8"],
packages = ["cv2"],
install_command = "sudo apt-get install -y python3-opencv2",
)
I just don't know how this would interact with pants' interpreter searchinghappy-kitchen-89482
10/14/2021, 9:28 PMenough-analyst-54434
10/14/2021, 9:37 PM--inherit-path {false,prefer,fallback}
or PEX_INHERIT_PATH
and then PEX_EXTRA_SYS_PATH
to add arbitrary thingshappy-kitchen-89482
10/14/2021, 9:48 PMplain-carpet-73994
10/14/2021, 11:42 PMrequirements.txt
docs say the only solution is to fork the .whl: https://pip.pypa.io/en/stable/topics/dependency-resolution/#loosen-the-requirements-of-your-dependencies. Does pants give me a way to say "loosen opencv's constraint on numpy"?enough-analyst-54434
10/14/2021, 11:46 PMplain-carpet-73994
10/14/2021, 11:46 PMenough-analyst-54434
10/14/2021, 11:48 PMplain-carpet-73994
10/14/2021, 11:49 PMenough-analyst-54434
10/14/2021, 11:50 PMflat-zoo-31952
10/15/2021, 12:17 AMhappy-kitchen-89482
10/15/2021, 12:18 AMenough-analyst-54434
10/15/2021, 12:35 AM--inherit-path
adds everything in the interpreter's natural sys.path
right back.happy-kitchen-89482
10/15/2021, 1:01 AMplain-carpet-73994
10/15/2021, 1:33 AMWhat about a route of doing local wheel build off a source tarball or python sdist?That would be super helpful!! I've already hit another case where I had to manually build a wheel 'cause all I could find was an sdist. And then I had to set up a private pypi repo just for that one dependency, figure out the config, etc. In the end it cost me a day or two where an sdist option would have eliminated all that work.
Or we could just do it globally, that would solve the user's problem in practice
at the expense of hermeticityPersonally, I'd prefer a targetted "just this one dependency" thing to limit the potential impact of loss of hermeticity.
enough-analyst-54434
10/15/2021, 3:46 PM--no-binary opencv-python
and plumbing for it in Pants, because today I can turn of wheels globally with Pex just fine (this uses Pip's --no-binary :all:
under the covers) albeit at a glacial pace since opencv-python and numpy get compiled from sdists:
$ pex --no-wheel opencv-python -o opencv-python.pex
$ pex-tools opencv-python.pex info -i4
{
"bootstrap_hash": "8fb7e13eebcc79897c76921aa95cab8d49a4f71b",
"build_properties": {
"pex_version": "2.1.52"
},
"code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
"distributions": {
"numpy-1.21.2-cp39-cp39-linux_x86_64.whl": "01f5a73964e180ad641ad496e8d412501eba2ae5",
"opencv_python-4.5.3.56-cp39-cp39-linux_x86_64.whl": "32593f38258b17ce59ec9395aba04a9991f6a64f"
},
"emit_warnings": true,
"ignore_errors": false,
"includes_tools": false,
"inherit_path": "false",
"interpreter_constraints": [],
"pex_hash": "a669293b22db7cbd5dd3c11f9d225e859a701658",
"pex_path": null,
"requirements": [
"opencv-python"
],
"strip_pex_env": true,
"venv": false,
"venv_bin_path": "false",
"venv_copies": false,
"pex_root": "/home/jsirois/.pex"
}
--build / --no-build / --wheel / --no-wheel
though in Pants: https://github.com/pantsbuild/pants/issues/5862. That still just gets you universal sdist use though (via --no-wheel
like I used above). For just selected sdist use, there's https://github.com/pantsbuild/pants/issues/12090.