wonderful-boots-93625
02/16/2023, 10:22 PMenough-analyst-54434
02/16/2023, 10:26 PMIf I use built distributions (wheels, sdist), the specific locked dependencies wont be followed in the docker build.I don't follow this bit.
wonderful-boots-93625
02/16/2023, 10:38 PMpip install
ing, which doesn’t obey any version constraints.enough-analyst-54434
02/16/2023, 10:40 PMpex_binary
field equivalents), you'll get a Mac PEX - it will never run on Linux.zipinfo my.pex
to get a quick take on what is inside. Or unzip -qc my.pex PEX-INFO | jq .
for a more concise summary. Either should make it somewhat clear to the eyeball what platforms the PEX will be compatible with.pex_binary
target, the base of the Docker image and the output of unzip -qc my.pex PEX-INFO | jq .
?wonderful-boots-93625
02/17/2023, 2:09 AMpex_binary(
name="bin",
dependencies=[
'iai_coral_api:dist'
]
)
where the dist
is a sdist
PEX-INFO
{
"bootstrap_hash": "00f63393678d425a75e44dcc8b4450b9efa581b0",
"build_properties": {
"pex_version": "2.1.108"
},
"code_hash": "969e4c40a854f5409021dce2a0d2c76fd3734026",
"distributions": {
"Flask-2.2.3-py3-none-any.whl": "7a0ed67f0c93cdb22b805e540d677c8cbeb89677ecc08d8dd975e35617ae6736",
"Jinja2-3.1.2-py3-none-any.whl": "ee8bc6748d864a510e8298d9de49c5e41b75faed18b5649089a1a09295a6c8ef",
"MarkupSafe-2.1.2-cp39-cp39-macosx_10_9_universal2.whl": "4596866d200e738c0e5dd211689c548ebecabfbe67f3ff794abc36da32cd3d11",
"MarkupSafe-2.1.2-cp39-cp39-macosx_10_9_x86_64.whl": "0474294d2f6f68d895eb163194e4e84cf44b7692c1974bd16fa8ca4cb2deed23",
"PyJWT-2.6.0-py3-none-any.whl": "803818df6174700b6f66d6844605deb4b6e63e509134c908b86121754cb2969f",
"PyYAML-6.0-cp39-cp39-macosx_10_9_x86_64.whl": "1a08337aaa0f63f4f94da401c57ec31408221212f19a7a4e10b32aeb82179233",
"PyYAML-6.0-cp39-cp39-macosx_11_0_arm64.whl": "e54244f0d511bf3413b517205880d0ae8131fa29c5e8fd91aeffbdc68312ea46",
"SQLAlchemy-2.0.3-cp39-cp39-macosx_10_9_x86_64.whl": "2eb45b1935cd46b2c133a9cb8d6d572a4903ed9d4d9b570b8fcafecbc192aff7",
"SQLAlchemy-2.0.3-cp39-cp39-macosx_11_0_arm64.whl": "0f36b8e7526fdbff27d95ffc4de58adec9224bd9c2fa3c6b5894d4a471dae0b1",
"Werkzeug-2.2.3-py3-none-any.whl": "144d40c05c7cb27d013739707020c617516c33c7bdfa58d96edce31db0802d15",
"attrs-22.2.0-py3-none-any.whl": "b7d965a018ed5fbe33529f7ffdd208e203bd38ae6313018bfef8b830e329941c",
"bcrypt-4.0.1-cp36-abi3-macosx_10_10_universal2.whl": "a7f1e8c691cd5626284833b106bb6e2084c19a07a0635a876992582d5cd56bde",
"boto3-1.24.59-py3-none-any.whl": "0d0ef28407f50a812c6cdbe43b2d6681650d470e4cada694ec3e97b85c7a97f8",
"botocore-1.27.59-py3-none-any.whl": "81bc68c3656887489cd9e2f6a0316bf4bb1c7e91baee53dee8262c8ae6983c8a",
"cachetools-5.3.0-py3-none-any.whl": "d6f15c6c02c2247d78a97e9dca4e23315bd6aa08cde7c5811281c19d374f121e",
"certifi-2022.12.7-py3-none-any.whl": "fa660808b834d51826f2d58643aad338e42b6d2bae69df8c8e5a4829dbeabdcf",
"charset_normalizer-3.0.1-cp39-cp39-macosx_10_9_x86_64.whl": "bb452e537f286bc912072293afcb53d87d5da9772067caa230ceb120e31511de",
"charset_normalizer-3.0.1-cp39-cp39-macosx_11_0_arm64.whl": "eb4dde55cb763c7620c53d5e302388f2029805c483893d832d2b5f7026c72b4f",
"click-8.1.3-py3-none-any.whl": "78086359bc4a576338dbcaacad4a42784cdd0755b6327b984812fe0913265abf",
"greenlet-2.0.2-cp39-cp39-macosx_11_0_x86_64.whl": "f486be16d5e3bb3800ff45694c838e38a4a01bf53fe669435df440313bf1bb3a",
"idna-3.4-py3-none-any.whl": "3baea0ffbdea9f349783bacbdb82dbd50eadfec75339388aaf1597754786e3ed",
"importlib_metadata-6.0.0-py3-none-any.whl": "ec3e6928e91d705f6ddc074bfa9ad570aa0f6a24d3a91fd71a0f89d9d7893674",
"isodate-0.6.1-py2.py3-none-any.whl": "466f37180566c0c09052e26da1928cffbebb8d6e924abeeb2415319c5d8a685c",
"itsdangerous-2.1.2-py3-none-any.whl": "d48f15bd9bedb89c478f00bd312dd8ffd8f8938b021d6e9f5931bfce833484ae",
"jmespath-1.0.1-py3-none-any.whl": "299c3a18595a39d4d54252e98a3c39343899b9bd1997e3eebfee63ea5757588b",
"jsonschema-4.17.3-py3-none-any.whl": "4f7345f7d4ddca08263841847e6a1dec970ce3003a8d6ec494cec2dd8553e843",
"jsonschema_spec-0.1.3-py3-none-any.whl": "77508a0f07354c60a01b0dd7e3ef75c778a78ecc600596c36c6b2a38d12633b2",
"lazy_object_proxy-1.9.0-cp39-cp39-macosx_10_9_x86_64.whl": "e8df964751b03bdc4469113b77c2468e318e17756d8e5f1e1751c156d436b28f",
"lazy_object_proxy-1.9.0-cp39-cp39-macosx_12_0_arm64.whl": "42c1a3a1bf6437454eba8c53718183b5d4d1e9a5b39ff751f34e986208411353",
"more_itertools-9.0.0-py3-none-any.whl": "e66b271a2ee3226b5a837c6a3404d830383d29531ac7a7dc94e888d78739b396",
"openapi_core-0.16.5-py3-none-any.whl": "875aee26a0e717c2b2d7ddfcf2050cb2c1f71db9547a2c0333adf612a9a3e8a7",
"openapi_schema_validator-0.4.3-py3-none-any.whl": "6fa063cb862b3e4622fe0bbf7c738e0890e5ea845b4df097d1fd0eba89e728c1",
"openapi_spec_validator-0.5.5-py3-none-any.whl": "82cd5d73f9d2f00e1630a2c5416827b62a91d2a7c672760e74323d5d67f0451b",
"packaging-23.0-py3-none-any.whl": "df561c71c9b367292028c9550ff03a8b3e86e87d512bdc038df97082ffd0c54e",
"parse-1.19.0-py3-none-any.whl": "191f6ce597e97678f39aeb7d86ed85e329f4eb8a7d182e7a6645541e5e12f69f",
"pathable-0.4.3-py3-none-any.whl": "b89676ae1e5df9738805871011313b39b18927b3692ad46a8de3f6a1aae58913",
"pynamodb-5.4.0-py3-none-any.whl": "c9dfb33bc00207901c8c738c651395a595e113fa86221a9900a3d2199d8637e4",
"pyrsistent-0.19.3-cp39-cp39-macosx_10_9_universal2.whl": "322d3a03a001072ae656437423c5eb4ecde22da8e4f661a111ae80f82e429440",
"python_dateutil-2.8.2-py2.py3-none-any.whl": "202c27a293331dd8fa9d41d1fcdd5ba4f4d6d2de0a2f00fa8547adc7c1aac629",
"pytz-2022.7.1-py2.py3-none-any.whl": "c869f82236c4cc4908fa51a57583cbf7dc66f2640740b206fc80c49644fabd86",
"requests-2.28.2-py3-none-any.whl": "bd5fbf4ad4e1d61ce4de27aeb31e0ba74111cd5869ffb0ad8f6e0caa99719d4d",
"rfc3339_validator-0.1.4-py2.py3-none-any.whl": "45ff252ce6f84b5576ff6a73b7c4a23e71e9c761de725276393a65252aa23b06",
"s3transfer-0.6.0-py3-none-any.whl": "be056641510d7effc911aad62944b4daf8b26281d992a395feb7e1046ed023b3",
"setuptools-67.3.2-py3-none-any.whl": "e33efc7291d8d297afb56a5445c7b9154035fd51948629a68c4799952c1237b5",
"six-1.16.0-py2.py3-none-any.whl": "3e1c439c88d2e7681372427bab751b3fc99969891e95a714fed9604bf7710213",
"typing_extensions-4.5.0-py3-none-any.whl": "b46af7f121ca528ad7180c389d9ace75b11294dc75989f8d553eb6474bafadd8",
"urllib3-1.26.14-py2.py3-none-any.whl": "4ac66d91e22d55749a6cfe1fe599d1e7d39d2f16e2b8309ada5a3b57074153e2",
"watchtower-3.0.1-py3-none-any.whl": "243912354d24754c3d4c1bdfb5583802f5933c3d9f1dde133643cbb362059f99",
"zipp-3.13.0-py3-none-any.whl": "1e51538915a09fc2e49ef6fa620916fb1d0d7c15ad61131e65e555d5b9bd3202"
},
"emit_warnings": false,
"ignore_errors": false,
"includes_tools": false,
"inherit_path": "false",
"interpreter_constraints": [
"CPython==3.9.*"
],
"pex_hash": "41148ecf8b1f29a47b754349832b13b41c38bac0",
"pex_path": "",
"pex_paths": [],
"requirements": [
"Flask",
"PyJWT",
"PyYAML",
"SQLAlchemy",
"bcrypt",
"boto3",
"botocore",
"cachetools",
"openapi-core",
"packaging",
"pynamodb",
"python-dateutil",
"pytz",
"requests",
"setuptools",
"watchtower"
],
"strip_pex_env": true,
"venv": false,
"venv_bin_path": "false",
"venv_copies": false,
"venv_site_packages_copies": false
}
$ docker run -it --rm iai/iai-coral-api bash
root@ef2d1c886001:/# /bin/iai_coral_api.pex
Failed to find compatible interpreter on path /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin.
Examined the following interpreters:
1.) /usr/local/bin/python3.9 CPython==3.9.16
No interpreter compatible with the requested constraints was found:
A distribution for pyyaml could not be resolved for /usr/local/bin/python3.9.
Found 2 distributions for pyyaml that do not apply:
1.) The wheel tags for PyYAML 6.0 are cp39-cp39-macosx_10_9_x86_64 which do not match the supported tags of /usr/local/bin/python3.9:
cp39-cp39-manylinux_2_28_aarch64
... 304 more ...
2.) The wheel tags for PyYAML 6.0 are cp39-cp39-macosx_11_0_arm64 which do not match the supported tags of /usr/local/bin/python3.9:
cp39-cp39-manylinux_2_28_aarch64
... 304 more ...
enough-analyst-54434
02/17/2023, 2:17 AMmacosx_*
in it - see many in your PEX-INFO.
When the PEX goes to boot up on Linux, it tries to resolve Linux platform specific dependencies.
A line like this, then, hopefully makes more sense then:
1.) The wheel tags for PyYAML 6.0 are cp39-cp39-macosx_10_9_x86_64 which do not match the supported tags of /usr/local/bin/python3.9:
cp39-cp39-manylinux_2_28_aarch64
That says - "Hey, we're on Linux using a local python3.9 interpreter and you're trying to tell us to use a macosx wheel!? Good luck!"platforms
or complete_platforms
fields on your pex_binary
where the platform you're specifying is the foreign Linux platform inside your container. See: https://www.pantsbuild.org/docs/reference-pex_binary#codecomplete_platformscodecomplete_platforms
at.wonderful-boots-93625
02/17/2023, 2:25 AMpip install
from within the Dockerfile. Doesn’t cover hash checks, but probably ok for us for now.enough-analyst-54434
02/17/2023, 2:26 AMpex3 lock export
will get you a requirements.txt with hashes. Unfortunately, Pants gets in the way here. You will 1st need to strip off the invalid JSON comment header it adds 😕pex3
by pip installing pex
.wonderful-boots-93625
02/17/2023, 2:28 AMenough-analyst-54434
02/17/2023, 2:29 AMwonderful-boots-93625
02/17/2023, 2:43 AM--platform
to the docker_image
target?enough-analyst-54434
02/17/2023, 2:53 AMpex --lock monorepo.lock req1 req2 -o subset.pex --include-tools && PEX_TOOLS=1 ./subset.pex venv create/a/venv/from/a/subset/of/the/lock/right/here
wonderful-boots-93625
02/17/2023, 3:35 AMPEX_TOOLS=1 /bin/iai_coral_api.pex venv my-venv
doesn’t seem to work because that would be an easy fix.enough-analyst-54434
02/17/2023, 3:48 AMPEX_TOOLS=1 /this/python/please /bin/iai_coral_api.pex venv my-venv
.wonderful-boots-93625
02/17/2023, 3:52 AMroot@bcb2e6148c65:/# PEX_TOOLS=1 /usr/local/bin/python /bin/iai_coral_api.pex -h
Failed to find compatible interpreter on path /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin.
Examined the following interpreters:
1.) /usr/local/bin/python3.9 CPython==3.9.16
No interpreter compatible with the requested constraints was found:
A distribution for pyyaml could not be resolved for /usr/local/bin/python3.9.
Found 2 distributions for pyyaml that do not apply:
1.) The wheel tags for PyYAML 6.0 are cp39-cp39-macosx_10_9_x86_64 which do not match the supported tags of /usr/local/bin/python3.9:
cp39-cp39-manylinux_2_28_aarch64
... 304 more ...
2.) The wheel tags for PyYAML 6.0 are cp39-cp39-macosx_11_0_arm64 which do not match the supported tags of /usr/local/bin/python3.9:
cp39-cp39-manylinux_2_28_aarch64
... 304 more ...
I believe the check is still being done first. Regardless I don’t see how this will work, since I don’t see version information for the packages in the INFO, just a listing of wheels, and requirements without version pins.
ultimately - I’m trying to get the version pinned list of requirements out of the pex without any of the platform constraints, and then call a pip install
to install the compatible versions.enough-analyst-54434
02/17/2023, 3:54 AMpex --lock monorepo.lock req1 req2 -o subset.pex --include-tools
I mentioned. That is required 1st. And that is to be done inside the container.wonderful-boots-93625
02/17/2023, 3:55 AMreq1 req2….reqN
from the pex info?enough-analyst-54434
02/17/2023, 3:55 AMwonderful-boots-93625
02/17/2023, 3:55 AMenough-analyst-54434
02/17/2023, 3:55 AM&&
chained command is to be run inside the container: https://pantsbuild.slack.com/archives/C046T6T9U/p1676602414889059?thread_ts=1676586163.756759&cid=C046T6T9Uwonderful-boots-93625
02/17/2023, 3:56 AMenough-analyst-54434
02/17/2023, 4:00 AMdependencies
goal (https://www.pantsbuild.org/docs/reference-dependencies) and the peek
goal (https://www.pantsbuild.org/docs/reference-peek). I think this can maybe be done with peek
but I'm not sure. Definitely Approaches #1 & #2 I mentioned above are way more standard.wonderful-boots-93625
02/17/2023, 4:04 AMenough-analyst-54434
02/17/2023, 4:07 AMwonderful-boots-93625
02/17/2023, 4:09 AMenough-analyst-54434
02/17/2023, 4:09 AM