https://pantsbuild.org/ logo
Join Slack
Powered by
I ran into the pex > docker platform challenge ...
# general
a
I ran into the pex > docker platform challenge today. I am working from a Mac but distributing via Docker. I was able to find answers by searching this Slack workspace and succeeded in configuring working Docker images derived from pex packaged python code. I'd love to produce a universal pex file that can be run on Mac x86, Mac Silicon or Linux hosted via Docker. Is that possible? My BUILD looks like so:
Copy code
# Works inside Docker
pex_binary(
    name="eeg_processor",
    entry_point="app.py",
    complete_platforms = [
        'lib_third_party/platforms:linux-py38',
    ],
)

# Does not work: A distribution for confluent-kafka could not be resolved
pex_binary(
    name="eeg_processor_mac",
    entry_point="app.py",
    complete_platforms = [
        'lib_third_party/platforms:macos_x86_64',
    ],
)

pex_binary(
    name="eeg_processor_default",
    entry_point="app.py",
)

docker_image(
    name="neuroedge_eeg_processor",
    image_tags=[
        "neuroedge_eeg_processor", 
        # TODO: is this tag useful? "<http://optios.jfrog.io/default-docker-local/neuroedge_eeg_processor:0.1.0|optios.jfrog.io/default-docker-local/neuroedge_eeg_processor:0.1.0>"
    ],
)
platform _macos_x86_64_ was produced using 
pex3 interpreter inspect --markers --tags
It does not produce a working pex file. platform linux-py38 was produced using 
docker run --rm -it python:3.8 sh -c 'python -mvenv venv && venv/bin/pip -q install pex && venv/bin/pex3 interpreter inspect --markers --tags -i2'
and works inside a simple linux base Docker container. linux-py38 works correctly when run from inside a Docker container. 
./pants run realtime/eeg_processor:neuroedge_eeg_processor
./pants run realtime/eeg_processor:eeg_processor_mac
(using macos_x86_64) fails with: A distribution for confluent-kafka could not be resolved... 
./pants run realtime/eeg_processor:eeg_processor_default
works without issue. Is there some way to produce a multi-platform pex distributable that can be executed from both mac and linux? Example project attached
pants_pex_docker.zip
e
The short answer is yes, but you need to have pre-built wheels for all transitive dependencies for all targeted platforms. Pex does not have cross build magic. So for all missing wheels, you'll need to do something like 
pip wheel --no-deps X
using a Pip in a venv built from the targeted interpreter for each distribution / platform with a missing wheel. You then need to gather up all these new wheels and make them available via flat directory and 
[python-repos] find_links
or via a custom index server and 
[python-repos] indexes
.
This will produce, sometimes ridiculously so, large PEXes.
But they will work on multiple platforms. An example of this are the Pants release PEXes themselves: https://github.com/pantsbuild/pants/releases/tag/release_2.16.0.dev3
Those are 6 platform PEXes (Python 3.{7,8,9} x {linux,mac}):
Copy code
$ pex-tools pants.2.16.0.dev3.pex info -i2
{
  "bootstrap_hash": "0d4227c34ba758e299322a6027844574530fbe24",
  "build_properties": {
    "pex_version": "2.1.116"
  },
  "code_hash": "9468e945a0bf2ce2824340a168a74c88293515d5",
  "distributions": {
    "PyYAML-6.0-cp37-cp37m-macosx_10_9_x86_64.whl": "6dfb2596afa3b26d0ca475dd1a53dee6137b9a6986e6fa055f8a7e723c243818",
    "PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl": "bd28c4892342d6d96457678e754324bdc6950abcde46e9735d7e498a51a96ab4",
    "PyYAML-6.0-cp38-cp38-macosx_10_9_x86_64.whl": "1b645a947c41a735abee60a7ca17cfc4475d1c8736a9c9c4a5f8b2ccd5674048",
    "PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl": "d68055cf5bd35222deaed312f2c7a256e60734ba598313016ae8e165503d3913",
    "PyYAML-6.0-cp39-cp39-macosx_10_9_x86_64.whl": "1a08337aaa0f63f4f94da401c57ec31408221212f19a7a4e10b32aeb82179233",
    "PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl": "31f242146847b7c77e3919e7f3456333e9d2f8b2a7738bdc8a445e5c98b1ee91",
    "ansicolors-1.1.8-py2.py3-none-any.whl": "949f7f911c7f36e9e84afa0729510629aaf1ebf3f8772649a350bb698305172b",
    "certifi-2022.12.7-py3-none-any.whl": "fa660808b834d51826f2d58643aad338e42b6d2bae69df8c8e5a4829dbeabdcf",
    "charset_normalizer-2.1.1-py3-none-any.whl": "c4dcf068d12625d4115f5e98cca5b90c18403837015e82bdf2a5cbc80f01d32d",
    "chevron-0.14.0-py3-none-any.whl": "a6c98dc1c656027f380390263d63ad3fb7455a8c8493bc2c543ea5d59217efd5",
    "fasteners-0.16.3-py2.py3-none-any.whl": "abf091a25a85c301dbbb0a76a1300876b7b36692c0e30c7abba42a94139e5b68",
    "humbug-0.2.7-py3-none-any.whl": "c27e82b88a29222994a01ef293018815d12ee43987e7646fbd3a3af10cd15956",
    "idna-3.4-py3-none-any.whl": "3baea0ffbdea9f349783bacbdb82dbd50eadfec75339388aaf1597754786e3ed",
    "ijson-3.1.4-cp37-cp37m-macosx_10_9_x86_64.whl": "77020f8fc9782c851ae3b8a2ea825501ca9286be676536fa24b61831216630e8",
    "ijson-3.1.4-cp37-cp37m-manylinux2010_x86_64.whl": "802607880f1735d65dfa7a5e48a9659234cd1b3faeb46e3ee99425aba4b0a4fe",
    "ijson-3.1.4-cp38-cp38-macosx_10_9_x86_64.whl": "4d0264b3be861c7afcbcdae62b6f4a8831d35deb54d04d8f5712ddde8da43969",
    "ijson-3.1.4-cp38-cp38-manylinux2010_x86_64.whl": "656a8798e9c2358f6764f291822680616bbcb048800a400d9e7088297a743ffb",
    "ijson-3.1.4-cp39-cp39-macosx_10_9_x86_64.whl": "78701013b845adf4477ac0a1a77362c4cdf756a0614b47f0770b60dcb53e81b5",
    "ijson-3.1.4-cp39-cp39-manylinux2010_x86_64.whl": "47960e816a8c5431fd07354c0f444c5e47f136743a659ca8aa0a144b268c7987",
    "importlib_resources-5.0.7-py3-none-any.whl": "55a1c6ae3511a774647ce849ae4dd92a34a3004cce4e12b3b35deb3345928599",
    "packaging-21.3-py3-none-any.whl": "a728bb5d35998463aee41e8a69df75fe81e027553b019f51236950d765d9f132",
    "pantsbuild.pants-2.16.0.dev3+gitdc243479-cp37-cp37m-macosx_10_16_x86_64.whl": "ffdc9588e4270132b85be624ce6dc6f51551e3574f060a5aa5c8891a90347bbf",
    "pantsbuild.pants-2.16.0.dev3+gitdc243479-cp37-cp37m-manylinux2014_x86_64.whl": "8eb380112b933627b368f95db7bbef636cab3d02863b955e976b961c5424a058",
    "pantsbuild.pants-2.16.0.dev3+gitdc243479-cp38-cp38-macosx_11_0_x86_64.whl": "f77524c444454c3e1d48467ae25c6a1aaadc66bca926820c4fe0d79c64d0fd63",
    "pantsbuild.pants-2.16.0.dev3+gitdc243479-cp38-cp38-manylinux2014_x86_64.whl": "e96f0d7aca64789970777572832df490e4b1c6deeae5957a1e5c379f1027aee7",
    "pantsbuild.pants-2.16.0.dev3+gitdc243479-cp39-cp39-macosx_10_16_x86_64.whl": "26fddd1c5de3d01fdadd8d33a785fa629f365c56452019d80d8a8bff7398aa9c",
    "pantsbuild.pants-2.16.0.dev3+gitdc243479-cp39-cp39-manylinux2014_x86_64.whl": "a9ffa8eaf7cb5a65a51ab86b65e776354d5f8827e6408e5c28448275322cdcfe",
    "pex-2.1.116-py2.py3-none-any.whl": "1c1f825902ea10ba55b25993a8a2882bd0ca3e50bce4b53827c8df6cdcd8910f",
    "psutil-5.9.0-cp37-cp37m-macosx_10_9_x86_64.whl": "13afed758b3f96dba7bb74113311edd48fcf9ae4fe369752d331ec6808000eda",
    "psutil-5.9.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "76890e37ad396288d10b40918c5665c80f472ea07258232eeb43d06a2eccab96",
    "psutil-5.9.0-cp38-cp38-macosx_10_9_x86_64.whl": "929e7efda07e1ff0306f988741cf180eb170c3d4bbb85e1ede691874ea7d8732",
    "psutil-5.9.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "1409ffca6cb2442101efa3607c28337e2fc4522968ae100f891d6028693c959e",
    "psutil-5.9.0-cp39-cp39-macosx_10_9_x86_64.whl": "000d5a923178ce1f9180649d7a92d163195287d8a07c7e1972b087b7a404766e",
    "psutil-5.9.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "447973e6d8fd0c2b91d3e95377bd721d43b6a159271a8c490a3a981b2feb3b69",
    "pyparsing-3.0.9-py3-none-any.whl": "a0d0acca03cd18219d6c8d6af5f1804e7142b9df0e3e05f9f7e6ae8f0b157511",
    "python_lsp_jsonrpc-1.0.0-py3-none-any.whl": "4c4574cd647dd03e65a52e076f7b4e8622dac54cd9c9f9a6986537d9adff2ff0",
    "requests-2.28.1-py3-none-any.whl": "aceadd0c9564fd6a3ded05afec41e816beee638b94feeccf30ec61da94a6c91d",
    "setproctitle-1.3.2-cp37-cp37m-macosx_10_9_x86_64.whl": "43121e76aa2d9da28e29413f02a5a3924c649d7295521cdc5e0b3aa24831038c",
    "setproctitle-1.3.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "c394374a391915038e4b681ab6ffe6bf1d3a17f982aa766061c37b11661e080a",
    "setproctitle-1.3.2-cp38-cp38-macosx_10_9_x86_64.whl": "bee333f56983b7336edc8e463d07009d83c2ec8704a444436fbcaf20e4453794",
    "setproctitle-1.3.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "8208232dafaed7d0f17c5dd4d3269b7e53294d648f7a50e5b52cf1b8097aa9d7",
    "setproctitle-1.3.2-cp39-cp39-macosx_10_9_x86_64.whl": "7b62c021b6f48927907cb7c48a368ce7e471a97aeb3f8deabb4bc3ab3f8741ed",
    "setproctitle-1.3.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "6a793c53d0e3c148a551e6cb897009441ab9eb2af2a9540783b94d273b9116d9",
    "setuptools-63.4.3-py3-none-any.whl": "298e5a6717249f7762e2966ebac8ba69e57f63ec5a1d6044950f95c1c11aeae8",
    "six-1.16.0-py2.py3-none-any.whl": "3e1c439c88d2e7681372427bab751b3fc99969891e95a714fed9604bf7710213",
    "toml-0.10.2-py2.py3-none-any.whl": "3ac6b864ba1f1700c2a303eddbf23845d50c4d8b842400ba2480b5006bb1af83",
    "types_PyYAML-6.0.3-py3-none-any.whl": "45598639af4c17d97315895ad94613d901f40a17bff74803d0344c06a6fc778e",
    "types_setuptools-62.6.1-py3-none-any.whl": "df21693c80e134a80e658f238732fa4c39c829906992cbb7b09944e43abccfb3",
    "types_toml-0.10.8-py3-none-any.whl": "a538139c45557651a6e6eacfa13cf40466f45861c917256e2a97ef2d3f557916",
    "typing_extensions-4.3.0-py3-none-any.whl": "c52ea780f5287cd80062d9e31ea85307e2b7eb501f83e36fce6f8073449c5fbe",
    "ujson-5.6.0-cp37-cp37m-macosx_10_9_x86_64.whl": "781e84c2569848ae3a515081d7f34a5bda124bbd09b9f2706735d372d4cb37db",
    "ujson-5.6.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "9ae6daccb3ed79061fd5d67b69446d5843c33c7428028b85e514a2061213afcf",
    "ujson-5.6.0-cp38-cp38-macosx_10_9_x86_64.whl": "4eb17ad45c9a9112bf00f7d4a9d7b680d8afc381235d665dd2bfd4aea2a6fdec",
    "ujson-5.6.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "d1d494f0b3a9413d0592f3c586568898b0dec16af0a7511c243a67bfafcd64d8",
    "ujson-5.6.0-cp39-cp39-macosx_10_9_x86_64.whl": "a8173a2e120171c2cebe88f599a29df03e86858467d5d254be4eb8e5d589a9ab",
    "ujson-5.6.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "d6973eabc99a836e020157681f735cff5bb7eddf2c9c59e63c4569a2f41748c2",
    "urllib3-1.26.13-py2.py3-none-any.whl": "1eaf449f9b7f473072db8b825c74ffe5e6f970e95dc0127467ab719d76057f23",
    "zipp-3.11.0-py3-none-any.whl": "db05f31190867dbe2b766a8696989db0cf01f33fc3f1c6223d51477dfece4772"
  },
  "emit_warnings": true,
  "entry_point": "pants.bin.pants_loader:main",
  "ignore_errors": false,
  "includes_tools": true,
  "inherit_path": "false",
  "inject_args": [],
  "inject_env": {},
  "interpreter_constraints": [
    "CPython<3.10,>=3.7"
  ],
  "pex_hash": "8753a4155fb37e7bb5a5d22c58d86a74d414685f",
  "pex_path": "",
  "pex_paths": [],
  "requirements": [
    "pantsbuild.pants==2.16.0.dev3+gitdc243479"
  ],
  "strip_pex_env": false,
  "venv": true,
  "venv_bin_path": "false",
  "venv_copies": false,
  "venv_site_packages_copies": false,
  "pex_root": "/home/jsirois/.pex"
}
The key is, we pre-build all the needed wheels on each targeted platform in CI, then we produce the PEX with assured all-wheels-available after.
@aloof-appointment-30987 does that make sense? No magic, if you hit an sdist-only dep, you have to roll up your sleeves and pre-build wheels?
a
It makes sense. From what I can tell, all our dependencies are available as wheels but the resolution does not find them using abbreviated 
platform
configuration. Pre-building all the needed wheels sounds daunting. I may defer having a single pex definition until another time since the 
complete_platforms
configuration targeting linux works well and it's easy enough to use the generic 
pex_binary
to run locally.
e
I don't understand - it sounds like you are not using abbreviated platforms, you are using complete platforms. Is that wrong?
A complete platform enables resolving exactly as the described interpreter would resolve, with the 1 restriction of wheels only accepted, no sdists allowed.
So, looking at your zip, your lib_third_party/platforms/macos_x86_64_platform.json looks ~invalid. How did you generate that? The 1st tag, which should be the most specific tag, is 
cp32-abi3-macosx_10_10_fat32
- an abi3 tag is impossible (there is always something more specific than abi3) and surely you're not targeting CPython 3.2!?
The marker environment shows its CPython 3.8.13 ... so something is off there.
The 
lib_third_party/platforms/linux_py38_complete_platform.json
complete platform looks much better.
So, afaict `# Does not work: A distribution for confluent-kafka could not be resolved`is explained. Does that make sense @aloof-appointment-30987? Fix the mac complete platform file / re-generate it, it seems to be bogus.
You may want to run like this for sanity sake:
Copy code
pex3 interpreter inspect --python /this/exact/python/please --markers --tags
So, @aloof-appointment-30987 works for me:
Copy code
$ ./pants package realtime/eeg_processor:eeg_processor_mac
12:43:35.02 [INFO] Wrote dist/realtime.eeg_processor/eeg_processor_mac.pex
$ pex-tools dist/realtime.eeg_processor/eeg_processor_mac.pex info -i2
{
  "bootstrap_hash": "5ba276cd9f3b69e00c9cc8452f2c9a088d563169",
  "build_properties": {
    "pex_version": "2.1.108"
  },
  "code_hash": "b330812312c019ff18e1549a146a8d48cc00915f",
  "distributions": {
    "certifi-2022.12.7-py3-none-any.whl": "fa660808b834d51826f2d58643aad338e42b6d2bae69df8c8e5a4829dbeabdcf",
    "charset_normalizer-2.1.1-py3-none-any.whl": "c4dcf068d12625d4115f5e98cca5b90c18403837015e82bdf2a5cbc80f01d32d",
    "confluent_kafka-1.9.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "1a1c2a88579def2aff1db8fbfe5592fe9fcebd5bb4e80d6ab69f004062a6530f",
    "idna-3.4-py3-none-any.whl": "3baea0ffbdea9f349783bacbdb82dbd50eadfec75339388aaf1597754786e3ed",
    "numpy-1.22.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "a7c56a231a52ae36388029b5259b062e36e4a04e5fa3cabaa2c58c15bfb71c75",
    "protobuf-3.20.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl": "f3dcddc46e4098851abbb28fad355869f60d046ac9958dafbc369f5e4f51f2d7",
    "requests-2.28.1-py3-none-any.whl": "aceadd0c9564fd6a3ded05afec41e816beee638b94feeccf30ec61da94a6c91d",
    "urllib3-1.26.13-py2.py3-none-any.whl": "1eaf449f9b7f473072db8b825c74ffe5e6f970e95dc0127467ab719d76057f23"
  },
  "emit_warnings": false,
  "entry_point": "<http://realtime.eeg_processor.app|realtime.eeg_processor.app>",
  "ignore_errors": false,
  "includes_tools": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_hash": "fa2aec14f252748e309e882312f639fdffc495bd",
  "pex_path": "",
  "pex_paths": [],
  "requirements": [
    "confluent-kafka==1.9.2",
    "numpy==1.22.4",
    "protobuf~=3.20.0",
    "requests~=2.27"
  ],
  "strip_pex_env": true,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false,
  "venv_site_packages_copies": false,
  "pex_root": "/home/jsirois/.pex"
}
And I'm:
Copy code
$ uname -a
Linux Gill-Windows 5.15.79.1-microsoft-standard-WSL2 #1 SMP Wed Nov 23 01:01:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
So I'm not seeing the failure you're seeing trying to use the mac complete platform - despite it having strangely ordered compatible tags entries.
Aha, sorry - I missed the fact that you currently have both complete platforms files targets pointing at linux ... I'm guessing 1/2 way through debugging? Let me try fixing that up.
Ok, yeah - still works. Here's a multi-platform PEX using your setup with small edits:
Copy code
$ git diff
diff --git a/lib_third_party/platforms/BUILD.pants b/lib_third_party/platforms/BUILD.pants
index ddca320..ffd9f34 100644
--- a/lib_third_party/platforms/BUILD.pants
+++ b/lib_third_party/platforms/BUILD.pants
@@ -5,5 +5,5 @@ file(

 file(
   name="macos_x86_64",
-  source="./linux_py38_complete_platform.json"
-)
\ No newline at end of file
+  source="./macos_x86_64_platform.json"
+)
diff --git a/realtime/eeg_processor/BUILD.pants b/realtime/eeg_processor/BUILD.pants
index 222ee9c..44d3c3d 100644
--- a/realtime/eeg_processor/BUILD.pants
+++ b/realtime/eeg_processor/BUILD.pants
@@ -18,6 +18,15 @@ pex_binary(
     ],
 )

+pex_binary(
+    name="combined",
+    entry_point="app.py",
+    complete_platforms = [
+        'lib_third_party/platforms:linux-py38',
+        'lib_third_party/platforms:macos_x86_64',
+    ],
+)
+
 pex_binary(
     name="eeg_processor_default",
     entry_point="app.py",
jsirois@Gill-Windows:~/Downloads/pants_pex_docker (main *) $ ./pants package realtime/eeg_processor:combined
12:49:21.39 [INFO] Completed: Building realtime.eeg_processor/combined.pex with 4 requirements: confluent-kafka==1.9.2, numpy==1.22.4, protobuf~=3.20.0, requests~=2.27
12:49:21.41 [INFO] Wrote dist/realtime.eeg_processor/combined.pex
And:
Copy code
$ ./pants package realtime/eeg_processor:combined
12:49:21.39 [INFO] Completed: Building realtime.eeg_processor/combined.pex with 4 requirements: confluent-kafka==1.9.2, numpy==1.22.4, protobuf~=3.20.0, requests~=2.27
12:49:21.41 [INFO] Wrote dist/realtime.eeg_processor/combined.pex
$ pex-tools dist/realtime.eeg_processor/combined.pex info -i2
{
  "bootstrap_hash": "5ba276cd9f3b69e00c9cc8452f2c9a088d563169",
  "build_properties": {
    "pex_version": "2.1.108"
  },
  "code_hash": "b330812312c019ff18e1549a146a8d48cc00915f",
  "distributions": {
    "certifi-2022.12.7-py3-none-any.whl": "fa660808b834d51826f2d58643aad338e42b6d2bae69df8c8e5a4829dbeabdcf",
    "charset_normalizer-2.1.1-py3-none-any.whl": "c4dcf068d12625d4115f5e98cca5b90c18403837015e82bdf2a5cbc80f01d32d",
    "confluent_kafka-1.9.2-cp38-cp38-macosx_10_9_x86_64.whl": "1f59e8e2f2c702b65053fb10e950abaabd33412a2e15b5c9c722ba9b5037b0d0",
    "confluent_kafka-1.9.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "1a1c2a88579def2aff1db8fbfe5592fe9fcebd5bb4e80d6ab69f004062a6530f",
    "idna-3.4-py3-none-any.whl": "3baea0ffbdea9f349783bacbdb82dbd50eadfec75339388aaf1597754786e3ed",
    "numpy-1.22.4-cp38-cp38-macosx_10_15_x86_64.whl": "12b5ae0831d1919447e93f91a34f0e71c9c657ae430105e4864309e70296bf4c",
    "numpy-1.22.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "a7c56a231a52ae36388029b5259b062e36e4a04e5fa3cabaa2c58c15bfb71c75",
    "protobuf-3.20.3-cp38-cp38-macosx_10_9_x86_64.whl": "eb6647656f15455e410cd6a8135122942ddff693bbe8b9852bfc8092aa588967",
    "protobuf-3.20.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl": "f3dcddc46e4098851abbb28fad355869f60d046ac9958dafbc369f5e4f51f2d7",
    "requests-2.28.1-py3-none-any.whl": "aceadd0c9564fd6a3ded05afec41e816beee638b94feeccf30ec61da94a6c91d",
    "urllib3-1.26.13-py2.py3-none-any.whl": "1eaf449f9b7f473072db8b825c74ffe5e6f970e95dc0127467ab719d76057f23"
  },
  "emit_warnings": false,
  "entry_point": "<http://realtime.eeg_processor.app|realtime.eeg_processor.app>",
  "ignore_errors": false,
  "includes_tools": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_hash": "61846f95963ee390868326ede3cb211ef2af75b1",
  "pex_path": "",
  "pex_paths": [],
  "requirements": [
    "confluent-kafka==1.9.2",
    "numpy==1.22.4",
    "protobuf~=3.20.0",
    "requests~=2.27"
  ],
  "strip_pex_env": true,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false,
  "venv_site_packages_copies": false,
  "pex_root": "/home/jsirois/.pex"
}
So ... it looks like you should have this mostly working @aloof-appointment-30987 - I'm not sure why you're seeing resolve errors for mac.
a
@enough-analyst-54434. In fact, the file ref typo was the reason the mac pex failed. Sorry about that. Thanks very much for helping me resolve this. I really appreciate all the support! Once the platform was actually referenced, I was able to use the same pex output locally on my mac and inside docker running linux. Amazing!
...and a simple, consistent build config 👏🏽
e
Ok, great.
a
@enough-analyst-54434 Follow-up to the 
No matching distribution found for confluent-kafka==1.9.2
I was reading [about Pex](https://pex.readthedocs.io/en/v2.1.119/buildingpex.html#platform) and discovered...
Constraints: when 
--platform
is used the environment marker 
python_full_version
will not be available if 
PYVER
is not given as a three component dotted version since 
python_full_version
is meant to have 3 digits (e.g., 
3.8.10
). If a 
python_full_version
environment marker is encountered during a resolve, an 
UndefinedEnvironmentName
exception will be raised. To remedy this, either specify the full version in the platform (e.g, 
linux_x86_64-cp-3.8.10-cp38
) or use 
--complete-platform
instead.
This fails resolution
Copy code
platforms = [
        'current', 
        'linux_x86_64-cp-38-cp8m'
    ]
But this succeeds
Copy code
platforms = [
        'current', 
        'linux_x86_64-cp-3.8.10-cp38'
    ]
The available files for confluent kafka include 
confluent_kafka-1.9.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
I'm curious why this does not resolve for the first platform abbv but pleased it works with the second. I may, however, continue with the 
complete-platform
as it is reported to be more reliable.
e
@aloof-appointment-30987 fails is a bit underspecified, can you share the failure exactly? Or can I repro using your zip?
Ah, yeah - I repro. Just a typo @aloof-appointment-30987 'linux_x86_64-cp-38-cp8m' -> 'linux_x86_64-cp-38-cp38' and it works fine.
You should absolutely positively stick with 
complete_platforms
. Plain old (abbreviated) 
platforms
are just that - abbreviated. They are a guess. These came a bit before my time as a Pex maintainer. A complete platform is the actual real information needed by Pip or any other PEP-XYZ compliant resolver in the Python ecosystem. Pip did not believe me at 1st: https://github.com/pypa/pip/issues/10050 But they have since come around to realizing this is the only way to do it: https://github.com/pypa/pip/issues/11664#issuecomment-1356296497
a
Perfect. I’ll take that advice! I copied the string you identified as a typo from the pex documentation so I think your point is proven. Complete platform is the way to go.
e
Can you point me to the typo? I don't see it and I'd like to fix it.
a
Oh snap. Sorry, I see the typo in my post. I missed the 3 in 
cp38m
I had copied the `linux_x86_64-cp-36-cp36m`string from https://pex.readthedocs.io/en/v2.1.119/buildingpex.html#platform but updated from 36 to 38 Independent of the missing 3 typo, With the 
m
at the end, it fails resolution with "ERROR: No matching distribution found for confluent-kafka==1.9.2" But without the trailing 
m
, using the abbv 
linux_x86_64-cp-38-cp38
, resolution and packaging succeeds. Sorry about the confusion with the typo. I was not able to find what the m means. Thanks again for your help
e
And the m goes away starting with 38. Using cp37m for the ABI, good. Using cp38m - no such thing, use cp38
It means "malloc"
a
Gotcha. I found pep3149 proposal but not comprehensive as to what versions are available. I would love to have a link that explained how to determine the proper ABI but I am going to be using 
--complete-platform
going forward 😉
e
You can only determine the proper abi by asking the interpreter, which is what the complete platforms generation command does.
PEP-3149 gives you an indirect way:
Copy code
sysconfig.get_config_var('EXT_SUFFIX')
'.cpython-32mu.so'
So that's 
cp32mu
But, IIRC executing 
pex --platform help
will tell you 
help
is an invalid platform and give you a bunch of help including the platform of the current interpreter (the one whose venv 
pex
is installed in).
8 Views
Open in Slack
PreviousNext
Powered by