https://pantsbuild.org/ logo
Join Slack
Powered by
What should I do for dealing with dependency issue...
# general
b
What should I do for dealing with dependency issues due to the usage of an apple silicon mac? My error message:
Copy code
Failed to resolve requirements from PEX environment @ /root/.pex/unzipped_pexes/ce56906b160237e1254e7999de724a1bdd708b54.
  Needed cp311-cp311-manylinux_2_31_aarch64 compatible dependencies for:
   1: MarkupSafe>=2.0
      Required by:
        Jinja2 3.1.2
      But this pex had no ProjectName(raw='MarkupSafe', normalized='markupsafe') distributions.
   2: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
      Required by:
        jsonschema 4.17.3
      But this pex had no ProjectName(raw='pyrsistent', normalized='pyrsistent') distributions.
   3: numpy
      Required by:
        altair 4.2.2
        streamlit 1.19.0
      But this pex had no ProjectName(raw='numpy', normalized='numpy') distributions.
   4: pandas>=0.18
      Required by:
        altair 4.2.2
      But this pex had no ProjectName(raw='pandas', normalized='pandas') distributions.
   5: pandas>=0.25
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='pandas', normalized='pandas') distributions.
   6: pillow>=6.2.0
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='pillow', normalized='pillow') distributions.
   7: pyarrow>=4.0
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='pyarrow', normalized='pyarrow') distributions.
   8: charset-normalizer<4,>=2
      Required by:
        requests 2.28.2
      But this pex had no ProjectName(raw='charset-normalizer', normalized='charset-normalizer') distributions.
   9: numpy>=1.16.4
      Required by:
        pydeck 0.8.0
      But this pex had no ProjectName(raw='numpy', normalized='numpy') distributions.
   10: tornado>=6.0.3
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='tornado', normalized='tornado') distributions.
   11: watchdog; platform_system != "Darwin"
      Required by:
        streamlit 1.19.0
      But this pex had no ProjectName(raw='watchdog', normalized='watchdog') distributions.
   12: watchdog==2.3.0
      But this pex had no ProjectName(raw='watchdog', normalized='watchdog') distributions.
BUILD file:
Copy code
python_sources()

pex_binary(
    name="bin",
    script="streamlit",
    dependencies=["./main.py", "//:poetry#watchdog"],
    restartable=True,
)

docker_image(
    name="docker"
)
c
when you build a pex for consumption in a docker image, you need to build the pex for the target platform of the image. Search slack history to find previous threads on this topic.
there ought to be a page about this in the docs, it’s a common enough question with sometimes involved answers.
💯 1
b
I found very confusing by reading the docs. Very hard to write the full platform string.
could have been some sensible defaults or examples, like an intel mac, arm mac, intel linux
c
pex can help you out with the complete platform string, not sure of the details though.
b
but even if I use "current" as a platform it doesn't work
Copy code
pex_binary(
    name="bin",
    script="streamlit",
    dependencies=["./main.py", "//:poetry#watchdog"],
    restartable=True,
    platforms=["current"]
)
c
here’s a recent thread I think is related https://pantsbuild.slack.com/archives/C046T6T9U/p1676659450621659
Copy code
platforms=["current"]
yea, that won’t work if you run on a Mac.. as current will be you mac platform for the pex then and unusable in the docker image.
you need to put on a cross-building thinking hat 😉
r
https://pantsbuild.slack.com/archives/C046T6T9U/p1670366698746769?thread_ts=1670366698.746769&amp;cid=C046T6T9U This thread has the way to generate the string for 
complete_platforms
using appropriate docker image We should add it as somewhere inside pants doc
👆 1
🙏 1
b
trying to learn all of this at once.... 😫
c
it’s a lot, and you entered into the deep end 😮
r
c
it will be rewarding once it “clicks” 🙂
🙌 1
b
At least I have this great community to help. At some point I need to do blog posts describing all of this.
🙏 1
Ok. After adding the complete platforms, things started to come in place except for one dependency:
Copy code
17:57:42.43 [INFO] Completed: Building 2 requirements for src.streamlit.template/bin.pex from the 3rdparty/python/default.lock resolve: streamlit==1.19.0, watchdog==2.3.0
17:57:42.43 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

ProcessExecutionFailure: Process 'Building 2 requirements for src.streamlit.template/bin.pex from the 3rdparty/python/default.lock resolve: streamlit==1.19.0, watchdog==2.3.0' failed with exit code 1.
stdout:

stderr:
Failed to resolve compatible artifacts from lock 3rdparty/python/default.lock for 1 target:
1. cp311-cp311-manylinux_2_31_x86_64:
    Failed to resolve all requirements for complete platform cp311-cp311-manylinux_2_31_x86_64 from 3rdparty/python/default.lock:

Configured with:
    build: False
    use_wheel: True

Dependency on validators not satisfied, 1 incompatible candidate found:
1.) validators 0.20 (via: streamlit==1.19.0 -> validators>=0.2) does not have any compatible artifacts:
    <https://files.pythonhosted.org/packages/95/14/ed0af6865d378cfc3c504aed0d278a890cbefb2f1934bf2dbe92ecf9d6b1/validators-0.20.0.tar.gz>



Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
Looks like the package has no platform info, therefore not installing. What could be wrong?
e
So, 
validators-0.20.0.tar.gz
is an sdist - it must be built into a wheel. You cannot build a wheel for a foreign platform. The hint is above in the message:
Copy code
Configured with:
    build: False
    use_wheel: True
So, you need to either pre-build a wheel yourself and make it available in a find-links repo or try using the new 2.15 environments feature.
Another thing that never seems to connect is answering this question: "How would you do this without Pants?" The answer to that is often something like:
Copy code
RUN pip install poetry
RUN poetry ...
If that's true for you, then you do exactly the same thing with Pants:
Copy code
RUN ... pants install steps ...
RUN pants package ...
I'm not sure how many people use this latter technique - but it should be the obvious one I'd think. That said - environments is the way to go here. That will be more efficient.
c
this issue is still relevant too https://github.com/pantsbuild/pants/issues/13185 (to help guide users if nothing else)
e
Yeah, the guiding is clearly needed.
➕ 1
It looks like validators would also succumb to yolo building (turns out it is not platform specific IOW):
Copy code
$ pex pip wheel -c pip -- wheel validators
Collecting validators
  Using cached validators-0.20.0.tar.gz (30 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting decorator>=3.4.0
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pants/decorator-5.1.1-py3-none-any.whl
Building wheels for collected packages: validators
  Building wheel for validators (pyproject.toml) ... done
  Created wheel for validators: filename=validators-0.20.0-py3-none-any.whl size=19581 sha256=25e221f311e6e6b60b175b89900f4ec0f5b48e0a0aff0b1db9ae325a457d38d8
  Stored in directory: /home/jsirois/.cache/pip/wheels/f2/ed/dd/d3a556ad245ef9dc570c6bcd2f22886d17b0b408dd3bbb9ac3
Successfully built validators
I'll file a Pex issue about supporting that idea.
👍 2
c
guess when you get 
*py3-none-any.whl
wheels, that it should be ok
b
The idea of pants building pex binaries for docker images is quite compelling but those quirks don’t convince me yet that the developer experience will improve rather than building the venv individually in a polyrepo environment.
Python is still a way to go into monorepos. The JS ecosystem already polished all those kinds of quirks
But I’m still confident pants/pex will reach a point where most edge cases will start to be covered
But this specific use case of using streamlit is getting really difficult to tackle. It will make sense to me to put this off the monorepo
e
b
Re https://pantsbuild.slack.com/archives/C046T6T9U/p1677515258195679?thread_ts=1677513871.527529&amp;cid=C046T6T9U blogging about Pants is always welcome and much appreciated, wherever it's posted. We're always happy to give feedback on a draft, if you want. Please do give me a ping if you do on post somewhere, so we can add a link to Resource Hub. And if you are interested in guest posting at Pants Blog, DM me anytime to discuss topics. Cheers!
🙌 1
f
@breezy-apple-27122, I found that using the new environments feature helped a lot with these issues. Then you can build your Pex in an x86 environment in a Docker container, so you won't have to worry about your host system at all. Also great since it means that everyone working in your repo will have a reliable build.
b
I'm trying that now
will post more results in the main chat
e
It turns out "yolo" support was very easy:
Copy code
$ git sdiff
 pex/pip/foreign_platform.py | 3 ---
 pex/pip/tool.py             | 3 +--
 2 files changed, 1 insertion(+), 5 deletions(-)
Just 2 old tests asserting sdists don't work for 
--platform
and 
--complete-platform
that fail. I'll get out a Pex release with the "yolo" support ~today so Pants 2.16.x has it at least.
👀 1
👖 1
🔥 2
And you can try this out with the following 
pants.toml
snippet:
Copy code
[pex-cli]
version = "v2.1.125"
known_versions = [
    "v2.1.125|macos_arm64|1da1ef933429f15b218c98c6b960f30adfd0221fc5284c1d8facac09923692f8|4080732",
    "v2.1.125|macos_x86_64|1da1ef933429f15b218c98c6b960f30adfd0221fc5284c1d8facac09923692f8|4080732",
    "v2.1.125|linux_x86_64|1da1ef933429f15b218c98c6b960f30adfd0221fc5284c1d8facac09923692f8|4080732",
    "v2.1.125|linux_arm64|1da1ef933429f15b218c98c6b960f30adfd0221fc5284c1d8facac09923692f8|4080732"
]
18 Views
Open in Slack
PreviousNext
Powered by