https://pantsbuild.org/ logo
Join Slack
Powered by
One of my coworkers is running into issues with `g...
# general
s
One of my coworkers is running into issues with 
grpcio
. Unfortunately I'm not able to reproduce the error exactly (for whatever reason it works on my machine). If I remove the version in my 
requirements.txt
I can get it to fail (I'm not sure if it's failing for the same reason) but I wanted to take this chance to provide some feedback on how a novice pants user would try to resolve this. I'm only interested in figuring out why grpcio isn't working so let's make a dummy target
Copy code
pex_binary(
    name="foo",
    dependencies=["//:reqs#grpcio"]
)
Since grpcio logs are noisy I'll run 
./pants --keep-sandboxes=on_failure package related-titles:foo > logs.txt 2>&1
since in my terminal I can't scroll up enough. Okay the logs show me
Copy code
16:59:50.77 [INFO] Preserving local process execution dir /private/var/folders/m8/cmsvrd417n39w5xjzjhmnznm0000gq/T/pants-sandbox-ql5kML for Test binary /Users/yjabri/.pyenv/shims/python3.
16:59:50.84 [INFO] Preserving local process execution dir /private/var/folders/m8/cmsvrd417n39w5xjzjhmnznm0000gq/T/pants-sandbox-UIqbuV for Test binary /Users/yjabri/.pyenv/shims/python.
16:59:50.92 [INFO] Preserving local process execution dir /private/var/folders/m8/cmsvrd417n39w5xjzjhmnznm0000gq/T/pants-sandbox-iY4rld for Test binary /usr/local/bin/python.
16:59:50.93 [INFO] Starting: Building 1 requirement for related-titles/foo.pex from the 3rdparty/python/default.lock resolve: grpcio
17:00:58.99 [INFO] Preserving local process execution dir /private/var/folders/m8/cmsvrd417n39w5xjzjhmnznm0000gq/T/pants-sandbox-zwQ29T for Building 1 requirement for related-titles/foo.pex from the 3rdparty/python/default.lock resolve: grpcio
17:00:58.99 [INFO] Completed: Building 1 requirement for related-titles/foo.pex from the 3rdparty/python/default.lock resolve: grpcio
So far so good.
Copy code
Engine traceback:
  in `package` goal

ProcessExecutionFailure: Process 'Building 1 requirement for related-titles/foo.pex from the 3rdparty/python/default.lock resolve: grpcio' failed with exit code 1.
stdout:
This is kind of odd. The error is the following command 
exited with 1 and STDERR:
Copy code
/Users/yjabri/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/389476d3d6dcf639f35a0bd43926d6b11532a553/bin/python \
  -sE /Users/yjabri/.cache/pants/named_caches/pex_root/venvs/41379f21c5956cff0d35ab1ef014bb72e1fe14dc/389476d3d6dcf639f35a0bd43926d6b11532a553/pex \
  --disable-pip-version-check \
  --no-python-version-warning \
  --exists-action a \
  --no-input \
  --use-deprecated legacy-resolver \
  --isolated \
  -q \
  --cache-dir /Users/yjabri/.cache/pants/named_caches/pex_root/pip_cache wheel \
  --no-deps \
  --wheel-dir /Users/yjabri/.cache/pants/named_caches/pex_root/built_wheels/sdists/grpcio-1.49.1.tar.gz/d4725fc9ec8e8822906ae26bb26f5546891aa7fbc3443de970cc556d43a5c99f/cp310-cp310-macosx_12_0_arm64.d2f5ef92ac104f0ab9de11c572b97357 \
  /Users/yjabri/.cache/pants/named_caches/pex_root/downloads/d4725fc9ec8e8822906ae26bb26f5546891aa7fbc3443de970cc556d43a5c99f/grpcio-1.49.1.tar.gz \
  --index-url <https://pypi.org/simple/> \
  --extra-index-url https://**********:@pypi.fury.io/amlg/ \  (almost posted the credentials in plaintext...)
  --retries 5 \
  --timeout 15
failed. At this point I'm not really sure where this is running. If I try to run it in the current directory I get 
ModuleNotFoundError: No module named 'pex'
. Same thing if I run it in the last sandbox pants told me about (/private/var/folders/m8/cmsvrd417n39w5xjzjhmnznm0000gq/T/pants-sandbox-zwQ29T) At this point what should I look for?
✅ 1
Furthermore if I run 
./related-titles/foo.pex
from the sandbox I can do stuff like
Copy code
>>> from grpc import server
>>> help(server)
and it all works
Of course after going through this exercise the command 
./pants --keep-sandboxes=on_failure package related-titles:foo
appears to just work
If I do 
rm -fr ~/.cache/pants && rm -fr .pids && rm -fr .pants.d
I can reproduce the above
This time it doesn't have the sandbox with 
./related-titles/foo.pex
w
For which python version are you building against?
Rather, what are your interpreter_constraints?
s
interpreter_constraints = ["CPython==3.10.*"]
I'm running 3.10.12 and my coworker is running python 3.10.11
w
Few sanity checks here just for my own sake: You mentioned your colleague's running into errors that you may/may not be running into. Are you on the same OS? I also noted that you're using 
./pants
in lieu of the 
pants
runner. Any particular reason? Or just historical?
s
He's got macos 12.6 and I have 12.3.
I'm afraid to ask my coworkers to download pants 😛
w
lol, fair enough
And just as a last sanity, for my sake, what version of Python is being used as the Pants runner? 3.7, 3.8, or 3.9? For you and your colleague? Before really digging into the grpcio stuff, I typically like to enumerate the platform/setup differences first, in order to debug
s
Good question. I'm not entirely sure how to check the runner
w
Good question, it's been a while since I've had to do it. I'm trying to recall if the Pants script prints it out when you do a first run, or if it would be in your cache. There should be an easier way, as I think one of the environment variables or debug mode will tell you. I'd have to look up one of my repos that still has the script BRB
🙏 1
s
I decided to try 
brew install pantsbuild/tap/pants
and give that a shot
👍 1
w
Yeah, at the very least I think you can see it when you install? 
green "Bootstrapping Pants using ${python}"
I vaguely recall a debug mechanism too
s
My 
./pants
appears to be using 
Bootstrapping Pants using /Users/yjabri/.pyenv/shims/python3.9
. Thanks I'll see what my coworker gets tomorrow! Appreciate the help so far
w
So, the 
pants
runner will automatically install the correct Python (3.9) to run pants, turning that into an implementation detail - way better way forward. Ah, okay, you were using pyenv - that makes sense. Another note: Does GRPC have wheels for Python 3.10, for MacOS ARM, on 1.49.1 ? That's another thing to maybe look into - if that might be causing some problems between yours and your colleague's setups?
s
I don't believe so https://pypi.org/project/grpcio/1.49.1/#files. They were able to run 
pip install --no-cache-dir --isolated grpcio~=1.49.1
(I need to do more research into what 
isolated
does)
Random observation. When clearing cache and running 
./pants --version
I get
Copy code
Bootstrapping Pants using /Users/yjabri/.pyenv/shims/python3.9
Creating the virtualenv PEX.
Downloading the Pex PEX.
SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.103/pex> verified.
Installing pantsbuild.pants==2.15.0 into a virtual environment at /Users/yjabri/.cache/pants/setup/bootstrap-Darwin-arm64/2.15.0_py39
New virtual environment successfully created at /Users/yjabri/.cache/pants/setup/bootstrap-Darwin-arm64/2.15.0_py39.
...
they get
Copy code
Bootstrapping Pants using /Users/dongp0509/opt/anaconda3/bin/python3.9
Creating the virtualenv PEX.
Downloading the Pex PEX.
SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.103/pex> verified.
Scrubbing PYTHONPATH
Installing pantsbuild.pants==2.15.0 into a virtual environment at /Users/dongp0509/.cache/pants/setup/bootstrap-Darwin-arm64/2.15.0_py39
Scrubbing PYTHONPATH
New virtual environment successfully created at /Users/dongp0509/.cache/pants/setup/bootstrap-Darwin-arm64/2.15.0_py39.
...
Why don't I get 
Scrubbing PYTHONPATH
?
That's probably not that important. One thing I didn't realize was that 
__run.sh
isn't populated initially when the sandbox is created? The second is that if I remove
Copy code
[subprocess-environment]
env_vars = [
  'GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1',
  'GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1',
  'GRPC_BUILD_WITH_BORING_SSL_ASM=',
]
in the logs I see
Copy code
22:17:33.32 [INFO] Starting: Building 1 requirement for related-titles/foo.pex from the 3rdparty/python/default.lock resolve: grpcio~=1.49.1
22:17:33.32 [INFO] Preserving local process execution dir /private/var/folders/m8/cmsvrd417n39w5xjzjhmnznm0000gq/T/pants-sandbox-zK9DIG for Building 1 requirement for related-titles/foo.pex from the 3rdparty/python/default.lock resolve: grpcio~=1.49.1
22:18:33.12 [INFO] Completed: Building 1 requirement for related-titles/foo.pex from the 3rdparty/python/default.lock resolve: grpcio~=1.49.1
there is no 
related-titles/foo.pex
. At this point I'm supposed to debug via 
__run.sh
but if I run 
bash __run.sh
it works and 
related-titles/foo.pex
is populated?
okay to reproduce I need to run 
env -i ./__run.sh
(looks like I don't need bash as its an executable)
Indeed 
env -i GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 GRPC_BUILD_WITH_BORING_SSL= ./__run.sh
works So this is more or less what I was expecting to happen because that's what folks on this form said would happen lol. This is just a little more spelled out. I think I just botched the execution at the end of the work day (and was thrown off by things like __run.sh missing initially)
So after running 
pyenv install 3.9
things worked 🤷‍♂️
But we also set
Copy code
CPPFLAGS=-I/opt/homebrew/opt/openssl@3/include
LDFLAGS=-L/opt/homebrew/opt/openssl@3/lib
I wonder if those have a side effect when installing python? Then again I don't think pyenv installed anything from source...
Open in Slack
PreviousNext
Powered by