https://pantsbuild.org/ logo
Join Slack
Powered by
Sorry to bother again: I'm having a problem with i...
# general
b
Sorry to bother again: I'm having a problem with installing/building dependencies in Pants v2 inside of a 
python:3.7-buster
container. While building wheels gcc throws 
fatal error: Python.h: No such file or directory
. Python.h is indeed available under the following path: 
/usr/local/includes/python3.7m/
, however, from the gcc call that fails I can see that only 
/usr/includes/python3.7m/
is included. Any idea what the cause of this could be? Shouldn't it use the includes from the 
~/.cache/pants/setup
directory anyway?
Specifically, it happens because one of my projects is depending on the 
hdbscan
package. Here is the corresponding call:
Copy code
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python3.7m -I/tmp/pip-build-env-x1_hs1mx/overlay/lib/python3.7/site-packages/numpy/core/include -c hdbscan/_hdbscan_tree.c -o build/temp.linux-x86_64-3.7/hdbscan/_hdbscan_tree.o
👍 1
h
No need to apologize ever for asking questions! What command are you running?
I’m thinking to debug it would be helpful to play around with the 
interpreter_search_paths
and/or 
interpreter_constraints
options: interpreter_search_paths For example, with constraints, if you know the precise version of the interpreter you intend to use, you could set 
interpreter_constraints == ["==3.7.4"]
Although, taking a step back, what is the output of 
which -l python
? The above suggestion only helps if there are two Python 3.7 interpreters on your machine, and Pants is choosing the wrong one.
b
This is the output of 
which -a python
:
Copy code
/usr/local/bin/python
/usr/local/bin/python
/usr/bin/python
The one located at 
/usr/bin/python
is Python 2.7.16, the other one is 3.7.7. I'm confused why it appears twice though
h
Indeed, that is confusing.. @witty-crayon-22786 any idea what that could be?
Also, what happens if you run 
/usr/local/bin/pip install hdbscan
? (I want to double check that the install would work so that we can isolate if this issue is with Pants vs. the underlying Python installation)
(warning that that will pollute your global 
site-packages
. But it sounds like this is docker at least)
w
yea, it sounds like Eric is on the right track.
the python that pants uses to run itself (ie 
~/.cache/pants/setup
) is independent from what it will use to build your dependencies, which is controlled by the interpreter search path option
👍 1
b
/usr/local/bin/pip install hdbscan
does indeed build the wheel without a problem 🤔
👍 1
w
if this is a relatively small build, you can pass 
-ldebug
which will dump “a lot” of information about what pex is doing with pip
but, actually.
b
Curiously, the interpreter that is used to execute the pip.pex (
/usr/bin/python3.7 /root/.cache/pants/named_caches/pex_root/pip.pex/240b20f98d345b70b316f7e734d4cea20bde73b6
) is called using 
/usr/bin/python3.7
w
for a failing run like this, there should actually be a 
pip
command rendered in the output
do you see that?
👍 1
h
(irrelevant suggestion) Hm, idea. There are two flags we use to control the LDFLAGS and CPPFLAGS passed to Python processes. 
ld_flags
and 
cpp_flags
, both in the 
[python-native-code]
options scope. They default to reading in the env vars What does 
echo $LDFLAGS
and 
echo $CPPFLAGS
do?
w
@better-nail-12700: yea, that is highly relevant.
👍 1
b
Yes I see the command:
Copy code
/usr/bin/python3.7 /root/.cache/pants/named_caches/pex_root/pip.pex/240b20f98d345b70b316f7e734d4cea20bde73b6 --disable-pip-version-check --isolated --no-python-version-warning -q --cache-dir /root/.cache/pants/named_caches/pex_root wheel --no-deps --wheel-dir /root/.cache/pants/named_caches/pex_root/built_wheels/sdists/hdbscan-0.8.26.tar.gz/4e23f82a649125d3f93c58dbe9e0d190bf291947/cp37-cp37m.fd92366b7396429f905761e9ca34406c --index-url <https://pypi.org/simple/> --header Cache-Control:max-age=3600 --retries 5 --timeout 15 /tmp/tmpg5kitjbw/cp37-cp37m/hdbscan-0.8.26.tar.gz raised Executing /usr/bin/python3.7 /root/.cache/pants/named_caches/pex_root/pip.pex/240b20f98d345b70b316f7e734d4cea20bde73b6 --disable-pip-version-check --isolated --no-python-version-warning -q --cache-dir /root/.cache/pants/named_caches/pex_root wheel --no-deps --wheel-dir /root/.cache/pants/named_caches/pex_root/built_wheels/sdists/hdbscan-0.8.26.tar.gz/4e23f82a649125d3f93c58dbe9e0d190bf291947/cp37-cp37m.fd92366b7396429f905761e9ca34406c --index-url <https://pypi.org/simple/> --header Cache-Control:max-age=3600 --retries 5 --timeout 15 /tmp/tmpg5kitjbw/cp37-cp37m/hdbscan-0.8.26.tar.gz
w
so yea… that’s the python it’s using.
👍 1
so eric’s link to the interpreter search paths docs are the path forward i think.
👍 1
h
What does 
/usr/bin/python3.7 --version
give you? Vs. 
/usr/local/bin/python3.7 --version
b
😂
Copy code
root@14c340f404ce:/# /usr/bin/python3.7 --version
Python 3.7.3
root@14c340f404ce:/# /usr/local/bin/python3.7 --version
Python 3.7.7
Seems like I have two Python3.7s afterall
h
Indeed. So, to confirm this will fix it, try setting this:
Copy code
[python-setup]
interpreter_constraints = ["==3.7.7"]
Or, you can set:
Copy code
[python-setup]
interpreter_search_paths = ["/usr/local/bin"]
(You possibly would need to include more values in this than just 
/usr/local/bin
)
b
Ok, give me a minute, I need to roll back some other changes I made to test it
👍 1
As a side note, I'm really confused why the official Python 3.7 Docker image comes with two separate Python installations
😓 1
👍 1
interpreter_constraints = ["==3.7.7"]
That did the trick! 👍
❤️ 2
h
Likely, the 
/usr/bin
one is installed by the OS. Those are hard to remove, and they’re usually out-of-date and also missing some key parts of a proper Python installation. So, many containers install a more modern Python. Which results in two 🤷‍♂️
That did the trick! 👍
Yay! Is this an acceptable solution for your use case? Usually people don’t like putting something as specific 
==3.7.7
in their config files because it risks things not working on other machines, e.g. if they have 
3.7.8
installed. But if everyone is using the same docker image, then that’s irrelevant
h
The official 3.8 image also has a 3.7 lying around, for some reason. Fun times!
b
Yeah, I fixed the docker image to use 3.7.7 as a base so setting the constraint to that is not a problem. Thanks for your help!
❤️ 1
6 Views
Open in Slack
PreviousNext
Powered by