https://pantsbuild.org/ logo
#general
Title
# general
b

better-nail-12700

06/18/2020, 7:03 PM
Sorry to bother again: I'm having a problem with installing/building dependencies in Pants v2 inside of a
python:3.7-buster
container. While building wheels gcc throws
fatal error: Python.h: No such file or directory
. Python.h is indeed available under the following path:
/usr/local/includes/python3.7m/
, however, from the gcc call that fails I can see that only
/usr/includes/python3.7m/
is included. Any idea what the cause of this could be? Shouldn't it use the includes from the
~/.cache/pants/setup
directory anyway?
Specifically, it happens because one of my projects is depending on the
hdbscan
package. Here is the corresponding call:
Copy code
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python3.7m -I/tmp/pip-build-env-x1_hs1mx/overlay/lib/python3.7/site-packages/numpy/core/include -c hdbscan/_hdbscan_tree.c -o build/temp.linux-x86_64-3.7/hdbscan/_hdbscan_tree.o
👍 1
h

hundreds-father-404

06/18/2020, 7:07 PM
No need to apologize ever for asking questions! What command are you running?
I’m thinking to debug it would be helpful to play around with the
interpreter_search_paths
and/or
interpreter_constraints
options: interpreter_search_paths For example, with constraints, if you know the precise version of the interpreter you intend to use, you could set
interpreter_constraints == ["==3.7.4"]
Although, taking a step back, what is the output of
which -l python
? The above suggestion only helps if there are two Python 3.7 interpreters on your machine, and Pants is choosing the wrong one.
b

better-nail-12700

06/18/2020, 7:14 PM
This is the output of
which -a python
:
Copy code
/usr/local/bin/python
/usr/local/bin/python
/usr/bin/python
The one located at
/usr/bin/python
is Python 2.7.16, the other one is 3.7.7. I'm confused why it appears twice though
h

hundreds-father-404

06/18/2020, 7:21 PM
Indeed, that is confusing.. @witty-crayon-22786 any idea what that could be?
Also, what happens if you run
/usr/local/bin/pip install hdbscan
? (I want to double check that the install would work so that we can isolate if this issue is with Pants vs. the underlying Python installation)
(warning that that will pollute your global
site-packages
. But it sounds like this is docker at least)
w

witty-crayon-22786

06/18/2020, 7:30 PM
yea, it sounds like Eric is on the right track.
the python that pants uses to run itself (ie
~/.cache/pants/setup
) is independent from what it will use to build your dependencies, which is controlled by the interpreter search path option
👍 1
b

better-nail-12700

06/18/2020, 7:32 PM
/usr/local/bin/pip install hdbscan
does indeed build the wheel without a problem 🤔
👍 1
w

witty-crayon-22786

06/18/2020, 7:33 PM
if this is a relatively small build, you can pass
-ldebug
which will dump “a lot” of information about what pex is doing with pip
but, actually.
b

better-nail-12700

06/18/2020, 7:34 PM
Curiously, the interpreter that is used to execute the pip.pex (
/usr/bin/python3.7 /root/.cache/pants/named_caches/pex_root/pip.pex/240b20f98d345b70b316f7e734d4cea20bde73b6
) is called using
/usr/bin/python3.7
w

witty-crayon-22786

06/18/2020, 7:34 PM
for a failing run like this, there should actually be a
pip
command rendered in the output
do you see that?
👍 1
h

hundreds-father-404

06/18/2020, 7:34 PM
(irrelevant suggestion) Hm, idea. There are two flags we use to control the LDFLAGS and CPPFLAGS passed to Python processes.
ld_flags
and
cpp_flags
, both in the
[python-native-code]
options scope. They default to reading in the env vars
What does
echo $LDFLAGS
and
echo $CPPFLAGS
do?
w

witty-crayon-22786

06/18/2020, 7:34 PM
@better-nail-12700: yea, that is highly relevant.
👍 1
b

better-nail-12700

06/18/2020, 7:34 PM
Yes I see the command:
Copy code
/usr/bin/python3.7 /root/.cache/pants/named_caches/pex_root/pip.pex/240b20f98d345b70b316f7e734d4cea20bde73b6 --disable-pip-version-check --isolated --no-python-version-warning -q --cache-dir /root/.cache/pants/named_caches/pex_root wheel --no-deps --wheel-dir /root/.cache/pants/named_caches/pex_root/built_wheels/sdists/hdbscan-0.8.26.tar.gz/4e23f82a649125d3f93c58dbe9e0d190bf291947/cp37-cp37m.fd92366b7396429f905761e9ca34406c --index-url <https://pypi.org/simple/> --header Cache-Control:max-age=3600 --retries 5 --timeout 15 /tmp/tmpg5kitjbw/cp37-cp37m/hdbscan-0.8.26.tar.gz raised Executing /usr/bin/python3.7 /root/.cache/pants/named_caches/pex_root/pip.pex/240b20f98d345b70b316f7e734d4cea20bde73b6 --disable-pip-version-check --isolated --no-python-version-warning -q --cache-dir /root/.cache/pants/named_caches/pex_root wheel --no-deps --wheel-dir /root/.cache/pants/named_caches/pex_root/built_wheels/sdists/hdbscan-0.8.26.tar.gz/4e23f82a649125d3f93c58dbe9e0d190bf291947/cp37-cp37m.fd92366b7396429f905761e9ca34406c --index-url <https://pypi.org/simple/> --header Cache-Control:max-age=3600 --retries 5 --timeout 15 /tmp/tmpg5kitjbw/cp37-cp37m/hdbscan-0.8.26.tar.gz
w

witty-crayon-22786

06/18/2020, 7:35 PM
so yea… that’s the python it’s using.
👍 1
so eric’s link to the interpreter search paths docs are the path forward i think.
👍 1
h

hundreds-father-404

06/18/2020, 7:36 PM
What does
/usr/bin/python3.7 --version
give you? Vs.
/usr/local/bin/python3.7 --version
b

better-nail-12700

06/18/2020, 7:36 PM
😂
Copy code
root@14c340f404ce:/# /usr/bin/python3.7 --version
Python 3.7.3
root@14c340f404ce:/# /usr/local/bin/python3.7 --version
Python 3.7.7
Seems like I have two Python3.7s afterall
h

hundreds-father-404

06/18/2020, 7:37 PM
Indeed. So, to confirm this will fix it, try setting this:
Copy code
[python-setup]
interpreter_constraints = ["==3.7.7"]
Or, you can set:
Copy code
[python-setup]
interpreter_search_paths = ["/usr/local/bin"]
(You possibly would need to include more values in this than just
/usr/local/bin
)
b

better-nail-12700

06/18/2020, 7:40 PM
Ok, give me a minute, I need to roll back some other changes I made to test it
👍 1
As a side note, I'm really confused why the official Python 3.7 Docker image comes with two separate Python installations
😓 1
👍 1
interpreter_constraints = ["==3.7.7"]
That did the trick! 👍
❤️ 2
h

hundreds-father-404

06/18/2020, 7:47 PM
Likely, the
/usr/bin
one is installed by the OS. Those are hard to remove, and they’re usually out-of-date and also missing some key parts of a proper Python installation. So, many containers install a more modern Python. Which results in two 🤷‍♂️
That did the trick! 👍
Yay! Is this an acceptable solution for your use case? Usually people don’t like putting something as specific
==3.7.7
in their config files because it risks things not working on other machines, e.g. if they have
3.7.8
installed. But if everyone is using the same docker image, then that’s irrelevant
h

happy-kitchen-89482

06/18/2020, 8:09 PM
The official 3.8 image also has a 3.7 lying around, for some reason. Fun times!
b

better-nail-12700

06/19/2020, 2:50 PM
Yeah, I fixed the docker image to use 3.7.7 as a base so setting the constraint to that is not a problem. Thanks for your help!
❤️ 1
6 Views