https://pantsbuild.org/ logo
#general
Title
# general
r

ripe-cpu-85141

10/21/2022, 8:34 PM
Hello. I have a weird issue with
./pants generate-lockfiles
, The command takes forever as it is trying to compile NumPy. I can't find why it can't use a wheel instead of compiling the package and why it tries to install NumPy in the first place. I tried to
pip install -r reqs.txt
on a clean venv to see if something was pulling numpy but nothing appeared. Any idea why numpy appears and how I could debug that?
w

wide-midnight-78598

10/21/2022, 9:02 PM
What OS and interpreter_constraints? I ran into some numpy weirdness while generating lockfiles yesterday too, but in my case, it was the version attempted to be resolved by a newer python version (no clue even how that happened though)
r

ripe-cpu-85141

10/21/2022, 9:07 PM
oh I think I found why
I'm using snowflake-connector-python. I didn't put any extra so the installation went well. But if I put the pandas extra, it tries to install pyarrow 0.17 which wants numpy 1.14.* and cython
I wonder if pants figured out I needed the extra and included the deps
I guess I'm not completely clear about what's going on 🙂
c

curved-television-6568

10/21/2022, 9:19 PM
I’ve found it instructive to look at the pip logs as the data for the lockfile is being compiled. While the generate lockfiles goal is running I
ps ax | grep pants
and look for the
…/pip.log
file arg to a process like this one:
Copy code
49518   ??  S      0:00.17 /Users/x/.cache/pants/named_caches/pex_root/venvs/c9e55cc98846b062ba9676b3c1a5214512602544/b804a7d1f8bd9cd2ff386af6a48d90174070910e/bin/python -sE /Users/x/.cache/pants/named_caches/pex_root/venvs/c9e55cc98846b062ba9676b3c1a5214512602544/b804a7d1f8bd9cd2ff386af6a48d90174070910e/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /Users/x/.cache/pants/named_caches/pex_root/pip_cache --log /private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/pants-sandbox-HFUKU3/.tmp/pex-pip-log.3bwh4h5q/pip.log download --dest /private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/pants-sandbox-HFUKU3/.tmp/tmphlyaj6u0/Users.x..pyenv.versions.3.8.13.bin.python3.8 --requirement __pip_args.txt Brotli==1.0.9 Django==3.2.16 JPype1==0.7.1 J...
Namely
/private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/pants-sandbox-HFUKU3/.tmp/pex-pip-log.3bwh4h5q/pip.log
in this case, then tail that 🙂
running with increased pex verbosity is also frutiful
e

enough-analyst-54434

10/21/2022, 9:59 PM
@ripe-cpu-85141 can you answer @wide-midnight-78598's question? The interpreter constraints in particular would be useful to know.
r

ripe-cpu-85141

10/22/2022, 12:55 PM
@curved-television-6568 Ah good idea, I'll take a look
@wide-midnight-78598 @enough-analyst-54434 Sorry the interpreter constraints is as follow:
CPython==3.9.*
w

wide-midnight-78598

10/22/2022, 5:05 PM
Oh wow, yeah, if numpy 1.14 is getting lumped in, compilation makes more sense... So I guess the question is why that version is being pulled? Are you able to share your requirements.txt and/or pants.toml/BUILD (if there is python deps specified there)? Are you locking an old version of snowflake connector?
r

ripe-cpu-85141

10/24/2022, 2:57 PM
no, pretty recent
@wide-midnight-78598 but it seems to lock pyarrow to pyarrow 8.0
wait no, I don't find the dependency to numpy
well that's actually the PyArrow dependency that triggers the compilation 😕 It requires 0.17.*
pff, false lead. Compilation happened but that wasn't the real issue. It's the new dependency resolver that was too long. I put a minimal version on the requirements and it works better now. Thanks for the help!
5 Views