https://pantsbuild.org/ logo
Join Slack
Powered by
I have a target depend on a complex (in house pack...
# general
p
I have a target depend on a complex (in house package), it works on 
test
(load and list out version) - but when using the same test goal with 
repl
It said 
AssertionError: Failed to cache distribution: .deps/idna-2.10-py2.py3-none-any.whl
Any idea how I can go deeper on this?
Copy code
❯ ./pants -linfo repl  examples/test/python/greetings/util/test_cbml.py
Traceback (most recent call last):
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmptbz6uhg8/requirements.pex/.bootstrap/pex/pex.py", line 482, in execute
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmptbz6uhg8/requirements.pex/.bootstrap/pex/pex.py", line 138, in activate
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmptbz6uhg8/requirements.pex/.bootstrap/pex/pex.py", line 125, in _activate
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmptbz6uhg8/requirements.pex/.bootstrap/pex/environment.py", line 428, in activate
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmptbz6uhg8/requirements.pex/.bootstrap/pex/environment.py", line 784, in _activate
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmptbz6uhg8/requirements.pex/.bootstrap/pex/environment.py", line 608, in resolve
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmptbz6uhg8/requirements.pex/.bootstrap/pex/environment.py", line 614, in resolve_dists
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmptbz6uhg8/requirements.pex/.bootstrap/pex/environment.py", line 384, in _update_candidate_distributions
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmptbz6uhg8/requirements.pex/.bootstrap/pex/environment.py", line 344, in _load_internal_cache
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmptbz6uhg8/requirements.pex/.bootstrap/pex/environment.py", line 325, in _write_zipped_internal_cache
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmptbz6uhg8/requirements.pex/.bootstrap/pex/util.
The package seems to be there
Copy code
❯ find . |grep idna-2.10-py2.py3-none-any.whl
./.cache/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl
./.cache/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl/idna
./.cache/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl/idna/__pycache__
./.cache/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl/idna/__pycache__/__init__.cpython-38.pyc
./.cache/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl/idna/__pycache__/intranges.cpython-38.pyc
./.cache/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl/idna/__pycache__/package_data.cpython-38.pyc
./.cache/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl/idna/__pycache__/core.cpython-38.pyc
./.cache/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl/idna/__pycache__/idnadata.cpython-38.pyc
h
Hey @plain-sundown-25537 pardon the delay. What Pants version is this? I see GitHub in the paths - are you running 
repl
in CI?
p
2.5.0rc0
👍 1
Nope I don’t I was doing that in commandline
h
Hmm, is it reproducible if you 
mv ~/.cache/pants
to have a new name like 
~/.cache/pants2
? (Rename rather than delete so we can grab relevant files)
p
Let me try - I am outside now.
h
No rush, I'm logging off for the night too
👋 1
p
Tried to removed the cache - doesn’t seem to work
Copy code
.cache/pants//named_caches/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl
.cache/pants//named_caches/pex_root/installed_wheels/c63d97c0a0865ea8d73fb42c11069d2326d2545a/pip-21.1.1-py3-none-any.whl/pip/_vendor/idna
.cache/pants//named_caches/pex_root/installed_wheels/c63d97c0a0865ea8d73fb42c11069d2326d2545a/pip-21.1.1-py3-none-any.whl/pip/_vendor/idna/intranges.py
.cache/pants//named_caches/pex_root/installed_wheels/c63d97c0a0865ea8d73fb42c11069d2326d2545a/pip-21.1.1-py3-none-any.whl/pip/_vendor/idna/package_data.py
.cache/pants//named_caches/pex_root/installed_wheels/c63d97c0a0865ea8d73fb42c11069d2326d2545a/pip-21.1.1-py3-none-any.whl/pip/_vendor/idna/compat.py
.cache/pants//named_caches/pex_root/installed_wheels/c63d97c0a0865ea8d73fb42c11069d2326d2545a/pip-21.1.1-py3-none-any.whl/pip/_vendor/idna/idnadata.py
.cache/pants//named_caches/pex_root/installed_wheels/c63d97c0a0865ea8d73fb42c11069d2326d2545a/pip-21.1.1-py3-none-any.whl/pip/_vendor/idna/__init__.py
.cache/pants//named_caches/pex_root/installed_wheels/c63d97c0a0865ea8d73fb42c11069d2326d2545a/pip-21.1.1-py3-none-any.whl/pip/_vendor/idna/core.py
.cache/pants//named_caches/pex_root/installed_wheels/c63d97c0a0865ea8d73fb42c11069d2326d2545a/pip-21.1.1-py3-none-any.whl/pip/_vendor/idna/codec.py
.cache/pants//named_caches/pex_root/installed_wheels/c63d97c0a0865ea8d73fb42c11069d2326d2545a/pip-21.1.1-py3-none-any.whl/pip/_vendor/idna/uts46data.py
^^ all seems normal to me
h
👀
Can you post the full stacktrace? The one above doesn't seem to include the actual error.
@enough-analyst-54434 This seems like it happens at pex bootstrapping time, in case you have any debugging advice.
e
@plain-sundown-25537 as a quick sanity check are permissions what you would expect given the user you're running this as vs the perms in ....cache/pants/named_caches/pex?
p
let me take a look
these files are in 755 permission
e
Best is probably just the full output of: 
ls -lR .cache/pants/named_caches/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl/
(I'm using your output but it looks funky - presumably 
.cache/pants
has a prefix.
If possible.
p
Copy code
❯ ls -lR ~/.cache/pants/named_caches/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl/
total 0
drwxr-xr-x  10 zhicongliang  staff  320 May  8 14:41 idna
drwxr-xr-x   9 zhicongliang  staff  288 May  8 14:41 idna-2.10.dist-info

/Users/zhicongliang/.cache/pants/named_caches/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl//idna:
total 552
-rw-r--r--  2 zhicongliang  staff      58 May  8 14:41 __init__.py
-rw-r--r--  2 zhicongliang  staff    3299 May  8 14:41 codec.py
-rw-r--r--  2 zhicongliang  staff     232 May  8 14:41 compat.py
-rw-r--r--  2 zhicongliang  staff   11951 May  8 14:41 core.py
-rw-r--r--  2 zhicongliang  staff   42350 May  8 14:41 idnadata.py
-rw-r--r--  2 zhicongliang  staff    1749 May  8 14:41 intranges.py
-rw-r--r--  2 zhicongliang  staff      22 May  8 14:41 package_data.py
-rw-r--r--  2 zhicongliang  staff  202084 May  8 14:41 uts46data.py

/Users/zhicongliang/.cache/pants/named_caches/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl//idna-2.10.dist-info:
total 64
-rw-r--r--  2 zhicongliang  staff     4 May  8 14:41 INSTALLER
-rw-r--r--  2 zhicongliang  staff  1565 May  8 14:41 LICENSE.rst
-rw-r--r--  2 zhicongliang  staff  9104 May  8 14:41 METADATA
-rw-r--r--  2 zhicongliang  staff  1116 May  8 14:41 RECORD
-rw-r--r--  2 zhicongliang  staff     0 May  8 14:41 REQUESTED
-rw-r--r--  2 zhicongliang  staff   110 May  8 14:41 WHEEL
-rw-r--r--  2 zhicongliang  staff     5 May  8 14:41 top_level.txt
^^ would this help?
e
Ok. Another sanity check: What do you get for?:
Copy code
PYTHONPATH=~/.cache/pants/named_caches/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl python3.8 -c 'import idna; print(idna.__file__)'
Here you can substitute the full path to the python you think or you know Pants is using when attempting to run repl against this target.
p
Copy code
❯ PYTHONPATH=~/.cache/pants/named_caches/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl python3.8 -c 'import idna; print(idna.__file__)'

/Users/zhicongliang/.cache/pants/named_caches/pex_root/installed_wheels/0e8ff7cceb2848e5adb0010e6d710edf33748117/idna-2.10-py2.py3-none-any.whl/idna/__init__.py
e
Ok, out of ideas using stock Pants and Pex. I can make a patch that prints out more information that you could configure Pants to use to debug this. I could either send you a Pex PEX with the patch or else post the PR and have you build a Pex PEX from that. Does one of those options work for you?
p
sure
pls let me know what I can help - you may push a PEX branch and I build on my mac
e
Ok. Coming right up...
p
np
e
Thanks a bunch @plain-sundown-25537. Please try this out: https://github.com/pantsbuild/pex/pull/1348
That will give you a bunch of output and a copy of the problematic PEX file saved off to a tmp dir. I'd definitely like all the output, but if the PEX file is sensitive, hold that back and just let me know you can't provide it.
p
@enough-analyst-54434 I think I accidentally fixed it. It is good news and bad news. Here is what I did: I add 
idna==2.10
to requirements.txt then run 
./pants repl //:idna
on the repo and I saw some errors very similar (see below), then I found out a 
.cache
folder and I removed 
.cache
- then the issue went away. Bad news is I didn’t archive the 
.cache
so the evidence may have gone.
Copy code
New virtual environment successfully created at /Users/zhicongliang/.cache/pants/setup/bootstrap-Darwin-x86_64/2.5.0.dev3_py37.
22:42:57.54 [INFO] initializing scheduler...
22:42:58.62 [INFO] scheduler initialized.
22:43:20.66 [INFO] Completed: Resolving constraints.txt
22:43:22.29 [INFO] Completed: Building requirements.pex with 1 requirement: idna==2.10.0
Traceback (most recent call last):
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmp6ku0p2rn/requirements.pex/.bootstrap/pex/pex.py", line 482, in execute
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmp6ku0p2rn/requirements.pex/.bootstrap/pex/pex.py", line 138, in activate
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmp6ku0p2rn/requirements.pex/.bootstrap/pex/pex.py", line 125, in _activate
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmp6ku0p2rn/requirements.pex/.bootstrap/pex/environment.py", line 428, in activate
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmp6ku0p2rn/requirements.pex/.bootstrap/pex/environment.py", line 784, in _activate
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmp6ku0p2rn/requirements.pex/.bootstrap/pex/environment.py", line 608, in resolve
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmp6ku0p2rn/requirements.pex/.bootstrap/pex/environment.py", line 614, in resolve_dists
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmp6ku0p2rn/requirements.pex/.bootstrap/pex/environment.py", line 384, in _update_candidate_distributions
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmp6ku0p2rn/requirements.pex/.bootstrap/pex/environment.py", line 344, in _load_internal_cache
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmp6ku0p2rn/requirements.pex/.bootstrap/pex/environment.py", line 325, in _write_zipped_internal_cache
  File "/Users/zhicongliang/Documents/GitHub/pynest/.pants.d/tmp6ku0p2rn/requirements.pex/.bootstrap/pex/util.py", line 222, in cache_distribution
AssertionError: Failed to cache distribution: .deps/idna-2.10-py2.py3-none-any.whl
In pants 1.x, when I had issues, I did 
./pants cleanup
and it removes cache/rekick pants.d etc. However in Pants 2.x, we don’t have similar command. I didn’t know and forgot to delete 
.cache
in pants folder (I cannot remember we had .cache in Pants 1.x) Apparently, this issue has been happening on my machine for days and survived through reboots. It seems my cache got polluted (for whatever reasons). I would suggest we implement some command that allow users to reset “pants” if we haven’t had that yet
e
I'm glad you're unblocked @plain-sundown-25537 but the loss of debugging context is unfortunate. We definitely want to avoid adding any form of fix-it hammer like 
clean
in v2, but we also want to avoid issues like you hit here. What's confusing is 
<buildroot>/.cache
- which sounds like what you deleted to get things working - is a Pants v1 only directory. It's the default location for 
--cache-read-from
and 
--cache-read-to
for v1 style caching. Pants v2 does not use this style of caching and so couldn't read contents from that directory even if it wanted to. Do you have any Pants v2 configuration that specifies 
<buildroot>/.cache
as the location for anything?
p
Interesting .. and I use 
./pants help-all |grep cache
and found out
Copy code
"alias": "always_write_cache",
          "description": "Whether PEX should always write the .deps cache of the .pex file to disk or not. This can use less memory in RAM-constrained environments.",
          "default": "/Users/zhicongliang/.cache/pants/lmdb_store",
                "value": "/Users/zhicongliang/.cache/pants/lmdb_store"
I do not think I have any cache settings from my pants.toml which is largely copy from example repo
e
An unrelated issue (https://github.com/pantsbuild/pex/issues/1349) had me discovering where 
<buildroot>/.cache
comes from just now. We leak this for the 
./pants run|repl ...
cases. Basically, for all non-interactive processes - like those spawned by 
./pants test ...
we assemble all needed files in a 
/tmp/process-executionXYZ
chroot. We also symlink 
/tmp/process-executionXYZ/.cache/pex_root -> ~/.cache/pants/named_caches/pex_root
. That symlink is not set up for interactive processes and so a directory is created in CWD (the buildroot) for those. That explains your 
.cache/pex_root
directory. If we had a long listing of that Pex cache directory before it was deleted, I'm pretty sure it would have either had permissions issues or missing files explaining the issue you encountered. I'll file a dedicated issue to get this leak fixed up here in a bit. Thanks for uncovering this @plain-sundown-25537.
👍 1
p
@enough-analyst-54434 orthogonal to this - can we have a command Pantsv2 to build/export a python root of a target - this will help tons for cases like notebook/IDE Current way to do that seems complex for me. https://www.pantsbuild.org/v2.5/docs/setting-up-an-ide
h
Yes, that's a popular user request @plain-sundown-25537: https://github.com/pantsbuild/pants/issues/11152
p
I vote for this one! In fact arguable this is adoption blockers (for DS … )
h
e
If we supported building a PEX for arbitrary python targets this would be 2 steps: 
./pants pex -o dist/arbitrary (targets or files or 3rdparty deps) && PEX_TOOLS=1 dist/arbitrary venv dist/create/venv/right/here
Of course 1 step would be nicer.
h
@rough-minister-58256 has opinions I believe...
p
I usually do 
./pants repl //:xyz
to experiment, and always love to export the venv and start with the editor.
e
That's actually a great idea @plain-sundown-25537. There may be complications because 
repl
is a top level goal and not Python specific, but in principle, the handwave: 
./pants repl --export=create/the/repl/venv/here //:xyz foo:bar
is exactly the right context for any venv we generate. It is (or should be) completely agnostic to the underlying target(s) types beyond they be pythony.
👍 1
r
imho, we need to figure out how to get an “index” facade around pants such that pants learns to talk in terms of coherent collections of “dists” at a given sha or fingerprint vs only terminating at artifacts like pex. at that point going pypa vs pex is a matter of:
Copy code
pex -r requirements.txt -o blah.pex --no-pypi -i http://<magic endpoint>
or just as trivially:
Copy code
pip install -r requirements.txt --index http://<magic endpoint>
if we can do this, its basically a grand unification for python + build systems. this interface would reap dividends across IDEs, notebooks, app deploys, tools, etc.
“reproducible incrementalism”
generating a venv is problematic when you’re starting from another venv. this is an incredibly common case in DS/ML + Docker land
🎉 1
h
Yeah I think this is potentially a great and groundbreaking idea. Are you thinking that "magic endpoint" is a remote service, or is it pantsd?
p
@rough-minister-58256 the current way to solve the problem is keep updating a docker, and reloading the docker - I believe this is not the right way of self contained + repeatable build
r
yeah, I think it’s a service component (HTTPS) that is aligned to pants on top of git (i.e. sha-mapped). topology/how it’s served is less important than the interface existing, I think.
a centralized/resolvable index endpoint would be the most fluid from a UX perspective (i.e. you don’t have to worry about whether a local pantsd/magic endpoint is running), but it could be done in different ways. you’d have to consider e.g. central (pushed to origin) vs local sha vs uncommitted, etc.
p
Should we move this discussion to a GitHub ticket?
a
Hi! Super late to the game. So if 
~/.cache/pants
is a pants v1 directory, what is the recommended way for deleting cache in pants v2?
Ahh found some docs here
h
Great, glad you found the answer 🙂
Open in Slack
PreviousNext
Powered by