I have a target depend on a complex in house package it work

Question

I have a target depend on a complex in house package it works on `test` load and list out version but when using the same test goal with `repl` It said `AssertionError Failed to cache distribution deps idna 2 10 py2 py3 none any whl` Any idea how I can go deeper on this

plain-sundown-25537 · Answer

``` pants linfo repl examples test python greetings util test cbml py Traceback most recent call last File Users zhicongliang Documents GitHub pynest pants d tmptbz6uhg8 requirements pex bootstrap pex pex py line 482 in execute File Users zhicongliang Documents GitHub pynest pants d tmptbz6uhg8 requirements pex bootstrap pex pex py line 138 in activate File Users zhicongliang Documents GitHub pynest pants d tmptbz6uhg8 requirements pex bootstrap pex pex py line 125 in activate File Users zhicongliang Documents GitHub pynest pants d tmptbz6uhg8 requirements pex bootstrap pex environment py line 428 in activate File Users zhicongliang Documents GitHub pynest pants d tmptbz6uhg8 requirements pex bootstrap pex environment py line 784 in activate File Users zhicongliang Documents GitHub pynest pants d tmptbz6uhg8 requirements pex bootstrap pex environment py line 608 in resolve File Users zhicongliang Documents GitHub pynest pants d tmptbz6uhg8 requirements pex bootstrap pex environment py line 614 in resolve dists File Users zhicongliang Documents GitHub pynest pants d tmptbz6uhg8 requirements pex bootstrap pex environment py line 384 in update candidate distributions File Users zhicongliang Documents GitHub pynest pants d tmptbz6uhg8 requirements pex bootstrap pex environment py line 344 in load internal cache File Users zhicongliang Documents GitHub pynest pants d tmptbz6uhg8 requirements pex bootstrap pex environment py line 325 in write zipped internal cache File Users zhicongliang Documents GitHub pynest pants d tmptbz6uhg8 requirements pex bootstrap pex util ```

plain-sundown-25537 · Answer

The package seems to be there ``` find |grep idna 2 10 py2 py3 none any whl cache pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl cache pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl idna cache pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl idna pycache cache pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl idna pycache init cpython 38 pyc cache pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl idna pycache intranges cpython 38 pyc cache pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl idna pycache package data cpython 38 pyc cache pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl idna pycache core cpython 38 pyc cache pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl idna pycache idnadata cpython 38 pyc```

hundreds-father-404 · Answer

Hey < plain sundown 25537> pardon the delay What Pants version is this I see GitHub in the paths are you running `repl` in CI

plain-sundown-25537 · Answer

2 5 0rc0

plain-sundown-25537 · Answer

Nope I don t I was doing that in commandline

hundreds-father-404 · Answer

Hmm is it reproducible if you `mv ~ cache pants` to have a new name like `~ cache pants2` Rename rather than delete so we can grab relevant files

plain-sundown-25537 · Answer

Let me try I am outside now

hundreds-father-404 · Answer

No rush I m logging off for the night too

plain-sundown-25537 · Answer

Tried to removed the cache doesn t seem to work

plain-sundown-25537 · Answer

``` cache pants named caches pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl cache pants named caches pex root installed wheels c63d97c0a0865ea8d73fb42c11069d2326d2545a pip 21 1 1 py3 none any whl pip vendor idna cache pants named caches pex root installed wheels c63d97c0a0865ea8d73fb42c11069d2326d2545a pip 21 1 1 py3 none any whl pip vendor idna intranges py cache pants named caches pex root installed wheels c63d97c0a0865ea8d73fb42c11069d2326d2545a pip 21 1 1 py3 none any whl pip vendor idna package data py cache pants named caches pex root installed wheels c63d97c0a0865ea8d73fb42c11069d2326d2545a pip 21 1 1 py3 none any whl pip vendor idna compat py cache pants named caches pex root installed wheels c63d97c0a0865ea8d73fb42c11069d2326d2545a pip 21 1 1 py3 none any whl pip vendor idna idnadata py cache pants named caches pex root installed wheels c63d97c0a0865ea8d73fb42c11069d2326d2545a pip 21 1 1 py3 none any whl pip vendor idna init py cache pants named caches pex root installed wheels c63d97c0a0865ea8d73fb42c11069d2326d2545a pip 21 1 1 py3 none any whl pip vendor idna core py cache pants named caches pex root installed wheels c63d97c0a0865ea8d73fb42c11069d2326d2545a pip 21 1 1 py3 none any whl pip vendor idna codec py cache pants named caches pex root installed wheels c63d97c0a0865ea8d73fb42c11069d2326d2545a pip 21 1 1 py3 none any whl pip vendor idna uts46data py```

plain-sundown-25537 · Answer

^^ all seems normal to me

happy-kitchen-89482 · Answer

eyes

happy-kitchen-89482 · Answer

Can you post the full stacktrace The one above doesn t seem to include the actual error

happy-kitchen-89482 · Answer

< enough analyst 54434> This seems like it happens at pex bootstrapping time in case you have any debugging advice

enough-analyst-54434 · Answer

< plain sundown 25537> as a quick sanity check are permissions what you would expect given the user you re running this as vs the perms in cache pants named caches pex

plain-sundown-25537 · Answer

let me take a look

plain-sundown-25537 · Answer

these files are in 755 permission

enough-analyst-54434 · Answer

Best is probably just the full output of `ls lR cache pants named caches pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl ` I m using your output but it looks funky presumably ` cache pants` has a prefix

enough-analyst-54434 · Answer

If possible

plain-sundown-25537 · Answer

``` ls lR ~ cache pants named caches pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl total 0 drwxr xr x 10 zhicongliang staff 320 May 8 14 41 idna drwxr xr x 9 zhicongliang staff 288 May 8 14 41 idna 2 10 dist info Users zhicongliang cache pants named caches pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl idna total 552 rw r r 2 zhicongliang staff 58 May 8 14 41 init py rw r r 2 zhicongliang staff 3299 May 8 14 41 codec py rw r r 2 zhicongliang staff 232 May 8 14 41 compat py rw r r 2 zhicongliang staff 11951 May 8 14 41 core py rw r r 2 zhicongliang staff 42350 May 8 14 41 idnadata py rw r r 2 zhicongliang staff 1749 May 8 14 41 intranges py rw r r 2 zhicongliang staff 22 May 8 14 41 package data py rw r r 2 zhicongliang staff 202084 May 8 14 41 uts46data py Users zhicongliang cache pants named caches pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl idna 2 10 dist info total 64 rw r r 2 zhicongliang staff 4 May 8 14 41 INSTALLER rw r r 2 zhicongliang staff 1565 May 8 14 41 LICENSE rst rw r r 2 zhicongliang staff 9104 May 8 14 41 METADATA rw r r 2 zhicongliang staff 1116 May 8 14 41 RECORD rw r r 2 zhicongliang staff 0 May 8 14 41 REQUESTED rw r r 2 zhicongliang staff 110 May 8 14 41 WHEEL rw r r 2 zhicongliang staff 5 May 8 14 41 top level txt``` ^^ would this help

enough-analyst-54434 · Answer

Ok Another sanity check What do you get for ```PYTHONPATH=~ cache pants named caches pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl python3 8 c import idna print idna file ``` Here you can substitute the full path to the python you think or you know Pants is using when attempting to run repl against this target

plain-sundown-25537 · Answer

``` PYTHONPATH=~ cache pants named caches pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl python3 8 c import idna print idna file Users zhicongliang cache pants named caches pex root installed wheels 0e8ff7cceb2848e5adb0010e6d710edf33748117 idna 2 10 py2 py3 none any whl idna init py```

enough-analyst-54434 · Answer

Ok out of ideas using stock Pants and Pex I can make a patch that prints out more information that you could configure Pants to use to debug this I could either send you a Pex PEX with the patch or else post the PR and have you build a Pex PEX from that Does one of those options work for you

plain-sundown-25537 · Answer

sure

plain-sundown-25537 · Answer

pls let me know what I can help you may push a PEX branch and I build on my mac

enough-analyst-54434 · Answer

Ok Coming right up

plain-sundown-25537 · Answer

np

enough-analyst-54434 · Answer

Thanks a bunch < plain sundown 25537> Please try this out

enough-analyst-54434 · Answer

That will give you a bunch of output and a copy of the problematic PEX file saved off to a tmp dir I d definitely like all the output but if the PEX file is sensitive hold that back and just let me know you can t provide it

plain-sundown-25537 · Answer

< enough analyst 54434> I think I accidentally fixed it It is good news and bad news Here is what I did I add `idna==2 10` to requirements txt then run ` pants repl idna` on the repo and I saw some errors very similar see below then I found out a ` cache` folder and I removed ` cache` then the issue went away Bad news is I didn t archive the ` cache` so the evidence may have gone ```New virtual environment successfully created at Users zhicongliang cache pants setup bootstrap Darwin x86 64 2 5 0 dev3 py37 22 42 57 54 INFO initializing scheduler 22 42 58 62 INFO scheduler initialized 22 43 20 66 INFO Completed Resolving constraints txt 22 43 22 29 INFO Completed Building requirements pex with 1 requirement idna==2 10 0 Traceback most recent call last File Users zhicongliang Documents GitHub pynest pants d tmp6ku0p2rn requirements pex bootstrap pex pex py line 482 in execute File Users zhicongliang Documents GitHub pynest pants d tmp6ku0p2rn requirements pex bootstrap pex pex py line 138 in activate File Users zhicongliang Documents GitHub pynest pants d tmp6ku0p2rn requirements pex bootstrap pex pex py line 125 in activate File Users zhicongliang Documents GitHub pynest pants d tmp6ku0p2rn requirements pex bootstrap pex environment py line 428 in activate File Users zhicongliang Documents GitHub pynest pants d tmp6ku0p2rn requirements pex bootstrap pex environment py line 784 in activate File Users zhicongliang Documents GitHub pynest pants d tmp6ku0p2rn requirements pex bootstrap pex environment py line 608 in resolve File Users zhicongliang Documents GitHub pynest pants d tmp6ku0p2rn requirements pex bootstrap pex environment py line 614 in resolve dists File Users zhicongliang Documents GitHub pynest pants d tmp6ku0p2rn requirements pex bootstrap pex environment py line 384 in update candidate distributions File Users zhicongliang Documents GitHub pynest pants d tmp6ku0p2rn requirements pex bootstrap pex environment py line 344 in load internal cache File Users zhicongliang Documents GitHub pynest pants d tmp6ku0p2rn requirements pex bootstrap pex environment py line 325 in write zipped internal cache File Users zhicongliang Documents GitHub pynest pants d tmp6ku0p2rn requirements pex bootstrap pex util py line 222 in cache distribution AssertionError Failed to cache distribution deps idna 2 10 py2 py3 none any whl```

plain-sundown-25537 · Answer

In pants 1 x when I had issues I did ` pants cleanup` and it removes cache rekick pants d etc However in Pants 2 x we don t have similar command I didn t know and forgot to delete ` cache` in pants folder I cannot remember we had cache in Pants 1 x Apparently this issue has been happening on my machine for days and survived through reboots It seems my cache got polluted for whatever reasons I would suggest we implement some command that allow users to reset pants if we haven t had that yet

enough-analyst-54434 · Answer

I m glad you re unblocked < plain sundown 25537> but the loss of debugging context is unfortunate We definitely want to avoid adding any form of fix it hammer like `clean` in v2 but we also want to avoid issues like you hit here What s confusing is ` lt buildroot gt cache` which sounds like what you deleted to get things working is a Pants v1 only directory It s the default location for ` cache read from` and ` cache read to` for v1 style caching Pants v2 does not use this style of caching and so couldn t read contents from that directory even if it wanted to Do you have any Pants v2 configuration that specifies ` lt buildroot gt cache` as the location for anything

plain-sundown-25537 · Answer

Interesting and I use ` pants help all |grep cache` and found out ``` alias always write cache description Whether PEX should always write the deps cache of the pex file to disk or not This can use less memory in RAM constrained environments default Users zhicongliang cache pants lmdb store value Users zhicongliang cache pants lmdb store ```

plain-sundown-25537 · Answer

I do not think I have any cache settings from my pants toml which is largely copy from example <https github com pantsbuild example python|repo>

enough-analyst-54434 · Answer

An unrelated issue had me discovering where ` lt buildroot gt cache` comes from just now We leak this for the ` pants run|repl ` cases Basically for all non interactive processes like those spawned by ` pants test ` we assemble all needed files in a ` tmp process executionXYZ` chroot We also symlink ` tmp process executionXYZ cache pex root gt ~ cache pants named caches pex root` That symlink is not set up for interactive processes and so a directory is created in CWD the buildroot for those That explains your ` cache pex root` directory If we had a long listing of that Pex cache directory before it was deleted I m pretty sure it would have either had permissions issues or missing files explaining the issue you encountered I ll file a dedicated issue to get this leak fixed up here in a bit Thanks for uncovering this < plain sundown 25537>

enough-analyst-54434 · Answer

plain-sundown-25537 · Answer

< enough analyst 54434> orthogonal to this can we have a command Pantsv2 to build export a python root of a target this will help tons for cases like notebook IDE Current way to do that seems complex for me

hundreds-father-404 · Answer

Yes that s a popular user request < plain sundown 25537>

plain-sundown-25537 · Answer

I vote for this one In fact arguable this is adoption blockers for DS

hundreds-father-404 · Answer

It s on the roadmap for later this year <https groups google com u 1 g pants devel c F8Saug3BrFw m ZtWDP4YmDwAJ>

enough-analyst-54434 · Answer

If we supported building a PEX for arbitrary python targets this would be 2 steps ` pants pex o dist arbitrary targets or files or 3rdparty deps amp amp PEX TOOLS=1 dist arbitrary venv dist create venv right here` Of course 1 step would be nicer

happy-kitchen-89482 · Answer

< rough minister 58256> has opinions I believe

plain-sundown-25537 · Answer

I usually do ` pants repl xyz` to experiment and always love to export the venv and start with the editor

enough-analyst-54434 · Answer

That s actually a great idea < plain sundown 25537> There may be complications because `repl` is a top level goal and not Python specific but in principle the handwave ` pants repl export=create the repl venv here xyz foo bar` is exactly the right context for any venv we generate It is or should be completely agnostic to the underlying target s types beyond they be pythony

rough-minister-58256 · Answer

imho we need to figure out how to get an index facade around pants such that pants learns to talk in terms of coherent collections of dists at a given sha or fingerprint vs only terminating at artifacts like pex at that point going pypa vs pex is a matter of ```pex r requirements txt o blah pex no pypi i http lt magic endpoint gt ``` or just as trivially ```pip install r requirements txt index http lt magic endpoint gt ```

rough-minister-58256 · Answer

if we can do this its basically a grand unification for python + build systems this interface would reap dividends across IDEs notebooks app deploys tools etc

rough-minister-58256 · Answer

reproducible incrementalism

rough-minister-58256 · Answer

generating a venv is problematic when you re starting from another venv this is an incredibly common case in DS ML + Docker land

happy-kitchen-89482 · Answer

Yeah I think this is potentially a great and groundbreaking idea Are you thinking that magic endpoint is a remote service or is it pantsd

plain-sundown-25537 · Answer

< rough minister 58256> the current way to solve the problem is keep updating a docker and reloading the docker I believe this is not the right way of self contained + repeatable build

rough-minister-58256 · Answer

yeah I think it s a service component HTTPS that is aligned to pants on top of git i e sha mapped topology how it s served is less important than the interface existing I think

rough-minister-58256 · Answer

a centralized resolvable index endpoint would be the most fluid from a UX perspective i e you don t have to worry about whether a local pantsd magic endpoint is running but it could be done in different ways you d have to consider e g central pushed to origin vs local sha vs uncommitted etc

plain-sundown-25537 · Answer

Should we move this discussion to a GitHub ticket

ambitious-student-81104 · Answer

Hi Super late to the game So if `~ cache pants` is a pants v1 directory what is the recommended way for deleting cache in pants v2

ambitious-student-81104 · Answer

Ahh found some docs <https www pantsbuild org docs troubleshooting cache or pantsd invalidation issues|here>

happy-kitchen-89482 · Answer

Great glad you found the answer slightly smiling face