I ve got a `pants jupyter plugin` related question I am usin

Question

I ve got a `pants jupyter plugin` related question I am using `pants jupyter plugin` in AWS SageMaker Studio I have a failure with ` pants load` but there is little information in the error log and I am struggling to investigate

bland-father-19717 · Answer

` pants load root xxx src python lib xxx pex binary` Build pants package root xxx experimental src python lib xxx pex binary ```$ cd root xxx experimental amp amp pants pants distdir= root xxxx experimental dist tmpammo36v6 package root media analyzer experimental src python lib xxx pex binary 08 01 04 42 INFO Initialization options changed reinitializing scheduler 08 01 06 83 INFO Scheduler initialized 08 01 18 21 INFO Wrote dist tmpammo36v6 src python lib xxx pex binary pex Successfully built root xxx experimental dist tmpammo36v6 src python lib media analyzer pex binary pex```

bland-father-19717 · Answer

Bootstrap pex binary pex ```Scrubbing sys path and sys modules in preparation for pex bootstrap sys path contains 8 items sys modules contains 987 keys sys path now contains 8 items sys modules now contains 987 keys CalledProcessError Traceback most recent call last usr local lib python3 8 site packages pants jupyter plugin plugin py in accordion widget self title height collapsed 147 Capture the output context 148 with outputter gt 149 yield expand collapse set output glyph 150 151 def stream binary build with output usr local lib python3 8 site packages pants jupyter plugin plugin py in bootstrap pex self pex path 295 296 Bootstrap pex gt 297 for path in self pex manager mount pex path 298 self display line f added sys path entry path n 299 except Exception usr local lib python3 8 site packages pants jupyter plugin pex py in mount self pex to mount 170 171 venv tmp = venv parent f venv name uuid4 hex gt 172 run pex tool args= venv str venv tmp 173 venv tmp rename venv 174 usr local lib python3 8 site packages pants jupyter plugin pex py in run pex tool args subprocess args 131 def run pex tool args Iterable str subprocess args Any gt bytes 132 return gt 133 subprocess run 134 args= 135 sys executable usr local lib python3 8 subprocess py in run input capture output timeout check popenargs kwargs 510 retcode = process poll 511 if check and retcode gt 512 raise CalledProcessError retcode process args 513 output=stdout stderr=stderr 514 return CompletedProcess process args retcode stdout stderr CalledProcessError Command usr local bin python root cache pants jupyter plugin pex exes pex 2 1 56 pex m pex tools root xxx experimental dist tmpammo36v6 src python lib xxx pex binary pex venv root cache pants jupyter plugin pex venvs e64be65ccbc3df pex binary pex 80bcffde returned non zero exit status 1 ```

bland-father-19717 · Answer

BUILD file ```pex binary name= pex binary dependencies= src python lib xxx core lib src python lib xxx datastore lib src python lib xxx ml lib src python lib xxx transform lib ```

bland-father-19717 · Answer

However if I run the following in other cells it will succeed ``` cd root xxx experimental amp amp pants run root xxx experimental src python lib xxx pex binary```

bland-father-19717 · Answer

There are many dependent libraries and it takes a long time to install them so I m guessing that s the cause

bland-father-19717 · Answer

<https github com pantsbuild pants jupyter plugin blob main pants jupyter plugin pex py L131 L149> There doesn t seem to be a timeout setting so that guess may be wrong

enough-analyst-54434 · Answer

< bland father 19717> can you try running the PEX directly to see if that reveal more Basically just run the exact command shown as failing ``` usr local bin python root cache pants jupyter plugin pex exes pex 2 1 56 pex m pex tools root xxx experimental dist tmpammo36v6 src python lib xxx pex binary pex venv root cache pants jupyter plugin pex venvs e64be65ccbc3df pex binary pex test``` Here I just changed the original ` root cache pants jupyter plugin pex venvs e64be65ccbc3df pex binary pex 80bcffde` to ` root cache pants jupyter plugin pex venvs e64be65ccbc3df pex binary pex test`in case that original venv target directory was created as part of the error

bland-father-19717 · Answer

Thank you for the quick reply I got the following error ```$ usr local bin python root cache pants jupyter plugin pex exes pex 2 1 56 pex m pex tools root media analyzer experimental dist tmpammo36v6 src python lib media analyzer pex binary pex venv root cache pants jupyter plugin pex venvs e64be65ccbc3df pex binary pex test Traceback most recent call last File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources vendor packaging requirements py line 106 in init req = REQUIREMENT parseString requirement string File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources vendor pyparsing py line 1654 in parseString raise exc File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources vendor pyparsing py line 1644 in parseString loc tokens = self parse instring 0 File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources vendor pyparsing py line 1402 in parseNoCache loc tokens = self parseImpl instring preloc doActions File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources vendor pyparsing py line 3417 in parseImpl loc exprtokens = e parse instring loc doActions File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources vendor pyparsing py line 1402 in parseNoCache loc tokens = self parseImpl instring preloc doActions File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources vendor pyparsing py line 3739 in parseImpl return self expr parse instring loc doActions callPreParse=False File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources vendor pyparsing py line 1402 in parseNoCache loc tokens = self parseImpl instring preloc doActions File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources vendor pyparsing py line 3400 in parseImpl loc resultlist = self exprs 0 parse instring loc doActions callPreParse=False File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources vendor pyparsing py line 1406 in parseNoCache loc tokens = self parseImpl instring preloc doActions File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources vendor pyparsing py line 2711 in parseImpl raise ParseException instring loc self errmsg self pex vendor vendored setuptools pkg resources vendor pyparsing ParseException Expected W abcd at char 0 line 1 col 1 During handling of the above exception another exception occurred Traceback most recent call last File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources init py line 3133 in init super Requirement self init requirement string File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources vendor packaging requirements py line 108 in init raise InvalidRequirement pex vendor vendored setuptools pkg resources extern packaging requirements InvalidRequirement Invalid requirement parse error at root me During handling of the above exception another exception occurred Traceback most recent call last File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex requirements py line 498 in parse requirement line return PyPIRequirement line Requirement parse processed text editable=editable File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources init py line 3180 in parse req = parse requirements s File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources init py line 3126 in parse requirements yield Requirement line File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex vendor vendored setuptools pkg resources init py line 3135 in init raise RequirementParseError str e pex vendor vendored setuptools pkg resources RequirementParseError Invalid requirement parse error at root me During handling of the above exception another exception occurred Traceback most recent call last File root pex unzipped pexes 47ec9dea46acc4e6373d07558f9eb2ad1c131f91 bootstrap pex pex py line 476 in execute exit value = self wrap coverage self wrap profiling self execute File root pex unzipped pexes 47ec9dea46acc4e6373d07558f9eb2ad1c131f91 bootstrap pex pex py line 401 in wrap coverage return runner args File root pex unzipped pexes 47ec9dea46acc4e6373d07558f9eb2ad1c131f91 bootstrap pex pex py line 432 in wrap profiling return runner args File root pex unzipped pexes 47ec9dea46acc4e6373d07558f9eb2ad1c131f91 bootstrap pex pex py line 532 in execute return self execute entry self pex info entry point File root pex unzipped pexes 47ec9dea46acc4e6373d07558f9eb2ad1c131f91 bootstrap pex pex py line 668 in execute entry return self execute pkg resources entry point File root pex unzipped pexes 47ec9dea46acc4e6373d07558f9eb2ad1c131f91 bootstrap pex pex py line 700 in execute pkg resources return runner File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex bin pex py line 666 in main do main File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex bin pex py line 687 in do main pex builder = build pex File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex bin pex py line 578 in build pex result = resolve File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex resolver py line 896 in resolve direct requirements = parse reqs requirements requirement files network configuration File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex resolver py line 782 in parse reqs return requirement configuration parse requirements network configuration=network configuration File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex resolve requirement configuration py line 29 in parse requirements parsed requirements extend parse requirement strings self requirements File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex requirements py line 646 in parse requirement strings yield parse requirement line File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex requirements py line 500 in parse requirement line raise ParseError pex requirements ParseError lt string gt line 1 root media analyzer experimental dist tmpammo36v6 src python lib media analyzer pex binary pex Problem parsing root media analyzer experimental dist tmpammo36v6 src python lib media analyzer pex binary pex as a requirement Invalid requirement parse error at root me ```

bland-father-19717 · Answer

< enough analyst 54434>

enough-analyst-54434 · Answer

My apologies I missed one important detail You need to export PEX TOOLS=1 so ``` PEX TOOLS=1 usr local bin python root cache pants jupyter plugin pex exes pex 2 1 56 pex m pex tools root xxx experimental dist tmpammo36v6 src python lib xxx pex binary pex venv root cache pants jupyter plugin pex venvs e64be65ccbc3df pex binary pex test``` I m not sure exactly how ` ` works in a notebook though That example uses a bash shell ism with the `ENV VAR=VALUE rest` structure There may be some other way you need to do this in a notebook

enough-analyst-54434 · Answer

Oh wait sorry again that s not it

enough-analyst-54434 · Answer

Just a minute

enough-analyst-54434 · Answer

``` PEX INTERPRETER=1 PEX PYTHON PATH= usr local bin python usr local bin python root cache pants jupyter plugin pex exes pex 2 1 56 pex m pex tools root xxx experimental dist tmpammo36v6 src python lib xxx pex binary pex venv root cache pants jupyter plugin pex venvs e64be65ccbc3df pex binary pex test```

enough-analyst-54434 · Answer

That s exactly the environment + command being run

bland-father-19717 · Answer

Thanks it is running

bland-father-19717 · Answer

```Traceback most recent call last File root pex unzipped pexes 47ec9dea46acc4e6373d07558f9eb2ad1c131f91 bootstrap pex pex py line 476 in execute exit value = self wrap coverage self wrap profiling self execute File root pex unzipped pexes 47ec9dea46acc4e6373d07558f9eb2ad1c131f91 bootstrap pex pex py line 401 in wrap coverage return runner args File root pex unzipped pexes 47ec9dea46acc4e6373d07558f9eb2ad1c131f91 bootstrap pex pex py line 432 in wrap profiling return runner args File root pex unzipped pexes 47ec9dea46acc4e6373d07558f9eb2ad1c131f91 bootstrap pex pex py line 517 in execute return self execute interpreter File root pex unzipped pexes 47ec9dea46acc4e6373d07558f9eb2ad1c131f91 bootstrap pex pex py line 579 in execute interpreter return self execute module module alter sys=True File root pex unzipped pexes 47ec9dea46acc4e6373d07558f9eb2ad1c131f91 bootstrap pex pex py line 691 in execute module runpy run module module name run name= main alter sys=alter sys File usr local lib python3 8 runpy py line 206 in run module return run module code code init globals run name mod spec File usr local lib python3 8 runpy py line 96 in run module code run code code mod globals init globals File usr local lib python3 8 runpy py line 86 in run code exec code run globals File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex tools main py line 10 in lt module gt sys exit tools File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex tools main py line 90 in main result = catch pex command run pex File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex commands command py line 130 in catch return func args kwargs File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex tools commands venv py line 416 in run populate venv with pex File root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex tools commands venv py line 136 in populate venv with pex raise CollisionError message pex tools commands venv CollisionError Encountered collision building venv at root cache pants jupyter plugin pex venvs e64be65ccbc3df pex binary pex test from root media analyzer experimental dist tmpammo36v6 src python lib media analyzer pex binary pex 1 root cache pants jupyter plugin pex venvs e64be65ccbc3df pex binary pex test bin tensorboard was provided by root pex installed wheels 807998e8c36dd0abcbfc1c121fabb56c0bcea942 tensorflow 2 4 1 cp38 cp38 manylinux2010 x86 64 whl bin tensorboard root pex installed wheels 2c63db6945005bcefe2b55eddd09560ae5e96c4f tensorboard 2 8 0 py3 none any whl bin tensorboard```

bland-father-19717 · Answer

tensorboard is already installed on SageMaker Studio One of the dependent libraries is tensorflow and tensorflow is already installed in SageMaker Studio

enough-analyst-54434 · Answer

Hrm

enough-analyst-54434 · Answer

So the PEX TOOLS=1 venv command does support saying ` collisions ok` but the pants jupyter plugin doesn t currently pass this Thinking what s best to do here

enough-analyst-54434 · Answer

< bland father 19717> so in your case is that binary unused irrelevant `bin tensorboard`

bland-father-19717 · Answer

Yes

bland-father-19717 · Answer

` usr local bin tensorboard`

enough-analyst-54434 · Answer

No the collision here is purely inside the PEX If you look at those two paths one instance is provided by `tensorflow 2 4 1 cp38 cp38 manylinux2010 x86 64 whl bin tensorboard` and one by `tensorboard 2 8 0 py3 none any whl bin tensorboard` Pex doesn t care about the 3rd ` usr local bin tensorboard` instance you point out

enough-analyst-54434 · Answer

The fact that tensorflow and tensorboard both use the same binary name `tensorboard` is generally problematic

enough-analyst-54434 · Answer

OK let me see what I can think up I ll start with an issue It may be that just saying ` collisions ok` is the way to go IIRC that still issues a warning so you won t be totally blind in the notebook when the collision later turns out to be important if it does

bland-father-19717 · Answer

OK Thank you for your support

bland-father-19717 · Answer

``` PEX INTERPRETER=1 PEX PYTHON PATH= usr local bin python usr local bin python root cache pants jupyter plugin pex exes pex 2 1 56 pex m pex tools root media analyzer experimental dist tmpammo36v6 src python lib media analyzer pex binary pex venv collisions ok root cache pants jupyter plugin pex venvs e64be65ccbc3df pex binary pex test``` = gt ``` root pex installed wheels dae57b1c3877fc0567acc0103c6437da021bcfdf pex 2 1 56 py2 py3 none any whl pex tools commands venv py 137 PEXWarning Encountered collision building venv at root cache pants jupyter plugin pex venvs e64be65ccbc3df pex binary pex test from root media analyzer experimental dist tmpammo36v6 src python lib media analyzer pex binary pex 1 root cache pants jupyter plugin pex venvs e64be65ccbc3df pex binary pex test bin tensorboard was provided by root pex installed wheels 807998e8c36dd0abcbfc1c121fabb56c0bcea942 tensorflow 2 4 1 cp38 cp38 manylinux2010 x86 64 whl bin tensorboard root pex installed wheels 2c63db6945005bcefe2b55eddd09560ae5e96c4f tensorboard 2 8 0 py3 none any whl bin tensorboard pex warnings warn message ```

bland-father-19717 · Answer

Warning only probably successful

enough-analyst-54434 · Answer

Ok I ll swing focus to this to get out a quick fix and release Thanks for confirming < bland father 19717>

bland-father-19717 · Answer

Thanks for the quick response too

enough-analyst-54434 · Answer

You can follow along here <https github com pantsbuild pants jupyter plugin issues 41>

bland-father-19717 · Answer

Thanks

bland-father-19717 · Answer

I have been watching your quick fixes Thank you I tried it immediately and it worked

bland-father-19717 · Answer

` pip install git+<https github com pantsbuild pants jupyter plugin>`

enough-analyst-54434 · Answer

Ah excellent Thanks for confirming that The 0 0 6 release will go out shortly

bland-father-19717 · Answer

Nice Thank you

enough-analyst-54434 · Answer

And it s live <https pypi org project pants jupyter plugin 0 0 6 >

bland-father-19717 · Answer

I have never seen an OSS that goes from question to release so quickly Thank you for your support