https://pantsbuild.org/ logo
Join Slack
Powered by
I’ve got a `pants-jupyter-plugin` related question...
# general
b
I’ve got a 
pants-jupyter-plugin
 related question. I am using 
pants-jupyter-plugin
  in AWS SageMaker Studio. I have a failure with 
%pants_load
, but there is little information in the error log and I am struggling to investigate.
%pants_load /root/xxx/src/python/lib/xxx:pex_binary
✓ [Build] ./pants package /root/xxx-experimental/src/python/lib/xxx:pex_binary
Copy code
$ cd /root/xxx-experimental && ./pants --pants-distdir='/root/xxxx-experimental/dist/tmpammo36v6' package /root/media-analyzer-experimental/src/python/lib/xxx:pex_binary
08:01:04.42 [INFO] Initialization options changed: reinitializing scheduler...
08:01:06.83 [INFO] Scheduler initialized.
08:01:18.21 [INFO] Wrote dist/tmpammo36v6/src.python.lib.xxx/pex_binary.pex

Successfully built /root/xxx-experimental/dist/tmpammo36v6/src.python.lib.media_analyzer/pex_binary.pex
✗ [Bootstrap] pex_binary.pex
Copy code
Scrubbing sys.path and sys.modules in preparation for pex bootstrap
sys.path contains 8 items, sys.modules contains 987 keys
sys.path now contains 8 items, sys.modules now contains 987 keys
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
/usr/local/lib/python3.8/site-packages/pants_jupyter_plugin/plugin.py in _accordion_widget(self, title, height, collapsed)
    147         # Capture the output context.
    148         with outputter:
--> 149             yield expand, collapse, set_output_glyph
    150 
    151     def _stream_binary_build_with_output(

/usr/local/lib/python3.8/site-packages/pants_jupyter_plugin/plugin.py in _bootstrap_pex(self, pex_path)
    295 
    296                     # Bootstrap pex.
--> 297                     for path in self._pex_manager.mount(pex_path):
    298                         self._display_line(f"added sys.path entry {path}\n")
    299             except Exception:

/usr/local/lib/python3.8/site-packages/pants_jupyter_plugin/pex.py in mount(self, pex_to_mount)
    170                     )
    171                 venv_tmp = venv.parent / f"{venv.name}.{uuid4().hex}"
--> 172                 run_pex_tool(args=["venv", str(venv_tmp)])
    173                 venv_tmp.rename(venv)
    174 

/usr/local/lib/python3.8/site-packages/pants_jupyter_plugin/pex.py in run_pex_tool(args, **subprocess_args)
    131                 def run_pex_tool(args: Iterable[str], **subprocess_args: Any) -> bytes:
    132                     return (
--> 133                         subprocess.run(
    134                             args=[
    135                                 sys.executable,

/usr/local/lib/python3.8/subprocess.py in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    510         retcode = process.poll()
    511         if check and retcode:
--> 512             raise CalledProcessError(retcode, process.args,
    513                                      output=stdout, stderr=stderr)
    514     return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['/usr/local/bin/python', '/root/.cache/pants_jupyter_plugin/pex/exes/pex-2.1.56.pex', '-m', 'pex.tools', '/root/xxx-experimental/dist/tmpammo36v6/src.python.lib.xxx/pex_binary.pex', 'venv', '/root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.80bcffde']' returned non-zero exit status 1.
BUILD file
Copy code
pex_binary(
    name="pex_binary",
    dependencies=[
        "src/python/lib/xxx/core:lib",
        "src/python/lib/xxx/datastore:lib",
        "src/python/lib/xxx/ml:lib",
        "src/python/lib/xxx/transform:lib",
    ]
)
However, if I run the following in other cells, it will succeed.
Copy code
!cd /root/xxx-experimental && ./pants run /root/xxx-experimental/src/python/lib/xxx:pex_binary
There are many dependent libraries and it takes a long time to install them, so I’m guessing that’s the cause.
https://github.com/pantsbuild/pants-jupyter-plugin/blob/main/pants_jupyter_plugin/pex.py#L131-L149 There doesn’t seem to be a timeout setting, so that guess may be wrong.
e
@bland-father-19717 can you try running the PEX directly to see if that reveal more? Basically just run the exact command shown as failing:
Copy code
!/usr/local/bin/python /root/.cache/pants_jupyter_plugin/pex/exes/pex-2.1.56.pex -m pex.tools /root/xxx-experimental/dist/tmpammo36v6/src.python.lib.xxx/pex_binary.pex venv /root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.test
Here I just changed the original 
/root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.80bcffde
to `/root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.test`in case that original venv target directory was created as part of the error.
b
Thank you for the quick reply. I got the following error.
Copy code
$ !/usr/local/bin/python /root/.cache/pants_jupyter_plugin/pex/exes/pex-2.1.56.pex -m pex.tools /root/media-analyzer-experimental/dist/tmpammo36v6/src.python.lib.media_analyzer/pex_binary.pex venv /root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.test

Traceback (most recent call last):
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/requirements.py", line 106, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1654, in parseString
    raise exc
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1644, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 3417, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 3739, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 3400, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1406, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 2711, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pex.vendor._vendored.setuptools.pkg_resources._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 3133, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/requirements.py", line 108, in __init__
    raise InvalidRequirement(
pex.vendor._vendored.setuptools.pkg_resources.extern.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'/root/me'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/requirements.py", line 498, in _parse_requirement_line
    return PyPIRequirement(line, Requirement.parse(processed_text), editable=editable)
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 3180, in parse
    req, = parse_requirements(s)
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 3126, in parse_requirements
    yield Requirement(line)
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 3135, in __init__
    raise RequirementParseError(str(e))
pex.vendor._vendored.setuptools.pkg_resources.RequirementParseError: Invalid requirement, parse error at "'/root/me'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 476, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 532, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 668, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 700, in execute_pkg_resources
    return runner()
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/bin/pex.py", line 666, in main
    do_main(
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/bin/pex.py", line 687, in do_main
    pex_builder = build_pex(
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/bin/pex.py", line 578, in build_pex
    result = resolve(
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/resolver.py", line 896, in resolve
    direct_requirements = _parse_reqs(requirements, requirement_files, network_configuration)
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/resolver.py", line 782, in _parse_reqs
    return requirement_configuration.parse_requirements(network_configuration=network_configuration)
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/resolve/requirement_configuration.py", line 29, in parse_requirements
    parsed_requirements.extend(parse_requirement_strings(self.requirements))
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/requirements.py", line 646, in parse_requirement_strings
    yield _parse_requirement_line(
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/requirements.py", line 500, in _parse_requirement_line
    raise ParseError(
pex.requirements.ParseError: <string> line 1:
/root/media-analyzer-experimental/dist/tmpammo36v6/src.python.lib.media_analyzer/pex_binary.pex
Problem parsing '/root/media-analyzer-experimental/dist/tmpammo36v6/src.python.lib.media_analyzer/pex_binary.pex' as a requirement: Invalid requirement, parse error at "'/root/me'"
@enough-analyst-54434
e
My apologies. I missed one important detail! You need to export PEX_TOOLS=1, so?:
Copy code
!PEX_TOOLS=1 /usr/local/bin/python /root/.cache/pants_jupyter_plugin/pex/exes/pex-2.1.56.pex -m pex.tools /root/xxx-experimental/dist/tmpammo36v6/src.python.lib.xxx/pex_binary.pex venv /root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.test
I'm not sure exactly how 
!
works in a notebook though. That example uses a bash / shell-ism with the 
ENV_VAR=VALUE ...rest
structure. There may be some other way you need to do this in a notebook.
Oh wait - sorry again, that's not it.
Just a minute...
👌 1
Copy code
!PEX_INTERPRETER=1 PEX_PYTHON_PATH=/usr/local/bin/python /usr/local/bin/python /root/.cache/pants_jupyter_plugin/pex/exes/pex-2.1.56.pex -m pex.tools /root/xxx-experimental/dist/tmpammo36v6/src.python.lib.xxx/pex_binary.pex venv /root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.test
That's exactly the environment + command being run.
b
Thanks. it is running.
Copy code
Traceback (most recent call last):
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 476, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 517, in _execute
    return self.execute_interpreter()
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 579, in execute_interpreter
    return self.execute_module(module, alter_sys=True)
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 691, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=alter_sys)
  File "/usr/local/lib/python3.8/runpy.py", line 206, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/local/lib/python3.8/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/local/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/tools/__main__.py", line 10, in <module>
    sys.exit(tools())
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/tools/main.py", line 90, in main
    result = catch(pex_command.run, pex)
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/commands/command.py", line 130, in catch
    return func(*args, **kwargs)
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/tools/commands/venv.py", line 416, in run
    populate_venv_with_pex(
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/tools/commands/venv.py", line 136, in populate_venv_with_pex
    raise CollisionError(message)
pex.tools.commands.venv.CollisionError: Encountered collision building venv at /root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.test from /root/media-analyzer-experimental/dist/tmpammo36v6/src.python.lib.media_analyzer/pex_binary.pex:
1. /root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.test/bin/tensorboard was provided by:
	/root/.pex/installed_wheels/807998e8c36dd0abcbfc1c121fabb56c0bcea942/tensorflow-2.4.1-cp38-cp38-manylinux2010_x86_64.whl/bin/tensorboard
	/root/.pex/installed_wheels/2c63db6945005bcefe2b55eddd09560ae5e96c4f/tensorboard-2.8.0-py3-none-any.whl/bin/tensorboard
tensorboard is already installed on SageMaker Studio. One of the dependent libraries is tensorflow, and tensorflow is already installed in SageMaker Studio
e
Hrm.
So the PEX_TOOLS=1 ... venv command does support saying 
--collisions-ok
but the pants-jupyter-plugin doesn't currently pass this. Thinking what's best to do here...
@bland-father-19717 so in your case is that binary unused / irrelevant? 
bin/tensorboard
?
b
Yes!
/usr/local/bin/tensorboard
e
No, the collision here is purely inside the PEX. If you look at those two paths, one instance is provided by 
tensorflow-2.4.1-cp38-cp38-manylinux2010_x86_64.whl/bin/tensorboard
and one by 
tensorboard-2.8.0-py3-none-any.whl/bin/tensorboard
. Pex doesn't care about the 3rd 
/usr/local/bin/tensorboard
instance you point out.
The fact that tensorflow and tensorboard both use the same binary name 
tensorboard
is generally problematic!
OK, let me see what I can think up. I'll start with an issue. It may be that just saying 
--collisions-ok
is the way to go. IIRC that still issues a warning; so you won't be totally blind in the notebook when the collision later turns out to be important (if it does).
b
OK, Thank you for your support.
Copy code
!PEX_INTERPRETER=1 PEX_PYTHON_PATH=/usr/local/bin/python /usr/local/bin/python /root/.cache/pants_jupyter_plugin/pex/exes/pex-2.1.56.pex -m pex.tools /root/media-analyzer-experimental/dist/tmpammo36v6/src.python.lib.media_analyzer/pex_binary.pex venv --collisions-ok /root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.test
=>
Copy code
/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/tools/commands/venv.py:137: PEXWarning: Encountered collision building venv at /root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.test from /root/media-analyzer-experimental/dist/tmpammo36v6/src.python.lib.media_analyzer/pex_binary.pex:
1. /root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.test/bin/tensorboard was provided by:
	/root/.pex/installed_wheels/807998e8c36dd0abcbfc1c121fabb56c0bcea942/tensorflow-2.4.1-cp38-cp38-manylinux2010_x86_64.whl/bin/tensorboard
	/root/.pex/installed_wheels/2c63db6945005bcefe2b55eddd09560ae5e96c4f/tensorboard-2.8.0-py3-none-any.whl/bin/tensorboard
  pex_warnings.warn(message)
Warning only, probably successful.
e
Ok. I'll swing focus to this to get out a quick fix and release. Thanks for confirming @bland-father-19717.
b
Thanks for the quick response too.
e
b
Thanks!
I have been watching your quick fixes. Thank you. I tried it immediately and it worked.
!pip install git+<https://github.com/pantsbuild/pants-jupyter-plugin>
e
Ah, excellent. Thanks for confirming that! The 0.0.6 release will go out shortly.
b
Nice!!! Thank you.
e
b
I have never seen an OSS that goes from question to release so quickly. Thank you for your support.
5 Views
Open in Slack
PreviousNext
Powered by