https://pantsbuild.org/ logo
Join Slack
Powered by
I want to install a package from the subdirectory ...
# general
b
I want to install a package from the subdirectory of the repo because of its none-standard structure. With pip I can do:
Copy code
pip install dgllife@ <https://github.com/awslabs/dgl-lifesci/archive/54649fb.zip#subdirectory=python>
But if I put 
dgllife@ <https://github.com/awslabs/dgl-lifesci/archive/54649fb.zip#subdirectory=python>
in requirements.txt, pants doesn't seem to recognize options after 
#
. Is there a way to make it work?
c
b
Copy code
STDERR:
    ERROR: Command errored out with exit status 1:
     command: /Users/inho/.cache/pants/named_caches/pex_root/venvs/c9e55cc98846b062ba9676b3c1a5214512602544/9d312f04d8089113e5d6d47efe2f73b6e63eef4d/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/inho/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-z_0sfcuv/setup.py'"'"'; __file__='"'"'/Users/inho/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-z_0sfcuv/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --no-user-cfg egg_info --egg-base /Users/inho/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-pip-egg-info-q9nei2be
         cwd: /Users/inho/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-z_0sfcuv/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/tokenize.py", line 394, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/Users/inho/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-z_0sfcuv/setup.py'
    ----------------------------------------
WARNING: Discarding file:///Users/inho/.cache/pants/named_caches/pex_root/downloads/resolver_download.bbab485z/opt.homebrew.Cellar.python%403.10.3.10.8.Frameworks.Python.framework.Versions.3.10.bin.python3.10/54649fb.zip. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
so it looks like its looking for 
setup.py
at the root dir level anyway
pex stacktrace:
Copy code
File "/Users/inho/.cache/pants/named_caches/pex_root/installed_wheels/d70444dc4c636a154fc77836d81e657d7ad8d148f2baee61c6a62095ecf3d504/pex-2.1.102-py2.py3-none-any.whl/pex/bin/pex.py", line 775, in main
    catch(
  File "/Users/inho/.cache/pants/named_caches/pex_root/installed_wheels/d70444dc4c636a154fc77836d81e657d7ad8d148f2baee61c6a62095ecf3d504/pex-2.1.102-py2.py3-none-any.whl/pex/result.py", line 103, in catch
    return func(*args, **kwargs)
  File "/Users/inho/.cache/pants/named_caches/pex_root/installed_wheels/d70444dc4c636a154fc77836d81e657d7ad8d148f2baee61c6a62095ecf3d504/pex-2.1.102-py2.py3-none-any.whl/pex/bin/pex.py", line 798, in do_main
    pex_builder = build_pex(
  File "/Users/inho/.cache/pants/named_caches/pex_root/installed_wheels/d70444dc4c636a154fc77836d81e657d7ad8d148f2baee61c6a62095ecf3d504/pex-2.1.102-py2.py3-none-any.whl/pex/bin/pex.py", line 681, in build_pex
    result = resolve(
  File "/Users/inho/.cache/pants/named_caches/pex_root/installed_wheels/d70444dc4c636a154fc77836d81e657d7ad8d148f2baee61c6a62095ecf3d504/pex-2.1.102-py2.py3-none-any.whl/pex/resolver.py", line 948, in resolve
    build_and_install_request.install_distributions(
  File "/Users/inho/.cache/pants/named_caches/pex_root/installed_wheels/d70444dc4c636a154fc77836d81e657d7ad8d148f2baee61c6a62095ecf3d504/pex-2.1.102-py2.py3-none-any.whl/pex/resolver.py", line 634, in install_distributions
    build_results = self._wheel_builder.build_wheels(
  File "/Users/inho/.cache/pants/named_caches/pex_root/installed_wheels/d70444dc4c636a154fc77836d81e657d7ad8d148f2baee61c6a62095ecf3d504/pex-2.1.102-py2.py3-none-any.whl/pex/resolver.py", line 535, in build_wheels
    for build_result in execute_parallel(
  File "/Users/inho/.cache/pants/named_caches/pex_root/installed_wheels/d70444dc4c636a154fc77836d81e657d7ad8d148f2baee61c6a62095ecf3d504/pex-2.1.102-py2.py3-none-any.whl/pex/jobs.py", line 522, in execute_parallel
    raise error
  File "/Users/inho/.cache/pants/named_caches/pex_root/installed_wheels/d70444dc4c636a154fc77836d81e657d7ad8d148f2baee61c6a62095ecf3d504/pex-2.1.102-py2.py3-none-any.whl/pex/jobs.py", line 544, in execute_parallel
    je_result = handler.handle_job_error(spawn_result.item, e)
  File "/Users/inho/.cache/pants/named_caches/pex_root/installed_wheels/d70444dc4c636a154fc77836d81e657d7ad8d148f2baee61c6a62095ecf3d504/pex-2.1.102-py2.py3-none-any.whl/pex/jobs.py", line 423, in handle_job_error
    raise self._raise_type(self.job_error_message(item, job_error))
I'm on pants 2.13.0 btw.
c
Thanks, that’s a little more to go on, I hope some of the other maintainers will pick it up from here 🙂
h
cc @happy-kitchen-89482 I think you were looking into this by adding Pip-style support, vs PEP 440?
e
Eric alludes to this, but the summary is: + 
dgllife@ https://...#subdirectory=python
is PEP-440 compliant but PEP-440 does not recognize 
#subdirectory
as anything special. + `https://...#egg=dgllife&subdirectory=python`is Pip-proprietary and does recognize subdirectory. Pex supports both (see https://github.com/pantsbuild/pex/issues/1751 for an example of a workaround for fbprophet enabled by Pip proprietary support), Pants only supports PEP-440.
Ok, interesting. I forgot about this: https://github.com/pantsbuild/pants/pull/15097#discussion_r847385181 Benjy's experiments match the OP claim. I wonder if older Pip does not support these franken requirements that are half PEP-440 half pip-proprietary but newer Pip does? Pex uses Pip 20.3.4 by default. I'll check this.
Nope, Benjy's example works using Pex directly with the default Pip:
Copy code
$ pex "sdev_logging_utils @ git+<https://github.com/SerialDev/sdev_py_utils.git@bd4d36a0#egg=sdev_logging_utils&subdirectory=sdev_logging_utils>"
Python 3.10.6 (main, Aug 10 2022, 11:40:04) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import sdev_logging_utils
>>> sdev_logging_utils.__file__
'/home/jsirois/.pex/installed_wheels/0e774b3b34ba18676a3e58e2b6972fa6ae1cf6e7c439f4bf3f6779374d9b126c/sdev_logging_utils-0.1-py3-none-any.whl/sdev_logging_utils/__init__.py'
>>>
now exiting InteractiveConsole...
@boundless-zebra-79556 can you try that?:
Copy code
pex "dgllife@ <https://github.com/awslabs/dgl-lifesci/archive/54649fb.zip#subdirectory=python>"
That should drop you in a REPL session where you can try to import symbols from the project.
b
I'm getting the same error:
Copy code
command: /Users/inho/.pex/venvs/3b4fe9e0bf2976bac979194c6c4bb8cf08d13bd7/9d312f04d8089113e5d6d47efe2f73b6e63eef4d/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/inho/.pex/pip_cache/.tmp/pip-req-build-g7to8y1c/setup.py'"'"'; __file__='"'"'/Users/inho/.pex/pip_cache/.tmp/pip-req-build-g7to8y1c/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --no-user-cfg egg_info --egg-base /Users/inho/.pex/pip_cache/.tmp/pip-pip-egg-info-kdwh_f3n
         cwd: /Users/inho/.pex/pip_cache/.tmp/pip-req-build-g7to8y1c/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/tokenize.py", line 394, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/Users/inho/.pex/pip_cache/.tmp/pip-req-build-g7to8y1c/setup.py'
I'm using pex==2.1.112 btw
doing 
#egg=dgllife&subdirectory=python
gives the same error btw
e
Hrm, ok. So, slowing down. This is not a VCS url - it's an archive URL.
b
yes. I did it to point to a particular commit
e
So, I think you've found a bug, but you can do this another way too. Can you try this instead?:
Copy code
dgllife@ git+<https://github.com/awslabs/dgl-lifesci@54649fb#subdirectory=python>
That should work in Pip and Pex and Pants.
b
yeah just tested with pex and it worked! I liked archive url solution since it doesn't require the git toolchain
e
Gotcha.
I'll double check the archive here and file an issue, but that's what you're stuck with for now.
🙏 1
b
Thanks for the help! Guess I have a workaround to fall back on for my need for now
5 Views
Open in Slack
PreviousNext
Powered by