https://pantsbuild.org/ logo
#general
Title
# general
h

high-yak-85899

02/09/2023, 8:48 PM
I want to run a
sphinx.application.Sphinx
in a script (I know a bit different use than normal). I'd like to do this with Pants, but it can't find
sphinxcontrib-serializinghtml
. This is a transitive dependency of
sphinx
. Are there any options other than adding that transitive dependency to my
requirements.txt
?
f

fast-nail-55400

02/09/2023, 8:51 PM
Not really. While it may be a transitive dependency for one requirement, your proposed use is as a direct dependency of your code, so you should be declaring it as such.
That is, add it as a requirement.
b

bitter-ability-32190

02/09/2023, 8:54 PM
There's some thoughts around transitive dependencies as targets. But nothing concrete
f

fast-nail-55400

02/09/2023, 8:55 PM
And although I imagine that
sphinx
would not remove this dependency from their own dependency graph, they are free to do so. Your code should not rely on assumptions built into sphinx when your code could just declare the dependency it needs.
h

high-yak-85899

02/09/2023, 9:00 PM
Yeah, that all makes sense to me. The only part where I slightly deviate in opinion is that calling
sphinx.application.Sphinx
has a valid input with
html
as the builder name which makes it seem like I shouldn't need to separately say "oh by the way I know internally you need this library to do the thing"
And for the same reasoning, they're free to change it.
So who am I to say that's the right library to use
f

fast-nail-55400

02/09/2023, 10:05 PM
yeah sounds like they are exposing what could be an implementation detail then
software is too hard
1
e

enough-analyst-54434

02/09/2023, 11:00 PM
@high-yak-85899 what version of Sphinx? The
sphinxcontrib-serializinghtml
project is direct dependency of Sphinx main: https://github.com/sphinx-doc/sphinx/blob/master/pyproject.toml#L62 and so you'd only need a proper module-mapping for that i- if even that.
h

high-yak-85899

02/09/2023, 11:03 PM
Yeah, it's already in the environment. We'ere on 4.5 at the moment. I'll try out a module mapping. I guess I didn't think to try that first since we don't have that package in our direct dependencies so I thought module mappings would complain.
e

enough-analyst-54434

02/09/2023, 11:05 PM
Copy code
$ pex sphinx -- -c 'import sphinxcontrib.serializinghtml; print(sphinxcontrib.serializinghtml.__file__)'
/tmp/tmpt_1zhtm0/.bootstrap/pex/environment.py:654: PEXWarning: The `pkg_resources` package was loaded from a pex vendored version when declaring namespace packages defined by:

1. sphinxcontrib-devhelp==1.0.2 namespace packages:
  sphinxcontrib

2. sphinxcontrib-jsmath==1.0.1 namespace packages:
  sphinxcontrib

3. sphinxcontrib-serializinghtml==1.1.5 namespace packages:
  sphinxcontrib

4. sphinxcontrib-qthelp==1.0.3 namespace packages:
  sphinxcontrib

These distributions should fix their `install_requires` to include `setuptools`
  pex_warnings.warn(
/home/jsirois/.pex/installed_wheels/352a9a00ae864471d3a7ead8d7d79f5fc0b57e8b3f95e9867eb9eb28999b92fd/sphinxcontrib_serializinghtml-1.1.5-py2.py3-none-any.whl/sphinxcontrib/serializinghtml/__init__.py
So, it totally works with latest Sphinx.
@high-yak-85899 you only need a module mapping if you import the sphinxcontrib code like I did in this example proof. If you don't, you don't.
h

high-yak-85899

02/09/2023, 11:07 PM
Right, so then 4.5 must not declare it as a proper dependency?
e

enough-analyst-54434

02/09/2023, 11:07 PM
Note the NS package warnings though. You'll want an execution_mode="venv" pex_binary to work around issues there.
Nope, works fine:
Copy code
$ pex sphinx==4.5 -- -c 'import sphinxcontrib.serializinghtml; print(sphinxcontrib.serializinghtml.__file__)'
/tmp/tmp361dpp9r/.bootstrap/pex/environment.py:654: PEXWarning: The `pkg_resources` package was loaded from a pex vendored version when declaring namespace packages defined by:

1. sphinxcontrib-devhelp==1.0.2 namespace packages:
  sphinxcontrib

2. sphinxcontrib-jsmath==1.0.1 namespace packages:
  sphinxcontrib

3. sphinxcontrib-serializinghtml==1.1.5 namespace packages:
  sphinxcontrib

4. sphinxcontrib-qthelp==1.0.3 namespace packages:
  sphinxcontrib

These distributions should fix their `install_requires` to include `setuptools`
  pex_warnings.warn(
/home/jsirois/.pex/installed_wheels/352a9a00ae864471d3a7ead8d7d79f5fc0b57e8b3f95e9867eb9eb28999b92fd/sphinxcontrib_serializinghtml-1.1.5-py2.py3-none-any.whl/sphinxcontrib/serializinghtml/__init__.py
h

high-yak-85899

02/09/2023, 11:08 PM
So then what am I doing wrong with my
python_source
? When I do
dependencies --transitive
, it's only showing that
sphinx
is a dependency
Copy code
//:reqs#click
//:reqs#jinja2
//:reqs#pyyaml
//:reqs#sortedcontainers
//:reqs#sphinx
//:reqs#sphinx-rtd-theme
//:reqs#types-PyYAML
//requirements.txt:reqs
e

enough-analyst-54434

02/09/2023, 11:09 PM
Pants
dependencies
goal only does 1st party and direct 3rdparty.
👍 1
It doesn't do resolves or incorporate their results.
Basically it only lists text in your repo.
h

high-yak-85899

02/09/2023, 11:10 PM
So perhaps I need to make a dedicate
pex_binary
target? I was hoping just the
python_source
would work now that running that is separated from a pex in newer pants versions
e

enough-analyst-54434

02/09/2023, 11:11 PM
It might help if you include the actual real problem. A command line with full output.
This should work with the latest machinations of the runnable python_source, but I'm not sure what versions of Pants have the latest fixes there (it recently uses venvs)
h

high-yak-85899

02/09/2023, 11:12 PM
A minimal repro is just instantiating
sphinx.application.Sphinx
with a builder set to
"html"
and you get
Copy code
Running Sphinx v4.5.0
Traceback (most recent call last):
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/sphinx/registry.py", line 425, in load_extension
    mod = import_module(extname)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/sphinx/builders/epub3.py", line 12, in <module>
    from sphinx.builders import _epub_base
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/sphinx/builders/_epub_base.py", line 15, in <module>
    from sphinx.builders.html import BuildInfo, StandaloneHTMLBuilder
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/sphinx/builders/html/__init__.py", line 1295, in <module>
    import sphinxcontrib.serializinghtml  # NOQA
ModuleNotFoundError: No module named 'sphinxcontrib.serializinghtml'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/pex", line 236, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/usr/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/nathanael/devel/monorepo/astranis-python/astranis/service/action_triggers/documenter/config_documenter.py", line 77, in <module>
    build()
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/nathanael/devel/monorepo/astranis-python/astranis/service/action_triggers/documenter/config_documenter.py", line 70, in build
    sphinx = application.Sphinx(
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/sphinx/application.py", line 219, in __init__
    self.setup_extension(extension)
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/sphinx/application.py", line 380, in setup_extension
    self.registry.load_extension(self, extname)
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/sphinx/registry.py", line 428, in load_extension
    raise ExtensionError(__('Could not import extension %s') % extname,
sphinx.errors.ExtensionError: Could not import extension sphinx.builders.epub3 (exception: No module named 'sphinxcontrib.serializinghtml')
e

enough-analyst-54434

02/09/2023, 11:13 PM
Great - it is running in a venv
Can you do
ls -1 /home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/
and provide the output?
h

high-yak-85899

02/09/2023, 11:14 PM
Untitled
So only one of the sphinxcontrib dependencies
e

enough-analyst-54434

02/09/2023, 11:15 PM
Ok:
Copy code
PEX_EXTRA_SYS_PATH.pth
That's the problem. Some tools don't handle this - which supports building venvs from symlink farms. You can turn that off with
pex_binary
👍 1
Just a sec for the details...
You need Pants >= 2.15.x and:
Copy code
pex_binary(
  ...
  execution_mode="venv",
  venv_site_packages_copies=True,
)
https://www.pantsbuild.org/v2.15/docs/reference-pex_binary#codevenv_site_packages_copiescode
h

high-yak-85899

02/09/2023, 11:18 PM
Sounds good! I'll try updating sometime soon.
6 Views