I want to run a `sphinx application Sphinx` in a script I kn Pants #general

I want to run a `sphinx.application.Sphinx` in a s...

high-yak-85899

02/09/2023, 8:48 PM

I want to run a

sphinx.application.Sphinx

in a script (I know a bit different use than normal). I'd like to do this with Pants, but it can't find

sphinxcontrib-serializinghtml

. This is a transitive dependency of

sphinx

. Are there any options other than adding that transitive dependency to my

requirements.txt

fast-nail-55400

02/09/2023, 8:51 PM

Not really. While it may be a transitive dependency for one requirement, your proposed use is as a direct dependency of your code, so you should be declaring it as such.

fast-nail-55400

02/09/2023, 8:52 PM

That is, add it as a requirement.

bitter-ability-32190

02/09/2023, 8:54 PM

There's some thoughts around transitive dependencies as targets. But nothing concrete

fast-nail-55400

02/09/2023, 8:55 PM

And although I imagine that

sphinx

would not remove this dependency from their own dependency graph, they are free to do so. Your code should not rely on assumptions built into sphinx when your code could just declare the dependency it needs.

high-yak-85899

02/09/2023, 9:00 PM

Yeah, that all makes sense to me. The only part where I slightly deviate in opinion is that calling

sphinx.application.Sphinx

has a valid input with

html

as the builder name which makes it seem like I shouldn't need to separately say "oh by the way I know internally you need this library to do the thing"

high-yak-85899

02/09/2023, 9:01 PM

And for the same reasoning, they're free to change it.

high-yak-85899

02/09/2023, 9:01 PM

So who am I to say that's the right library to use

fast-nail-55400

02/09/2023, 10:05 PM

yeah sounds like they are exposing what could be an implementation detail then

fast-nail-55400

02/09/2023, 10:05 PM

software is too hard

➕ 1

enough-analyst-54434

02/09/2023, 11:00 PM

@high-yak-85899 what version of Sphinx? The

sphinxcontrib-serializinghtml

project is direct dependency of Sphinx main: https://github.com/sphinx-doc/sphinx/blob/master/pyproject.toml#L62 and so you'd only need a proper module-mapping for that i- if even that.

high-yak-85899

02/09/2023, 11:03 PM

Yeah, it's already in the environment. We'ere on 4.5 at the moment. I'll try out a module mapping. I guess I didn't think to try that first since we don't have that package in our direct dependencies so I thought module mappings would complain.

enough-analyst-54434

02/09/2023, 11:05 PM

Copy code

$ pex sphinx -- -c 'import sphinxcontrib.serializinghtml; print(sphinxcontrib.serializinghtml.__file__)'
/tmp/tmpt_1zhtm0/.bootstrap/pex/environment.py:654: PEXWarning: The `pkg_resources` package was loaded from a pex vendored version when declaring namespace packages defined by:

1. sphinxcontrib-devhelp==1.0.2 namespace packages:
  sphinxcontrib

2. sphinxcontrib-jsmath==1.0.1 namespace packages:
  sphinxcontrib

3. sphinxcontrib-serializinghtml==1.1.5 namespace packages:
  sphinxcontrib

4. sphinxcontrib-qthelp==1.0.3 namespace packages:
  sphinxcontrib

These distributions should fix their `install_requires` to include `setuptools`
  pex_warnings.warn(
/home/jsirois/.pex/installed_wheels/352a9a00ae864471d3a7ead8d7d79f5fc0b57e8b3f95e9867eb9eb28999b92fd/sphinxcontrib_serializinghtml-1.1.5-py2.py3-none-any.whl/sphinxcontrib/serializinghtml/__init__.py

enough-analyst-54434

02/09/2023, 11:06 PM

So, it totally works with latest Sphinx.

enough-analyst-54434

02/09/2023, 11:07 PM

@high-yak-85899 you only need a module mapping if you import the sphinxcontrib code like I did in this example proof. If you don't, you don't.

high-yak-85899

02/09/2023, 11:07 PM

Right, so then 4.5 must not declare it as a proper dependency?

enough-analyst-54434

02/09/2023, 11:07 PM

Note the NS package warnings though. You'll want an execution_mode="venv" pex_binary to work around issues there.

enough-analyst-54434

02/09/2023, 11:08 PM

Nope, works fine:

Copy code

$ pex sphinx==4.5 -- -c 'import sphinxcontrib.serializinghtml; print(sphinxcontrib.serializinghtml.__file__)'
/tmp/tmp361dpp9r/.bootstrap/pex/environment.py:654: PEXWarning: The `pkg_resources` package was loaded from a pex vendored version when declaring namespace packages defined by:

1. sphinxcontrib-devhelp==1.0.2 namespace packages:
  sphinxcontrib

2. sphinxcontrib-jsmath==1.0.1 namespace packages:
  sphinxcontrib

3. sphinxcontrib-serializinghtml==1.1.5 namespace packages:
  sphinxcontrib

4. sphinxcontrib-qthelp==1.0.3 namespace packages:
  sphinxcontrib

These distributions should fix their `install_requires` to include `setuptools`
  pex_warnings.warn(
/home/jsirois/.pex/installed_wheels/352a9a00ae864471d3a7ead8d7d79f5fc0b57e8b3f95e9867eb9eb28999b92fd/sphinxcontrib_serializinghtml-1.1.5-py2.py3-none-any.whl/sphinxcontrib/serializinghtml/__init__.py

high-yak-85899

02/09/2023, 11:08 PM

So then what am I doing wrong with my

python_source

? When I do

dependencies --transitive

, it's only showing that

sphinx

is a dependency

high-yak-85899

02/09/2023, 11:09 PM

Copy code

//:reqs#click
//:reqs#jinja2
//:reqs#pyyaml
//:reqs#sortedcontainers
//:reqs#sphinx
//:reqs#sphinx-rtd-theme
//:reqs#types-PyYAML
//requirements.txt:reqs

enough-analyst-54434

02/09/2023, 11:09 PM

Pants

dependencies

goal only does 1st party and direct 3rdparty.

👍 1

enough-analyst-54434

02/09/2023, 11:09 PM

It doesn't do resolves or incorporate their results.

enough-analyst-54434

02/09/2023, 11:09 PM

Basically it only lists text in your repo.

high-yak-85899

02/09/2023, 11:10 PM

So perhaps I need to make a dedicate

pex_binary

target? I was hoping just the

python_source

would work now that running that is separated from a pex in newer pants versions

enough-analyst-54434

02/09/2023, 11:11 PM

It might help if you include the actual real problem. A command line with full output.

enough-analyst-54434

02/09/2023, 11:12 PM

This should work with the latest machinations of the runnable python_source, but I'm not sure what versions of Pants have the latest fixes there (it recently uses venvs)

high-yak-85899

02/09/2023, 11:12 PM

A minimal repro is just instantiating

sphinx.application.Sphinx

with a builder set to

"html"

and you get

Copy code

Running Sphinx v4.5.0
Traceback (most recent call last):
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/sphinx/registry.py", line 425, in load_extension
    mod = import_module(extname)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/sphinx/builders/epub3.py", line 12, in <module>
    from sphinx.builders import _epub_base
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/sphinx/builders/_epub_base.py", line 15, in <module>
    from sphinx.builders.html import BuildInfo, StandaloneHTMLBuilder
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/sphinx/builders/html/__init__.py", line 1295, in <module>
    import sphinxcontrib.serializinghtml  # NOQA
ModuleNotFoundError: No module named 'sphinxcontrib.serializinghtml'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/pex", line 236, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/usr/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/nathanael/devel/monorepo/astranis-python/astranis/service/action_triggers/documenter/config_documenter.py", line 77, in <module>
    build()
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/nathanael/devel/monorepo/astranis-python/astranis/service/action_triggers/documenter/config_documenter.py", line 70, in build
    sphinx = application.Sphinx(
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/sphinx/application.py", line 219, in __init__
    self.setup_extension(extension)
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/sphinx/application.py", line 380, in setup_extension
    self.registry.load_extension(self, extname)
  File "/home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/sphinx/registry.py", line 428, in load_extension
    raise ExtensionError(__('Could not import extension %s') % extname,
sphinx.errors.ExtensionError: Could not import extension sphinx.builders.epub3 (exception: No module named 'sphinxcontrib.serializinghtml')

enough-analyst-54434

02/09/2023, 11:13 PM

Great - it is running in a venv

enough-analyst-54434

02/09/2023, 11:13 PM

Can you do

ls -1 /home/nathanael/.cache/pants/named_caches/pex_root/venvs/874a56f50430c8505af5832bda4ffa33c6fe9d8c/c4fe7bf08b2a1259d39d005551c5bf25de7c32ef/lib/python3.8/site-packages/

and provide the output?

high-yak-85899

02/09/2023, 11:14 PM

Untitled

high-yak-85899

02/09/2023, 11:15 PM

So only one of the sphinxcontrib dependencies

enough-analyst-54434

02/09/2023, 11:15 PM

Ok:

Copy code

PEX_EXTRA_SYS_PATH.pth

That's the problem. Some tools don't handle this - which supports building venvs from symlink farms. You can turn that off with

pex_binary

👍 1

enough-analyst-54434

02/09/2023, 11:15 PM

Just a sec for the details...

enough-analyst-54434

02/09/2023, 11:17 PM

You need Pants >= 2.15.x and:

Copy code

pex_binary(
  ...
  execution_mode="venv",
  venv_site_packages_copies=True,
)

https://www.pantsbuild.org/v2.15/docs/reference-pex_binary#codevenv_site_packages_copiescode

high-yak-85899

02/09/2023, 11:18 PM

Sounds good! I'll try updating sometime soon.

8 Views

Open in Slack

Previous Next