Does Pants offer the ability to run tests using tox (or Nox), or offer a similar option for running the tests in an isolated virtual env into which the packages are installed (so that the distribution is discoverable with _pkg_resources_ and _importlib_metadata_?

not sure I understand what you mean, but perhaps https://www.pantsbuild.org/v2.6/docs/reference-python_tests#coderuntime_package_dependenciescode is what you are looking for?

Importing works, but the packages don't seem to be registered (as if

pip install

ed). Foir example, this fails:

pkg_resources.get_distribution("<my-package-name>")

do you want to have 3rd party packages to be installed into a venv that is used to run your tests or do you want some part of your monorepo project source code to be “installed” into that venv?

That's the difference between adding a package to PYTHONPATH and installing it. Installing creates a

.dist-info

directory in

lib/site-packages

which stores package metadata.

I am very sorry that I perhaps am missing something obvious, but why would you want to put a directory with source code from your monorepo project into

site-packages

? The directory of your interest will be added automatically thanks to the dependency inference (happening by reading the

import

statement) if the test(s) need it.

As far as I can see, Pants does not install my package when running tests, but sets PYTHONPATH to make the code available for the tests (or equivalent). In contrast, Poetry installs my package in development (AKA editable) mode, so that it is discoverable by _pkg_resources_ or _importlib_metadata_. That more closely resembles an installation from a distribution archive and thus allows for testing also these aspects.

I think I was after something similar, and came up with creating a virtualenv and installing my packages in edit mode from there. See this issue comment: https://github.com/pantsbuild/pants/issues/11812#issuecomment-810831415 Beaware, that it’s a hackish work-around, though.

tox/nox - no, just pytest for now. pkg_resources/importlib_metadata - others addressed already. Personally, I think relying on PYTHONPATH is dangerous and should be avoided, but I'm willing to ignore pants use of it because of all the other benefits pants provides. I would prefer that packages under test be "installed" (in editable mode) as described here.

when Pants runs almost any Python code it does so using

PEX

, and the default execution mode of

PEX

is to create a virtualenv containing the package contents. but for

pytest

, only 3rdparty code is packaged inside the

PEX

, while the sources are added via a scrubbed sys.path as you all have mentioned: https://github.com/pantsbuild/pants/blob/fcb5b5cc71301465c4a0f87807191f75011e24f6/src/python/pants/backend/python/goals/pytest_runner.py#L261-L302 … so this is quite a bit more isolated than typical PYTHONPATH usage. i believe that we add loose sources for performance reasons, to avoid wrapping a

PEX

around the sources. but we also began doing this before

PEX

even supported a virtualenv execution mode, so it’s possible that we could 1) add support to PEX for adding sources as installed in “editable” mode, 2) have Pants use that support rather than adding via PEX_EXTRA_SYS_PATH cc @enough-analyst-54434

If your test depends on a

python_distribution

target then that distribution will be built and placed in the test's sandbox. But it won't be installed. That is something we can possibly add, if important. I have actually started looking at a similar problem for installing those sources in editable mode for

run

@witty-crayon-22786 Pex already supports installing sources in ~editable mode (editable mode makes no sense in a PEX since editable mode is essentially a normal dist install, but using symlinks to the source directory so you can edit the files live). Pex in fact dogfoods this to build the Pex PEX using

pex .

instead of

pex -D ...

(https://github.com/pantsbuild/pex/blob/100b687938293a022e5bb842118b662f5037069e/scripts/package.py#L25-L57). The problem @quaint-gold-40000 is running into is Pants does neither of these currently: 1. If you have defined a

python_distribution

target owning some set of source files, when generating a PEX to run tests with or what have you, expose an option (say on

python_library

) to use the owning distribution instead of the sources when creating any venvs to run Pants actions in. 2. If you have not defined a

python_distribution

, allow the same configuration as in 1, but with Pants auto-generting a distribution based on the owning target name? and with a fake version. I think though @quaint-gold-40000 probably just needs #1. Either way, these can both be purely accomplished from the Pants side with Pex as-is today.

#1 does remind me of this observation: I didn't look far into creating distribution archives, but I was surprised to learn that there was a separate python_distribution (besides python_library). I guess that makes sense, but I did find it annoying that a python_distribution cannot have the same name as a python_library.

It could have the same name, if you’re able to put it in a separate BUILD file from your python_library. (Well, it wouldn’t have the same target address, though.)

@quaint-gold-40000: that is in part because Pants’ ancestry assumes fairly large wheels, with lots of internal libraries/modules per external wheel.

I have a branch in which I am noodling on part #1! Specifically on installing the dist in

develop

mode

Unless I'm missing something, develop mode is not useful here, the useful thing is just installing it. The develop mode is about symlinks to source being setup in a venv site-libraries.

Isn't develop mode how you install it without bumping versions and such?

We're generating a wheel and doing a local wheel install so there are no version bumping concerns or publishing concerns IIUC.

What about the pip cache?

Not actually sure - thats a quick CLI experiment with Pex though.

So when would we want to use

develop

mode? Possibly never?

I think never. It's antithetical to snadboxing.

Perhaps this is not relevant to the above, but I've had issues with pip's cache before. When having Nox (~ tox) install my in-development package from sdist, pip will first create a wheel from the sdist. This wheel is being cached based on the name and version only. Some more info here: • https://github.com/cjolowicz/nox-poetry/issues/377 • https://github.com/pypa/pip/issues/4183

Its relevant, but I think we'll be immune to this since the Pex cache for distributions is based on a content hash of the distribution and not the name and version.