https://pantsbuild.org/ logo
#general
Title
# general
q

quaint-gold-40000

07/27/2021, 12:31 PM
Does Pants offer the ability to run tests using tox (or Nox), or offer a similar option for running the tests in an isolated virtual env into which the packages are installed (so that the distribution is discoverable with _pkg_resources_ and _importlib_metadata_?
f

fresh-cat-90827

07/27/2021, 12:35 PM
not sure I understand what you mean, but perhaps https://www.pantsbuild.org/v2.6/docs/reference-python_tests#coderuntime_package_dependenciescode is what you are looking for?
you can specify what you need to have when running the tests and those files will be available
q

quaint-gold-40000

07/27/2021, 12:38 PM
Importing works, but the packages don't seem to be registered (as if
pip install
ed). Foir example, this fails:
Copy code
pkg_resources.get_distribution("<my-package-name>")
f

fresh-cat-90827

07/27/2021, 12:41 PM
do you want to have 3rd party packages to be installed into a venv that is used to run your tests or do you want some part of your monorepo project source code to be “installed” into that venv?
q

quaint-gold-40000

07/27/2021, 12:41 PM
That's the difference between adding a package to PYTHONPATH and installing it. Installing creates a
.dist-info
directory in
lib/site-packages
which stores package metadata.
Part of the monorepo project.
f

fresh-cat-90827

07/27/2021, 12:47 PM
I am very sorry that I perhaps am missing something obvious, but why would you want to put a directory with source code from your monorepo project into
site-packages
? The directory of your interest will be added automatically thanks to the dependency inference (happening by reading the
import
statement) if the test(s) need it.
or is it the case that you use
importlib.import_module
and thus Pants doesn’t include that package in the test environment?
q

quaint-gold-40000

07/27/2021, 12:56 PM
As far as I can see, Pants does not install my package when running tests, but sets PYTHONPATH to make the code available for the tests (or equivalent). In contrast, Poetry installs my package in development (AKA editable) mode, so that it is discoverable by _pkg_resources_ or _importlib_metadata_. That more closely resembles an installation from a distribution archive and thus allows for testing also these aspects.
c

curved-television-6568

07/27/2021, 1:03 PM
I think I was after something similar, and came up with creating a virtualenv and installing my packages in edit mode from there. See this issue comment: https://github.com/pantsbuild/pants/issues/11812#issuecomment-810831415 Beaware, that it’s a hackish work-around, though.
p

proud-dentist-22844

07/27/2021, 3:05 PM
tox/nox - no, just pytest for now. pkg_resources/importlib_metadata - others addressed already. Personally, I think relying on PYTHONPATH is dangerous and should be avoided, but I'm willing to ignore pants use of it because of all the other benefits pants provides. I would prefer that packages under test be "installed" (in editable mode) as described here.
w

witty-crayon-22786

07/27/2021, 5:52 PM
when Pants runs almost any Python code it does so using
PEX
, and the default execution mode of
PEX
is to create a virtualenv containing the package contents. but for
pytest
, only 3rdparty code is packaged inside the
PEX
, while the sources are added via a scrubbed sys.path as you all have mentioned: https://github.com/pantsbuild/pants/blob/fcb5b5cc71301465c4a0f87807191f75011e24f6/src/python/pants/backend/python/goals/pytest_runner.py#L261-L302 … so this is quite a bit more isolated than typical PYTHONPATH usage. i believe that we add loose sources for performance reasons, to avoid wrapping a
PEX
around the sources. but we also began doing this before
PEX
even supported a virtualenv execution mode, so it’s possible that we could 1) add support to PEX for adding sources as installed in “editable” mode, 2) have Pants use that support rather than adding via PEX_EXTRA_SYS_PATH cc @enough-analyst-54434
h

happy-kitchen-89482

07/27/2021, 8:18 PM
If your test depends on a
python_distribution
target then that distribution will be built and placed in the test's sandbox. But it won't be installed. That is something we can possibly add, if important. I have actually started looking at a similar problem for installing those sources in editable mode for
run
e

enough-analyst-54434

07/29/2021, 4:43 PM
@witty-crayon-22786 Pex already supports installing sources in ~editable mode (editable mode makes no sense in a PEX since editable mode is essentially a normal dist install, but using symlinks to the source directory so you can edit the files live). Pex in fact dogfoods this to build the Pex PEX using
pex .
instead of
pex -D ...
(https://github.com/pantsbuild/pex/blob/100b687938293a022e5bb842118b662f5037069e/scripts/package.py#L25-L57). The problem @quaint-gold-40000 is running into is Pants does neither of these currently: 1. If you have defined a
python_distribution
target owning some set of source files, when generating a PEX to run tests with or what have you, expose an option (say on
python_library
) to use the owning distribution instead of the sources when creating any venvs to run Pants actions in. 2. If you have not defined a
python_distribution
, allow the same configuration as in 1, but with Pants auto-generting a distribution based on the owning target name? and with a fake version. I think though @quaint-gold-40000 probably just needs #1. Either way, these can both be purely accomplished from the Pants side with Pex as-is today.
The downside of what @quaint-gold-40000 wants is test caching now becomes as fat as the owning dist.
q

quaint-gold-40000

07/30/2021, 8:40 AM
#1 does remind me of this observation: I didn't look far into creating distribution archives, but I was surprised to learn that there was a separate python_distribution (besides python_library). I guess that makes sense, but I did find it annoying that a python_distribution cannot have the same name as a python_library.
c

curved-television-6568

07/30/2021, 8:43 AM
It could have the same name, if you’re able to put it in a separate BUILD file from your python_library. (Well, it wouldn’t have the same target address, though.)
🆒 1
w

witty-crayon-22786

07/30/2021, 3:09 PM
@quaint-gold-40000: that is in part because Pants’ ancestry assumes fairly large wheels, with lots of internal libraries/modules per external wheel.
h

happy-kitchen-89482

07/30/2021, 3:26 PM
I have a branch in which I am noodling on part #1! Specifically on installing the dist in
develop
mode
This assumes a
python_distribution
target
e

enough-analyst-54434

07/30/2021, 3:52 PM
Unless I'm missing something, develop mode is not useful here, the useful thing is just installing it. The develop mode is about symlinks to source being setup in a venv site-libraries.
h

happy-kitchen-89482

07/30/2021, 4:06 PM
Isn't develop mode how you install it without bumping versions and such?
and publishing?
e

enough-analyst-54434

07/30/2021, 4:09 PM
We're generating a wheel and doing a local wheel install so there are no version bumping concerns or publishing concerns IIUC.
The local install being in a process execution sandbox.
h

happy-kitchen-89482

07/30/2021, 4:09 PM
What about the pip cache?
the named cache
e

enough-analyst-54434

07/30/2021, 4:10 PM
Not actually sure - thats a quick CLI experiment with Pex though.
The cache is properly busted.
Its based on the content hash of the wheel.
So good to go with a straight up install.
h

happy-kitchen-89482

07/30/2021, 4:33 PM
So when would we want to use
develop
mode? Possibly never?
e

enough-analyst-54434

07/30/2021, 4:34 PM
I think never. It's antithetical to snadboxing.
We'd need to symlink to live sources in the buildroot to get the desired effect - live edits.
We'd need something like a persistent worker process that we ~rsynced sources to. At any rate, if I understand @quaint-gold-40000 need, it wasn't actually for the develope mode part, just the metadata that comes along with (any) install.
👍 1
q

quaint-gold-40000

08/02/2021, 9:15 AM
Perhaps this is not relevant to the above, but I've had issues with pip's cache before. When having Nox (~ tox) install my in-development package from sdist, pip will first create a wheel from the sdist. This wheel is being cached based on the name and version only. Some more info here: • https://github.com/cjolowicz/nox-poetry/issues/377https://github.com/pypa/pip/issues/4183
e

enough-analyst-54434

08/02/2021, 6:24 PM
Its relevant, but I think we'll be immune to this since the Pex cache for distributions is based on a content hash of the distribution and not the name and version.
6 Views