Hey, everyone! Is there a way of making a test-only Python package always be bundled together with another one? I have Python tests that use fixtures provided by

pytest-spark

(github.com/malexer/pytest-spark). For it to work properly, PySpark must be installed. PySpark isn't a dep on

pytest-spark

, tho, so our tests that happen to use PySpark directly pass, while others don't. Any way to make this work?

message has been deleted

Thx for the response! But yeah... Unfortunately the repo is pretty stale and, while trying to convince people to migrate from this dep is possible, it will probably take some time. This "peer" relationship between those libs indeed seem weird, but I guess that the fact that a Spark session can exist outside the scope of Python (or something like that) is what this design decision is based on.

@careful-postman-55465 re: > I have Python tests that use fixtures provided by

pytest-spark

(github.com/malexer/pytest-spark). For it to work properly, PySpark must be installed. PySpark isn't a dep on

pytest-spark

, tho ... > Thx for the response! But yeah... Unfortunately the repo is pretty stale I don't think many Python users realize VCS requirements are a thing. If the repo is stale, you can freshen it: https://github.com/malexer/pytest-spark/pull/30 You should be able to switch your

pytest-spark

dep to

pytest-spark[pyspark] <https://git>+<http://github.com/jsirois/pytest-spark@f152c6170b406f3f068cafd717b7dfebed9e7821|github.com/jsirois/pytest-spark@f152c6170b406f3f068cafd717b7dfebed9e7821>

. For example:

:; pex "pytest-spark[pyspark] @ git+<https://github.com/jsirois/pytest-spark@f152c6170b406f3f068cafd717b7dfebed9e7821>" -- -c 'import pytest_spark; import pyspark; print(pytest_spark.__file__); print(pyspark.__file__)'
/home/jsirois/dev/malexer/pytest-spark/pytest_spark/__init__.py
/home/jsirois/.cache/pex/installed_wheels/2/e7ece249e4614762222735ea8679b198a8abe6a18df68abc2065d0dbdd98d95a/pyspark-4.0.1-py2.py3-none-any.whl/pyspark/__init__.py

I'm not sure the PR will be accepted, but it doesn't really matter; since you can unblock yourself in this manner anytime the code is available in a public git repo.

You can also do this in your own repo:

poetry_requirements(
    name="poetry",
    overrides={
        "pytest": {"dependencies": [":poetry#pytest-cov", ":poetry#pytest-timeout"]},
    },
)

Easy way to add dependencies to a particular requirement. I did this to force pytest to understand some of the plugins I had installed for it. (My understanding is that they are normally found by some kind of filepath discovery, which doesn't work when pants has already done some dependency resolution into a sandbox). So you could similarly add a (locally known) dependency from

pytest-spark

to

pyspark