hi all, i have a question about testing python wheels produced by pants. i have a monorepo / and in /ponder/BUILD I have a

python_distribution

rule producing a wheel we call “ponder”. the goal is for users to be able to

pip install

the wheel and then use the

ponder

library in other python code. I have a unit test in /pushdown_service/test/subdir_b/test_astype.py that depends on /pushdown_service/test/subdir_b/common.py, which has the line

import ponder.subdir

, but the

ponder

there comes directly from the source files, specifically via

ponder/subdir/__init__.py

. i want to do 2 things: 1. have the test use the ponder wheel instead of the source files within

ponder

. My goal here is to check that the wheel is being built correctly. I have already seen an error where we didn’t specify the wheel’s dependencies correctly, so the wheel was broken, but our tests, which use the ponder from the snowflake folder directly rather than from wheel, didn’t detect the failure. a. i tried adding

"ponder:ponder_wheel"

as a dependency to

python_tests

. but then I got the errors shown at the end of this message. what was wrong? the msising depdencies are all transitive dependencies of the ponder wheel 2. prevent the test from being able to access targets in

ponder/

that are not the wheel, such as the

ponder/subdir/_init_.py

target that it’s using currently is testing the wheel this way reasonable? are 1) and 2) possible?

pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /Users/maheshvashishtha/.cache/pants/named_caches/pex_root/unzipped_pexes/0b173c486d327c83c634831eb9c4ac34b76e61ae.
Needed cp310-cp310-macosx_12_0_x86_64 compatible dependencies for:
 1: boto3
    Required by:
      ponder 0.0.3
    But this pex had no ProjectName(raw='boto3', normalized='boto3') distributions.
 2: fsspec[http]
    Required by:
      ponder 0.0.3
    But this pex had no ProjectName(raw='fsspec', normalized='fsspec') distributions.
 3: lz4==4.0.2
    Required by:
      ponder 0.0.3
    But this pex had no ProjectName(raw='lz4', normalized='lz4') distributions.
 4: modin@ git+<https://github.com/devin-petersohn/modin.git@2ad8813ccf3cb263857478df951b8c7b4b7d4396>
    Required by:
      ponder 0.0.3
    But this pex had no ProjectName(raw='modin', normalized='modin') distributions.
 5: rpyc==5.2.3
    Required by:
      ponder 0.0.3
    But this pex had no ProjectName(raw='rpyc', normalized='rpyc') distributions.

cc @clever-hamburger-59716

Re “the goal is for users to be able to pip install the wheel and then use the ponder library in other python code.” - is that “other python code” in the repo, or outside it?

It’s in the same repo @happy-kitchen-89482 . @glamorous-accountant-97217 I would think we can use the same sources target that we use as dependency in the python_distribution target as a dependency in the service as well. This way there is 1:1; however, there is bloat. I don’t think this should be necessary though. We include a target (the only one needed from Ponder/) as a dependency in pushdown_service. I would argue this alone will import the sources and all the dependencies.

Gotcha. So my first question would be, why consume

ponder

via a distribution at all? Why not let the client and service just depend on it in the repo? Does there actually need to be a

ponder

wheel for other reasons?

Sorry, that wasn't clear. In the monorepo, yes, they do. The `ponder`wheel has a python distribution target that depends on some

python_sources

target. say a,b, and c and the

ponder servcice

depends on b. Our goal is to distribute the client independently from the service. We would like to be able to inject the

ponder

wheel we are building as a dependency into the test targets of the `ponder service`so that we can do integration testing with the wheel (rather than have the sources be picked up from the repo).

Ah, that makes sense. So there is code outside the repo that uses the client.

I tried that in first post on this thread, but had some dependency issues I Couldn’t figure out. any chance you can tell what’s wrong from the error messages?

So then only that integration test needs that explicit dep, and you can do a bunch of smoke testing in that integration test

@happy-kitchen-89482 we have several python_test targets that are almost integration tests, because they’re using the python sources instead of the wheel. we run all of them with

./pants test

in our CI. wouldn’t pants build the wheel just once for all the tests?

Oh, hmm, maybe we don’t pull in 3rdparty deps in this case

we actually don’t have a setup.py. code generating the wheel is like

python_distribution(
    name="ponder_wheel",
    entry_points={
     "console_scripts": {"ponder":"ponder.command_line:main"},
    },
    dependencies=["//:soda", ":command_line", "ponder/subdir:subdir", "ponder/utils:utils"],
    provides=python_artifact(
        name="ponder",
        version="0.0.3",
    ),
    sdist=False,
)

Which version of Pants is this? And are you using a lockfile?

pants v2.13

I think the best way to debug is with a small, sanitized repo on github that reproduces the problem

Which version of Pants is this? And are you using a lockfile?

we do

./pants generate-lockfiles --resolve=pytest

before tests

Ah, that is the lockfile for

pytest

itself, I mean a lockfile for your code (named typically something like

python-default

)

Hi @happy-kitchen-89482, we are short of producing a stand alone repo yet; however, we do have an update. I was able to get this to work by using the wheel explicitly as a python_requirement and using that target as a dependency (and excluding the sources that were being pulled in). We will try to get a stand-alone repo going to close the loop here.

Glad it’s working! Would still love a repro when you have a moment

For sure, after all its in our own interest 🙂!