glamorous-accountant-97217
11/30/2022, 3:53 AMpython_distribution
rule producing a wheel we call “ponder”. the goal is for users to be able to pip install
the wheel and then use the ponder
library in other python code. I have a unit test in /pushdown_service/test/subdir_b/test_astype.py that depends on /pushdown_service/test/subdir_b/common.py, which has the line import ponder.subdir
, but the ponder
there comes directly from the source files, specifically via ponder/subdir/__init__.py
. i want to do 2 things:
1. have the test use the ponder wheel instead of the source files within ponder
. My goal here is to check that the wheel is being built correctly. I have already seen an error where we didn’t specify the wheel’s dependencies correctly, so the wheel was broken, but our tests, which use the ponder from the snowflake folder directly rather than from wheel, didn’t detect the failure.
a. i tried adding "ponder:ponder_wheel"
as a dependency to python_tests
. but then I got the errors shown at the end of this message. what was wrong? the msising depdencies are all transitive dependencies of the ponder wheel
2. prevent the test from being able to access targets in ponder/
that are not the wheel, such as the ponder/subdir/_init_.py
target that it’s using currently
is testing the wheel this way reasonable? are 1) and 2) possible?
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /Users/maheshvashishtha/.cache/pants/named_caches/pex_root/unzipped_pexes/0b173c486d327c83c634831eb9c4ac34b76e61ae.
Needed cp310-cp310-macosx_12_0_x86_64 compatible dependencies for:
1: boto3
Required by:
ponder 0.0.3
But this pex had no ProjectName(raw='boto3', normalized='boto3') distributions.
2: fsspec[http]
Required by:
ponder 0.0.3
But this pex had no ProjectName(raw='fsspec', normalized='fsspec') distributions.
3: lz4==4.0.2
Required by:
ponder 0.0.3
But this pex had no ProjectName(raw='lz4', normalized='lz4') distributions.
4: modin@ git+<https://github.com/devin-petersohn/modin.git@2ad8813ccf3cb263857478df951b8c7b4b7d4396>
Required by:
ponder 0.0.3
But this pex had no ProjectName(raw='modin', normalized='modin') distributions.
5: rpyc==5.2.3
Required by:
ponder 0.0.3
But this pex had no ProjectName(raw='rpyc', normalized='rpyc') distributions.
cc @clever-hamburger-59716happy-kitchen-89482
11/30/2022, 5:43 AMclever-hamburger-59716
11/30/2022, 5:52 AMhappy-kitchen-89482
11/30/2022, 7:18 PMponder
via a distribution at all? Why not let the client and service just depend on it in the repo? Does there actually need to be a ponder
wheel for other reasons?clever-hamburger-59716
11/30/2022, 8:54 PMpython_sources
target. say a,b, and c and the ponder servcice
depends on b. Our goal is to distribute the client independently from the service. We would like to be able to inject the ponder
wheel we are building as a dependency into the test targets of the `ponder service`so that we can do integration testing with the wheel (rather than have the sources be picked up from the repo).happy-kitchen-89482
11/30/2022, 11:35 PMpython_tests
target has an explicit dependency on the python_distribution
then it will build the wheel and depend on it, and subtract the corresponding loose sources from the deps, so they aren’t present twice on the sys.path.glamorous-accountant-97217
11/30/2022, 11:46 PMhappy-kitchen-89482
11/30/2022, 11:46 PMglamorous-accountant-97217
11/30/2022, 11:47 PM./pants test
in our CI. wouldn’t pants build the wheel just once for all the tests?happy-kitchen-89482
11/30/2022, 11:47 PMglamorous-accountant-97217
11/30/2022, 11:50 PMpython_distribution(
name="ponder_wheel",
entry_points={
"console_scripts": {"ponder":"ponder.command_line:main"},
},
dependencies=["//:soda", ":command_line", "ponder/subdir:subdir", "ponder/utils:utils"],
provides=python_artifact(
name="ponder",
version="0.0.3",
),
sdist=False,
)
happy-kitchen-89482
11/30/2022, 11:51 PMclever-hamburger-59716
11/30/2022, 11:54 PMhappy-kitchen-89482
11/30/2022, 11:54 PMglamorous-accountant-97217
11/30/2022, 11:54 PMWhich version of Pants is this? And are you using a lockfile?we do
./pants generate-lockfiles --resolve=pytest
before testsAnd when you run the tests against the local sources, presumably those third party deps are present and the tests pass?that’s right
python_tests
happy-kitchen-89482
12/01/2022, 12:02 AMpytest
itself, I mean a lockfile for your code (named typically something like python-default
)i do see the third party dependencies as transitive dependencies when we add the wheel as a dependency of python_testsWhen you run
./pants dependencies path/to/test.py
you see the third-party deps listed?clever-hamburger-59716
12/07/2022, 12:10 AMhappy-kitchen-89482
12/07/2022, 12:18 AMclever-hamburger-59716
12/07/2022, 12:25 AM