stocky-helmet-22655
03/21/2023, 5:44 PMpython_distribution
and I have tests that I want to run against this whl. I setup 2 resolves, one for building the package and one for running the tests. The tests resolve depends on the generated whl by setting python-repos.find_links = ["file://%(buildroot)s/dist"]
, and this works if the whl has already been made. Pants doesn’t seem to infer that it should always package the whl before running the tests, and I don’t see a way to tell pants that it should other than making a custom target. I can do both steps manually in my CI, but that to me defeats the purpose of using pants. Is there an easy way to do this?stocky-helmet-22655
03/21/2023, 5:53 PMwitty-crayon-22786
03/21/2023, 5:55 PMpython_distribution
witty-crayon-22786
03/21/2023, 5:56 PMstocky-helmet-22655
03/21/2023, 5:57 PMwitty-crayon-22786
03/21/2023, 5:58 PMpython_distribution
target in the dependencies
field of a python_test
target will cause it to be loaded as if it had been pip installed, iirc.stocky-helmet-22655
03/21/2023, 6:00 PMpython_test
is in the same resolve as the python_sources
used by the python_distribution
, what tells the python_test
to not infer a dependency on the python_sources
?stocky-helmet-22655
03/21/2023, 6:00 PMpython_test
target ignore the `python_distribution`’s dependencies?witty-crayon-22786
03/21/2023, 6:01 PMhappy-kitchen-89482
03/21/2023, 6:29 PMpython_tests
on python_distribution
is interpreted as "build the dist and consume the relevant code from it, instead of from the underlying sources". Under the hood, python subtracts the sources from the dists from the direct sources.stocky-helmet-22655
03/21/2023, 7:30 PMpython_tests
on python_distribution
using runtime_package_dependencies
. It appears to still find the files - I don’t know if by source or by whl - but I’m getting a ModuleNotFoundError
. Per the stacktrace a test imports code in the whl which imports code in another whl which tries to import code in another whl and that last one is not found. When I ran the tests with a direct dependency on 3rd party dependencies it ran fine, but not running against the whl.
I don’t see a way to add modules or module_mapping in python_testshappy-kitchen-89482
03/21/2023, 9:04 PMhappy-kitchen-89482
03/21/2023, 9:04 PMruntime_package_dependencies
but just regular dependencies=
happy-kitchen-89482
03/21/2023, 9:04 PMhappy-kitchen-89482
03/21/2023, 9:05 PMstocky-helmet-22655
03/22/2023, 3:27 AMruntime_package_dependencies
instead of dependencies=
, though using that brings up another unfortunate issue….More background context here but I’ll try to summarize: my project has different 3rd party dependency versions depending on whether python 3.7 is used or python 3.10. I tried to solve this with environment markers, but Pants does not seem to like those so I used separate resolves in addition. I’m using a python_distribution
which depends on multiple resolves to make a whl that works whether you use python 3.7 or 3.10 (through environment markers) that all seems to work up until where we are now - if I have the python_tests
depend on the python_distribution
using dependencies=
, it complains saying the target uses one resolve but has dependencies in the other resolve and throws NoCompatibleResolveException
. Conceptually this error does make sense, but it makes me believe either this can’t be done - or I’m doing something wrong along the way..stocky-helmet-22655
03/22/2023, 3:50 AMhappy-kitchen-89482
03/22/2023, 1:53 PMhappy-kitchen-89482
03/22/2023, 1:54 PMhappy-kitchen-89482
03/22/2023, 1:55 PMhappy-kitchen-89482
03/22/2023, 1:56 PMhappy-kitchen-89482
03/22/2023, 1:56 PMstocky-helmet-22655
03/22/2023, 2:23 PMThanks for the extra detail. Can you explain why you want A to depend on “the output of B” rather than just on B’s sources?From what I’m told, doing it this way can help prevent some strange types of errors particularly with extension modules
The idiomatic way here would be for A to depend on B from source when you run tests, even though you also publish BThis is what I’m used to doing - the requirement is coming from others and I’ll admit I don’t fully understand it - hopefully the above response makes sense
Re environment markers, I would have expected those to work, can you elaborate on what error you were seeing with them? And also, are you using a lockfile?I am using a lockfile - I have a demo of this not working here: https://github.com/nikwotton/Broken-Pants-Demo/tree/broken-lockfiles I agree it should work and pants’ documentation suggests it would, but trying to generate the lockfile fails. Note it has the same failure whether the requirements are in requirements.txt or in
python_requirement
blocksstocky-helmet-22655
03/22/2023, 7:57 PMhappy-kitchen-89482
03/22/2023, 8:05 PMpython_distribution
, does it?)stocky-helmet-22655
03/22/2023, 8:32 PMand yes, if B contains extensions then you would need to represent it as aThis specific package I’m working on does not, but I’m working on getting this setup in such a way that it can be a blueprint for adding pants to all packages in the monorepo so I want to set this up as if it did, does it?python_distribution
Yeah, I think filing an issue would be the good next stephttps://github.com/pantsbuild/pants/issues/18555
enough-analyst-54434
03/23/2023, 2:21 AMpython_distribution
that contains the split requirement:
numpy >=1.16.2,<1.20; python_version == '3.7'
numpy >=1.21.3,<1.22.0; python_version == '3.10'
This is unlockable in a single resolve - Pex will never be able to lock something like this in a universal lock. Pip fundamentally does 1 resolve and that set of requirements is unsolvable since the req range is disjoint. So this forces 2 resolves. The issue there is Pants needs to be able to allow creating a single python_distrubution
that combines requirements strings from 2 resolves then. It doesn't even come close to allowing this today.enough-analyst-54434
03/23/2023, 2:25 AMpython_distribution
in Pants.stocky-helmet-22655
03/28/2023, 7:57 PMpython_distribution
per resolve
I put a comment on the GitHub issue though, not sure if you saw it - testing against the wheel even within a single resolve seems to run into issues. I don’t understand the error message (linked here), but it seems like running a test against a python_distribution
which has any 3rd party dependencies cannot be done as of now - hoping I’m just doing something wrong