https://pantsbuild.org/ logo
#general
Title
# general
s

stocky-helmet-22655

03/21/2023, 5:44 PM
I have a project that builds a whl through
python_distribution
and I have tests that I want to run against this whl. I setup 2 resolves, one for building the package and one for running the tests. The tests resolve depends on the generated whl by setting
python-repos.find_links = ["file://%(buildroot)s/dist"]
, and this works if the whl has already been made. Pants doesn’t seem to infer that it should always package the whl before running the tests, and I don’t see a way to tell pants that it should other than making a custom target. I can do both steps manually in my CI, but that to me defeats the purpose of using pants. Is there an easy way to do this?
It also appears to not be the right path because if I change anything and re-run, pants complains that the hash doesn’t match what’s in the lockfile. Having to re-build the lockfiles every time a change is made seems bad
w

witty-crayon-22786

03/21/2023, 5:55 PM
you probably want to be using https://www.pantsbuild.org/docs/reference-python_tests#coderuntime_package_dependenciescode to declare the dependency from the test to the
python_distribution
although. hm. you want it to actually be loaded by the test’s intepreter? that’s not currently possible with two different resolves. why do you need to use two resolves?
s

stocky-helmet-22655

03/21/2023, 5:57 PM
I’m not sure that two resolves is needed. My thought process was/is that I want to be able to run tests depending on the generated whl without the tests knowing about the source files used to build the whl. Can this be done with one resolve?
w

witty-crayon-22786

03/21/2023, 5:58 PM
yes: including an explicit dependency on the
python_distribution
target in the
dependencies
field of a
python_test
target will cause it to be loaded as if it had been pip installed, iirc.
s

stocky-helmet-22655

03/21/2023, 6:00 PM
since the
python_test
is in the same resolve as the
python_sources
used by the
python_distribution
, what tells the
python_test
to not infer a dependency on the
python_sources
?
Are you saying the explicit dependency makes the
python_test
target ignore the `python_distribution`’s dependencies?
w

witty-crayon-22786

03/21/2023, 6:01 PM
yes, effectively. i’ll let @happy-kitchen-89482 explain that one… i’m failing to find the docs for it.
🙏 1
h

happy-kitchen-89482

03/21/2023, 6:29 PM
Yes, an explicit dep from
python_tests
on
python_distribution
is interpreted as "build the dist and consume the relevant code from it, instead of from the underlying sources". Under the hood, python subtracts the sources from the dists from the direct sources.
s

stocky-helmet-22655

03/21/2023, 7:30 PM
Ok I removed the second resolve and added a dependency from
python_tests
on
python_distribution
using
runtime_package_dependencies
. It appears to still find the files - I don’t know if by source or by whl - but I’m getting a
ModuleNotFoundError
. Per the stacktrace a test imports code in the whl which imports code in another whl which tries to import code in another whl and that last one is not found. When I ran the tests with a direct dependency on 3rd party dependencies it ran fine, but not running against the whl. I don’t see a way to add modules or module_mapping in python_tests
h

happy-kitchen-89482

03/21/2023, 9:04 PM
Where are those other wheels? First-party or third-party? There should be dependencies inferred on them?
Oh, and I don't think you want to use
runtime_package_dependencies
but just regular
dependencies=
I could be wrong about that though
The functionality I described above exists to support building first-party native code, so it's not very full-featured, and in particular it doesn't deal with 3rdparty deps of the wheel in the way you'd assume (but it should still pull them in via dep inference from the original sources that the wheel shadowed)
s

stocky-helmet-22655

03/22/2023, 3:27 AM
The issue might be using
runtime_package_dependencies
instead of
dependencies=
, though using that brings up another unfortunate issue….More background context here but I’ll try to summarize: my project has different 3rd party dependency versions depending on whether python 3.7 is used or python 3.10. I tried to solve this with environment markers, but Pants does not seem to like those so I used separate resolves in addition. I’m using a
python_distribution
which depends on multiple resolves to make a whl that works whether you use python 3.7 or 3.10 (through environment markers) that all seems to work up until where we are now - if I have the
python_tests
depend on the
python_distribution
using
dependencies=
, it complains saying the target uses one resolve but has dependencies in the other resolve and throws
NoCompatibleResolveException
. Conceptually this error does make sense, but it makes me believe either this can’t be done - or I’m doing something wrong along the way..
To help explain/demonstrate my current situation, I’ve simplified an example here: https://github.com/nikwotton/Broken-Pants-Demo/
h

happy-kitchen-89482

03/22/2023, 1:53 PM
Thanks for the extra detail. Can you explain why you want A to depend on "the output of B" rather than just on B's sources?
The idiomatic way here would be for A to depend on B from source when you run tests, even though you also publish B
Re environment markers, I would have expected those to work, can you elaborate on what error you were seeing with them? And also, are you using a lockfile?
You should be able to generate a lockfile that supports multiple python versions, even with environment markers in your requirements.txt
I think
s

stocky-helmet-22655

03/22/2023, 2:23 PM
Thanks for the extra detail. Can you explain why you want A to depend on “the output of B” rather than just on B’s sources?
From what I’m told, doing it this way can help prevent some strange types of errors particularly with extension modules
The idiomatic way here would be for A to depend on B from source when you run tests, even though you also publish B
This is what I’m used to doing - the requirement is coming from others and I’ll admit I don’t fully understand it - hopefully the above response makes sense
Re environment markers, I would have expected those to work, can you elaborate on what error you were seeing with them? And also, are you using a lockfile?
I am using a lockfile - I have a demo of this not working here: https://github.com/nikwotton/Broken-Pants-Demo/tree/broken-lockfiles I agree it should work and pants’ documentation suggests it would, but trying to generate the lockfile fails. Note it has the same failure whether the requirements are in requirements.txt or in
python_requirement
blocks
@happy-kitchen-89482 If I should be able to generate a lockfile that supports multiple python versions but cannot, would it make more sense for me to file this as an issue on GitHub and continue the conversation there?
h

happy-kitchen-89482

03/22/2023, 8:05 PM
Yeah, I think filing an issue would be the good next step (and yes, if B contains extensions then you would need to represent it as a
python_distribution
, does it?)
s

stocky-helmet-22655

03/22/2023, 8:32 PM
and yes, if B contains extensions then you would need to represent it as a
python_distribution
, does it?
This specific package I’m working on does not, but I’m working on getting this setup in such a way that it can be a blueprint for adding pants to all packages in the monorepo so I want to set this up as if it did
Yeah, I think filing an issue would be the good next step
https://github.com/pantsbuild/pants/issues/18555
e

enough-analyst-54434

03/23/2023, 2:21 AM
I've commented on the issue, but Pants just can't handle this at all today. The crux, IIUC, is @stocky-helmet-22655 wants a single
python_distribution
that contains the split requirement:
Copy code
numpy >=1.16.2,<1.20; python_version == '3.7'
numpy >=1.21.3,<1.22.0; python_version == '3.10'
This is unlockable in a single resolve - Pex will never be able to lock something like this in a universal lock. Pip fundamentally does 1 resolve and that set of requirements is unsolvable since the req range is disjoint. So this forces 2 resolves. The issue there is Pants needs to be able to allow creating a single
python_distrubution
that combines requirements strings from 2 resolves then. It doesn't even come close to allowing this today.
So this is down to a magical new feature in Pex that I have no idea how to implement but I'm sure would be hard / require extensive re-work of how locks are done, or support for merging resolves in a
python_distribution
in Pants.
s

stocky-helmet-22655

03/28/2023, 7:57 PM
I’m in favor of supporting this, but I’m also not blocked by this - I can use a different
python_distribution
per resolve I put a comment on the GitHub issue though, not sure if you saw it - testing against the wheel even within a single resolve seems to run into issues. I don’t understand the error message (linked here), but it seems like running a test against a
python_distribution
which has any 3rd party dependencies cannot be done as of now - hoping I’m just doing something wrong
2 Views