I have a project that builds a whl through

python_distribution

and I have tests that I want to run against this whl. I setup 2 resolves, one for building the package and one for running the tests. The tests resolve depends on the generated whl by setting

python-repos.find_links = ["file://%(buildroot)s/dist"]

, and this works if the whl has already been made. Pants doesn’t seem to infer that it should always package the whl before running the tests, and I don’t see a way to tell pants that it should other than making a custom target. I can do both steps manually in my CI, but that to me defeats the purpose of using pants. Is there an easy way to do this?

you probably want to be using https://www.pantsbuild.org/docs/reference-python_tests#coderuntime_package_dependenciescode to declare the dependency from the test to the

python_distribution

I’m not sure that two resolves is needed. My thought process was/is that I want to be able to run tests depending on the generated whl without the tests knowing about the source files used to build the whl. Can this be done with one resolve?

yes: including an explicit dependency on the

python_distribution

target in the

dependencies

field of a

python_test

target will cause it to be loaded as if it had been pip installed, iirc.

since the

python_test

is in the same resolve as the

python_sources

used by the

python_distribution

, what tells the

python_test

to not infer a dependency on the

python_sources

?

yes, effectively. i’ll let @happy-kitchen-89482 explain that one… i’m failing to find the docs for it.

Yes, an explicit dep from

python_tests

on

python_distribution

is interpreted as "build the dist and consume the relevant code from it, instead of from the underlying sources". Under the hood, python subtracts the sources from the dists from the direct sources.

Ok I removed the second resolve and added a dependency from

python_tests

on

python_distribution

using

runtime_package_dependencies

. It appears to still find the files - I don’t know if by source or by whl - but I’m getting a

ModuleNotFoundError

. Per the stacktrace a test imports code in the whl which imports code in another whl which tries to import code in another whl and that last one is not found. When I ran the tests with a direct dependency on 3rd party dependencies it ran fine, but not running against the whl. I don’t see a way to add modules or module_mapping in python_tests

Where are those other wheels? First-party or third-party? There should be dependencies inferred on them?

The issue might be using

runtime_package_dependencies

instead of

dependencies=

, though using that brings up another unfortunate issue….More background context here but I’ll try to summarize: my project has different 3rd party dependency versions depending on whether python 3.7 is used or python 3.10. I tried to solve this with environment markers, but Pants does not seem to like those so I used separate resolves in addition. I’m using a

python_distribution

which depends on multiple resolves to make a whl that works whether you use python 3.7 or 3.10 (through environment markers) that all seems to work up until where we are now - if I have the

python_tests

depend on the

python_distribution

using

dependencies=

, it complains saying the target uses one resolve but has dependencies in the other resolve and throws

NoCompatibleResolveException

. Conceptually this error does make sense, but it makes me believe either this can’t be done - or I’m doing something wrong along the way..

Thanks for the extra detail. Can you explain why you want A to depend on "the output of B" rather than just on B's sources?

Thanks for the extra detail. Can you explain why you want A to depend on “the output of B” rather than just on B’s sources?

From what I’m told, doing it this way can help prevent some strange types of errors particularly with extension modules

The idiomatic way here would be for A to depend on B from source when you run tests, even though you also publish B

This is what I’m used to doing - the requirement is coming from others and I’ll admit I don’t fully understand it - hopefully the above response makes sense

Re environment markers, I would have expected those to work, can you elaborate on what error you were seeing with them? And also, are you using a lockfile?

I am using a lockfile - I have a demo of this not working here: https://github.com/nikwotton/Broken-Pants-Demo/tree/broken-lockfiles I agree it should work and pants’ documentation suggests it would, but trying to generate the lockfile fails. Note it has the same failure whether the requirements are in requirements.txt or in

python_requirement

blocks

Yeah, I think filing an issue would be the good next step (and yes, if B contains extensions then you would need to represent it as a

python_distribution

, does it?)

and yes, if B contains extensions then you would need to represent it as a

python_distribution

, does it?

This specific package I’m working on does not, but I’m working on getting this setup in such a way that it can be a blueprint for adding pants to all packages in the monorepo so I want to set this up as if it did

Yeah, I think filing an issue would be the good next step

https://github.com/pantsbuild/pants/issues/18555

I've commented on the issue, but Pants just can't handle this at all today. The crux, IIUC, is @stocky-helmet-22655 wants a single

python_distribution

that contains the split requirement:

numpy >=1.16.2,<1.20; python_version == '3.7'
numpy >=1.21.3,<1.22.0; python_version == '3.10'

This is unlockable in a single resolve - Pex will never be able to lock something like this in a universal lock. Pip fundamentally does 1 resolve and that set of requirements is unsolvable since the req range is disjoint. So this forces 2 resolves. The issue there is Pants needs to be able to allow creating a single

python_distrubution

that combines requirements strings from 2 resolves then. It doesn't even come close to allowing this today.

I’m in favor of supporting this, but I’m also not blocked by this - I can use a different

python_distribution

per resolve I put a comment on the GitHub issue though, not sure if you saw it - testing against the wheel even within a single resolve seems to run into issues. I don’t understand the error message (linked here), but it seems like running a test against a

python_distribution

which has any 3rd party dependencies cannot be done as of now - hoping I’m just doing something wrong

@stocky-helmet-22655 FWIW, Pex has now learned how to lock this:

:; python -mpex.cli lock create \
    --pip-version latest-compatible \
    --style universal \
    --target-system linux \
    --target-system mac \
    --interpreter-constraint "CPython>=3.7,!=3.8.*,!=3.9.*,<3.11" \
    "numpy >=1.16.2,<1.20; python_version == '3.7'" \
    "numpy >=1.21.3,<1.22.0; python_version == '3.10'" \
    --indent 2 -o lock.json \
    --elide-unused-requires-dist

:; jq '.locked_resolves[] | {marker: .marker, numpy: .locked_requirements[] | select(.project_name == "numpy")}' lock.json 
{
  "marker": "(platform_system == \"Linux\" or platform_system == \"Darwin\") and platform_python_implementation == \"CPython\" and python_full_version == \"3.7.*\"",
  "numpy": {
    "artifacts": [
      {
        "algorithm": "sha256",
        "hash": "50a4a0ad0111cc1b71fa32dedd05fa239f7fb5a43a40663269bb5dc7877cfd28",
        "url": "<https://files.pythonhosted.org/packages/fe/93/b0b203c693d621d6ee9577fc650bab49434c6d23c809dc4eb49db6a339af/numpy-1.19.5-cp37-cp37m-manylinux2014_aarch64.whl>"
      },
      {
        "algorithm": "sha256",
        "hash": "6149a185cece5ee78d1d196938b2a8f9d09f5a5ebfbba66969302a778d5ddd1d",
        "url": "<https://files.pythonhosted.org/packages/08/d6/a6aaa29fea945bc6c61d11f6e0697b325ff7446de5ffd62c2fa02f627048/numpy-1.19.5-cp37-cp37m-manylinux2010_x86_64.whl>"
      },
      {
        "algorithm": "sha256",
        "hash": "06fab248a088e439402141ea04f0fffb203723148f6ee791e9c75b3e9e82f080",
        "url": "<https://files.pythonhosted.org/packages/0b/2c/b20e49628109f1f6aa56c5a283dc6001276b0b8853712dc66f2243abddfe/numpy-1.19.5-cp37-cp37m-manylinux2010_i686.whl>"
      },
      {
        "algorithm": "sha256",
        "hash": "603aa0706be710eea8884af807b1b3bc9fb2e49b9f4da439e76000f3b3c6ff0f",
        "url": "<https://files.pythonhosted.org/packages/39/ba/f3db2b5bb36fee2478b1cf47e7f9664b9c27241be80acd31c844404da297/numpy-1.19.5-cp37-cp37m-macosx_10_9_x86_64.whl>"
      },
      {
        "algorithm": "sha256",
        "hash": "a76f502430dd98d7546e1ea2250a7360c065a5fdea52b2dffe8ae7180909b6f4",
        "url": "<https://files.pythonhosted.org/packages/51/60/3f0fe5b7675a461d96b9d6729beecd3532565743278a9c3fe6dd09697fa7/numpy-1.19.5.zip>"
      },
      {
        "algorithm": "sha256",
        "hash": "36674959eed6957e61f11c912f71e78857a8d0604171dfd9ce9ad5cbf41c511c",
        "url": "<https://files.pythonhosted.org/packages/b1/e1/8c4c5632adaffc18dba4e03e97458dc1cb00583811e6982fc620b9d88515/numpy-1.19.5-cp37-cp37m-manylinux1_x86_64.whl>"
      },
      {
        "algorithm": "sha256",
        "hash": "cae865b1cae1ec2663d8ea56ef6ff185bad091a5e33ebbadd98de2cfa3fa668f",
        "url": "<https://files.pythonhosted.org/packages/ca/8d/cfeab2119d6c94e1c6ad7753853773a142109a5f8f56e1d8c940886814bc/numpy-1.19.5-cp37-cp37m-manylinux1_i686.whl>"
      }
    ],
    "project_name": "numpy",
    "requires_dists": [],
    "requires_python": ">=3.6",
    "version": "1.19.5"
  }
}
{
  "marker": "(platform_system == \"Linux\" or platform_system == \"Darwin\") and platform_python_implementation == \"CPython\" and python_full_version == \"3.10.*\"",
  "numpy": {
    "artifacts": [
      {
        "algorithm": "sha256",
        "hash": "5f30427731561ce75d7048ac254dbe47a2ba576229250fb60f0fb74db96501a1",
        "url": "<https://files.pythonhosted.org/packages/6f/7b/036000a55680e6c7eb81502b0aa27ce0ed65d4d8805613909967d9f8baf6/numpy-1.21.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl>"
      },
      {
        "algorithm": "sha256",
        "hash": "3820724272f9913b597ccd13a467cc492a0da6b05df26ea09e78b171a0bb9da6",
        "url": "<https://files.pythonhosted.org/packages/26/e7/4a6f579af8186372b03e8480e47df309520d91cfead8759b64dd5ac62688/numpy-1.21.6-cp310-cp310-macosx_11_0_arm64.whl>"
      },
      {
        "algorithm": "sha256",
        "hash": "ecb55251139706669fdec2ff073c98ef8e9a84473e51e716211b41aa0f18e656",
        "url": "<https://files.pythonhosted.org/packages/45/b7/de7b8e67f2232c26af57c205aaad29fe17754f793404f59c8a730c7a191a/numpy-1.21.6.zip>"
      },
      {
        "algorithm": "sha256",
        "hash": "fdffbfb6832cd0b300995a2b08b8f6fa9f6e856d562800fea9182316d99c4e8e",
        "url": "<https://files.pythonhosted.org/packages/4a/72/a3379f83172f1431d7949138373e3a24beed68184c9362dab1b4d465be26/numpy-1.21.6-cp310-cp310-macosx_10_9_x86_64.whl>"
      },
      {
        "algorithm": "sha256",
        "hash": "f17e562de9edf691a42ddb1eb4a5541c20dd3f9e65b09ded2beb0799c0cf29bb",
        "url": "<https://files.pythonhosted.org/packages/57/ba/d8cbdfd507b541bb247beff24d9d7304ac8ffc379cf585701187d45d4512/numpy-1.21.6-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl>"
      },
      {
        "algorithm": "sha256",
        "hash": "8737609c3bbdd48e380d463134a35ffad3b22dc56295eff6f79fd85bd0eeeb25",
        "url": "<https://files.pythonhosted.org/packages/ff/c6/05ae3c7f75b596e1bb3d78131c331eada9376a03d1af9801bd40e4675023/numpy-1.21.6-cp310-cp310-macosx_10_9_universal2.whl>"
      }
    ],
    "project_name": "numpy",
    "requires_dists": [],
    "requires_python": "<3.11,>=3.7",
    "version": "1.21.6"
  }
}

That will be released in Pex 2.62.0 in a few hours via https://github.com/pex-tool/pex/pull/2940