stocky-helmet-22655
02/14/2023, 4:18 PMpython_distribution()
for that - and I have package A which depends on package B locally. I’m struggling to figure out how to get A to depend on the output of B - I could add B’s source files to A’s resolve but then A wouldn’t inherit B’s 3rd party dependencies. Depending on the generated whl doesn’t seem to be working (not even sure if it should). Package A is using poetry and pants is using poetry_requirements()
. Happy to provide further info but I’m not sure what the “right” way to do it is to know which errors/logs to shareenough-analyst-54434
02/14/2023, 4:39 PMpython_sources(name="lib")
python_distribution(name="wheel", dependencies=[":lib"])
The A should depend on "lib" and not "wheel".stocky-helmet-22655
02/15/2023, 2:38 AMpython_sources(name="app", dependencies=[":pyproject"])
it doesn’t seem to find any 3rd party dependenciesenough-analyst-54434
02/15/2023, 2:40 AMstocky-helmet-22655
02/15/2023, 2:55 AMenough-analyst-54434
02/15/2023, 3:05 AM$ pants --python-infer-unowned-dependency-behavior=error test :
19:10:43.03 [INFO] Initializing scheduler...
19:10:43.21 [INFO] Scheduler initialized.
19:10:44.99 [ERROR] 1 Exception encountered:
UnownedDependencyError: Pants cannot infer owners for the following imports in the target //test_file.py:tests:
* numpy (line: 1)
If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/v2.14/docs/troubleshooting#import-errors-and-missing-dependencies> for common problems.
And then:
$ git grep numpy
setup.py:setup(install_requires=["numpy >=1.16.2"])
test_file.py:import numpy
There is no current way to get Pants to read 3rdparty dependencies from setup.py
, you need one of:
1. https://www.pantsbuild.org/docs/reference-python_requirements
2. https://www.pantsbuild.org/docs/reference-pipenv_requirements
3. https://www.pantsbuild.org/docs/reference-poetry_requirements
All 3 of those read requirements from corresponding file type (requirements.txt file, Poetry pyproject.toml, Pipenv.lock). The 1st though allows you to just write the requirement strings inline if you have no such file.stocky-helmet-22655
02/15/2023, 3:40 AMI’m too used to folks not providing something helpful like that.I feel your pain - I’ve been on both sides of that 😉 so a couple responses - I’m aware of the 3 solutions you posted and I’ve used all 3 so far (I’m trying to pantsify a massive repo using a different pattern in each package because the repo’s a decade old). I can migrate to one of those 3 if need be - I want to normalize everything if I can. The only reason I want to keep stuff in the setup.py as above is because that’s how it is now and I want pants to run in parallel initially, before replacing long-term regarding the dynamic-metadata - huge thanks for the link! The use case here is to have multiple requirements blocks depending on a flag and it looks like I can set it up to use requirements.txt files using a flag as well - I’ll play around with that now The thing that led me down this path originally is this bit which gave me the impression that if pants can build a whl based on a pyproject.toml, it must be able to pull dependency info as well. Clearly I was wrong - I think it could be worth documenting this limitation and recommended alternatives on that page such as with this change: https://github.com/pantsbuild/pants/pull/18253
enough-analyst-54434
02/15/2023, 4:14 AMThe thing that led me down this path originally is this bit which gave me the impression that if pants can build a whl based on a pyproject.toml, it must be able to pull dependency info as well. Clearly I was wrongWell, that's unclear. You example repo does not follow those directions either. I.E. you do not declare a `python_distribution`target at all. There is an example setuptools one there at the bottom of the doc section. Had you tried doing that and depending on that target in your test target?
stocky-helmet-22655
02/15/2023, 4:29 AMYou example repo does not follow those directions either. I.E. you do not declare a `python_distribution`target at all.My example repo does not - my actual repo does. As I said above:
It doesn’t include the python_distribution or separation of modules, not sure if including those would be more or less helpfulIn my project I did do a
python_distribution
target and tried depending on both that and the resource
target in my tests - the original problem being that nothing led to 3rd party dependencies working and nothing in the docs telling me that it shouldn’t workFWIW, modern setuptools supports reading from a requirements.txt file with no setup.py code; so you may want to use that method and it will keep your requirements DRY.Just to confirm, pants cannot infer from the pyproject.toml that it should look in a requirements.txt. You are suggesting the dependencies go in a requirements.txt which is then separately depended on by pants directly as well as the pyproject.toml unrelated to pants, correct?
enough-analyst-54434
02/15/2023, 5:19 AMhappy-kitchen-89482
02/16/2023, 11:31 PMstocky-helmet-22655
02/16/2023, 11:53 PMpython_sources
depending on a resource
, in this case a pyproject.toml resourcehappy-kitchen-89482
02/16/2023, 11:55 PMpoetry_requirements
that wraps that same pyproject.toml things work, right?stocky-helmet-22655
02/16/2023, 11:58 PMhappy-kitchen-89482
02/17/2023, 12:25 AMstocky-helmet-22655
02/17/2023, 12:30 AM[build-system]
requires = ["setuptools>=61.0.0", "wheel"]
build-backend = "setuptools.build_meta"
happy-kitchen-89482
02/17/2023, 12:53 AMstocky-helmet-22655
02/17/2023, 2:43 PMpip install -r requirements.txt
it installs the appropriate one. If I try to use it through pants though, it appears to ignore all but the last entry and if that entry doesn’t match the environment marker it just doesn’t install the dependencyenough-analyst-54434
02/17/2023, 2:52 PMstocky-helmet-22655
02/17/2023, 4:29 PMenough-analyst-54434
02/17/2023, 4:42 PMstocky-helmet-22655
02/17/2023, 4:52 PMenough-analyst-54434
02/17/2023, 4:52 PMstocky-helmet-22655
02/17/2023, 4:52 PMenough-analyst-54434
02/17/2023, 4:53 PMstocky-helmet-22655
02/17/2023, 4:53 PMenough-analyst-54434
02/17/2023, 4:53 PMstocky-helmet-22655
02/17/2023, 4:54 PMenough-analyst-54434
02/17/2023, 4:55 PMstocky-helmet-22655
02/17/2023, 4:56 PMOk. I suspect that misses your single setup.py constraint (edited)maybe not, though it’s getting complicated. I believe setup.py can pull in multiple requirements.txt files, meaning I can do one per resolve
enough-analyst-54434
02/17/2023, 4:56 PMstocky-helmet-22655
02/17/2023, 4:57 PMSo, what do you do today - publish 2 totally different projects / artifacts?In this case only 1 is published externally directly, but it’s a different setup when used locally vs when packaged/published as part of a different local package
enough-analyst-54434
02/17/2023, 4:58 PMstocky-helmet-22655
02/17/2023, 5:01 PMenough-analyst-54434
02/17/2023, 5:01 PMstocky-helmet-22655
02/17/2023, 5:02 PMenough-analyst-54434
02/17/2023, 5:02 PMstocky-helmet-22655
02/17/2023, 5:03 PMenough-analyst-54434
02/17/2023, 5:06 PM--find-links
support for example, and so it didn't suit for internal lock use since it didn't cover all Pants cases. That leaves using it as an external input, which I think could be done, but no-one has asked for it, or at least done so and followed up with the muscle to implement.stocky-helmet-22655
02/17/2023, 5:07 PMenough-analyst-54434
02/17/2023, 5:08 PMstocky-helmet-22655
02/17/2023, 5:10 PMenough-analyst-54434
02/17/2023, 5:10 PMstocky-helmet-22655
02/17/2023, 5:13 PM./pants install numpy==1.2.3
which would automatically add the block to a BUILD file?enough-analyst-54434
02/17/2023, 5:13 PMstocky-helmet-22655
02/17/2023, 5:14 PMenough-analyst-54434
02/17/2023, 5:14 PMstocky-helmet-22655
02/17/2023, 5:15 PMenough-analyst-54434
02/17/2023, 5:15 PMstocky-helmet-22655
02/17/2023, 5:18 PMI think nearly all ideas that “save dev time” tend to be misused anyhow. Most of them seem to make you know less at the end of the day.This said by a dev working on pants - a tool used to save dev time 😂
enough-analyst-54434
02/17/2023, 5:30 PMthing[extra1, extra2]>=1.0
and get the optional dependencies implied by thing1
and by thing2
.optional-thing; extra = "extra1"
Now Pants does support that requirement string, but it does not offer support for injecting the "extra1" extra into the marker evaluation environment.thing[extra1, extra2]>=1.0
for a 3rd party dependency the standard Python machinery below Pants injects the "extra1" and "extra2" markers. It's just the 1st party case where Pants would have to provide that mechanism and does not today.It’s a tightrope walk - when things are 100% rock solid and completely remove any need for doing a thing (removing pointers in higher level languages for example), they’re great and should be used - because they “can’t” be mis-used.I guess, but I fooled myself by doing this for, say,
ld
. It always works and you don't know it's there, but you pay a big price for it. I still don't understand linking and loading completely, but I've since read a book on linkers & loaders and I'm slightly less dumb. I think tools are definitely dangerous and I am wary of them at least. Writing them is another story! That forces you to learn how things work. That I like.stocky-helmet-22655
02/17/2023, 6:30 PMenough-analyst-54434
02/17/2023, 6:32 PMstocky-helmet-22655
02/17/2023, 6:34 PMenough-analyst-54434
02/17/2023, 6:44 PM