#general
Title
s
stocky-helmet-22655
02/14/2023, 4:18 PMI have a use case where I’m struggling to understand the proper process using pants. I have packages A and B. I want to publish B as a whl - I’m using
python_distribution()poetry_requirements()e
enough-analyst-54434
02/14/2023, 4:39 PMCan A just depend on the dependencies declared by B instead of B itself? IOW A does not need a wheel (presumably), it just needs sources + 3rdparty deps ... so eliminate the middleman.
So, given:
Copy code
python_sources(name="lib")
python_distribution(name="wheel", dependencies=[":lib"])Some folks do have semi-contorted legitimate reasons to actually depend on wheel middlemen in their own repo from their own repo, but if you can avoid this - great. Wheels are for pulishing only - leave it that way when you can.
s
stocky-helmet-22655
02/15/2023, 2:38 AMok I think I’ve narrowed down why I’m having issues with that solution - the dependencies in B are declared in a pyproject.toml file, and I can’t seem to get A to depend on that. In generating a whl it works fine, but when I do
python_sources(name="app", dependencies=[":pyproject"])e
enough-analyst-54434
02/15/2023, 2:40 AMAnd what type of target is ":pyproject"? Can you share the BUILD files in-play?
s
stocky-helmet-22655
02/15/2023, 2:55 AMHere’s a mostly simplified version of what I’m trying to do: https://github.com/nikwotton/Broken-Pants-Demo
It doesn’t include the python_distribution or separation of modules, not sure if including those would be more or less helpful
e
enough-analyst-54434
02/15/2023, 3:05 AMOk, I'm blind here. I really mean can you show as much of your work as possible. 20 questions is a possibility, but its a very long road to debugging,
Oops!
I just read last sentence. Thanks you very much for the repo!!
I'm too used to folks not providing something helpful like that.
I see:
Copy code
$ pants --python-infer-unowned-dependency-behavior=error test :
19:10:43.03 [INFO] Initializing scheduler...
19:10:43.21 [INFO] Scheduler initialized.
19:10:44.99 [ERROR] 1 Exception encountered:
UnownedDependencyError: Pants cannot infer owners for the following imports in the target //test_file.py:tests:
* numpy (line: 1)
If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/v2.14/docs/troubleshooting#import-errors-and-missing-dependencies> for common problems. Copy code
$ git grep numpy
setup.py:setup(install_requires=["numpy >=1.16.2"])
test_file.py:import numpysetup.pyAn overview is here: https://www.pantsbuild.org/docs/python-third-party-dependencies
And thanks so much for the example repo. That is gold and I appreciate it.
FWIW, modern setuptools supports reading from a requirements.txt file with no setup.py code; so you may want to use that method and it will keep your requirements DRY.
s
stocky-helmet-22655
02/15/2023, 3:40 AMI’m too used to folks not providing something helpful like that.I feel your pain - I’ve been on both sides of that 😉 so a couple responses - I’m aware of the 3 solutions you posted and I’ve used all 3 so far (I’m trying to pantsify a massive repo using a different pattern in each package because the repo’s a decade old). I can migrate to one of those 3 if need be - I want to normalize everything if I can. The only reason I want to keep stuff in the setup.py as above is because that’s how it is now and I want pants to run in parallel initially, before replacing long-term regarding the dynamic-metadata - huge thanks for the link! The use case here is to have multiple requirements blocks depending on a flag and it looks like I can set it up to use requirements.txt files using a flag as well - I’ll play around with that now The thing that led me down this path originally is this bit which gave me the impression that if pants can build a whl based on a pyproject.toml, it must be able to pull dependency info as well. Clearly I was wrong - I think it could be worth documenting this limitation and recommended alternatives on that page such as with this change: https://github.com/pantsbuild/pants/pull/18253
(I also just submitted that through readme.com because I just thought to check the contributing.md)
actually regarding the dynamic metadata bit that may not be my solution - in my actual codebase the dependencies are different depending on programmatic logic which afaik I cannot run in pyproject.toml
e
enough-analyst-54434
02/15/2023, 4:14 AMAnd presumably the logic cannot be replaced by environment markers?: https://peps.python.org/pep-0508/#environment-markers
I.E.: the logic probes for system libraries installed or something more complex like that?
👀 1
The thing that led me down this path originally is this bit which gave me the impression that if pants can build a whl based on a pyproject.toml, it must be able to pull dependency info as well. Clearly I was wrongWell, that's unclear. You example repo does not follow those directions either. I.E. you do not declare a `python_distribution`target at all. There is an example setuptools one there at the bottom of the doc section. Had you tried doing that and depending on that target in your test target?
s
stocky-helmet-22655
02/15/2023, 4:29 AMYou example repo does not follow those directions either. I.E. you do not declare a `python_distribution`target at all.My example repo does not - my actual repo does. As I said above:
It doesn’t include the python_distribution or separation of modules, not sure if including those would be more or less helpfulIn my project I did do a
python_distributionresourceGranted nothing implied it should work either - imo it’s left ambiguous right now
FWIW, modern setuptools supports reading from a requirements.txt file with no setup.py code; so you may want to use that method and it will keep your requirements DRY.Just to confirm, pants cannot infer from the pyproject.toml that it should look in a requirements.txt. You are suggesting the dependencies go in a requirements.txt which is then separately depended on by pants directly as well as the pyproject.toml unrelated to pants, correct?
Fine if so, just confirming my understanding
e
enough-analyst-54434
02/15/2023, 5:19 AMYup, exactly.
👍 1
h
happy-kitchen-89482
02/16/2023, 11:31 PMHi @stocky-helmet-22655, thanks for the suggested edit to the documentation, which I assume was in the context of this conversation. Do I understand correctly that your suggestion is specifically for the case of a python_sources depending on a python_distribution (instead of on the python_distribution's underlying python_sources)?
s
stocky-helmet-22655
02/16/2023, 11:53 PMIt’s actually for the case of a
python_sourcesresourceh
happy-kitchen-89482
02/16/2023, 11:55 PMI see, but if you have a
poetry_requirementsSo I'm not sure that page is the right place for this info? That page is talking about how to package wheels, and this seems unrelated directly to that (although that happens to be how you encountered the question)?
s
stocky-helmet-22655
02/16/2023, 11:58 PMnot if the pyproject.toml doesn’t use poetry. In my case I have a pyproject.toml that uses setuptools, and the dependencies themselves are in setup.py which pants cannot parse (I’m told).
This page is where I happened to encounter the question, and I didn’t see answers for it anywhere else. If you think there’s a better place to document that then I’m happy to suggest an edit to a different page
h
happy-kitchen-89482
02/17/2023, 12:25 AMWell, we now generate docs from the repo, so I'm looking for where to apply that edit there, rather than directly in readme.com's UI
How does the pyproject.toml use setuptools? It points to it as a PEP-517 backend?
s
stocky-helmet-22655
02/17/2023, 12:30 AMcorrect
Copy code
[build-system]
requires = ["setuptools>=61.0.0", "wheel"]
build-backend = "setuptools.build_meta"h
happy-kitchen-89482
02/17/2023, 12:53 AMGotcha, that makes sense
s
stocky-helmet-22655
02/17/2023, 2:43 PM@enough-analyst-54434 regarding your comment about environment markers - Good news and bad news. The good news is as defined I believe they should work for what I want. The bad news is pants does not behave as I would expect with them. I have another example repo for you: https://github.com/nikwotton/Broken-Pants-Demo
(Same repo, new commit)
To describe what’s happening, I have the same 3rd party dependency listed twice in the requirements.txt with different versions for different environment markers. If I do
pip install -r requirements.txtI don’t see much documentation about environment markers in pants’ docs but what I do see seems to suggest pants supports them, so I’m assuming this is a bug in pants? Or I’m just doing something wrong
e
enough-analyst-54434
02/17/2023, 2:52 PMI'll get to a keyboard shortly and use your example repo to demonstrate what is almost certainly some missing Pants-foo. This whole exercise requires Python-foo with environment markers, which you now have, and a bit of Pants BUILD-foo.
Ok, long run you probably want: https://github.com/nikwotton/Broken-Pants-Demo/pull/1
In the narrow of this example though, this makes more sense / is less obfuscated: https://github.com/nikwotton/Broken-Pants-Demo/pull/2
I attached the explanation of what's going on here to PR#2.
Let me know if you have questions, but the short of it is Pants doesn't understand environment markers (Although Pex does), and so you need to convince Pants the two separate requirements are 1 for the purposes of resolving what would otherwise be an ambiguous source for the numpy package.
s
stocky-helmet-22655
02/17/2023, 4:29 PMSo that meets the requirements of the example I gave - that’s on me for not providing the full example. The original reason for wanting environment markers in a requirements.txt was so that the dependencies are shared between a pyproject.toml and pants. Since pants cannot parse requirements from a setup.py, I used a requirements.txt to be shared by pants and pyproject.toml. Putting the dependencies in BUILD defeats that purpose
Is pants’ not understanding environment markers a bug which I can/should file? Or is there an intentional reason to not support it?
I can also update the example in the repo if my wording doesn’t make sense
e
enough-analyst-54434
02/17/2023, 4:42 PM@stocky-helmet-22655 my example1 PR 1 achieves your ends, does it not? The requirements are, in fact, in requirements.txt. You only need to non-DRY the forks like numpy. Are you saying your forks outnumber single dependencies?
In other words, I am assuming you have 50 requirements, 3 pairs of which need special treatment like the treatment demonstrated in the overrides of PR#1. Is my expectation of the ratio off?
For example, see the Pants repo's own requirements.txt: https://github.com/pantsbuild/pants/blob/main/3rdparty/python/requirements.txt
And its BUILD - 1 override, all the rest "normal": https://github.com/pantsbuild/pants/blob/8384c5cfa73b148a761d6229b34b4205cb24e19d/3rdparty/python/BUILD#L4-L13
s
stocky-helmet-22655
02/17/2023, 4:52 PM@enough-analyst-54434 Unfortunately my forks do in fact outnumber my single dependencies - in this case I have nothing but forks in fact..
e
enough-analyst-54434
02/17/2023, 4:52 PMJesus
Ok, well ... that sounds like multiple resolves.
s
stocky-helmet-22655
02/17/2023, 4:52 PMit’s a small number of dependencies, but all forks
e
enough-analyst-54434
02/17/2023, 4:53 PMDo these cleanly divide into, say 2 or 3 classes?
s
stocky-helmet-22655
02/17/2023, 4:53 PM2, yes
3.7 and 3.10 as in the example, just more dependencies within those
e
enough-analyst-54434
02/17/2023, 4:53 PMDid we already discuss multiple resolves?
s
stocky-helmet-22655
02/17/2023, 4:54 PMnot in this conversation, but I have used resolves and know how to set that up here
e
enough-analyst-54434
02/17/2023, 4:55 PMOk. I suspect that misses your single setup.py constraint
So, what do you do today - publish 2 totally different projects / artifacts?
s
stocky-helmet-22655
02/17/2023, 4:56 PMOk. I suspect that misses your single setup.py constraint (edited)maybe not, though it’s getting complicated. I believe setup.py can pull in multiple requirements.txt files, meaning I can do one per resolve
e
enough-analyst-54434
02/17/2023, 4:56 PMYou can't be publishing 1
s
stocky-helmet-22655
02/17/2023, 4:57 PMSo, what do you do today - publish 2 totally different projects / artifacts?In this case only 1 is published externally directly, but it’s a different setup when used locally vs when packaged/published as part of a different local package
so 2
e
enough-analyst-54434
02/17/2023, 4:58 PMOk. Yeah, I think requirement file per resolve and then using the multiple requirment file dynamic install_requires setup.py approach has to be it then. If that is too complicated or unattractive for whatever reason, Pants currently does not bend enough to fit your case.
The requirements files would look a bit funny, each requirement will have the same boilerplate environment marker, but it would be maybe a clean separation, if repetitive.
s
stocky-helmet-22655
02/17/2023, 5:01 PMThe requirements files shouldn’t duplicate anything, it’ll just look unnecessarily split up when it could be in one file (without pants)
It’s complicated/unattractive, but my project has ~30 packages and only 3 of them use setuptools. I’d rather suffer in those 3 and get to use the benefits of pants elsewhere than forego pants entirely, at least so far
e
enough-analyst-54434
02/17/2023, 5:01 PMI assume, though, that you have complication today, just elsewhere, in Makefiles?
You don't have some magic that makes this all tidy do you?
s
stocky-helmet-22655
02/17/2023, 5:02 PMmakefiles in some places, pipfiles in others, poetry in some, and setuptools in these…. I have no tools to make this all tidy and organized - my hope is pants can help be that tool
e
enough-analyst-54434
02/17/2023, 5:02 PMGotcha
s
stocky-helmet-22655
02/17/2023, 5:03 PMUnrelated but semi-related - do you know if there’s a reason when using poetry why pants reads pyproject.toml instead of poetry.lock?
e
enough-analyst-54434
02/17/2023, 5:06 PMI think this started with history - Pants supported requirements declared in Poetry before it supported lock files as a concept, regardless of provenance. And then when we added support for lock files as a concept, Poetry couldn't handle the world - no
--find-linkss
stocky-helmet-22655
02/17/2023, 5:07 PMgotcha
e
enough-analyst-54434
02/17/2023, 5:08 PMStraight-up, our support for Poetry has been haphazard as things tend to be in OSS. There was basically no company pushing a polished feature-set there, just folks adding what they needed when they needed it without a comprehensive view, etc.
s
stocky-helmet-22655
02/17/2023, 5:10 PMMakes sense. Part of me wants to request/help with poetry, but few enough of my packages use poetry and it’s a stepping stone to hopefully getting rid of it in favor of pants anyways that I can just deal with it as is for now
e
enough-analyst-54434
02/17/2023, 5:10 PMI think that's what most folks using Pants have done.
The only missing feature that seems to keep coming up is a CLI helper to add dependencies. Now I personally find that a misfeature / silly feature for Poetry to even have, but folks like it.
s
stocky-helmet-22655
02/17/2023, 5:13 PMyou mean something like
./pants install numpy==1.2.3e
enough-analyst-54434
02/17/2023, 5:13 PMYeah, something like that. But Poetry I think lets you guess the name, be wrong and suggest near match (attack vector!) and ditto versions, etc.
s
stocky-helmet-22655
02/17/2023, 5:14 PMoh that sounds so bad
e
enough-analyst-54434
02/17/2023, 5:14 PMIt simply pays to be slow and know what PyPI is and look up stuff there and etc. The 5 minutes saved is just not a thing.
s
stocky-helmet-22655
02/17/2023, 5:15 PMNormally I’m in favor of saving dev time whenever possible, but in this case I agree with you - the attack vector is an issue, even similarly named packages become an issue
e
enough-analyst-54434
02/17/2023, 5:15 PMBut people really like it.
I think nearly all ideas that "save dev time" tend to be misused anyhow. Most of them seem to make you know less at the end of the day.
s
stocky-helmet-22655
02/17/2023, 5:18 PMIt’s a tightrope walk - when things are 100% rock solid and completely remove any need for doing a thing (removing pointers in higher level languages for example), they’re great and should be used - because they “can’t” be mis-used. When they leave edge cases or introduce new possible issues though, they’re better off not being used until they’re more solid
My goto analogy is autopilot in planes - it makes the pilot’s life easier but then when something it can’t handle comes up, the pilot is less practiced/experienced and now doesn’t know how to handle it any better than the computer
On the other hand if we never use tools until they’re 100%, they may never become 100% because they don’t have users to find the edge cases
I think nearly all ideas that “save dev time” tend to be misused anyhow. Most of them seem to make you know less at the end of the day.This said by a dev working on pants - a tool used to save dev time 😂
Back to the solution above, it’ll actually be 4 requirements files - 3.7/3.10 by required/optional which brings my to a realization/question - does pants support optional dependencies through requirements.txt files?
e
enough-analyst-54434
02/17/2023, 5:30 PMWell, they Python world does through extras. But that's just like environment markers, Pants basically knows nothing about them. Of course extras only work for packaged artifacts that declare extras metadata. With that though you can say
thing[extra1, extra2]>=1.0thing1thing2Basically, for 1st party optionals Pants would need to support this:
Copy code
optional-thing; extra = "extra1"When you say
thing[extra1, extra2]>=1.0That's all a bit of words but hopefully it makes some sense @stocky-helmet-22655.
It’s a tightrope walk - when things are 100% rock solid and completely remove any need for doing a thing (removing pointers in higher level languages for example), they’re great and should be used - because they “can’t” be mis-used.I guess, but I fooled myself by doing this for, say,
lds
stocky-helmet-22655
02/17/2023, 6:30 PMRegarding the optionals, I’ll admit it’s a little over my head right now - I’m in the process of learning by doing with so much of this at the moment. I think it makes sense, but once I implement and compare to the current I’ll no doubt have more questions to bring back - hopefully I’m not getting too annoying yet
e
enough-analyst-54434
02/17/2023, 6:32 PMNope, its ok. The shared document of an example, or ideally real, public repo is invaluable of course.
s
stocky-helmet-22655
02/17/2023, 6:34 PMregarding tools - completely agree on writing them. I love making the tools as a way of learning, when time permits
As far as being dumb for using tools though, I think it’s a numbers game - a few potentially very smart developers might become a bit dumber for using the tools but far more - let’s say less interested - developers will be empowered to do things they wouldn’t have figured out how to do without those tools. They may not always use them right or understand what’s happening, but without the tools they would never get it right at all so overall I think tools are generally a win and it’s up to the developers to decide whether they’re made smarter or dumber by using them
Plus one of the better ways to learn about a concept is by looking into why a tool does that thing wrong or why it doesn’t support it - sucks in the short term but educational in the long term
e
enough-analyst-54434
02/17/2023, 6:44 PMYeah, I'm pretty sold on empowerment, but as you say there is a fine line. It may be true that empowering less interested developers nets you - say - 5% of them converting to more interested. I'm just pretty bearish on how often that happens. It seems like if you don't make it past 14 curious, you're chances of picking that crucial bit up later is really small. You are curious and probing and learning and all of those good things, which is great. I hope to see more of that from more people, but I can't shake the feeling I'm contributing to the problem.
8 Views