https://pantsbuild.org/ logo
Join Slack
Powered by
I have a use case where I’m struggling to understa...
# general
s
I have a use case where I’m struggling to understand the proper process using pants. I have packages A and B. I want to publish B as a whl - I’m using 
python_distribution()
for that - and I have package A which depends on package B locally. I’m struggling to figure out how to get A to depend on the output of B - I could add B’s source files to A’s resolve but then A wouldn’t inherit B’s 3rd party dependencies. Depending on the generated whl doesn’t seem to be working (not even sure if it should). Package A is using poetry and pants is using 
poetry_requirements()
. Happy to provide further info but I’m not sure what the “right” way to do it is to know which errors/logs to share
e
Can A just depend on the dependencies declared by B instead of B itself? IOW A does not need a wheel (presumably), it just needs sources + 3rdparty deps ... so eliminate the middleman.
So, given:
Copy code
python_sources(name="lib")
python_distribution(name="wheel", dependencies=[":lib"])
The A should depend on "lib" and not "wheel".
Some folks do have semi-contorted legitimate reasons to actually depend on wheel middlemen in their own repo from their own repo, but if you can avoid this - great. Wheels are for pulishing only - leave it that way when you can.
s
ok I think I’ve narrowed down why I’m having issues with that solution - the dependencies in B are declared in a pyproject.toml file, and I can’t seem to get A to depend on that. In generating a whl it works fine, but when I do 
python_sources(name="app", dependencies=[":pyproject"])
it doesn’t seem to find any 3rd party dependencies
e
And what type of target is ":pyproject"? Can you share the BUILD files in-play?
s
Here’s a mostly simplified version of what I’m trying to do: https://github.com/nikwotton/Broken-Pants-Demo
It doesn’t include the python_distribution or separation of modules, not sure if including those would be more or less helpful
e
Ok, I'm blind here. I really mean can you show as much of your work as possible. 20 questions is a possibility, but its a very long road to debugging,
Oops!
I just read last sentence. Thanks you very much for the repo!!
I'm too used to folks not providing something helpful like that.
I see:
Copy code
$ pants --python-infer-unowned-dependency-behavior=error test :
19:10:43.03 [INFO] Initializing scheduler...
19:10:43.21 [INFO] Scheduler initialized.
19:10:44.99 [ERROR] 1 Exception encountered:

  UnownedDependencyError: Pants cannot infer owners for the following imports in the target //test_file.py:tests:

  * numpy (line: 1)

If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/v2.14/docs/troubleshooting#import-errors-and-missing-dependencies> for common problems.
And then:
Copy code
$ git grep numpy
setup.py:setup(install_requires=["numpy >=1.16.2"])
test_file.py:import numpy
There is no current way to get Pants to read 3rdparty dependencies from 
setup.py
, you need one of: 1. https://www.pantsbuild.org/docs/reference-python_requirements 2. https://www.pantsbuild.org/docs/reference-pipenv_requirements 3. https://www.pantsbuild.org/docs/reference-poetry_requirements All 3 of those read requirements from corresponding file type (requirements.txt file, Poetry pyproject.toml, Pipenv.lock). The 1st though allows you to just write the requirement strings inline if you have no such file.
An overview is here: https://www.pantsbuild.org/docs/python-third-party-dependencies And thanks so much for the example repo. That is gold and I appreciate it.
FWIW, modern setuptools supports reading from a requirements.txt file with no setup.py code; so you may want to use that method and it will keep your requirements DRY.
s
I’m too used to folks not providing something helpful like that.
I feel your pain - I’ve been on both sides of that 😉 so a couple responses - I’m aware of the 3 solutions you posted and I’ve used all 3 so far (I’m trying to pantsify a massive repo using a different pattern in each package because the repo’s a decade old). I can migrate to one of those 3 if need be - I want to normalize everything if I can. The only reason I want to keep stuff in the setup.py as above is because that’s how it is now and I want pants to run in parallel initially, before replacing long-term regarding the dynamic-metadata - huge thanks for the link! The use case here is to have multiple requirements blocks depending on a flag and it looks like I can set it up to use requirements.txt files using a flag as well - I’ll play around with that now The thing that led me down this path originally is this bit which gave me the impression that if pants can build a whl based on a pyproject.toml, it must be able to pull dependency info as well. Clearly I was wrong - I think it could be worth documenting this limitation and recommended alternatives on that page such as with this change: https://github.com/pantsbuild/pants/pull/18253
(I also just submitted that through readme.com because I just thought to check the contributing.md)
actually regarding the dynamic metadata bit that may not be my solution - in my actual codebase the dependencies are different depending on programmatic logic which afaik I cannot run in pyproject.toml
e
And presumably the logic cannot be replaced by environment markers?: https://peps.python.org/pep-0508/#environment-markers I.E.: the logic probes for system libraries installed or something more complex like that?
👀 1
The thing that led me down this path originally is this bit which gave me the impression that if pants can build a whl based on a pyproject.toml, it must be able to pull dependency info as well. Clearly I was wrong
Well, that's unclear. You example repo does not follow those directions either. I.E. you do not declare a `python_distribution`target at all. There is an example setuptools one there at the bottom of the doc section. Had you tried doing that and depending on that target in your test target?
s
You example repo does not follow those directions either. I.E. you do not declare a `python_distribution`target at all.
My example repo does not - my actual repo does. As I said above:
It doesn’t include the python_distribution or separation of modules, not sure if including those would be more or less helpful
In my project I did do a 
python_distribution
target and tried depending on both that and the 
resource
target in my tests - the original problem being that nothing led to 3rd party dependencies working and nothing in the docs telling me that it shouldn’t work
Granted nothing implied it should work either - imo it’s left ambiguous right now
FWIW, modern setuptools supports reading from a requirements.txt file with no setup.py code; so you may want to use that method and it will keep your requirements DRY.
Just to confirm, pants cannot infer from the pyproject.toml that it should look in a requirements.txt. You are suggesting the dependencies go in a requirements.txt which is then separately depended on by pants directly as well as the pyproject.toml unrelated to pants, correct?
Fine if so, just confirming my understanding
e
Yup, exactly.
👍 1
h
Hi @stocky-helmet-22655, thanks for the suggested edit to the documentation, which I assume was in the context of this conversation. Do I understand correctly that your suggestion is specifically for the case of a python_sources depending on a python_distribution (instead of on the python_distribution's underlying python_sources)?
s
It’s actually for the case of a 
python_sources
depending on a 
resource
, in this case a pyproject.toml resource
h
I see, but if you have a 
poetry_requirements
that wraps that same pyproject.toml things work, right?
So I'm not sure that page is the right place for this info? That page is talking about how to package wheels, and this seems unrelated directly to that (although that happens to be how you encountered the question)?
s
not if the pyproject.toml doesn’t use poetry. In my case I have a pyproject.toml that uses setuptools, and the dependencies themselves are in setup.py which pants cannot parse (I’m told).
This page is where I happened to encounter the question, and I didn’t see answers for it anywhere else. If you think there’s a better place to document that then I’m happy to suggest an edit to a different page
h
Well, we now generate docs from the repo, so I'm looking for where to apply that edit there, rather than directly in readme.com's UI
How does the pyproject.toml use setuptools? It points to it as a PEP-517 backend?
s
correct
Copy code
[build-system]
requires = ["setuptools>=61.0.0", "wheel"]
build-backend = "setuptools.build_meta"
h
Gotcha, that makes sense
s
@enough-analyst-54434 regarding your comment about environment markers - Good news and bad news. The good news is as defined I believe they should work for what I want. The bad news is pants does not behave as I would expect with them. I have another example repo for you: https://github.com/nikwotton/Broken-Pants-Demo (Same repo, new commit)
To describe what’s happening, I have the same 3rd party dependency listed twice in the requirements.txt with different versions for different environment markers. If I do 
pip install -r requirements.txt
it installs the appropriate one. If I try to use it through pants though, it appears to ignore all but the last entry and if that entry doesn’t match the environment marker it just doesn’t install the dependency
I don’t see much documentation about environment markers in pants’ docs but what I do see seems to suggest pants supports them, so I’m assuming this is a bug in pants? Or I’m just doing something wrong
e
I'll get to a keyboard shortly and use your example repo to demonstrate what is almost certainly some missing Pants-foo. This whole exercise requires Python-foo with environment markers, which you now have, and a bit of Pants BUILD-foo.
Ok, long run you probably want: https://github.com/nikwotton/Broken-Pants-Demo/pull/1 In the narrow of this example though, this makes more sense / is less obfuscated: https://github.com/nikwotton/Broken-Pants-Demo/pull/2 I attached the explanation of what's going on here to PR#2. Let me know if you have questions, but the short of it is Pants doesn't understand environment markers (Although Pex does), and so you need to convince Pants the two separate requirements are 1 for the purposes of resolving what would otherwise be an ambiguous source for the numpy package.
s
So that meets the requirements of the example I gave - that’s on me for not providing the full example. The original reason for wanting environment markers in a requirements.txt was so that the dependencies are shared between a pyproject.toml and pants. Since pants cannot parse requirements from a setup.py, I used a requirements.txt to be shared by pants and pyproject.toml. Putting the dependencies in BUILD defeats that purpose Is pants’ not understanding environment markers a bug which I can/should file? Or is there an intentional reason to not support it?
I can also update the example in the repo if my wording doesn’t make sense
e
@stocky-helmet-22655 my example1 PR 1 achieves your ends, does it not? The requirements are, in fact, in requirements.txt. You only need to non-DRY the forks like numpy. Are you saying your forks outnumber single dependencies?
In other words, I am assuming you have 50 requirements, 3 pairs of which need special treatment like the treatment demonstrated in the overrides of PR#1. Is my expectation of the ratio off?
s
@enough-analyst-54434 Unfortunately my forks do in fact outnumber my single dependencies - in this case I have nothing but forks in fact..
e
Jesus
Ok, well ... that sounds like multiple resolves.
s
it’s a small number of dependencies, but all forks
e
Do these cleanly divide into, say 2 or 3 classes?
s
2, yes
3.7 and 3.10 as in the example, just more dependencies within those
e
Did we already discuss multiple resolves?
s
not in this conversation, but I have used resolves and know how to set that up here
e
Ok. I suspect that misses your single setup.py constraint
So, what do you do today - publish 2 totally different projects / artifacts?
s
Ok. I suspect that misses your single setup.py constraint (edited)
maybe not, though it’s getting complicated. I believe setup.py can pull in multiple requirements.txt files, meaning I can do one per resolve
e
You can't be publishing 1
s
So, what do you do today - publish 2 totally different projects / artifacts?
In this case only 1 is published externally directly, but it’s a different setup when used locally vs when packaged/published as part of a different local package
so 2
e
Ok. Yeah, I think requirement file per resolve and then using the multiple requirment file dynamic install_requires setup.py approach has to be it then. If that is too complicated or unattractive for whatever reason, Pants currently does not bend enough to fit your case.
The requirements files would look a bit funny, each requirement will have the same boilerplate environment marker, but it would be maybe a clean separation, if repetitive.
s
The requirements files shouldn’t duplicate anything, it’ll just look unnecessarily split up when it could be in one file (without pants) It’s complicated/unattractive, but my project has ~30 packages and only 3 of them use setuptools. I’d rather suffer in those 3 and get to use the benefits of pants elsewhere than forego pants entirely, at least so far
e
I assume, though, that you have complication today, just elsewhere, in Makefiles?
You don't have some magic that makes this all tidy do you?
s
makefiles in some places, pipfiles in others, poetry in some, and setuptools in these…. I have no tools to make this all tidy and organized - my hope is pants can help be that tool
e
Gotcha
s
Unrelated but semi-related - do you know if there’s a reason when using poetry why pants reads pyproject.toml instead of poetry.lock?
e
I think this started with history - Pants supported requirements declared in Poetry before it supported lock files as a concept, regardless of provenance. And then when we added support for lock files as a concept, Poetry couldn't handle the world - no 
--find-links
support for example, and so it didn't suit for internal lock use since it didn't cover all Pants cases. That leaves using it as an external input, which I think could be done, but no-one has asked for it, or at least done so and followed up with the muscle to implement.
s
gotcha
e
Straight-up, our support for Poetry has been haphazard as things tend to be in OSS. There was basically no company pushing a polished feature-set there, just folks adding what they needed when they needed it without a comprehensive view, etc.
s
Makes sense. Part of me wants to request/help with poetry, but few enough of my packages use poetry and it’s a stepping stone to hopefully getting rid of it in favor of pants anyways that I can just deal with it as is for now
e
I think that's what most folks using Pants have done.
The only missing feature that seems to keep coming up is a CLI helper to add dependencies. Now I personally find that a misfeature / silly feature for Poetry to even have, but folks like it.
s
you mean something like 
./pants install numpy==1.2.3
which would automatically add the block to a BUILD file?
e
Yeah, something like that. But Poetry I think lets you guess the name, be wrong and suggest near match (attack vector!) and ditto versions, etc.
s
oh that sounds so bad
e
It simply pays to be slow and know what PyPI is and look up stuff there and etc. The 5 minutes saved is just not a thing.
s
Normally I’m in favor of saving dev time whenever possible, but in this case I agree with you - the attack vector is an issue, even similarly named packages become an issue
e
But people really like it.
I think nearly all ideas that "save dev time" tend to be misused anyhow. Most of them seem to make you know less at the end of the day.
s
It’s a tightrope walk - when things are 100% rock solid and completely remove any need for doing a thing (removing pointers in higher level languages for example), they’re great and should be used - because they “can’t” be mis-used. When they leave edge cases or introduce new possible issues though, they’re better off not being used until they’re more solid
My goto analogy is autopilot in planes - it makes the pilot’s life easier but then when something it can’t handle comes up, the pilot is less practiced/experienced and now doesn’t know how to handle it any better than the computer
On the other hand if we never use tools until they’re 100%, they may never become 100% because they don’t have users to find the edge cases
I think nearly all ideas that “save dev time” tend to be misused anyhow. Most of them seem to make you know less at the end of the day.
This said by a dev working on pants - a tool used to save dev time 😂
Back to the solution above, it’ll actually be 4 requirements files - 3.7/3.10 by required/optional which brings my to a realization/question - does pants support optional dependencies through requirements.txt files?
e
Well, they Python world does through extras. But that's just like environment markers, Pants basically knows nothing about them. Of course extras only work for packaged artifacts that declare extras metadata. With that though you can say 
thing[extra1, extra2]>=1.0
and get the optional dependencies implied by 
thing1
and by 
thing2
.
Basically, for 1st party optionals Pants would need to support this:
Copy code
optional-thing; extra = "extra1"
Now Pants does support that requirement string, but it does not offer support for injecting the "extra1" extra into the marker evaluation environment.
When you say 
thing[extra1, extra2]>=1.0
for a 3rd party dependency the standard Python machinery below Pants injects the "extra1" and "extra2" markers. It's just the 1st party case where Pants would have to provide that mechanism and does not today.
That's all a bit of words but hopefully it makes some sense @stocky-helmet-22655.
It’s a tightrope walk - when things are 100% rock solid and completely remove any need for doing a thing (removing pointers in higher level languages for example), they’re great and should be used - because they “can’t” be mis-used.
I guess, but I fooled myself by doing this for, say, 
ld
. It always works and you don't know it's there, but you pay a big price for it. I still don't understand linking and loading completely, but I've since read a book on linkers & loaders and I'm slightly less dumb. I think tools are definitely dangerous and I am wary of them at least. Writing them is another story! That forces you to learn how things work. That I like.
s
Regarding the optionals, I’ll admit it’s a little over my head right now - I’m in the process of learning by doing with so much of this at the moment. I think it makes sense, but once I implement and compare to the current I’ll no doubt have more questions to bring back - hopefully I’m not getting too annoying yet
e
Nope, its ok. The shared document of an example, or ideally real, public repo is invaluable of course.
s
regarding tools - completely agree on writing them. I love making the tools as a way of learning, when time permits As far as being dumb for using tools though, I think it’s a numbers game - a few potentially very smart developers might become a bit dumber for using the tools but far more - let’s say less interested - developers will be empowered to do things they wouldn’t have figured out how to do without those tools. They may not always use them right or understand what’s happening, but without the tools they would never get it right at all so overall I think tools are generally a win and it’s up to the developers to decide whether they’re made smarter or dumber by using them
Plus one of the better ways to learn about a concept is by looking into why a tool does that thing wrong or why it doesn’t support it - sucks in the short term but educational in the long term
e
Yeah, I'm pretty sold on empowerment, but as you say there is a fine line. It may be true that empowering less interested developers nets you - say - 5% of them converting to more interested. I'm just pretty bearish on how often that happens. It seems like if you don't make it past 14 curious, you're chances of picking that crucial bit up later is really small. You are curious and probing and learning and all of those good things, which is great. I hope to see more of that from more people, but I can't shake the feeling I'm contributing to the problem.
8 Views
Open in Slack
PreviousNext
Powered by