https://pantsbuild.org/ logo
Join Slack
Powered by
Is there a way to mark one `python_requirements` s...
# general
p
Is there a way to mark one 
python_requirements
so that it has higher priority than another? I would like a global way to resolve 
>1 target exports this module, so it is ambiguous which to use
, because the global file should always win. There are a couple of packages in the monorepo that, due to how they get deployed, depend on having their own requirements files. So, I added BUILD with 
python_requirements
for each of them. But some of those requirements overlap with the global requirements.
Maybe a way to filter the requirements file and say 
python_requirements(except=["requests"])
?
Geez. I only scratched the surface with the 3rd party deps… There are a lot of warnings like that when I run 
./pants py-constraints --summary
huh. That warning only showed up the first time. Is it bogus?
Copy code
09:15:27.46 [WARN] The target st2common/tests/unit/test_persistence_change_revision.py:tests imports `tests.unit.base.ChangeRevFakeModel`, but Pants cannot safely infer a dependency because >1 target exports this module, so it is ambiguous which to use: ['contrib/runners/orquesta_runner/tests/unit/base.py', 'st2common/tests/unit/base.py'].

Please explicitly include the dependency you want in the `dependencies` field of st2common/tests/unit/test_persistence_change_revision.py:tests, or ignore the ones you do not want by prefixing with `!` or `!!` so that <=1 targets are left.

Alternatively, you can remove the ambiguity by deleting/changing some of the targets so that only 1 target exports this module. Refer to <https://www.pantsbuild.org/v2.4/docs/troubleshooting#import-errors-and-missing-dependencies>.
How can I get rid of warnings like this?
h
The warning isn't bogus, it's the result of having the same module exported by multiple targets. There isn't currently a way to say "if there's a conflict pick this one". It's an interesting idea, but I'd have to think through the implications. Basically, "if there's no explicit dep on the other one, use this one"?
👍 1
There are a couple of packages in the monorepo that, due to how they get deployed, depend on having their own requirements files. So, I added BUILD with python_requirements for each of them.
Can you elaborate on that? And point to the code?
Why is that separate requirements.txt file required?
And are its versions always in sync with a global one, or could they actually drift?
Pants will generate requirements for setup.py from the true requirements of the wheel you're building, so I'm trying to see what the need for a separate actual requirements file is, and how to accommodate it.
For example, can Pants generate it
p
StackStorm has a concept of “packs” of content. It builds a virtualenv for each pack, and installs any dependencies in a requirements.txt file in the root of the pack.
So, those don’t get deployed as wheels, they’re a StackStorm-specific thing.
And in the case of the packs embedded in the main repo, they are mostly just for testing.
I went ahead and skipped including the requirements.txt file for the packs that have overlapping deps, and just added a 
python_requirement_library
in BUILD for the unique deps.
But that still leaves all of the internal deps.
Hmm. OK. I think I figured out how to add the relevant dependencies. I edited some BUILD files which seems to have fixed it. I look for global knobs to adjust things first before editing lots of files in lots of places. It still feels really odd to have the pants config spread all across the repo.
h
Yeah, I'd like to take a look at that branch again, I feel like there should be a better way.
For example, do those requirements.txt files in the packs need corresponding BUILD files with 
python_requirements()
at all?
If they're just there for manually building virtualenvs?
Could you describe what these "packs" are for? What the use-cases are etc?
Are these deployed?
I'll go read the documentation on them as well
p
https://docs.stackstorm.com/packs.html StackStorm has a pack index (similar to pypi or ansible galaxy): https://exchange.stackstorm.org/
I just pushed my BUILD changes
StackStorm is kind of like IFTTT for Ops. Each “pack” allows for integrating with one service (like github, hashicorp vault, jira, jenkins, …). When running things (actions or sensors) that are distributed via a pack, StackStorm manages building the virtualenv (with 
virtualenv
) and then sets PYTHONPATH when running the action or sensor to enable a few features advertised to pack authors.
Hmm. This thread has a lot of diverging lines of thought/questions… I think this should probably split out into different threads 😛
h
🙂
So, a pack is distributed as a wheel? or some special format?
p
Can you elaborate on that? And point to the code?
Here are the commits where I added things to the BUILD files to disambiguate the imports. This one is for the packs: https://github.com/st2sandbox/st2/commit/aaccd6a25b05ce70f5ea1d5f4afb2c5bdd866931 And this one is for test file deps: https://github.com/st2sandbox/st2/commit/92d2b14cf070038254318001790c21affe03e2a0
A pack is cloned from the git repository
h
will take a look later today or tomorrow, but off the top of my head it sounds like what we want is a custom rule to generate those 
requirements.txt
files, as they are (if I understand correctly) not a source of truth, but a technical means for creating a venv from source.
I have an idea or two
OK, done some reading on Packs
As I understand it, packs typically each live in their own repo or directory, with a specific structure, and a 
requirements.txt
is part of that structure.
I see a handful of those in the s2 repo itself, and those appear to be mostly for testing?
In other words, what I'm getting at is that the problem here isn't packs in general but the handful of example packs that happen to be embedded in the st2 repo?
I think this is the best way to handle this: the 
requirements.txt
for those in-repo packs don't need to be managed by Pants at all, so they don't need companion BUILD stanzas. They are technically part of a deployment artifact, not a build time artifact. At build time the pack can consume deps from the global `requirements.txt`/`contraints.txt` as any other code does. The separate, subset 
requirements.txt
exists only so that at runtime StackStorm can install the pack in a virtualenv. It is not needed at build time.
In fact it could even be generated, from the information pants already has.
But that's maybe a later step.
For now I think the issue with ambiguous deps is solved by not having BUILD file stanzas for the pack's requirements.txt, and instead consuming deps from the global one (this assumes they are in version sync, which generating the pack's requirements would ensure, but that in-sync requirement is true today, presumably this is done manually?)
👍 1
p
Yeah. It’s manual today.
So, I’ll move those deps into the top-level requirements-pants.txt with a comment about what it’s for.
And yes, most of them are for testing. There are a couple packs that get deployed for use in production systems.
4 Views
Open in Slack
PreviousNext
Powered by