Hi everyone I have couple of apps in my repository...
# general
b
Hi everyone I have couple of apps in my repository that use the same libs(also in the repo) and each app can have different interpreter and different 3rd party package versions The libs themselves also have their own requirements I'm trying to export a virtualenv using
pants export
to use it in Pycharm but I'm not using lockfiles, instead I use
ambiguity_resolution = "by_source_root"
I'm not sure if this option conflicts with having lockfiles, I tried to do it but with no success Using lockfiles on its own works great and allows me to export virtualenv-s but it makes everything so much more complex It makes me declare so many things that are automatically inferred when using
ambiguity_resolution = "by_source_root"
Is there a way to create a virtualenv according to a target instead of resolve? Or maybe is there a way to combine the 2 methods? Let me know if you want me to supply a detailed example
h
cc @steep-eve-20716 since I think this sounds related to something he's brought up
But specifically re ambiguity_resolution, it is not instead of lockfiles, it works with lockfiles, but not quite how you want it to
πŸ’― 1
this is the discussion in question, I think your issue, if I understand it correctly, boils down to this? Or am I misunderstanding?
b
Yeah youre right its really similar to my issue I actually now got my repo to work with both lockfiles and "by_source_root" So now the only issue for me is that we need to parametrize targets to add all libraries to all resolves But its not that bad and I probably can make it work with a plugin so people wont have to do it themselves Initially I thought that when using lockfiles pants cant infer 1st party dependencies and aggregate their 3rd party dependencies but that now works for me thanks
by the way in our repo we decided that in CI if someone changed code of a lib we would run the lib tests for each of the resolves the lib "supports" so the lib doesnt have a lockfile of its own, its always using some service's lockfile
s
@breezy-cricket-45254 be warned, I thought the same thing about parametrizing libraries. But be warned, if you do any overriding of the libraries dependencies, it won’t behave as expected
b
what do you mean by overriding? do you mean that the service has a different requirement for a package than the library requirement? something like this?
Copy code
service -> library

# service/pyproject.toml
[tool.poetry.dependencies]
requests = "2.20.0"


# library/pyproject.toml
[tool.poetry.dependencies]
requests = ">=2.20,<2.33"
because this does work for me
πŸ‘€ 1
s
It β€œworks” but with different versions than perhaps expected. Running your service with poetry will use requests 2.20, running with pants will use requests 2.33
b
its using 2.20 as expected here is the output from my example:
Copy code
18:04:18.65 [INFO] Completed: Building 3 requirements for app_1.pex from the python/apps/app_group_1/app_1/lock.json resolve: numpy==1.21.1, requests<2.33,>=2.20, requests==2.20.0
Hello, app_1!
requests version: 2.20.0
python 3.7.17 (default, Jul 21 2024, 16:26:57) 
[Clang 15.0.0 (clang-1500.3.9.4)]
numpy version: 1.21.1
requests version: 2.20.0
πŸ‘€ 1
I can create a repo with our solution if you want so you can take a look at it
s
That would be awesome
b
ok will send here a link once I finish
πŸ‘ 1
s
This repo doesn't have any unit tests, which is best at surfacing the challenges of resolves. I added unit tests here: https://github.com/jasondamour/pants-by-source-resolve/tree/jason/add-unit-tests β€’ You had a configuration issue in your repo, where lib2 was in both resolves
app_1
and
app_2
. β—¦ That causes unit tests to break, because
lib2@resolve=app_1
can't find
lib3
since lib3 is only in
lib3@resolve=app_2
β—¦ this is an easy fix, but it shows how difficult/complex it is working with resolves, as opposed to the intuitive config of poetry β€’ Unit tests run per resolve β—¦ Our core "common" library has unit tests which take 10 mins. Its used in 24 services πŸ™ƒ
But overall this repo was pretty helpful for me
πŸ’― 1
b
right lib_2 isnt supposed to support app_1 I intend to use a plugin to handle this so mistakes like this wont happen but I see your point
h
@steep-eve-20716 I'm not entirely sure about your unit tests scenario, but did you see this? https://www.pantsbuild.org/2.20/docs/python/overview/lockfiles#lockfiles-for-tools
s
What about it?
h
Looks like you can control the lock files of pytest which is probably what your configuration is missing
s
Thats not the issue. The issue is that
pants test lib/::
run tests for all resolves that use the library
h
Isn't that effectively what this line does?
Copy code
__defaults__(all=dict(resolve=parametrize("libA", "libB", "service1", "service2")))
s
But thats not the desired behavior. Tests for a library should only run against its own resolve *IMO, *configurably
h
Oh, I see your point now...
s
@breezy-cricket-45254 an enormous "thank you". Your repo helped me understand resolves better than the docs, and unblocked my project ❀️ (If you happen to get a plugin working to simplify parametrization, I'd be interested in seeing that too, but you've done enough already πŸ˜„ )
b
happy to hear that! actually I got my solution working only after reading yours so thank you as well I'll post here the plugin once I have it but it can take a while
πŸ‘ 1
h
This makes me very happy! Thanks for being part of a constructive and helpful community folks!
This makes Pants better for everyone
Also, it sounds like you now understand resolves better than the docs do (maybe better than I do), so any PRs to improve the docs (which are in the repo) would be extremely welcome.
πŸ‘ 2
s
I can still see a few clear things which could improve the current experience with resolves: 1. [Question] Is it possible to filter targets by resolve/parametrize? i.e.
pants test :: --filter-target-resolve=app1
? I don't see any indication here a. Use-case here is that our libraries take 10+ mins to test, so I would like to just pick a single resolve and test against that b. Maybe extending the current filters syntax? like
--filter-target-type=python_tests(resolve=="app1", interpreter_constraints=="3.10.*")
c. Or maybe this should be (or already is) in target expansion? i.e.
pants test libs::@resolve=app1
? 2. [Feature Request] Manually adding libraries to resolves is error-prone a. "pushing" targets to a resolve means a library needs to know everywhere its consumed by, which breaks the typical development workflow. b. An amazing feature would be if pants could automatically add transitive targets to resolves 3. [Feature Request] Configure resolves/lockfiles more granularly a. Some subdirectories don't need resolves at all, and for some resolves I would still prefer the "wing it" dependency resolution b. This is a broader request and is already documented, but just noting it here
One approach for #1 that ig would work today is parametrizing the resolve and tags, like so:
Copy code
**parametrize('app_1', resolve='app_1', tags=["app1_resolve"]),
then in theory I could filter by tags...
πŸ’― 1
well I found the open issue for it: https://github.com/pantsbuild/pants/issues/17856 but much more obvious for use case: just dont parameterize tests πŸ™ƒ
Oh actually this ^ isn't a valid workaround for typechecking. So we definitely need resolve filtering
And also need some way to set the
[mypy].config
per directory, OR update
config_discovery
logic to search within the source_root tree
@breezy-cricket-45254 Sharing an update, since I've made a lot of progress. I've created the following macro, which works fairly well so far:
Copy code
def python_module(name: str, resolves: set[str] = None, **kwargs):
    all_defaults: dict[str, str | list | dict] = {"tags": [name]}
    target_defaults: dict[str, dict] = {"python_sources": {}, "python_tests": {}}
    # If a module belongs to multiple resolves..
    if resolves:
        # Then for each resolve in the list...
        for r in resolves:
            # Check if the resolve is for this specific module
            is_resolve_owner = r == name
            # Default to skip all checks for all resolves except the owning resolve
            python_defaults = {
                "skip_mypy": not is_resolve_owner,
                "skip_bandit": not is_resolve_owner,
                "skip_black": not is_resolve_owner,
                "skip_docformatter": not is_resolve_owner,
                "skip_flake8": not is_resolve_owner,
                "skip_isort": not is_resolve_owner,
                "skip_mypy": not is_resolve_owner,
            }
            # Add python_sources to the resolve, and skip checks for all resolves except the owning resolve
            # i.e. only run mypy for common/file.py@resolve=common and not common/file.py@resolve=activity_logger
            target_defaults["python_sources"].update(**parametrize(r, resolve=r, **python_defaults))
            # Add python_tests to the resolve, and skip tests for all resolves except the owning resolve
            # i.e. only run tests for common/file_test.py@resolve=common and not common/file_test.py@resolve=activity_logger
            target_defaults["python_tests"].update(
                **parametrize(r, resolve=r, skip_tests=(not is_resolve_owner), **python_defaults)
            )
            # Default all other targets to use resolves
            all_defaults.update(**parametrize(r, resolve=r))
    else:
        all_defaults["resolve"] = name
    # Update all_defaults with kwargs
    all_defaults.update(kwargs)
    __defaults__(target_defaults, all=all_defaults, ignore_unknown_fields=True)
Ugh, I spoke too soon. This actually isn't able to resolve dependencies...
b
thanks for the update we are thinking about creating a rule that checks dependents and adds resolves according to it so each time pants runs a lib it will run that rule essentially doing the same as pants does for figuring out requirements but for resolves as well we dont have anything yet though