https://pantsbuild.org/ logo
Join Slack
Powered by
Would anyone be able to give me a few hints at wha...
# general
n
Would anyone be able to give me a few hints at what I've done wrong with my setup? I have some basics working but I'm not sure how to fix the rough edges I have. https://github.com/Incognito/python-architecture-linter/pull/14 1. It will infer 
click
but not 
astroid
, did I do something to confuse it? 2. I can't seem to run tests Also any general "this is bad style" feedback is welcome, I'm unfamiliar with the library still and open to find out more. The ultimate goal is to release 3 pypi packages via a CI/CD pipeline once I'm done.
I made a lot of good progress, I can now release multiple pypi packages and I'm able to integrate them into a unrelated project. I'm still a bit unsure about some of the quirks I ran into, but it looks like I could make a lot of improvements to how testing works, possibly even the release process. Any help/guidance would be appreciated. If I get more common use-cases working I can cross-reference it as an example for the poetry community as an example of how to release many packages from a poetry-run project. ( https://github.com/python-poetry/poetry/issues/936 )
e
It would be really helpful if you - here or on the PR - commented on the exact commands you ran, the result and what you expect. For instance, the fact your using 
PANTS_SHA=0182de9814c7694eed2b9407e3577684db26ea2e ./pants ...
is off in another thread.
For example with this diff:
Copy code
$ git diff
diff --git a/pants.toml b/pants.toml
index 3229711..a37c531 100644
--- a/pants.toml
+++ b/pants.toml
@@ -2,3 +2,6 @@
 pants_version = "2.5.1rc3"
 
 backend_packages = ["pants.backend.python"]
+
+[python-setup]
+interpreter_constraints = ["==3.9.*"]
I get:
Copy code
$ $ PANTS_SHA=0182de9814c7694eed2b9407e3577684db26ea2e ./pants test modular_provider_architecture_definition/::
09:03:59.47 [INFO] Initializing scheduler...
09:03:59.62 [INFO] Scheduler initialized.
09:04:00.76 [WARN] Unmatched glob from python_architecture_linter:tests's `sources` field: "python_architecture_linter/**/test_*.py"

Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to (<https://www.pantsbuild.org/v2.6/docs/troubleshooting#pants-cannot-find-a-file-in-your-project>).
09:04:03.81 [INFO] Completed: Building requirements.pex with 1 requirement: astroid<3.0.0,>=2.5.8
09:04:06.90 [INFO] Completed: Building pytest.pex with 2 requirements: pytest-cov>=2.10.1,<2.12, pytest>=6.0.1,<6.3
09:04:07.96 [INFO] Completed: Building pytest_runner.pex
09:04:08.63 [WARN] Completed: test - modular_provider_architecture_definition/tests/test_definition.py:../tests failed (exit code 1).
============================= test session starts ==============================
platform linux -- Python 3.9.5, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /tmp/process-executionnzdAfn
plugins: cov-2.11.1
collected 1 item

modular_provider_architecture_definition/tests/test_definition.py F      [100%]

=================================== FAILURES ===================================
_______________________________ test_definition ________________________________

    def test_definition():
        path = os.path.dirname(os.path.realpath(__file__)) + "/cases/modular_provider_architecture"  # noqa: E501
        linter = Linter(project_definition)
        results = list(linter.lint(path))
    
>       assert results == [
            ValidationResult(
                explanation="No issues found",
                is_valid=True,
                location=path + "",
                validator="must_only_have_provider_in_module_root",
            ),
...
            ValidationResult(
                explanation="No issues found",
                is_valid=True,
                location=path + "/modular_provider_architecture/module_runtime/run.py",
                validator="must_only_be_in_run_modules",
            ),
        ]
E       AssertionError: assert [ValidationRe...module_root')] == [ValidationRe...method'), ...]
E         Right contains 33 more items, first extra item: ValidationResult(explanation='No issues found', is_valid=True, location='/tmp/process-executionnzdAfn/modular_provider...0', validator='python_architecture_linter.ast_validators.nodeng_validator.validate_node_children_exclusive_allow_list')
E         Use -v to get the full diff

modular_provider_architecture_definition/tests/test_definition.py:14: AssertionError
=========================== short test summary info ============================
FAILED modular_provider_architecture_definition/tests/test_definition.py::test_definition
============================== 1 failed in 0.25s ===============================



š„‚ modular_provider_architecture_definition/tests/test_definition.py:../tests failed.
That much looks legit I think.
Tob be clear on the diff - Pants only derives requirements from poetry configuration - it does not derive python constraints. So I repeat your constraints in Pants configuration.
Ok, there are a few problems here from a random inspection of things: 1. Pants cannot yet handle dependency inference + multiple overlapping requirements files. Since you have, for example, astroid defined as a dependency in two seperate pyproject.toml that are 
poetry_requirements()
, it doesn't know which requirement to use (never mind that these reqs happend to be identical - Pants isn't peering that deep when doing dep inference). As such, as you discovered, you needed to add manual 
':astroid'
dependencies. 2. Pants doesn't correctly special case the 
python
dependency. If you 
$ PANTS_SHA=0182de9814c7694eed2b9407e3577684db26ea2e ./pants repl python_architecture_linter::
things go boom trying to resolve the 
python
distribution. 3. Pants doesn't correctly special case path dependencies. Presumably as a result, your 
python_architecture_linter_cli/pyproject.toml
has to lie to appease Pants which presumably breaks Poetry since 
python_architecture_linter_cli
depends on 
python_architecture_linter
which Pants can infer but I'm pretty sure Poetry cannot.
Issues coming up. 1 is known and I'll find that issue and link here - it's not a poetry-specific issue, but a larger problem we have in our sights. 2 and 3 are poetry-support-specific issues.
Ok, the 1st issue should fall out of tackling https://github.com/pantsbuild/pants/issues/11165.
For now you need to do what you did adding manual 
:astroid
dependencies in BUILD files.
Item 3 just got an issue over here: https://github.com/pantsbuild/pants/issues/12272
And I filed and issue for item 2 here: https://github.com/pantsbuild/pants/issues/12276
h
Also, I would add that it's more idiomatic in Pants to have a single set of requirements for the repo wherever you can (assuming you can do so without conflict).
So for example if that 
astroid
requirement was only defined once, there would be no ambiguity and Pants could infer the dependency automatically.
The paradigm shift is this: Without Pants, your requirements (whether poetry or requirements.txt) represent what actually goes into the metadata for your binary/distribution. So if you have multiple of the latter you need multiple of the former. But with Pants, the system itself figures out, based on dependencies (explicit or, preferably, inferred), what the true requirements are. So your poetry/pip requirements files now represent the universe of potential requirements that Pants may select from.
šŸ‘ 1
And therefore you don't need multiple requirements files, one per deployable binary/distribution.
You can still have multiple files if it's easier for you to manage, or you can have one big one, but conceptually they represent one big bucket of potential requirements.
The only time you must have multiple files is if you have conflicting versions.
šŸ‘ 1
And then Pants can no longer infer which version you mean, so you have to manually add those deps.
But in cases (such as, it seems, 
astroid
) where you don't have conflicts, you might as well list that req in just one place, and let Pants do the work for you.
Does that make sense?
e
The kicker here, and I think the thing we (Pants developers) don't understand is how Poetry users use Poetry (and not Pants) day to day in a multi-project repo.
Item 3 (https://github.com/pantsbuild/pants/issues/12272) points to that same conceptual grasp problem we have.
n
Thanks for the response, I have to admit It will take me some time to read through it and try the suggestions. I got the SHA tip from my former post, it seems there was a recent bugfix I want to explain the dependency "duplication" a bit here. In this style of monorepo I want to maintain a collection of many small releasable packages, none of them are monolithich in design. The design objective here is strictly to decouple and permit package users the most permissive range of possible installs possible. It means if I have packages A, B and C, in the structure of: A depends on C B depends on C It is only a coincidence of the package naming that these are called A, B, C. It's actually not relevant (but it makes it easier for us humans to think about it). If 
A
uses features introduced in 
C v1.2.0
and 
B
uses features introduced 
C v1.3.0
I should not force 
A
in my monorepo to use 
C v1.3.0
just by chance that another "unrelated" package might want it. The direct dependency is on a feature set of v1.2.0 and not v1.3.0. I've been burned a lot in the past when I would do this kind of "everyone gets the latest" moves and release hundreds of modules that depend on the "latest". My customers would write me long emails asking why I forced them into superficial upgrades for no real reason, so now I don't opt for "this is an easy model for me, the author of the repo to think about" but I opt for "this is the set of software releases that permits the most users the easiest installations because the constraints reflect what is actually used, not what I wrote in the file". I can't do it if I keep everything in the monorepo on one level or at "latest since I last ran update" The third diagram in a short post I wrote here explains a bit of the mental model I use when thinking about releasing packages: https://statagroup.com/articles/intuitive-guide-to-dependencies
e
So, I want to emphasize a split. There are the users of your published packages A, B and C and then there are the internal uses; ie: A uses C and B uses C. The internal uses are necessarily coupled by the current state of the git repo. That is, if you want your repo as a whole to be able to test green at a given sha. If you accept my split description, then for the internal use of C by B, if both B and C use astroid, they need to use compatible versions since the Python ecosystem has no "shading" (JVM term) or mechanism. In other words, for B and C to both work at a given sha, they will need to use a compatible version of astroid and they might as well share the astroid version constraint then. To not do so only invites errors, to do so just expresses the truth of the situation. Does that make sense?
Maybe more useful to discuss is how you actually use or intend to use Poetry in this project and how you intend to use Pants. Do you intend using one for some tasks and the other for other tasks? Or is the intention to migrate existing Poetry projects to Pants and only use Poetry for maintaining the poetry.lock file?
To make that question more clear: For example, you can use Poetry to maintain venvs for your projects. You can also do so with Pants (repl to get a venv repl, run to run a venv binary, test to run tests in a venv, etc.)
n
I think your explanation of internal/external makes sense, it's just based on the runtime env. Ultimately I don't want to deploy with version constraints that are based on what a global runtime in my mono-repo expects, but I understand the issue with needing everything on latest HEAD sharing that global state. I'm mostly interested in the releases and their version constraints, so if I can replace poetry entirely with pants I will but I'm too new to the feature set in pants to determine if it is possible or not. A lot of this depends on well-behaving semver releases by package maintainers, so if B and C need A ^1.2.0 and ^1.4.0, in any given project I might get tag 1.6.1 or something, so latest possible constraints isn't even testable on the latest sha, that's just the minimum constraint possible (eg, if i abandon the project and someone releases a new version, it should still work). At the moment I use poetry as a crutch, because I don't know how to make other things work as I would expect it.
h
Got it. So you do want explicit requirements for each deployable, so you can control those version constraints.
šŸ’Æ 1
e
Checking I'm on the same page. The concern then is just the internal version constraints of local project on local project in published form? I.E.: When B is published (we run the package 
package
goal on it), Brian wants to control the version range of exactly one of B's dependencies, the local project dependency on A. All the third-party dependencies just transfer over to the published package as written in pyproject.toml / inferred by Pants. This is in contrast to what Pants currently does, which is to pin B's dependency on A to the single version A has written down in its 
python_distribution
BUILD file target at the time of running the 
package
goal.
h
Hey @nice-france-74763, checking how things are going and if there's anything we can help with?
n
Hey Eric, all good, I have a stable deploy pipeline for my packages right now. I think if I run into practical issues I'll check in here for more information, but for now every is working. Thanks for the support!
šŸ‘Œ 1
šŸ™Œ 1
4 Views
Open in Slack
PreviousNext
Powered by