gentle-painting-24549
11/10/2022, 7:11 PMpackage-1
depends on `package-2`:
monorepo-example/
βββ 3rdparty
β βββ python
β βββ default.lock
βββ README.md
βββ packages
β βββ package-1
β β βββ BUILD
β β βββ package_1
β β β βββ __init__.py
β β βββ requirements.txt
β β βββ setup.py
β βββ package-2
β βββ BUILD
β βββ package_2
β β βββ __init__.py
β βββ requirements.txt
β βββ setup.py
βββ pants
βββ pants.toml
Before adopting pants weβve used pip-compile
to assemble our requirements.txt
files so package-2/requirements.txt
might look like this and pull from our private PyPi repository:
pandas==1.5.0
boto3==1.22.0
package-1>=1,<2
Itβs worth noting that all of our packages are being migrated from individual repos and weβd like to keep that directory structure the same with a single BUILD
file per project and using our own setup.py
files. Hereβs what package-2/BUILD
looks like right now:
python_requirements(
name="reqs",
)
python_sources(
name="lib",
dependencies=[
":reqs",
],
sources=[
"package_2/**/*.py",
"package_2/*.py",
],
)
python_distribution(
name="dist",
dependencies=[
":lib",
],
provides=setup_py(
name="package-2",
),
generate_setup=False,
)
Will pants determine that package-2
depends on package-1
based on directory structure, or do I need to add a packages/package-1:lib
dependency to packages/package-2:lib
? How can I keep package-1
out of my third party dependency lock files and have those imports resolve to my other package instead?happy-kitchen-89482
11/10/2022, 8:24 PMpackage-1
imports from package-2
, what does the import statement look like?gentle-painting-24549
11/10/2022, 8:26 PM[source]
root_patterns = [
"/packages/package-1/",
"/packages/package-2/",
]
from package_1 import foo
happy-kitchen-89482
11/10/2022, 8:28 PMgentle-painting-24549
11/10/2022, 8:32 PMhappy-kitchen-89482
11/10/2022, 8:52 PMgentle-painting-24549
11/10/2022, 8:54 PMpackages
is ending up to be a poor name for the top level directory but maybe a better directory structure looks like this where the final deployable is the app-1
docker image (and app-1
depends on package-1
and package-2
)
monorepo-example/
βββ 3rdparty
β βββ python
β βββ default.lock
βββ README.md
βββ packages
β βββ app-1
β β βββ BUILD
β β βββ Dockerfile
β β βββ app_1
β β β βββ __init__.py
β β βββ requirements.txt
β β βββ setup.py
β βββ package-1
β β βββ BUILD
β β βββ package_1
β β β βββ __init__.py
β β βββ requirements.txt
β β βββ setup.py
β βββ package-2
β βββ BUILD
β βββ package_2
β β βββ __init__.py
β βββ requirements.txt
β βββ setup.py
βββ pants
βββ pants.toml
happy-kitchen-89482
11/10/2022, 10:20 PMsetup.py
data for you, including the requirements. So, e.g., Pants sees that package-1 depends on package-2 (because of an import), and knows that package-2 is published as package-2==1.2.3
, and so adds that to package-1's setup.py
as a requirement, along with the third-party requirements package-1 imports from.gentle-painting-24549
11/11/2022, 5:22 PMpackage-1
consumes package-2
by including its dist in the requirements.txt
We have some challenges with publishing and versioning - we currently handle this with via semantic-release generating release tags like package-2 v1.2.3
. But for us collocating the packages into the same repo helps to find breaking changes and update inter-dependencies easier.
This all makes sense. I guess I was thinking of some kind of happy middleground between the two approaches. For example: package-1
could consume package-2
via its dist but still know that when code changes in packages/package-2/package_2
that it needs to re-run unit tests for both.
In the second scenario you outlined where pants decides on dependencies and generates its own setup.py - will pants always use the latest version of the dependent distribution? Iβm thinking of a scenario where I want to make a breaking change in package-2
and slowly roll out that release to the affected dependencies also managed by pants (weβre slowly doing this with things like Python Major Version and PyTorch upgrades). Maybe that goes against the principle of the managed monorepo though and Iβm just thinking about it wrong π€·