• We want to keep using Python virtual environment and Python editable installs for the development of a project

• We want to use Poetry

• We want to keep using VS Code as our primary IDE

We also need help in setting up the “commons” library such that it is available as an dependency for all our projects and is available as part of the project’s virtual environment. It should also be included in the bundled package for each project. How can this be achieved?

Currently our dependencies within a project have a high overlap with each other. What tradeoffs do we need to evaluate for a per project dependency management vs. global dependency management on the monorepo level?

We currently see that we will have to bump project versions manually. Is there an already included feature to auto update versions? And what is the recommendation when wanting to have semantic versioning? Because the current approach with conventional commits is not straightforward in the case of a monorepo, as semantic versioning libraries today are not monorepo aware…

• We “multi version” using conventional commits, semver - of all “publishable” packages.

◦ this is complex - but the principal is

▪︎ everything that needs to publish with a version, add to their BUILD, version_and_changelog() <-- custom

▪︎ this behind the scenes operates via “pants run //tgt/address:bump_version” - and then test/package can run etc

• versioning is handled by convention-commits, convco *( a rust CLI which has support for monorepos)

add to their BUILD

python_distribution.provides.python_artifact.version

version_and_changelog()

Pants does this naturally - that is

• either - the user is using “in repo” so it’s not versioned, but just available in a venv, from an install

pants package data_lake/data_asset_1

pants export

pants package ::

Had a quick read of the convco documentation and couldn’t figure out how monorepo versioning is supported. Probably need to look again.

./apps/arc/VERSION
  ./apps/devolver/VERSION
  ./apps/log_manager/VERSION
  ./lib/finder_2000/VERSION
  ./lib/knohh/VERSION
  ./tools/containers/cicd-general/VERSION
  ./tools/python/acme_cicd/VERSION

COPY /*.whl /tmp
RUN pip install /tmp/*.whl && rm /tmp/*.whl

apps/arc/v4.5.1, lib/knohh/v0.2.33

git tag -l

apps/arc

--prefix

convco version --prefix apps/arc/v  ...

--paths

convco version --prefix apps/arc/v --paths apps/arc

It would be great if you can elaborate a bit more about your setup here. I am guessing you have some custom scripts to glue things together…

When you say add to their BUILD, do you mean updating python_distribution.provides.python_artifact.version ?

When you say version_and_changelog(), is this something Pants offers?

def version_and_changelog():
    # this runs convco (inside a python script), and determines the next version of the package
    # the PWD of this command is the "directory" of the BUILD file.
    # the PANTS_BUILDROOT_OVERRIDE is the root of the workspace.
    # ${PWD#$PANTS_BUILDROOT_OVERRIDE/} will equal the relative path to the versionable thing

    # python $PANTS_BUILDROOT_OVERRIDE/tools/python/acme_cicd/src/acme_cicd/versioning_ops.py version lib/jlab VERSION

    #
    run_shell_command(
        name="bump_version",
        #
        command="python $PANTS_BUILDROOT_OVERRIDE/tools/python/acme_cicd/src/acme_cicd/versioning_ops.py version ${PWD#$PANTS_BUILDROOT_OVERRIDE/} VERSION",
    )

    run_shell_command(
        name="gen_changelog",
        command="python $PANTS_BUILDROOT_OVERRIDE/tools/python/acme_cicd/src/acme_cicd/versioning_ops.py changelog ${PWD#$PANTS_BUILDROOT_OVERRIDE/} CHANGELOG.md",
    )

either - the user is using “in repo” so it’s not versioned, but just available in a venv, from an install

The aim here is to have a shared library package that is published to our internal PyPi.

Additionally, when running pants package data_lake/data_asset_1, I want the shared library to also get packaged along with data_asset_1 source.

This is an optimisation I am wanting to do to reduce the effort of using a python package with as artifacts submitted to an Apache Spark cluster. Rather than providing data_asset_1.tar.gz and commons.tar.gz, I am wanting to provide only data_asset_1.tar.gz with the latest commons already part of this tar.

6.1.2-dev.cafe67677

My struggle currently is getting the pants configuration right for this. 😞

I was not able to make pants include the shared library to all project venvs that get created as part pants export and to all project packages that get created as part of pants package ::