#12421 Plugins: add `debian_package` target to create Debian packages from PEX files Issue created by alexey-tereshenkov-oxb Packaging PEX files as Debian packages PEX files: overview Pants currently provides support for packaging PEX files via the `pex_binary` target. This makes it possible to produce a file that can be later run on a machine that has a compatible version of Python interpreter:

$ ./example-pex.pex --args

Using PEX files, however, is associated with certain difficulties: • PEX files do contain package metadata, however, it is somewhat limited (e.g., PEX distributions do not have a notion of the version). Looking at the metadata involves either unpacking the PEX archive file or reading the metadata file straight out of the archive file:

$ tar xfO example-binary.pex PEX-INFO | jq
{
  "always_write_cache": false,
  "build_properties": {
    "class": "CPython",
    "pex_version": "2.1.42",
    "platform": "macosx_10_15_x86_64",
    "version": [
      3,
      6,
      9
    ]
  },
  "code_hash": "add39ea4f9ec7fe69c01311ffd8658581f674c25",
  "distributions": {
    "Jinja2-3.0.1-py3-none-any.whl": "5fed66ed790bb0d771ef5b6cc73c544315739ded",
    ...

• there are no standard tools for managing PEX distributions: in contrast to wheels, PEX files cannot be hosted on a public or private PyPI server and installed with

pip

or other package manager. If one would like to distribute multiple PEX files for individual project's artifacts, writing a fairly complex custom tool would be necessary (e.g., the tool would need to differentiate the versions of PEX distributions and for what runtime platforms they have been built, manage downloading, updating to a later version, and removing PEX distributions). • multiplatform PEX distributions require special handling: it may be required to find what PEX binary file should be obtained based on the operating system and architecture of the runtime environment user would like to use the PEX file on. • multiple identical PEX distributions may be built for different Python versions which requires special handling. To find out the path to a Python interpreter that will be searched when running the PEX file, the shebang will need to be inspected by looking at the head of the archive file:

$ head -1 example-pex.pex                    
#!/usr/bin/env python3.8

• making PEX files available on the

$PATH

on a target machine may be required (which would involve either placing the PEX file itself to a discoverable location or creating a symlink). Some difficulties mentioned can partially be solved by incorporating the metadata in the PEX file name by trying to follow Compatibility Tags for Built Distributions conventions: • `example-pex-v1.2.3-linux_x86_64-cp-38-cp38.pex`:

example-pex

executable of version

1.2.3

built to be run on a Linux machine of x86 architecture with assumption that it will have a system Python 3.8 interpreter available. • `example-pex-v1.2.3-macosx_10.15_x86_64-cp-36-cp36m.pex`:

example-pex

executable of version

1.2.3

built to be run on a MacOS machine of x86 architecture with assumption that it will have a system Python 3.6 interpreter available. Debian packaging: overview The idea to include Python code into a Debian package is not new, and a few tools have been developed for this such as dh-virtualenv. A tool like this can be used to take advantage of the Debian packaging and distribution management practices and at the same time be able to effortlessly distribute self-contained PEX files. Debian package for PEX file with an entry point Once having a PEX file, it can be put into a Debian package. During installation of the package, the PEX executable is placed into a user-defined installation location on a Linux machine (e.g., into the

/opt

directory which is used for installing packages that are not part of the OS distribution). Thereafter, a symlink is placed to make the PEX file available with a custom executable name. Here is an example of the directory that is being packaged into a Debian package archive file `example-pex-bionic-1.0.0_amd64.deb`:

├── example-pex
│   ├── DEBIAN
│   │   └── control
│   ├── opt
│   │   └── company
│   │       └── example
│   │           └── example-linux-cp3.6.pex
│   └── usr
│       └── bin
│           └── do-work -> /opt/company/example/example-linux-cp3.6.pex

After installing this Debian package, the

do-work

command becomes accessible to the terminal users. Debian packaging: benefits Packaging the PEX file as a Debian package has multiple advantages over distributing raw PEX files: • software exists for managing Debian packages on Linux such as

dpkg

and

apt

which can be used to search for, install, update, and uninstall Debian packages. • users can add an arbitrary amount of metadata to be baked into the Debian package and later discovered using the

dpkg-deb

and

dpkg-query

programs. • Debian packages have a native versioning notation which is understood by

apt

and other system tools. • Debian packages can be stored within the binary repository managers such as Artifactory or using

Personal Package Archive

(PPA) to distribute software and updates directly to Linux users. • Debian packages can contain symlinks that will be placed at a desired location on the system where package is being installed. • Debian packages for the relevant operating system versions can be discovered through using the `sources lists` where packages for different operating system versions (and different platform architecture such as

amd64

or

arm64

) are stored in separate pools. From the user perspective, given the correct system files configuration, the package for the user's system will be found and installed. • User does not need to be concerned with the target Python interpreter of the Debian package (with PEX file inside) because each Ubuntu distribution guarantees to have a Python interpreter of a certain version installed (e.g., Ubuntu 18.04 comes with Python 3.6.* and Ubuntu 20.04 comes with Python 3.8.*). Debian packaging with Pants: plugin Given an existing

pex_binary

target:

pex_binary(
    name="example-pex",
    entry_point="main.py",
)

A new target available via a plugin can be defined with

dependencies

containing one or more

pex_binary

targets (assuming each PEX file will have a corresponding symlink):

debian_package(
    name="example",
    description="Example Debian package.",
    dependencies=["example-pex"],
    control="project/example/debian-support/DEBIAN/control",
    symlinks="project/example/debian-support/symlinks",
    install_prefix="opt/company/example-pex",
    distribution=["bionic", "focal"],
    tags=["debian"],   
)

To cut on the boilerplate, the new target should support macros to declare targets for multiple system distributions and platform architectures. When running the

./pants package

goal, the dependencies of the

debian_package

will need to be created first and once they exist, Debian packages are produced for each

debian_package

target. With the help of tags, only certain Debian packages may be packaged, if desired:

$ ./pants filter --filter-target-type=debian_package --filter-tag-regex="+some-tag" --filter-sep=" " :: | xargs ./pants package

Debian packages will need to be produced on a Debian compatible Linux operating system. Creating a Debian package from the new target would require executing a few commands (either via Python code or by starting a subprocess). The workflow is outlined below: • the PEX file is placed in a temporary directory (that Pants provides) under the

name/install_prefix

location (that is created first… pantsbuild/pants

#12421 Plugins: add `debian_package` target to create Debian packages from PEX files Issue closed as completed by cognifloyd pantsbuild/pants