I ran into the pex > docker platform challenge today. I am working from a Mac but distributing via Docker. I was able to find answers by searching this Slack workspace and succeeded in configuring working Docker images derived from pex packaged python code. I'd love to produce a universal pex file that can be run on Mac x86, Mac Silicon or Linux hosted via Docker. Is that possible? My BUILD looks like so:

# Works inside Docker
pex_binary(
    name="eeg_processor",
    entry_point="app.py",
    complete_platforms = [
        'lib_third_party/platforms:linux-py38',
    ],
)

# Does not work: A distribution for confluent-kafka could not be resolved
pex_binary(
    name="eeg_processor_mac",
    entry_point="app.py",
    complete_platforms = [
        'lib_third_party/platforms:macos_x86_64',
    ],
)

pex_binary(
    name="eeg_processor_default",
    entry_point="app.py",
)

docker_image(
    name="neuroedge_eeg_processor",
    image_tags=[
        "neuroedge_eeg_processor", 
        # TODO: is this tag useful? "<http://optios.jfrog.io/default-docker-local/neuroedge_eeg_processor:0.1.0|optios.jfrog.io/default-docker-local/neuroedge_eeg_processor:0.1.0>"
    ],
)

platform _macos_x86_64_ was produced using

pex3 interpreter inspect --markers --tags

It does not produce a working pex file. platform linux-py38 was produced using

docker run --rm -it python:3.8 sh -c 'python -mvenv venv && venv/bin/pip -q install pex && venv/bin/pex3 interpreter inspect --markers --tags -i2'

and works inside a simple linux base Docker container. linux-py38 works correctly when run from inside a Docker container.

./pants run realtime/eeg_processor:neuroedge_eeg_processor

./pants run realtime/eeg_processor:eeg_processor_mac

(using macos_x86_64) fails with: A distribution for confluent-kafka could not be resolved...

./pants run realtime/eeg_processor:eeg_processor_default

works without issue. Is there some way to produce a multi-platform pex distributable that can be executed from both mac and linux? Example project attached

PSA, dark mode now enabled for Toolchain docs 🌚 Sounds like it will reflect your system defaults

Pants is the headliner of the most recent PycoderWeekly! https://pycoders.com/issues/556

Stuck on a pytest issue in this repo: https://github.com/BruceEckel/RethinkingObjects The

pytest.ini

in the root directory contains:

[pytest]
minversion = 7.0.1
python_files = *.py
addopts = -rA

The

python_files = *.py

should make pytest look in all files for tests, according to https://docs.pytest.org/en/7.1.x/example/pythoncollection.html#customizing-test-collection, but it doesn't seem to be; it only seems to look in `foo_test.py`:

bruce@groot:~/RethinkingObjects$ ./pants test ::
14:40:42.23 [ERROR] Completed: Run Pytest - test_experiments/foo_test.py:tests failed (exit code 5).
============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.0.1, pluggy-1.0.0
rootdir: /tmp/pants-sandbox-pDDAT5, configfile: pytest.ini
plugins: forked-1.4.0, cov-3.0.0, xdist-2.5.0
collected 0 items

- generated xml file: /tmp/pants-sandbox-pDDAT5/test_experiments.foo_test.py.tests.xml -
============================ no tests ran in 0.02s =============================

(Also it doesn't seem like

addopts = -rA

is having any effect, either).

Hey folks, quick one, is it possible to run

package

on all

docker_image

targets in my repo?

Hi there 👋 Does anyone have an script to visualize a dependency graph with pants? Mentioning this because I've just found this webpage, and on the Dependency Graph Visualization section says that we can do with other visualization tools... So just wondering how people are doing it, and using it to do that.

idk what happened on the last months, but the docs are incredibly fast now. Thanks for this 👍

Hey Team, we randomly had this occur in CI and unsure what the issue is. This happens when running

./pants peek :: --peek-output-file=/dev/null

with cache. When we regenerate the cache this was fixed. Is there any reason this would happen or do we have a way to prevent this (obviously remote caching but we're not there yet)

[2022-12-28T14:18:45Z] Exception: Could not identify a process to backtrack to for: Missing digest: Couldn't find file contents for "app/something/__init__.py": Was not present in the local store: Digest { hash: Fingerprint<5c836f5086164c2048428ca39322f7d6ae372bedf67de8f0788c61f5ee365499>, size_bytes: 35 }

I'm creating a monorepo, and they use poetry. If I remove one of the internal packages from the

pyproject.toml

dependencies, I don't see that package being added to the wheel metadata. Example: I have both packages:

data-types

and `entities`; and

entities

depends on

data-types

, if I remove

data-types

from the

pyprojec.toml

, the latter is not inferred anyway. I'm using the following macro:

def poetry_distribution(name, package, **kwargs):
    resources(name="package_data", sources=["pyproject.toml"])

    python_distribution(
        name="dist",
        dependencies=[":package_data", f"src/python/{name}/{package}"],
        provides=python_artifact(name=name),
        generate_setup=False,
    )

Does anyone know what I'm doing wrong?

Also, is there any public example with poetry? 🤔

I recently found out about https://cuelang.org/ and like it a lot. Planning on adding support for it in Pants. Now, my idea as a first primary use for cue is to make github actions more manageable. After toying with it locally (including writing a github action cue package) I found out it already exists, example use here: https://github.com/cue-lang/setup-cue/blob/main/workflows.cue implementation here: https://github.com/cue-lang/setup-cue/blob/main/cue.mod/pkg/json.schemastore.org/github/github-workflow.cue

Hello, We have a pants setup for python to build pex executable, that is copied into a Dockerfile. I am running into issues were one of the os level dependency is missing. I tried installing that dependency on the CI instance where the pex file is built. But that doesn't help. How do you add os level dependencies for pex?

Hey folks, I've got an Azure-grade problem. We've got an Azure Function App which we package into a docker container. We currently use poetry-based lockfiles so we can pip install dependencies in the docker container. We'd like to use PEX instead since it has better integration with pants, like dependency inference and future support. The problem is that Azure Functions don't really have an entry point to designate as a pex_binary target. Does anyone have ideas? I tried installing the dependencies with PEX directly (

pex -r requirements.txt --lock requirements.lock

), but it didn't like the header pants put on that file.

Hi folks! Toolchain is ringing in the new year with a new website! Check out https://www.toolchain.com/

Does Pants pay attention to the

conftest.py

file, or any part of it? Thanks...

📣 If you're interested in public speaking, this persistent thread is chock full of opportunities at tech conferences and meetups. If your talk topic is relevant for Pants community members (e.g. things like tooling, scaling, devops, data science, developer productivity, developer experience, open source management, etc), you are also welcome to DM me to get help from community members who are experienced speakers, conference organizers, and program committee members.

The first line of: https://www.pantsbuild.org/docs/reference-python_tests is "Generate a

python_test

target for each file in the

sources

field." Should that be

sources

"argument", rather than "field"? I've seen this in a number of places and it had me hunting for a "sources field" until I guessed that it probably meant the function argument.

This is probably a feature request. Looking at

[python-infer].string_imports

and `[python-infer].assets`: https://www.pantsbuild.org/docs/reference-python-infer#string_imports These flags work one way, but I would like to invert the signals given to the python dep inference system. So, the flag enables checking all springs for possible imports or assets, and then you use code comments or a

!

(negative) dependency to exclude false positives. My code base has a lot of strings that look similar to an import but they aren't. So, turning string import inference on globally sounds very problematic to me. So, instead of excluding false positives, could I use comments to say

# pants: string_imports on

and

# pants: string_imports off

? In other words, I want to enable string import inference selectively with code comments. I think something similar would be great for assets too.

I have an issue with pants not finding all installed pyenv versions.

pants.toml

...
[GLOBAL]
backend_packages = [
    "pants.backend.python",
    "pants.backend.python.mixed_interpreter_constraints",
    "pants.backend.project_info",
]

[python-bootstrap]
search_path = ["<PYENV>", "<PATH>"]

[python]
interpreter_constraints = ["CPython==3.10.2"]

.python-version

3.9.13  # This is only set because pants doesn't work with 3.10.*

pyenv versions

system
3.9.13 (set by .python-version)
3.10.2
3.10.6

now if I run anything with pants, it only find my system interpreter and 3.9.13, but the other two not. Basically it doesn't matter what I set under search_path, I always get the same result:

Examined the following interpreters:
1.)                                    /usr/bin/python3.10 CPython==3.10.8
2.)                 ~/.pyenv/versions/3.9.13/bin/python3.9 CPython==3.9.13

Any ideas someone?

I take everything back. If I install a global 3.9 version and use it with pants, then everything works.

Huh

<PYENV>

is supposed to see all the pyenv-installed interpreters in the versions dir, regardless of which ones are set as the current global

I used a local pyenv version already for pants.

This seemed to mess with pyenv somehow.

Because I did not have a 3.9 version installed on my computer, so I installed that version already through pyenv

I can see how that would mess with Pants itself, but once Pants is running I would have thought your config was correct

Is it normal that for pants it can take up to 15 seconds to restart a django server?

I think it is rather because it takes too long to build - If I run it without pants the server takes like 0.1ms to restart after a file change, if pants does it it take 15 seconds to rebuild and restart.

It feels like it rebuilds it completely everytime, could there be something wrong with caching?