One thing I'm evaluating is related to developer ergonomics. There's a few things that we've done in some other projects, as well as some tooling thoughts. Going to make a few posts here to allow for proper threading, sorry for any spammyness 🙂 🧵 First is that we're used to having the ability to specify different dependencies in development vs. our release environments. Specifically, this is useful for dependencies such as

psycopg2

which have significant burdens for local development (especially on some OSes) while realizing possible wins when the code is deployed. What I've done in my poetry projects is to have

pg8000

setup as a dev dependency and then have a specific extras section for a production only dependency after flagging it as optional.

[tool.poetry.dependencies]
psycopg2 = { version = "~2.9.3", optional = true }

[tool.poetry.dev-dependencies]
pg8000 = "~1.29.1"

[tool.poetry.extras]
production-only = ["psycopg2"]

This has reduced the burden on our developers while maintaining api compatibility and the code is largely unaware of the distinction. (I only have one place where I have to do multiple imports and that's for a try/except block). From what I can see there's not an obvious way to do a similar dichotomy when using pants and this is a considerable regression that I'm having to weigh. Is there anything I'm missing? Could a plugin enable such a pattern?

🧵 I have a similar dev dependency pain point that I'd like to find a better solution for. While working out a logging setup, I'm settling on using

structlog

to help generate structured json based logs. However, during development those logs are effectively useless. In general this is fine because I can vary the configuration to output easy to parse logs when you're in running locally. However, to get some extra high quality formatting in development we need to have an extra library available:

rich

To have

pants run

work and load up rich into the dev time pex file, I've had to add this to one fo the build files

python_sources(
    dependencies=['//:deps#rich']
)

This now works fine when running locally. However, that library has -zero- reason to exist in our hosted environments, but it will continue to be part of the pex file given the above configuration. Is there a lever to manage this that doesn't lead to significant overhead and duplication?

🧵I'm also wondering if there's any plans for IDE integrations. Right now I see a seemingly abandoned intellij plugin for pants v1, and nothing else really. Again, with the focus on a good developer experience. I'm just concerned that this will be a bit of a stumbling block given the several partial and manual steps needed to setup an environment.

Hi. I've got a qustion regarding accessing binaries installed in venv from outside pex. We're distributing docker images, built using pants, with .pexes inside, and deploying them on k8s cluster. Recently we wanted to start using celery, but we have to figure out a way to do liveliness checks in kubernetes. In order to do that, we have to invoke a celery command, namely

celery -A path.to.module inspect ping

. The problem here is that

celery

is a part of the .pex file, and it is unclear how to find it and execute it. One way to just find the binary would be to package the pexes in

execution_mode="venv"

, set

PEX_ROOT

to a known location and then to use

source $PEX_ROOT/venvs/s/*/venv/bin/activate

This way I can reach the binary. But the problem remains, because I can't reach the module I need (it is in user code). I suppose I'd also need to set up PYTHONPATH. Another way to reach the binary would be to just install

celery

itself separately in the system python distribution, but I would still need to reach the user code binary. Both the solutions seem like a lot of hackery and I'm not even sure if it's possible to pull any of them off. Is there an easier way to achieve this - accessing venv binaries without running the pex file itself? Or maybe you can override the entrypoint when running venv. Or maybe there is a better way to package docker images? Any help would be much appreciated

Hi everyone. I have a problem with my CD workflow on GitHub actions, everytime in the

./pants package

step it failed with the following error

Failed to digest inputs: "Error storing Digest { hash: Fingerprint<a682886bff95b6d0ffae4e1d3efe75bd9dd51cf54aa42a81247a2c7d1ceee4ca>, size_bytes: 95935573 }: Input/output error

. I don't know what to do. I tried to increase memory and storage in my runner, but that doesn't seem to be the problem. I also tried to play with global options (

local_store_files_max_size_bytes

and

local_store_directories_max_size_bytes

) but that doesn't change anything either. The

package

step builds PEX files and copy them on Docker images, this error happened during Docker builds. That's weird because it works on my machine.

hey folks. Is there a syntax to address all targets recursively excluding the targets in the current directory? I can exclude an arbitrary directory with

--filter-address-regex

but struggling to find the right syntax for something like

./pants dependencies :: --exclude-targets=":"

I want to embed some version metadata into a .pex in such a way that it can be accessed from within code packaged by the pex. Is there a recommended way to do that?

Small docs site wish: When I select a version of the docs explicitly, remember that for at least some time window so I don't have to reselect it every time I go back.

I've had a bit of an overall challenge when porting over our

uvicorn

usage over to a pex/pants friendly world and would like some thoughts. I'm worried that our usage of

dramatiq

are going to hit even more sever issues when I get that far. Previous to trying out pants we evoke

uvicorn

via the cli. Evocation looking something like

uvicorn --port 3001 --no-access-log --proxy-headers swizzler:app

I tried for a while to get it so the

pex_binary

target would be doing this basic thing for me. However I couldn't find anyway to both execute a cli as my entry point -and- give it options. I've gotten into a decent spot by instantiating unicorn programmatically, but I am concerned that for something like

dramatiq

I will need to be able to use the CLI with arguments. Is there a reasonable way to achieve that (or does anyone know if I can start

dramatiq

programmatically?)

However I couldn't find anyway to both execute a cli as my entry point -and- give it options.

Can you elaborate? Is

./pants run path/to/dir:target -- --port 3001 ...

not working?

well

uvicorn

and

dramatiq

are executables provided by the packages of the same name.

AH yeah. pretty sure that's doable in PEX but I'm not the PEX wizard 🧙‍♂️

Another example

dramatiq swizzler.tasks:redis_broker -p 1 -t 8

Josh has it right, but, in more detail:

./pants run path/to/dir:target -- --port 3001 --no-access-log --proxy-headers swizzler:app

Where path/to/dir/BUILD has:

pex_binary(
    name="target",
    script="uvicorn",
    ...
)

What would be nice is to be able to seal in the

swizzler:app

argument to the PEX binary such that it always ran your

swizzler:app

via

uvicorn

without you having to type that. That feature idea is tracked here but it's had no one yet motivated to actually add the feature: https://github.com/pantsbuild/pex/issues/987

@enough-analyst-54434 is there a way to have the PEX contain multiple packages with scripts, and allow for executing one of them via the cmd-line? I think thats the intent here. One PEX, but

./pants run <pex_binary> -- script1 --arg1

as well as

./pants run <pex_binary> -- script2 --arg1

If

PEX_SCRIPT=script_name ...

or

PEX_MODULE=entry:point ...

aren't appealing, there is always https://pypi.org/project/conscript/

I think the intent is actually exactly the pex issue I linked to.

Maintainers

John.Sirois

I know that guy!

I did try with the script flag, but didn't see anyway to embed the arguments into it or the like, which is what I was kinda looking for.

It didn't even occur to me that you could then pass in arguments like that as part of the pants run command

Right, exactly. Maybe you'll be the person motivated enough to contribute https://github.com/pantsbuild/pex/issues/987!

for production environments it would be fine to do it that way, but for dev it'd be pretty annoying to have to do it that way 😕

s/fine & annoying/dangerous & error prone/! in my opinion. As far as I know Pants users currently write a thin shim script to bake things in that should be baked in as a result.

Not awesome.

For any who listen to the Talk Python To Me podcast, I’ll be recording an episode with them on Thursday, talking about Pants of course, and it’ll be livestreamed on YouTube: https://www.youtube.com/c/Talkpythontomepodcast

TIL: MacBooks are slower when not plugged in!

I have a problem with ModuleNotFoundError even though I use module mapping on django app. Can anybody help out? I want to add package as app when loading settings.py. Currently, jazzmin, crispy_form are not working. BUILD file

python_requirements(
    name="requirements",
    module_mapping={
        "django-ninja": ["ninja"],
        "python-dotenv": ["dotenv"],
        "django_log_request_id": ["log_request_id"],
        "django-jazzmin": ["jazzmin"],
        "django_admin_cursor_paginator": ["admin_cursor_paginator"],
        "django_crispy_forms": ["crispy_forms"],
        "django-celery-beat": ["django_celery_beat"],
    },
)

settings.py

DJANGO_APPS = [
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.sites",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    # "django.contrib.humanize", # Handy template tags
    # "grappelli",  # this app must be before django admin app
    "jazzmin",
    "django.contrib.admin",
    "admin_cursor_paginator",
]
THIRD_PARTY_APPS = [
    "crispy_forms",
    "rest_framework",
    "django_celery_beat",
    "simple_history",
    "django_mysql",
    "corsheaders",
    "django_filters",
    "drf_yasg",
    "django_slack",
    "django_user_agents",
    "django_migration_linter",
    "taggit",
    "taggit_autosuggest",
]

error trace

17:12:40.71 [WARN] Failed to generate JUnit XML data for payhere/customers/tests/api/test_customer.py:tests.
17:12:40.71 [ERROR] Completed: Run Pytest - payhere/customers/tests/api/test_customer.py:tests failed (exit code 1).
Traceback (most recent call last):
  File "/private/var/folders/jd/znnczw0j2n9_hfy_9kj5_5sw0000gn/T/pants-sandbox-fqwLut/.cache/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/pex", line 245, in <module>
    sys.exit(func())
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/_pytest/config/__init__.py", line 185, in console_main
    code = main()
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/_pytest/config/__init__.py", line 143, in main
    config = _prepareconfig(args, plugins)
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/_pytest/config/__init__.py", line 318, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/_pytest/helpconfig.py", line 100, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1003, in pytest_cmdline_parse
    self.parse(args)
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1283, in parse
    self._preparse(args, addopts=addopts)
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1191, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/pytest_django/plugin.py", line 353, in pytest_load_initial_conftests
    _setup_django()
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/pytest_django/plugin.py", line 236, in _setup_django
    django.setup()
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/django/apps/registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
  File "/Users/szto/.cache/pants/named_caches/pex_root/venvs/6d4a9de78703d30bcbe4e7c4b80df143e68dc32c/91bd7b41fccc3fdcc0bbea862ae7f7b017df774c/lib/python3.9/site-packages/django/apps/config.py", line 224, in create
    import_module(entry)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'jazzmin'

Hello, I have built some PEX binaries that serve as data processing tasks, and they need some environment variables to run (they will be deployed in Amazon ECS where these environment variables are already set up). I would like to be able to test them locally by running

./pants run

but I don't find a way to pass environment variables in the subprocess. Does anyone have a recommendation for that?

Hi. I wonder if there's a way for pants to list all third-party imports that aren't declared in requirements (in a python project). I'd like to fix such errors, and it would be helpful to be able to automate the searching process

We depend on such dependencies in two ways: as transitive dependencies of other libraries, and also as our own imports. Even though it works right now, I'd like to explicitly list the libraries in requirements