Trying to produce a binary with

pants.backend.experimental.python.packaging.pyoxidizer

and hitting an issue with argsparse. I’ve been able to successfully run

pants package

, but when running the resulting binary I get an error that seems to stem from the use of

argsparse

❯ dist/src.python.mdevxp.code_ownership/code-ownership/aarch64-apple-darwin/debug/install/code-ownership
Traceback (most recent call last):
  File "runpy", line 196, in _run_module_as_main
  File "runpy", line 86, in _run_code
  File "mdevxp.code_ownership.code_ownership_mapper", line 258, in <module>
  File "mdevxp.code_ownership.code_ownership_mapper", line 50, in cli
  File "argparse", line 1717, in __init__
  File "posixpath", line 142, in basename
TypeError: expected str, bytes or os.PathLike object, not NoneType

The same entry point works with pex. If I replace my entrypoint to a hello_world.py script within the same pants package, it works, so it seems like something is not being configured correctly between pyoxidizer and what argsparse expected. Before I go down that rabbit hole, was wondering if anyone has already hit and solved this issue.

I've been trying to run Go goals for Go 1.20 on MacOS ARM64 arch and consistently get "could not import fmt" etc for all standard library imports, and I get the same result for Pants 2.13, 2.14, 2.15. Any idea what's going on here?

I'm running into this issue running the dependents goal. Except I set the max memory limit of the daemon to be 16 GB to be safe. It keeps downloading 3rd-party Go modules every single invocation. Then it seems to be running analysis on all external and internal packages, taking a total of 103 seconds to return 5 results. I did delete my

~/Library/Caches/*

to debug other issues but I would assume caches are regenerated cleanly after that.

I am developing pants-plugins using vscode and following Setting up an IDE. However, in python-default’s resolve, pants is not installed (“import pants” is not possible). What is the solution?

Hey everyone, I'm working on managing my Docker image tags in my repo. Right now, my tags look like ["service_name-{pants.hash}"], which is perfect for local runs. But when it comes to my CI/CD pipeline, I'd like to add the git commit message as an extra tag without losing the hash tag. So, it'd be like ["service_name-{pants.hash}", "service_name-{GIT_COMMIT}"]. Any tips on how to make this happen? Thanks!

I am looking to implement a

deploy-awslambda

goal that will execute

aws lambda update-function-code --function-name my-function --image-uri <image_ref>

by referencing its image ref after it is packaged + published in

docker_image

target. I have created a

deployment_awslambda

target and confirmed that it works up to the point where I get the DockerFieldSet from the DockerImageTarget in dependencies. After this I will need to run the DockerFieldSet.image_refs method, but it looks a bit complicated. helm deployment is what I am trying to reference, do I still have to implement these for this requirement? https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/helm/util_rules/post_renderer.py#L72-L155 Or is there an easier way to get the image_refs from the DockerFieldSet?

Hi all - I’m trying to see if there’s a way in pants to use a different interpreter constraint in pants based on the github runner python version matrix?

Hi all, I’m trying to configure the pants ignore in this way

[GLOBAL]
pants_version = "2.15.0"
pants_ignore.add = ["!.env"]

However, whenever I added that line, I see this error below

native_engine.PantsdClientException: The pantsd process was killed during the run.

If this was not intentionally done by you, Pants may have been killed by the operating system due to memory overconsumption (i.e. OOM-killed). If you keep seeing this error message, try the troubleshooting steps below. If none of those help, please consider filing a GitHub issue or reaching out on Slack so that we can investigate the possible memory overconsumption (<https://www.pantsbuild.org/docs/getting-help>).
 - Exit other applications, including applications running in the background.
 - Set the global option `--pantsd-max-memory-usage` to reduce Pantsd's memory consumption by retaining less in its in-memory cache (run `./pants help-advanced global`).
 - Disable pantsd with the global option `--no-pantsd` to avoid persisting memory across Pants runs, although you will miss out on additional caching.

i won’t see this error after removing the line. I’m curious is this a configuration error or something else

I observed this really strange error when running

./pants export

with

--no-pantsd

. Is this expected?

[BOOTSTRAP] [INFO] Sending command: /home/jtilahun/devel/monorepo/pants --print-stacktrace --no-export-symlink-python-virtualenv --no-pantsd export --resolve=python-default
[BOOTSTRAP] [WARNING] Received non-zero return code (-6) from /home/jtilahun/devel/monorepo/pants --print-stacktrace --no-export-symlink-python-virtualenv --no-pantsd export --resolve=python-default
[BOOTSTRAP] [WARNING] STDOUT: 
[BOOTSTRAP] [WARNING] STDERR: /github/home/.cargo/git/checkouts/lmdb-rs-369bfd26153a2575/6ae7a55/lmdb-sys/lmdb/libraries/liblmdb/m:2126: Assertion 'rc == 0' failed in mdb_page_dirty()

Hi, is it possible to somehow add pants exported virtualenv to pyenv virtualenv?

Hi there, how can make Pants run faster in the cloud? I'm using Pants to run my CI/CD pipeline. Using Google CloudBuild, I'm basically running pants

test

,

lint

,

check

, and

publish

commands as separate steps of a CloudBuild pipeline. When I run the same steps locally, they execute in a 3-5 seconds, thanks to some clever caching I assume. But on the cloud, it can easily take more than 10 minutes. I noticed from the logs that the most time is spent on Pants overhead rather then the actions of linting or testing themselves. For instance, pytest takes 15s to run the tests, but the entire

pants test

command needs 7 minutes to build the requirements.pex files etc. I'm wondering how can I make it faster: • Is it possible to run e.g. testing and linting in parallel? I cannot make it work, I'm getting the same error as the one discussed here • Is there anything I can do other than parallelization to accelerate individual pants commands, e.g. via caching?

@happy-kitchen-89482 ICYMI https://github.com/pantsbuild/pants/issues/18706

Yeah

Hi all, I'm having problems porting our repository to use pantsbuild, builds seems to be really slow and use a lot of disk space. This is a data science related repository, having dependencies on rather big python packages like tensorflow, pandas, etc. It also has around 170 test files containing unittests. Linting targets work fine, problems start when running tests. Current approach runs tests in docker container, as we need to a) run on linux b) have some system libraries present. In short, we do docker build & docker run pytest tests. The whole process - before migrating to pants - takes around 4 minutes: 3 minutes for building the image, 1 minute for running all tests using pytest-xdist plugin. With pants, the initial attempt was really slow,

pants test

took around 25 minutes, including pex builds for each test and test execution, and with parallelism set to 1. Because of 3rd party binary dependencies, tests were running within docker_environment, based on ubuntu image that contains all that we need. Lots of tests take a while to start, that is due to import of tensorflow taking some time, so I enabled

batch_compatibility_tag

for all tests and run them in a couple of batches - that makes using cache a bit pointless, but at least build times got to really decent 5 minutes. Next, I tried to use

run_against_entire_lockfile=true

setting (I am using lockfiles for my 3rd party requirements) to reuse single pex binary for whole codebase (as tensorflow installations were causing every build to be pretty slow). That reduced build times a bit more to around 3 minutes, with batch size of 64 tests (or are those tests files?)... However, after a while I run out of disk space and discovered that sandboxes are not removed properly. I use keep_sandboxes=never (default), but looks like when tests are run using docker_environment, sandboxes are not cleared. When I run tests on my host machine, they are properly removed from /tmp directory. Is that expected? As each sandbox has 0.5GB of tensorflow wheel installed, I'm running out of disk space pretty quickly when modyfing and rerunning tests... Ideally I'd like to avoid using batching as our aim was to run only changed tests, but looks like without reworking our dependencies it is not possible? Are there any more tricks I can use here to speed up pex builds?

not directly related to Pants itself, but still very important for large Pants repos with Python sources. I wonder if there's any tooling to identify unnecessary transitive imports in Python code? In practice,

A

depends on

B

because

B

provides

foo

, however, the

foo

is declared in

C

, so now

A

depends both on

B

and

C

(transitively).

A -> B -> C

is what we have but we could have

A -> C

and

B -> C

. 🧵

Pants dumps really sensitive info when errors happen

--index-url <https://nathanael>:<my whole token in plaintext>@artifactory.corp.astranis.space/artifactory/api/pypi/pypi-virtual/simple --retries 5 --timeout 15 exited with 1 and STDERR:

How would I prevent that?

what are people doing for versioning/tagging docker images built with pants? any favorite tools / methodologies you've incorporated? would be interested in the same question as it applied to python packages as well

Hello all, wondering if this is the right channel to ask the following: In evaluating Pants I'm wondering if it is feasible to have build artifacts that come from the scientific python stack (e.g. Numpy/Scipy). Some of our projects will wrap Fortran or C code. Is Pants going to be friend or foe here? ** For Fortran, we'll be using numpy's

f2py

module

Pants is having trouble finding the following in a private pypi repository that proxies pypi.org

ERROR: No matching distribution found for PyMCP2221A<2,>=1.4.0

I don't think it has anything to do with module_mappings. Any ideas for what could be the cause for issue?

Can I get some details on what all is happening when generating constraints and exporting a resolve? I would like to proxy pypi dependencies through our Artifactory server using their remote repository capabilities. I'm having trouble with two issues • Generating lockfiles seems to error out finding distributions. I think this is a chicken/egg problem because the distributions don't readily exist in the proxied cache until they've been `pip install`ed. • Retrieving artifacts specified in the lockfile seems to be dependent on the url in the artifacts section of a package. This doesn't trigger any caching in the remote repository because it doesn't seem to be using

pip install

. Do those observations match what the underlying tooling would be doing? I need to ask JFrog (Artifactory developers) how they expect remote repositories to be used for lockfile generation. One idea I had was to do lockfile generation without going through the proxy and then introduce the proxy when it came time to install. I don't think that works because of what I'm seeing with the second bullet above.

is this design doc the best thing to read on how the environment stuff works under the hood? I'm interested in understanding how it compares to platforms & transitions in bazel.

🎉 We're hiring engineers at Kiwibank! And we use Pants! https://kiwibankpeople.csod.com/ux/ats/careersite/1/home/requisition/434?c=kiwibankpeople

Hi all, trying to set up the django example to see how helpful pants will be in terms of packaging a project for distribution in the future and I'm running into some issues while attempting to use the manage.pex files on a different ip and port. Is that standard or is it specifically locked by the binary targets of the example?

Hi all! Is it normal for packages to be duplicated inside

.cache/pants/named_caches/pex_root/installed_wheels/

? It seems I have 5 identical copies of torch, each of size 3.5G (

du -hs

tells me the size of the

installed_wheels

directory is 21G). When installing the same requirements in a virtual environment, the total size of the venv is 7G, with torch being 3.5G. This is after nuking the

.cache/pants/

directory and running

pants test ::

If I remove the 4 tests which transitively import torch, then nuke the cache and run

pants test ::

again, I only have one torch directory under

.cache/pants/named_caches/pex_root/installed_wheels/

(but other packages are duplicated). So my current hypothesis is that pants creates a copy of torch (or other library) for every test importing them transitively, but I'm guessing it's a problem with my configuration.

I am currently attempting to populate the

python_artifact(...)

with a

version

value from a file. I am following the guide here - https://www.pantsbuild.org/docs/plugins-setup-py I am hitting a very opaque error, - but it is not obvious which area of the code it is not happy with. (it would seem something is not initialising properly)

❯ pants  --print-stacktrace
16:50:33.07 [ERROR] 'NoneType' object has no attribute '__name__'
Traceback (most recent call last):
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/bin/daemon_pants_runner.py", line 130, in single_daemonized_run
    scheduler, options_initializer = self._core.prepare(options_bootstrapper, complete_env)
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/pantsd/pants_daemon_core.py", line 130, in prepare
    build_config = self._options_initializer.build_config(options_bootstrapper, env)
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/init/options_initializer.py", line 111, in build_config
    return _initialize_build_configuration(self._plugin_resolver, options_bootstrapper, env)
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/init/options_initializer.py", line 57, in _initialize_build_configuration
    return load_car_switch_rail_and_plugins(
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/init/extension_loader.py", line 46, in load_car_switch_rail_and_plugins
    load_build_configuration_from_source(bc_builder, car_switch_rail)
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/init/extension_loader.py", line 128, in load_build_configuration_from_source
    load_backend(build_configuration, backend_package)
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/init/extension_loader.py", line 143, in load_backend
    module = importlib.import_module(backend_module)
  File "/home/user8/.cache/nce/cdc3a4cfddcd63b6cebdd75b14970e02d8ef0ac5be4d350e57ab5df56c19e85e/cpython-3.9.15+20221106-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/user8/projects/svcs.com/acme-eng/se/core/tools/python/pants-plugins/acme/publishing/register.py", line 4, in <module>
    from acme.publishing import versioned
  File "/home/user8/projects/svcs.com/acme-eng/se/core/tools/python/pants-plugins/acme/publishing/versioned.py", line 15, in <module>
    async def shunup_kwargs_plugin(request: CustomshunupKwargsRequest) -> shunupKwargs:
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/engine/rules.py", line 337, in rule
    return inner_rule(*args, **kwargs, rule_type=RuleType.rule, cacheable=True)
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/engine/rules.py", line 327, in inner_rule
    return rule_decorator(*args, **kwargs)
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/engine/rules.py", line 277, in rule_decorator
    return _make_rule(
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/engine/rules.py", line 103, in wrapper
    awaitables = FrozenOrderedshun(collect_awaitables(func))
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/util/memo.py", line 123, in memoize
    result = func(*args, **kwargs)
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/engine/internals/rule_visitor.py", line 134, in collect_awaitables
    return _AwaitableCollector(func).awaitables
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/engine/internals/rule_visitor.py", line 43, in __init__
    self.visit(ast.parse(source))
  File "/home/user8/.cache/nce/cdc3a4cfddcd63b6cebdd75b14970e02d8ef0ac5be4d350e57ab5df56c19e85e/cpython-3.9.15+20221106-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/home/user8/.cache/nce/cdc3a4cfddcd63b6cebdd75b14970e02d8ef0ac5be4d350e57ab5df56c19e85e/cpython-3.9.15+20221106-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/ast.py", line 415, in generic_visit
    self.visit(item)
  File "/home/user8/.cache/nce/cdc3a4cfddcd63b6cebdd75b14970e02d8ef0ac5be4d350e57ab5df56c19e85e/cpython-3.9.15+20221106-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/home/user8/.cache/nce/cdc3a4cfddcd63b6cebdd75b14970e02d8ef0ac5be4d350e57ab5df56c19e85e/cpython-3.9.15+20221106-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/ast.py", line 415, in generic_visit
    self.visit(item)
  File "/home/user8/.cache/nce/cdc3a4cfddcd63b6cebdd75b14970e02d8ef0ac5be4d350e57ab5df56c19e85e/cpython-3.9.15+20221106-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/home/user8/.cache/nce/cdc3a4cfddcd63b6cebdd75b14970e02d8ef0ac5be4d350e57ab5df56c19e85e/cpython-3.9.15+20221106-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/ast.py", line 417, in generic_visit
    self.visit(value)
  File "/home/user8/.cache/nce/cdc3a4cfddcd63b6cebdd75b14970e02d8ef0ac5be4d350e57ab5df56c19e85e/cpython-3.9.15+20221106-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/home/user8/.cache/nce/cdc3a4cfddcd63b6cebdd75b14970e02d8ef0ac5be4d350e57ab5df56c19e85e/cpython-3.9.15+20221106-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/ast.py", line 417, in generic_visit
    self.visit(value)
  File "/home/user8/.cache/nce/cdc3a4cfddcd63b6cebdd75b14970e02d8ef0ac5be4d350e57ab5df56c19e85e/cpython-3.9.15+20221106-x86_64-unknown-linux-gnu-install_only.tar.gz/python/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/engine/internals/rule_visitor.py", line 123, in visit_Call
    self.awaitables.append(self._get_awaitable(call_node))
  File "/home/user8/.cache/nce/6621291ed4120fa45607b367abf7786ce496f389f27240712a63c31f1c438c21/bindings/venvs/2.15.0/lib/python3.9/site-packages/pants/engine/internals/rule_visitor.py", line 85, in _get_awaitable
    is_effect = func.__name__ == "Effect"
AttributeError: 'NoneType' object has no attribute '__name__'

Use -ldebug for more logs. 
See <https://www.pantsbuild.org/v2.15/docs/troubleshooting> for common issues.
Consider reaching out for help: <https://www.pantsbuild.org/v2.15/docs/getting-help>

the file

versioned.py

is

# acme/publishing/versioned.py
# ref: <https://www.pantsbuild.org/docs/plugins-setup-py>
from pants.backend.python.goals.setup_py import SetupKwargs, SetupKwargsRequest
from pants.engine.rules import collect_rules, rule
from pants.engine.target import Target
from pants.engine.unions import UnionRule


class CustomSetupKwargsRequest(SetupKwargsRequest):
    @classmethod
    def is_applicable(cls, _: Target) -> bool:
        return True


@rule
async def setup_kwargs_plugin(request: CustomSetupKwargsRequest) -> SetupKwargs:
    digest_contents = await Get(
        DigestContents,
        PathGlobs(
            ["VERSION"],
            description_of_origin="`python_artifact()` plugin",
            glob_match_error_behavior=GlobMatchErrorBehavior.error,
        ),
    )
    version = digest_contents[0].content.decode()
    return SetupKwargs(
        {**request.explicit_kwargs, "version": version},
        address=request.target.address
    )

def rules():
    return [
        *collect_rules(),
        UnionRule(SetupKwargsRequest, CustomSetupKwargsRequest),
    ]

the

acme/publishing/register.py

is

from typing import Iterable

from pants.engine.target import Target
from acme.publishing import versioned


def rules():
    return [*versioned.rules()]

I've got a weird situation. I have moved the scie-pants binary to my repo and commited it, and it works fine for me. One of my coworkers tried to get it running, though, but she gets errors when accessing our internal pypi repo due to bad credentials. From what I can see, her configuration is identical, we have the pypi server in .netrc with the right credentials, and we also added PANTS_PYTHON_REPOS_INDEXES just to be sure. The installer fails when setting up the virtual env. Any ideas what I might be missing?

Hi all, quick question - pants complain about not defined 3rd party dependency:

Pants cannot infer owners for the following imports in the target <name>.pyi :
* typing_extensions (line: 17)

This is coming from an autogenerated type hints file:

if sys.version_info >= (3, 8):
    import typing as typing_extensions
else:
    import typing_extensions

Since this file was autogenerated, and our codebase is on python 3.8, is there a way to globally ignore this

typing_extensions

import for

.pyi

files? I can of course modify the file and add

no-infer-dep

comment there, but I was wondering if there is more global way of doing that? Thanks!

Having just now partially got the

setup_py / python_artifact

macro, reading a STATIC

VERSION

file via a plugin, i need to have it read a generated

VERSION

file from the

vcs_version(...)

Is there a simple method to make this work ? I did try adding it in the python_distribution, and the macro. but neither "generate it before the macro

python_artifact(...)

runs . How do I achieve this ? *below does not work" . the macro expects

VERSION

file to be there, static on the filesystem,

vcs_version(
    name = "tagged_version",
    generate_to = "VERSION",
    template = "{version}",
)

python_distribution(
    name = "myproject",
    dependencies = [
        ...

        ":tagged_version",    <<< add dependency here
    ],
    generate_setup = True,
    provides = python_artifact(
        name = "myproject",
        dependencies = [ 
            ":tagged_version",    <<< OR add dependency here
        ]
    ),
    ...
)

What is the appropriate way to run a pants

repl

within a

docker_environment

? If I start a docker container and try to run pants in it I run into the m1/m2 issues. I setup a

docker_environment

and it manages tests great, but if I run

pants repl

it runs on mac not in docker and I don't see a flag in the docs to tell it to do otherwise. I tried making a

pex_binary

and running that in docker, which appears to do what I want but it takes ~45s to start, which is way worse than running

pants repl

which takes ~4s.

Hi all, first time to use pyoxidizer, got an error complaining about

ERROR: Package 'xx-api' requires a different Python: 3.9.7 not in '==3.10.9'

, more details in the thread