#20200 Adding a wrapper to generate type coverage reports New discussion created by jorgegomezcq Hi, What do you think if we had a pants report goal wrapper around the type coverage reports that mypy offers? https://mypy.readthedocs.io/en/stable/command_line.html#report-generation pantsbuild/pants

#20202 Question about comment regarding dependency inferrence New discussion created by jorgegomezcq Hi, Does this comment still hold true today? https://github.com/pantsbuild/example-python/blob/main/helloworld/greet/BUILD#L6 pantsbuild/pants

#20205 `pex_binary` fails to locate `.so` files inside pex, `python_source` works Issue created by tgolsson Describe the bug Users using

pex_binary

together with

torch>=1.13

may encounter weird

_dl_open

errors in relation to CUDA libraries. The same lockfile, same code, works when using

python_source

instead. This indicates that there is some difference in how the pex and/or venv is built between these two cases that causes one to fail unexpectedly. The solution is simply to

import nvidia

before loading torch. For both

packed

and

loose

pex's this seems to cause all nvidia packages to install correctly, and work. This is a huge footgun, because transitive dependencies should be available no matter. I'm going to guess nothing actually imports these packages in

torch

either, as they contain no actual code... only `.so`s. Pants version 2.18 OS Can only occur on Linux. Additional info Runnable repro here: https://github.com/tgolsson/pants-repros/tree/main/torch-2 Slack thread here: https://pantsbuild.slack.com/archives/C046T6T9U/p1700184172636389 pantsbuild/pants

#20206 How To Run a Python Source That Uses Files In Repo Subdirectory New discussion created by justicefreed I'm likely missing something, but more or less, no matter what I do, pants seems to be running my scripts from repo root, without relocated_files targets present. Repo Structure (Monorepo):

repo-name
      pants.toml
      -- projects
                -- myservice
                          BUILD
                          config.json
                          -- src
                                    main.py   # imports mylibrary.foo, usually run from myservice dir
                -- myotherservice
                          BUILD
                          otherconfig.json
                          -- src
                                    othermain.py   # also imports mylibrary.foo
      -- library
                -- mylibrary
                          BUILD
                          -- mylibrary
                                   foo.py

The key idea is that

config.json

contains necessary data for `main.py`'s service, but is NOT bundled/deployed with the service and will be different for every deployment instance. This data is loaded via python, using

open("config.json")

. In the deployment context, each project/service lives in it's own somewhat isolated directory, and does not have any sense of the repo structure other than below the

myservice

directory. The goal is to deploy/distribute each service using PEX files. This all generally seems to work. However, I want to be able to use pants to run main.py during development/testing, without breaking the script's access to

config.json

. This is what I can't seem to make work. I've tried: • Using

files

and

relocated_files

to specify that

config.json

should be moved to root at runtime, along with

run_goal_use_sandbox=True

. The file gets successfully added to the sandbox, but running

os.listdir()

and

os.getcwd()

within my script show that the script's working directory is project root, NOT the sandbox as expected. • Same thing except using

pex_binary

instead of

python_source

. Same result. • Using

adhoc_tool

->

run_shell_command

as a hack to try to execute the script using

myservice

as the working directory, but really doesn't seem to be the right approach (though it's nice to be able pass in the default required args that way). At this point, my main question is - why doesn't the python script's working directory use the sandbox directory when that setting is applied? It seems this would resolve my issue. Secondarily, is there any way to run a target with the working directory equal to the BUILD file location for that target? This would be so, so much simpler for my use case I think, but I could not find any way to change the working directory in a way that doesn't break normal, deployed operation. I feel it's likely I'm missing something silly here, so any guidance on how to allow a simple relative file import to work as expected from a script & file with paths relative to their common subdirectory instead of root would be greatly appreciated. I've spent probably 6 hours with the docs trying to get this one thing working 😅 so I've been as thorough as I could. Note: this repo is all on python 3.12, and would not install on the latest stable release, hence the alpha.

pants.toml

[GLOBAL]
# pants_version = "2.18.0"
pants_version = "2.19.0a0"
backend_packages = [
    "pants.backend.shell",
    "pants.backend.python",
    # "pants.backend.experimental.python.lint.ruff",
    "pants.backend.python.typecheck.mypy",
    "pants.backend.experimental.adhoc"
]
pants_ignore=["!__pycache__/","!**/__pycache__/","!config.json"]

[source]
root_patterns = [
    '/',
    '/components/*',
    '/library/*',
    '/library-stubs/*',
    '/tools'
]

[python]
interpreter_constraints = ['==3.12']
enable_resolves = true
default_run_goal_use_sandbox = true

myservice -> BUILD

python_requirements(
    name="reqs",
    source="requirements.txt",
)

files(
    name="files_needed_at_cwd",
    sources=["dynamic_config_overrides.json"]
)

relocated_files(
    name="cwd_files",
    src="components/MTConnectClient",
    dest="",
    files_targets=[":files_needed_at_cwd"]
)

pex_binary(
    name="pex_binary",
    entry_point="main",
    dependencies=[":cwd_files"]
)

python_source(
    name="main",
    source="src/main.py",
    dependencies=[":reqs",":cwd_files"],
    run_goal_use_sandbox=True
)

pantsbuild/pants

#20207 JS dep inference: refine handling of `pants: no-infer-dep` on multi-line statements Issue created by huonw Is your feature request related to a problem? Please describe. In the JavaScript backend, we're not 100% sure the handling of

// pants: no-infer-dep

on multi-line statements is exactly what we want. Currently, it has to be applied to the last line of the statement. This might be confusing. Tests from #20190:

import {
     a
} from 'ignored'; // pants: no-infer-dep
export {
   a
} from 'ignored'; // pants: no-infer-dep
require(
   'ignored'
) // pants: no-infer-dep
import(
   'ignored'
) // pants: no-infer-dep


import { // pants: no-infer-dep
   a
} from 'accepted';
import {
   c  // pants: no-infer-dep
} from 'accepted';

export { // pants: no-infer-dep
   a
} from 'b';
export {
   c // pants: no-infer-dep
} from 'd';

require( // pants: no-infer-dep
   'accepted'
);
require(
   'accepted' // pants: no-infer-dep
)

import( // pants: no-infer-dep
   'accepted'
);
import(
   'accepted' // pants: no-infer-dep
)

Describe the solution you'd like Someone to make a decision about which of the above lines (and any others, like "partial" ignores below (which don't make sense) or extreme syntax) should ignore or not.

import {
   b,
   c  // pants: no-infer-dep
} from 'some-module';

Preferably, any

pants: no-infer-dep

comments that aren't used are an error and/or warning too, because that indicates that we're likely doing something unexpected: the user was trying to ignore a dep but didn't do it exactly right. We should probably also look at other tools in the JS ecosystem so that we can do something that is familiar to users and/or preserved by tools like auto-formatters. Describe alternatives you've considered N/A Additional context Discussed in #20190, and #20190 (comment) is of particular interest. pantsbuild/pants

#19819 JavaScript dependency inference does not read export statements Issue created by AlexTereshenkov Describe the bug Parser used to statically analyze JavaScript sources doesn't take into considerations the `export` statements:

export { default } from './LocalFile';
export { default as MyFoo } from './LocalFile'

Pants version 2.18.0a0 OS N/A Additional info The parser should be extended to list modules discovered via the

export

statement. pantsbuild/pants

#2294 Generate requirements.txt from lockfile.txt Issue created by IvanRibakov Is it possible to convert

lockfile.txt

to

requirement.txt

format so that Pants lock files can be used instead of

pip freeze

? We're using an SCA (software composition analysis) tool that does not support Pants but does support `setuptools`/`requirements.txt` format for specifying dependencies and we would like to ensure that we use the same set of dependencies for both the build process and SCA scans. pantsbuild/pex

#20210 helm 2.18.0 interpolation DEPRECATED warnings appear regardless actual use Issue created by cburroughs Describe the bug When using the helm backend these warnings are emitted

11:11:03.64 [WARN] DEPRECATED: pants.backend.helm.target_types.HelmDeploymentValuesField.format_with() is scheduled to be removed in version 2.19.0.dev0.
 ==> /home/ecsb/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.0/lib/python3.9/site-packages/pants/util/memo.py:123
                return memoized_results[key]
            result = func(*args, **kwargs)
            memoized_results[key] = result

11:11:03.64 [WARN] DEPRECATED: Using the {env.VAR_NAME} interpolation syntax is scheduled to be removed in version 2.19.0.dev0.

Even when the

{env.VAR_NAME}

interpolation syntax is not used anywhere in the target repository. (Besides

grep

, I also tried upgrading to the 2.19.0 series to make sure everything still worked.) Pants version 2.18.0 OS Linux Additional info I remain confused when pants chooses to emit DEPRECATED as I can't get it print consistently with

pants --no-local-cache --no-pantsd

https://pantsbuild.slack.com/archives/C046T6T9U/p1700496601965609 pantsbuild/pants

#19270 Don't use pex to run ruff Issue created by thejcannon Describe the bug I'm pretty sure ruff is just a binary. We should be able to use

Process

to run it for EVEN MORE PERF pantsbuild/pants

#20219 pants.util.frozendict.FrozenDict has error prone order-sensitive comparisons Issue created by huonw Describe the bug The

FrozenDict

type is order sensitive, which differs to a normal

dict

. It also gives

False

when comparing to a normal dict. This makes it easy to get unexpected behaviour, and likely leads to unnecessary cache misses. pants/src/python/pants/util/frozendict.py Lines 15 to 21 in </pantsbuild/pants/commit/57d1801c80f8273ca386fba198495a05a45f60ae|57d1801>

d1 = {'a': 0, 'b': 1}
d2 = {'b': 1, 'a': 0}
print(d1 == d2) # True
print(FrozenDict(d1) == FrozenDict(d2)) # False

print(FrozenDict(d1) == d1) # False

NB. this seems to apply to all methods that call

tuple(self.items())

or similar:

__eq__

,

__lt__

, `_calculate_hash`/`__hash__`. Pants version

main

OS N/A Additional info This caused half of #20210, see #20220. pantsbuild/pants

#20222 Terraform validate not applied with check goal Issue created by xaniasd Describe the bug A clear and concise description of the bug. when running the check goal on terraform modules, the command executes successfully (with exit code 0) but does not perform any actual validation. Additionally, no logs are shown even at debug log level. Version v2.17.0 works as expected. Pants version Which version of Pants are you using? v2.18.0 OS Are you encountering the bug on MacOS, Linux, or both? MacOS Additional info Add any other information about the problem here, such as attachments or links to gists, if relevant. This test repository can be used to reproduce the problem. Goals fmt and lint work as expected, check on the other hand not. pantsbuild/pants

#20224 Allow silencing dependency collision warnings when packaging `python_aws_lambda_function` Issue created by cedric-fauth Describe the bug When running a package goal to build a python_aws_lambda_function I'm getting the following error:

$ pants package services/qgis-error:qgis-error-lambda
16:27:53.62 [INFO] Completed: Build python_aws_lambda_function artifact for services/qgis-error:qgis-error-lambda
16:27:53.63 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

ProcessExecutionFailure: Process 'Build python_aws_lambda_function artifact for services/qgis-error:qgis-error-lambda' failed with exit code 1.
stdout:

stderr:
Encountered collisions populating services.qgis-error/qgis-error-lambda from PEX at faas_repository.pex:
1. services.qgis-error/qgis-error-lambda/tests/test_defaults.py was provided by:
        sha1:2c2019f4e377693cdee64161a78f525a318cf174 -> /tmp/pantsbuild/named_caches/pex_root/installed_wheels/56d14958defcd513098953931ea36e691a6a5b87340d281ff818bc81f0810075/django_hosts-5.1-py3-none-any.whl/tests/test_defaults.py
        sha1:5b1d41d271ef3a72762f5008aa8c5fcdced64cb0 -> /tmp/pantsbuild/named_caches/pex_root/installed_wheels/d415a9964690a2cddb66a31ef32dd46c277e9b80434b94e39e3043188ed83e33/draftjs_exporter-2.1.7-py3-none-any.whl/tests/test_defaults.py
2. services.qgis-error/qgis-error-lambda/tests/__init__.py was provided by:
        sha1:da39a3ee5e6b4b0d3255bfef95601890afd80709 -> /tmp/pantsbuild/named_caches/pex_root/installed_wheels/56d14958defcd513098953931ea36e691a6a5b87340d281ff818bc81f0810075/django_hosts-5.1-py3-none-any.whl/tests/__init__.py
        sha1:ec6017a2bb6aa49c4d51cacabb503f98670a56a6 -> /tmp/pantsbuild/named_caches/pex_root/installed_wheels/d415a9964690a2cddb66a31ef32dd46c277e9b80434b94e39e3043188ed83e33/draftjs_exporter-2.1.7-py3-none-any.whl/tests/__init__.py

The directory does not include a

test/

subdir, so I guess this is created by the

python_aws_lambda_function

but cannot find it's own dependencies? Target:

python_aws_lambda_function(
    name="qgis-error-lambda",
    runtime="python3.8",
    handler="lambda_handler.py:handler",
)

Pants version 2.18.0 and 2.18.1rc0 OS Windows 11 WSL Devcontainer pantsbuild/pants

Comment on #20206 How To Run a Python Source That Uses Files In Repo Subdirectory Discussion answered by justicefreed Wrapping my service inside another python script that also changes the working directory seems to satisfactorily do everything I need to here. Thanks @huonw for the assistance. For others, what I now have is the below files:

repo_name/projects/myservice/dev-main-wrapper.py

"""Simple wrapper function to run src/main.py in the right context"""
import os
from src.main import main

os.chdir("./components/MTConnectClient")

main()

`repo_name/projects/myservice/BUILD`:

python_requirements(
    name="reqs",
    source="requirements.txt",
)

pex_binary(
    name="pex_binary",
    entry_point="src/main.py",
    dependencies=[":reqs"]
)

python_source(
    name="runner",
    source="dev-main-wrapper.py",
    dependencies=[":reqs"]
)

This works well it seems, and packaging via PEX loads the file as expected too, and all the dependencies get loaded correctly. pantsbuild/pants

Comment on #20206 How To Run a Python Source That Uses Files In Repo Subdirectory Discussion answered by huonw Ah, well. Thanks for trying things.

I'd happily avoid running the goal in the sandbox if I could instead run the goal in the subdirectory for that specific project, but I could not find a way to get pants to change the effective working directory when running a goal. Any recommendations would certainly be appreciated!

I don't know of one... except for having the code that is being run itself do this. For instance, could you have a "local" version of the target itself that varies slightly from the production code? For instance, does a few set-up things like switching directories and/or loading the config before calling into the real

main()

. (For my work codebase, this is effectively what we do: have a "wrapper" entrypoint for local dev for things like API servers, that does the special set-up required when running locally.) pantsbuild/pants

#20227 Add `compress`/`uncompressed` flag to `pex_binary` Issue created by tgolsson Is your feature request related to a problem? Please describe. When iterating on

pex_binary

targets for large ML packages, the zip time can easily dominate the whole runtime. This is fine when shipping out the pex as is, but when using it for

run

, debugging, or immediately unpacking into a container, it's wasted effort. For example, pantsbuild/pex#2292 #20205 Describe the solution you'd like Would wrap the following

pex

option:

--compress, --compressed, --no-compress, --not-compressed, --no-compression
                        Whether to compress zip entries when creating either a
                        zipapp PEX file or a packed PEX's bootstrap and
                        dependency zip files. Does nothing for loose layout
                        PEXes. (default: True)

This would be exposed on the `pex_binaries`/`pex_binary` target. Additional context I'm planning to do this work, just filing an issue to have it tracked. pantsbuild/pants

#20229 Shading throws `NoSuchFileException` given a `deploy_jar` in a subdirectory Issue created by grihabor Describe the bug Given a

deploy_jar

target in a subdirectory shading process throws an exception

NoSuchFileException

Pants version 2.18.0 OS Ubuntu 22.04 Additional info `helloworld/HelloWorld.java`:

package helloworld;

public class HelloWorld {
    public static void main(String[] args) {
        System.out.println("Hello, World!");
    }
}

`helloworld/BUILD`:

java_sources(name="src")

`subdir/BUILD`:

deploy_jar(
    name="fat",
    main="helloworld.HelloWorld",
    dependencies=["//helloworld:src"],
    shading_rules=[
        shading_keep(pattern="helloworld.**"),
    ],
)

It's important to put deploy_jar into a subdirectory. It works without errors if you put it into root BUILD file. Steps to reproduce: • pants generate-lockfiles • pants subdir:fat I get this:

ProcessExecutionFailure: Process 'Shading JAR subdir/fat.jar' failed with exit code 899.
...
java.nio.file.NoSuchFileException: subdir/fat.jar
...

pantsbuild/pants

#20232 docker_image repository does not override default Issue created by kuza55 Describe the bug Setting the repository flag on a docker_image target does not seem to override the default defined in pants.toml:

docker_image(
    name="base",
    image_tags=["0.1-{pants.hash}"],
    registries=[
        "@gcp",
    ],
    repository = "project/image",
)

Does not override

[docker.registries.gcp]
default = true
address = "europe-west4-docker.pkg.dev"
repository = "project/{directory}/{name}"

Pants version 2.17.0 OS Linux via WSL pantsbuild/pants

#20242 Exporting project's dependency graph New discussion created by AlexTereshenkov Introduction Pants already provides a rich set of tools to query the dependency graph to find out information about dependencies and dependents of individual build targets (be it a file, a package, or a glob pattern): •

dependencies

goal lets you list the dependencies:

$ pants dependencies cheeseshop/cli/cli.py                 
cheeseshop/cli/cli.py
cheeseshop/cli/utils/utils.py
cheeseshop/repository/package.py
cheeseshop/repository/properties.py
cheeseshop/repository/query.py
cheeseshop/repository/repository.py
cheeseshop/version.py
cheeseshop:project-version
requirements#click
requirements#loguru

•

dependents

(formerly known as

dependees

) goal lets you list the dependents (also known as reverse dependencies):

$ pants dependents cheeseshop/repository/parsing/casts.py  
cheeseshop/repository/package.py
cheeseshop/repository/parsing:parsing
tests/repository/parsing/test_casts.py:tests

It is possible to list dependencies for multiple files, one after another, e.g.

$ pants dependencies cheeseshop/repository/*.py            
cheeseshop/configs.py
cheeseshop/repository/package.py
cheeseshop/repository/parsing/casts.py
cheeseshop/repository/parsing/exceptions.py
cheeseshop/repository/properties.py
requirements#loguru
requirements#packaging
requirements#requests
requirements#typing-extensions

Motivation Having the dependencies listed for multiple targets such as individual source files, you don't know what modules out of those files in the

cheeseshop/repository

package depends on what. Running Pants goal on each individual file is very inefficient: each invocation of Pants has an overhead, so it's more preferrable to get all the work done within a single Pants call. It is also possible that a command would be run in a environment without

pantsd

process already running and/or any cache available. So even though this works, it will prove to be unreasonably slow for even a medium sized codebase:

$ for filename in $(ls cheeseshop/repository/*.py); do 
  echo "--${filename}--"; pants dependencies ${filename}
done

--cheeseshop/repository/package.py--
cheeseshop/repository/parsing/casts.py
cheeseshop/repository/properties.py
requirements#packaging
requirements#typing-extensions

--cheeseshop/repository/query.py--
cheeseshop/repository/package.py
requirements#packaging

--cheeseshop/repository/repository.py--
cheeseshop/configs.py
cheeseshop/repository/package.py
cheeseshop/repository/parsing/exceptions.py
requirements#loguru
requirements#requests

It is therefore more helpful to list dependencies for multiple files individually to be able to distinguish them, using a new goal when construction of the graph happens only once:

$ pants <goal> <options> cheeseshop/repository/*.py
{
    "cheeseshop/repository/__init__.py": [],
    "cheeseshop/repository/package.py": [
        "cheeseshop/repository/parsing/casts.py",
        "cheeseshop/repository/properties.py",
        "requirements#packaging",
        "requirements#typing-extensions"
    ],
    "cheeseshop/repository/properties.py": [],
    "cheeseshop/repository/query.py": [
        "cheeseshop/repository/package.py",
        "requirements#packaging"
    ],
    "cheeseshop/repository/repository.py": [
        "cheeseshop/configs.py",
        "cheeseshop/repository/package.py",
        "cheeseshop/repository/parsing/exceptions.py",
        "requirements#loguru",
        "requirements#requests"
    ],
    "cheeseshop/repository/types.py": []
}

The information produced by this new goal would return adjacency representation of the dependency graph as a dictionary of lists. The output is JSON compatible which makes it trivial to filter and query the graph using standard tooling such as

jq

and standard library of most programming languages. More importantly, this data structure may be used to construct graphs using 3rd party tooling such as

networkx

to be able to query and manipulate it, see networkx.convert.from_dict_of_lists:

$ pants <goal> <options> cheeseshop/repository/*.py > depgraph.json
$ python3
>>> import json
>>> import networkx
>>> with open("depgraph.json") as fh:
...     g = networkx.from_dict_of_lists(json.load(fh), create_using=networkx.DiGraph)
>>> networkx.shortest_path(g, "cheeseshop/repository/query.py", "cheeseshop/repository/properties.py")
['cheeseshop/repository/query.py', 'cheeseshop/repository/package.py', 'cheeseshop/repository/properties.py']

Having the dependency graph exported makes it possible to cheaply answer a variety of useful questions such as: • are there any build targets that no one depends on? • what is the longest path in the graph? • what source module leads to most tests? • what test module has most dependencies? Having the graph exported also opens up the opportunity to visualize the whole graph or its parts using visualization libraries such as `graphviz`:

$ python3 -m venv .venv && source .venv/bin/activate
$ pip install networkx pydot

import json
import networkx
from networkx.drawing.nx_pydot import write_dot

with open("depgraph.json") as fh:
  g = networkx.from_dict_of_lists(json.load(fh), create_using=networkx.DiGraph)
  write_dot(g, "graph.dot")

$ dot -Tpng graph.dot > graph.png

graph Having the graph exported into a JSON data structure is enough to be able to perform any query/manipulation with the graph, but for practical reasons, it may be helpful to provide additional functionality available out-of-the-box to avoid forcing users to write additional programs. This could mean: • listing the dependents (reverse dependencies) • listing the dependencies/dependents transitively Implementation Practically, fetching dependencies (direct or transitive) is trivial:

direct_deps_request_result = await Get(Targets, DependenciesRequest(target[Dependencies]))
deps = [str(d.address) for d in FrozenOrderedSet(direct_deps_request_result)]
...
transitive_deps_request_result = await Get(TransitiveTargets, TransitiveTargetsRequest([target.address]))
dependencies = transitive_deps_request_result.dependencies

and so is fetching dependents:

dependees = await Get(
	Dependents,
	DependentsRequest(
		(target.address,),
		transitive=True,
		include_roots=False,
	),
)

Fetching dependencies for multiple targets is likely to happen in a

MultiGet

call to a rule, filling a mapping of build targets and their dependencies which will be the output of the new goal. With the naming of the goal and the options being subject to change, this is how the user interface may look like: ``` # fetching direct dependencies of two files $ pants dep-graph --dependencies cheeseshop/repository/query.py cheeseshop/repository/package.py { "cheeseshop/repository/package.py": [ "cheeseshop/repository/parsing/casts.py", "cheeseshop/repository/properties.p… pantsbuild/pants

#17221 ModuleNotFoundError: No module named 'pex' Issue created by engnatha Describe the bug When running pants in our CI system, we can have a container that has multiple clones of our repository. At various points, they will be running goals like

./pants export

or

./pants run

. At some point in the chain, we see errors like this

10:42:12.54 [INFO] Initialization options changed: reinitializing scheduler...
10:42:13.10 [INFO] Scheduler initialized.
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/buildbot/.pex/unzipped_pexes/7907cee4a80399696d31dbee748702a492b91a80/__main__.py", line 102, in <module>
    from pex.pex_bootstrapper import bootstrap_pex
ModuleNotFoundError: No module named 'pex'

The only way to remedy this seems to be clearing out

~/.pex

and trying the goal again. Pants version 2.13.0 OS Ubuntu 20.04 Additional info We've tried adding some steps to manage concurrency in the system and automatically wipe

~/.pex

when this error is detected, but that's unreliable to implement everywhere. Some initially debugging was kindly provided by @enough-analyst-54434 who filed #17176 to resolve. pantsbuild/pants

#20246 Django `manage.py` files should infer dependencies on migrations from `INSTALLED_APPS` Issue created by chrisjrn Currently, running any Django management command that depends on migrations requires running

manage.py

outside the sandbox, or (apparently) explicitly declaring migrations packages as dependencies of

manage.py

.

manage.py

knows which migrations to run by way of the apps declared in the

INSTALLED_APPS

value in a Django settings file. With the experimental Django plugin, it appears that Django migrations are able to infer dependencies on other migrations by way of understanding the

AppConfig.name

value in the

apps.py

file for those other migrations. It appears that the leaf migrations still need to be declared automatically to be picked up as a dependency. With that in mind, it looks like

INSTALLED_APPS

values can be inspected -- they either declare a package spec, or an

AppConfig

class, with a name that consists of a package spec. That should be enough information to infer dependencies on those migrations. Pants should do that :) UPDATE: It looks like management commands are in the same boat, and should also get picked up. pantsbuild/pants

#20247 wheel_config_settings, --global-option --python-tag breaks with setuptools &gt;= 69.0 Issue created by bradday4 Describe the bug Passing

--python-tag

as part of

--global-option

in the

wheel_config_settings

dictionary of a

python_distribution

target results in the following error if using setuptools version >= 69:

stderr:
usage: backend_shim.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: backend_shim.py --help [cmd1 cmd2 ...]
   or: backend_shim.py --help-commands
   or: backend_shim.py cmd --help

error: option --python-tag not recognized

Here's a minimal reproduction of an example

python_distribution target

for pants

python_distribution(
    name="mydist",  # this is the name given to the target
    provides=python_artifact(name="mylib")
    wheel_config_settings={"--global-option": ["--python-tag", "py38.py39.py310"]},
    generate_setup=False,
)

Along with the relevant portion of my

pyproject.toml

[build-system]
requires = ["setuptools ~= 69.0", "wheel"]
build-backend = "setuptools.build_meta"

The documentation for pants shows the

--global-option

parameter being used but should be updated to to note the breaking change with setuptools. Substitututing

--global-option

with

--build-option

fixes the issue and appears to work with earlier versions of setuptools back to 64.0 Pants version Which version of Pants are you using? 2.18 OS Linux Additional info Here is the PR for the change in setuptools And the discussion that precipitated it pantsbuild/pants

#20251 Unclear error message for pants.toml Issue created by ubmarco Describe the bug When I run any pants target like

pants package ::

I get the message

$ pants package ::
16:54:55.26 [INFO] waiting for pantsd to start...
16:54:56.47 [INFO] pantsd started
16:54:56.76 [INFO] Initializing scheduler...
16:55:01.56 [INFO] Scheduler initialized.
16:55:01.70 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

InvalidRequirement: Parse error at "'.*'": Expected string_end

It would be good if the message included a hint where the issue is. The problem is the following definition:

[python]
interpreter_constraints = ["3.11.*"]

This works:

[python]
interpreter_constraints = ["==3.11.*"]

Pants version 2.18.0 OS Linux pantsbuild/pants

#20259 `pants check ::` failed when `mypy` is configured with a separate lock file Issue created by felixkbb Describe the bug

pants check ::

failed with a lot of error messages when

mypy

is configured with a separate lock file. Pants version 2.17.0 and 2.18.0 OS Linux 5.4.0-167-generic 184-Ubuntu SMP Additional info Having an issue with

pants check ::

type checking when

mypy

is configured with separate lock file. As shown in the configuration below, the resolves are separated depending on which tool is running.

# pants.toml
[python]
interpreter_constraints = ["CPython==3.9.*"]
default_resolve = "python-default"
enable_resolves = true

[python.resolves]
flake8 = "3rdparty/python/flake8.lock"
pytest = "3rdparty/python/pytest.lock"
mypy = "3rdparty/python/mypy.lock"
python-default = "3rdparty/python/default.lock"

[flake8]
install_from_resolve = "flake8"
requirements = ["//3rdparty:flake8"]

[pytest]
install_from_resolve = "pytest"
requirements = ["//3rdparty:pytest"]

[mypy]
install_from_resolve = "mypy"
requirements = ["//3rdparty:mypy"]

[python-protobuf]
mypy_plugin = true
infer_runtime_dependency = false

# 3rdparty/BUILD
... ...
... ...
python_requirements(
    name="mypy",
    source="requirements-mypy.txt",
    resolve="mypy",
)

However, All works, except mypy. Seems mypy doesn't work out the resolves using the specified lock files. When I run

pants check ::

command, the output shows a lot type checking errors

14:48:09.10 [ERROR] Completed: Typecheck using MyPy - mypy - mypy failed (exit code 1).
service_pb2.pyi:7: error: Library stubs not installed for "google.protobuf.descriptor"  [import-untyped]
service_pb2.pyi:7: error: Library stubs not installed for "google.protobuf"  [import-untyped]
service_pb2.pyi:7: error: Skipping analyzing "google": module is installed, but missing library stubs or py.typed marker  [import-untyped]
service_pb2.pyi:8: error: Library stubs not installed for "google.protobuf.internal.containers"  [import-untyped]
service_pb2.pyi:8: error: Library stubs not installed for "google.protobuf.internal"  [import-untyped]
service_pb2.pyi:9: error: Library stubs not installed for "google.protobuf.internal.enum_type_wrapper"  [import-untyped]
service_pb2.pyi:10: error: Library stubs not installed for "google.protobuf.message"  [import-untyped]
tools/utils.py:3: error: Library stubs not installed for "colorama"  [import-untyped]
service_pb2_grpc.pyi:10: error: Name "grpc.Channel" is not defined  [name-defined]
service_pb2_grpc.pyi:11: error: Name "grpc.UnaryUnaryMultiCallable" is not defined  [name-defined]
service_pb2_grpc.pyi:21: error: Name "grpc.ServicerContext" is not defined  [name-defined]
service_pb2_grpc.pyi:24: error: Name "grpc.Server" is not defined  [name-defined]
client.py:4: error: Library stubs not installed for "colorama"  [import-untyped]
client.py:4: note: Hint: "python3 -m pip install types-colorama"
client.py:37: error: Module has no attribute "insecure_channel"  [attr-defined]
scripts/request_ext.py:4: error: Library stubs not installed for "requests"  [import-untyped]
scripts/request_ext.py:5: error: Library stubs not installed for "requests.exceptions"  [import-untyped]
unit/types.py:14: error: Library stubs not installed for "dateutil"  [import-untyped]
system/account_date.py:3: error: Library stubs not installed for "dateutil.relativedelta"  [import-untyped]
schedule.py:4: error: Library stubs not installed for "dateutil.relativedelta"  [import-untyped]
schedule.py:4: note: Hint: "python3 -m pip install types-python-dateutil"
... ...
... ...

The suggested workaround (#19757) for the problem is to use the default resolve for mypy. It does work. However, not sure if it's the proper way of solving the problem. pantsbuild/pants

#20260 Failure to Prepare JDK Environment Behind Proxy Issue created by NGustafson Describe the bug Commands on Scala targets that require a JDK, such as

pants check ::

and

pants lint ::

, fail when the JVM index used by Coursier is unavailable. Pants version 2.18.0 OS Linux Additional info The following is a sample of the output from a server running behind a proxy, and therefore unable to reach the JVM index used by Coursier:

pants --changed-since=### lint
  86.22s	Ensure download of JDK --jvm=zulu:8.0.372.
[INFO] Long running tasks:
  79.06s	Run `scalafmt` on 73 files.
  79.06s	Run `scalafmt` on 18 files.
  79.06s	Run `scalafmt` on 5 files.
[INFO] Long running tasks:
  109.15s	Run `scalafmt` on 73 files.
  109.15s	Run `scalafmt` on 18 files.
  109.15s	Run `scalafmt` on 5 files.
[ERROR] 1 Exception encountered:

Engine traceback:
  in `lint` goal

ProcessExecutionFailure: Process 'Run `scalafmt` on 5 files.' failed with exit code 1.
stdout:

stderr:
Downloading <https://github.com/coursier/jvm-index/raw/master/index.json>
Downloaded <https://github.com/coursier/jvm-index/raw/master/index.json>
Exception in thread "main" java.lang.Exception: Error while getting <https://github.com/coursier/jvm-index/raw/master/index.json>: download error: Caught java.net.ConnectException (Connection timed out) while downloading <https://github.com/coursier/jvm-index/raw/master/index.json>
	at coursier.jvm.JvmIndex$.$anonfun$load$5(JvmIndex.scala:142)
	at coursier.jvm.JvmIndex$.$anonfun$load$5$adapted(JvmIndex.scala:140)
	at coursier.util.Task$.$anonfun$flatMap$extension$1(Task.scala:14)
	at coursier.util.Task$.$anonfun$flatMap$extension$1$adapted(Task.scala:14)
	at coursier.util.Task$.wrap(Task.scala:82)
	at coursier.util.Task$.$anonfun$flatMap$2(Task.scala:14)
	at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
	at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base@17.0.5/java.lang.Thread.run(Thread.java:833)
	at com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775)
	at com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)

pantsbuild/pants

#20024 Type error on FileContent results in infinite loop Issue created by hopper-signifyd Describe the bug I'm working on a custom plugin and I did this:

content = FileContent(path=versions_file_path, content="some string")
digest = await Get(Digest, CreateDigest([content]))

That resulted in the infamous

Filesystem changed during run: retrying 'MyGoal'

infinite loop. I was really confused. Upon further inspection of the run logs, I found this:

09:14:56.95 [31m[ERROR][0m panic at 'called `Result::unwrap()` on an `Err` value: "Field `content` was not convertible to type alloc::vec::Vec<u8>: PyErr { type: <class 'TypeError'>, value: TypeError(\"'str' object cannot be interpreted as an integer\"), traceback: None }"', src/intrinsics.rs:422

So I updated the code above to call

.encode()

on the string before passing it as content and things started working. However, I feel like my mistake should have resulted in an error / failure instead of the infinite

Filesystem changed

loop as this just made things harder to debug. Pants version 2.16.0 OS MacOS Additional info I like pants pantsbuild/pants

#20263 Download statistics for release assets New discussion created by huonw I was curious, so I did an investigation of the download statistics of the released artifacts. I compared: • the PyPI downloads of

pantsbuild.pants

in the last 2 years • the PyPI downloads in the last 30 days • the GitHub Releases downloads across all time (all assets) This plot shows the values for all stable releases since the start of 2022: image Summary: • from the blue series (top), it's clear that 2.13.0 and 2.17.0 at the most popular "ever" releases • from the red series (middle), it's clear that most people are still on 2.17.0 • from the green series (bottom), there's far fewer downloads of 2.18.0 This plot shows the PyPI downloads per day for the last year, for the versions with the most download (we don't have this info available for GitHub assets, AFAICT): image There's hints to me that either: • people aren't transitioning to 2.18.0 as quickly as they did earlier releases • OR, GitHub download stats are fairly different to PyPI ones The raw data is available here: https://gist.github.com/huonw/112684339a5e6e1e947e75890233ff35 • `downloads-2stable.csv`: PyPI and GitHub releases stats for all 2.x.y stable versions • `downloads-all.csv`: PyPI and GitHub releases stats for all versions • `downloads-pypi-year-per-day.csv`: PyPI release stats per day per version for the last year How I computed this BigQuery queries following https://packaging.python.org/en/latest/guides/analyzing-pypi-package-downloads/ 30 days:

SELECT file.version, COUNT(*) AS num_downloads
FROM `bigquery-public-data.pypi.file_downloads`
WHERE file.project = 'pantsbuild-pants'
  -- Only query the last 30 days of history
  AND DATE(timestamp)
    BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
    AND CURRENT_DATE()
GROUP BY file.version

2 years:

SELECT file.version, COUNT(*) AS num_downloads
FROM `bigquery-public-data.pypi.file_downloads`
WHERE file.project = 'pantsbuild-pants'
  -- Only query the last 30 days of history
  AND DATE(timestamp)
    BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 730 DAY)
    AND CURRENT_DATE()
GROUP BY file.version

Downloads per day for the last year:

SELECT file.version, COUNT(*) AS num_downloads, DATE_TRUNC(DATE(timestamp), DAY) AS `day`
FROM `bigquery-public-data.pypi.file_downloads`
WHERE file.project = 'pantsbuild-pants'
  -- Only query the last 2 years days of history
  -- Only query the last 30 days of history
  AND DATE(timestamp)
    BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 365 DAY)
    AND CURRENT_DATE()
GROUP BY day, file.version
ORDER BY day DESC

Stats for assets:

gh api --paginate -X GET '/repos/pantsbuild/pants/releases' | jq -r '.[] | [.tag_name, ([.assets[].download_count] | add) // 0] | @csv'  |  sort -V

Release dates (from tags)

git tag -l 'release_*' --sort=taggerdate --format='%(refname:short),%(taggerdate:short-local)'

pantsbuild/pants

#20265 Using pip-tools with pants New discussion created by jorgegomezcq Hi, Does it make sense to use pip-tools with pants? pantsbuild/pants

#20266 `ruff` not executable as formatter Issue created by cedric-fauth Is your feature request related to a problem? Please describe. Hi, I am using the

ruff

linter and formatter and wanted to include it in my pants setup. However only linting and fixing seems to be available through the

lint

and

fix

goals. I couldn't find a way to run

ruff format

which does more than

ruff --fix

. This feature is missing I think. Describe the solution you'd like It would be great to have a

pants fmt

command which invokes

ruff format

. Describe alternatives you've considered I tried using the

[ruff].args

field to provide

format

but this did not work. pantsbuild/pants

#1540 Constrain the matrix of shebang, platforms and ICs as much as possible. Issue created by jsirois Building and booting a PEX can involve the interplay of 4 concepts currently: 1. The interpreter(s) the Pex CLI is run with. 2. The platforms specified. 3. The interpreter constraints specified. 4. The shebang the resulting PEX file is built with. These all need to be in alignment to produce a PEX that "boots" properly. Further, they need to be particular beyond that in order to squeeze performance out of some combinations of those items. Pex cannot ensure alignment, let along maximally performing alignment, in all cases. It can probably catch a few out-of-aligment cases though and - preferably - automatically bring them in alignment, or - less preferably - warn or potential issues. For the latter, care will be needed not to provided unwanted nannying for a knowledgeable user or a too sharp knife (the current situation) for the casual user. An example of automated alignment would be basing the default shebang for a single --platform PEX, or else multiplatform PEX where all platforms share the same major / minor versions in their platform tag, off that major / minor version pair. I.E.: for

--platform linux-x86_64-cp-37-cp37m --platform macosx-10.13-x86_64-cp-37-m

use a default shebang of

#!/usr/bin/env python3.7

. This is in contrast to the behavior today, which is the shebang defaulting to

#!/usr/bin/env pythonX.Y

where X / Y are the major / minor versions of the interpreter used to run the Pex CLI, and which need bear no relation to the platforms selected. The answer to what can be and should be constrained will change once #1020 is implemented, but even shy of that the status quo can be improved to prevent this style of unambiguous mis-aligment. pantsbuild/pex

#15651 BSP: Ensure that Scala compiler plugins are detected by the frontend Issue created by stuhood Currently, we do not expose any of `scalac`'s arguments via the

buildTarget/scalacOptions

message: pants/src/python/pants/backend/scala/bsp/rules.py Line 377 in </pantsbuild/pants/commit/be411561c0ea282c6557854e3519d07a26354b79|be41156> It's likely (although I'm attempting to confirm upstream), that correctly exposing the arguments (including plugin instantion arguments: computed here) would be sufficient for this detection. pantsbuild/pants