Splitting a single process result up into multiple cache entries is controversial - I am cautiously more in favor of it than most, but it is inherently more risky, and would have to be done with great caution.

It would mean decoupling cache entries from processes. I think it is worth doing for the use case you mention, but it is not as easy as it may seem.

That said, there are several other mechanisms that help here. One is to only run tests that are affected by changes: https://www.pantsbuild.org/docs/advanced-target-selection#running-over-changed-files-with---changed-since

(the observation being that if the tests were affected by changes they wouldn't be cached anyway...)

And you can run the remaining tests in a few batches, rather than one giant batch, so you get the concurrency benefits

What are the numbers in your example case: how many tests, how long do they take in each case?

I'm on my next step to extend

pants

usage at my client, publishing changed libraries automatically 🙂 I had in mind to use

--changed-since

and

package

for that. But I can't achieve to have

pants

build the changed libraries. Consider this example trying to build our

geometry

library: Here are the library's direct dependencies:

→ ./pants dependencies libs/geometry:kaiko-geometry-dist
libs/geometry/kaiko/geometry/__init__.py:../../kaiko-geometry
libs/geometry/kaiko/geometry/_point_utils.py:../../kaiko-geometry
libs/geometry/kaiko/geometry/geometry_box.py:../../kaiko-geometry
libs/geometry/kaiko/geometry/geometry_point.py:../../kaiko-geometry
libs/geometry/kaiko/geometry/geometry_polygon.py:../../kaiko-geometry
libs/geometry/kaiko/geometry/geometry_size.py:../../kaiko-geometry
libs/geometry/kaiko/geometry/geometry_square.py:../../kaiko-geometry

I have modified a file of the library in the last commit (added a print):

→ git diff --name-only HEAD~1
libs/geometry/kaiko/geometry/geometry_polygon.py

The library has a

python_distribution

stanza:

→ grep -A 2 python_distribution libs/geometry/BUILD 
python_distribution(
    name="kaiko-geometry-dist",
    dependencies=[":kaiko-geometry"],

So I am expecting this command would create the library's wheel:

→ ./pants --changed-since=HEAD~2 --changed-dependees=direct --filter-target-type=python_distribution package

but it leaves

dist/

empty 😞 (whereas

./pants --filter-target-type=python_distribution package ::

does generate the concerned library's wheel). What am I misunderstanding/doing wrong?

Hi all. I’m using pants (poorly, unfortunately) and I’d like to improve what I’m doing. I’ve got it installed and running and have been using the

./pants fmt ::

with success. I also use

./pants lint ::

, but with mixed results. It helps me find various code issues, but I get this: “E0401: Unable to import ‘<some required module>’ (import-error)” for all my required modules. My project is setup with a remote Python interpreter (Python 3.10.8) in a Docker container. I’m also using Poetry to manage the dependencies. I’m not sure what to do to resolve this, other than tell pylint to stop reporting on E0401 errors.

I've got 115 unit tests right now, so perhaps not that many compared to some. But there still seems to be a pretty large time gap. Running with python -m pytest test/unit takes 58.3 seconds. Running with ./pants --debug test/unit:: takes 4 minutes and 1 second. I realize that using the --debug option disables caching and parallelism, but I thought it was a good way to get a fair comparison. After all, we could get caching and parallelism outside of pants by using the pytest-testmon and pytest-xdist plugins.

Thanks for the information on the concerns with the safety of splitting process runs into multiple cache entries. I would certainly be in favour of allow the user to declare that the tests in a certain directory have no side effects, so that it would be safe to split a single process result into multiple cache entries when running them. I guess another option would be to add the ability to declare that certain tests should be run with the pytest-testmon plugin. Maybe then the testmon cache could be leveraged in the same way as the mypy cache is leveraged in this PR: https://github.com/pantsbuild/pants/pull/16276.

in a Python monorepo, how do you work with

conftest.py

fixtures and what are the best practices you've discovered? GitHub Discussion with more information is here

What do the numbers look like without

--debug

?

I'm trying to come up with a good way to manage local environment variables. I asked a question a couple days ago about using a

.env

file, but hardcoding the file into a

BUILD

file doesn't work well since

.env

isn't tracked in version control, so CI tests throw warnings. I'm wondering whether a better approach is a plugin that explicitly loads the

.env

variables as environment variables for the test, then run that plugin as a command line argument to test? That way I could run it locally, but not include it in CI. Two things I'm wondering about: 1. would this approach work? I know tests are "hermetic", so would building a plugin that loads the variables work? 2. Can you "stack" targets; e.g. call the

.env

plugin prior to test so that they work together?

We notice that minor changes to

pants.toml

invalidates lots of build caches. There are sections of

pants.toml

that we would expect to change more regularly (e.g. cli aliases). Are there any thoughts about how to separate/shield from that?

If there is anyone who wants to use `C`/`C++` in Pants, and willing to take a read through a bit of a chain here - I would like some more eyeballs on this

cc

dependency inference question: https://github.com/pantsbuild/pants/pull/17738#discussion_r1042519097 In short, would Source Roots be equivalent to

include_directories

in C land?

i have pants pytests that call a service that I’ve built with pants. I know how to use ipdb in pytest, but I want to use it within the service. pdb works, but when I try to add ipdb to the requirements I get “Could not find a version that satisfies the requirement ipdb” even though i’m in python 3.10 and ipdb should be available. Has pants done something special to prevent users from using ipdb? i know it has to go in via

extra_requirements

to pytest

Hi all, I’m trying to get started with pants in one of our existing repos. Following along with the getting started guide, using Go and I was able to install pants and run

pants tailor ::

to generate my build files. But now, no matter what target I do such as

pants check

or

pants list

, I get a crash with the following:

15:53:39.68 [ERROR] 1 Exception encountered:

  IncompleteJSONError: parse error: trailing garbage
          mod", 	"GoVersion": "1.19" } { 	"Path": "<http://cloud.google.com/go|cloud.google.com/go>
                     (right here) ------^

Anyone seen this before? Google isn’t turning up much and searching for that snippet with

ag

doesn’t pull up matching code in my repo. I assume this is some internal JSON that pants is generating, but I’m not sure why it would be malformed

if I run

./pants test --force test/unit::

it takes 1 minute 20 seconds. So concurrency can significantly but not completely mitigate the overhead

Hmm. And you're running with

./pants test --debug

?

i’m not in

pants test

at all-- just running an interactive python session that talks to this service built by pants. i don’t remember whether I had the same problem when running pants pytests that talked to the service, but i can’t imagine that would make a difference

I noticed that the

export-codegen

goal is part of the

pants.backend.docker

backend, but... I can't think of why it would need to be associated with docker. Am I missing something? https://www.pantsbuild.org/docs/reference-export-codegen

Ah, so

./pants run ...

?

You mentioned "pytests" above, so I'm not sure what the framing is here

sorry for the confusion. i want to use the service from pytest but also from ipython. but the problem is with debugging the service, not with debugging the test. let me try running the service with

./pants run

actually i have tried that earlier for some other reasons, and every time I get this

i get the error when i try calling into the service

my ipython session gives

EOFError: connection closed by peer

command is

./pants run pushdown_service/service.py

. normally I do

./pants package pushdown_service/::

and then

dist/pushdown_service.service.pex

i was using an rpyc

ForkingServer

. @clever-hamburger-59716 suggested

ThreadedServer

and that let me debug! I’m not sure why