Question: the VSZ of pantsd shows up as enormous values: like 539 G in my case

3642272 jreed      20   0  539G 92532 25036 S  0.0  0.3  0:00.00 pantsd [/home/jreed/.../repo-root]

Is this related to mmap or something used in caching? Or is something mapped in memory being counted multiple times

I’m trying to use the new docker

tools

config in 2.11 along with the amazon-ecr-credential-helper. I see the shim for

docker-credential-ecr-login

is being properly created in the build process’s tmp dir (

.shims/bin/docker-credential-ecr-login

exists, and I can invoke it manually and get the same results as the actual binary). When it’s set up, the build process is failing with:

failed to solve with frontend dockerfile.v0: failed to create LLB definition: rpc error: code = Unknown desc = error getting credentials - err: docker-credential-ecr-login resolves to executable in current directory (./.shims/bin/docker-credential-ecr-login), out: ``

if I update

__run.sh

to set

PATH=/bin:/usr/local/bin

, the build works. not sure if this weird “resolves to executable in current directory” error is a general docker thing or specific to docker-for-mac - I haven’t been able to find a config toggle to make it work

Trying to use the buf backend for linting proto files, and I'm having a hard time wiring up my config file ("buf.yaml"). Ideally I would put it in "build-support/proto/buf.yaml" and have

[buf].config = "build-support/proto/buf.yaml"

but I see that's not an option just yet. I tried

[buf].lint_args = ["--config", "build-support/proto/buf.yaml"]

but that didn't work as it seems buf is not running from the build root. Anyone have any suggestions?

It's possible I'm misunderstanding how the VCS dependency gets included. I was running

./pants export ::

with and without a VCS dependency in my

requirements.txt

(and implicitly the lockfile), but maybe it still got included?

export

, as most things in Pants, deals in the transitive closure of dependencies.

So if the VCS dep is in the lockfile, that implies it is in fact a transitive dep and so is being included in the venv export IIUC.

It wouldn't be included transitively. I can confirm that when I take it out of

requirements.txt

, it is also removed from the lockfile as expected.

Okay, I confirmed. With my VCS dependency in

requirements.txt

and the lockfile, it shows up in the exported venv. When I remove it and export again, it is not in the venv as expected.

OK. I'll admit that at this point I'm not at all tracking what's going on here. If you're continuing to see an unexpected slowdown @high-yak-85899 could you perhaps summarize, maybe in an issue?

I think the general summary is just that I'm surprised it takes, at a minimum, 45 seconds to export a venv (especially when none of the requirements have changed). All the noise above was to figure out if it was related to pants versions, the presence of vcs dependencies, and so on. I'm fairly confident that it's not related to having vcs dependencies and that @happy-kitchen-89482 correctly called out the difference between 2.10 and 2.11 since it's constructing a proper venv instead of symlinking.

since it's constructing a proper venv instead of symlinking.

Given that change, are you still surprised?

Yes

If there's no changes, it would be great if it didn't rebuild

But I have the luxury of treating caching like it's an easy problem because I don't understand it 🙂

Makes sense. FYI I'm pretty sure this is the culprit https://github.com/pantsbuild/pants/pull/15068 -- definitely curious if there is a better way to solve the original problem

Proposal for new error messages when you're using multiple resolves/lockfiles & deps aren't comaptible! Feedback very much appreciated before I start programming - this is gonna be non-trivial to implement https://github.com/pantsbuild/pants/issues/14864#issuecomment-1117648575

Hi pants, I am trying to capture summary output from a

test

run and having some trouble. Is there a convenient way to write the result summary at the end of a test run to a file or envvar, like`RESULTS=$(./pants test ::)` ? I also tried using pytest plugins to generate summary reports (like

pytest-html

) and they weren’t working, possibly because the full results would have to be aggregated from the parallel test runs. I noticed that the summary (i.e. the list of

𐄂 test:test failed in 1.32s.

at the end of the output) goes to stderr so I added a

2>&1

, but then it captures other errors too. Is there an easy way to just capture the summary?

A quick question, is there any config which we can temporally disable Pants to use cache?

I think I know the answer to this, but is there any way to do a

pip install

and point to the pex-based constraints file pants can generate? My context is that I'd really like to switch to the native lockfile generation pants offers, but we still have some tests that rely on a venv-style setup. We currently have

requirements.txt

,

requirements-dev.txt

(which contains things like

pytest

, etc.) and

constraints.txt

generated by

pip-tools

based on those two source files. Pants is happy to use

constraints.txt

even though it contains more information than it actually needs right now. If I generate the lockfile from

generate-lockfiles

, that only reflects

requirements.txt

. I can

./pants export

to get a venv, and then

pip install -r requirements-dev.txt

in that venv, but I'm losing some reproduceability there as that pip install command no longer has a lockfile to look at.

my understanding of -

-no-local-cache

from the doc is that the pants will not write the cache. Does it mean pants will not use the existing cache?

but I'm losing some reproduceability there as that pip install command no longer has a lockfile to look at.

What do you mean? The Virtual environment will have all pinned versions from your lock file.

Only for what was in

requirements.txt

(which doesn't include what is in

requirements-dev.txt

)

If I generate the lockfile from generate-lockfiles, that only reflects requirements.txt .

The lockfile will include every

python_requirement

in that resolve, even if your code does not use it. So create a second

python_requirements

target generator

Yes. Pants will not use the cache at all

Thanks! I'll try that out.

Hi all! Before any question, a little bit of context: Monorepo using Python, split into “libraries” and “lambdas” folders. A “lambda” folder can have a library dependency from “libraries” folder. Each lambda folder is meant to be zipped and used as, guess what: aws lambda package. In order to create each lambda package, we developped a tool that looks for dependency file and gather everything in a dist folder. This folder is then zipped and meant to be used as a aws lambda package. So, Pants looks like the perfect tool to replace our internal one. I already managed to have something working on a test repository (with dummy files, but with the same folder organization), but I have some questions: • Using python_awslambda seemed perfect but not really... As it using PEX, it needs to have a runtime defined. This means that I have to define it in the Pants target, but also again in my IaC tool (Using CDK). Is there any way to avoid it ? My guess is no. • My idea was then to generate a simple archive, without any PEX related stuff. After trying the python_awslambda target, I tried the archive, but I can’t get archive target to work with python_sources. Is it normal? I have a python_sources target, with dependencies. It works great when used in python_awslambda target, but don’t work at all with archive. It seems to consider that there are no files to zip/tar. Is such a thing possible? Feel free to ask for more details if needed! Thanks.

You were totally right. This makes everything great.

I tried the archive, but I can’t get archive target to work with python_sources. Is it normal?

yea, that’s correct: currently when sources are “typed” as python via

python_sources

, they are not treated as loose files which can be packaged into an

archive

files

targets are typed as loose files, and so can be… but then they are not typed as python, and so don’t have dependency inference, for example

As it using PEX, it needs to have a runtime defined. This means that I have to define it in the Pants target, but also again in my IaC tool (Using CDK). Is there any way to avoid it ? My guess is no.

what do you mean by “runtime” here?