https://pantsbuild.org/ logo
Join Slack
Powered by
Hi, I'm trying to find the source of slowness we a...
# general
h
Hi, I'm trying to find the source of slowness we are having in some cases by reading the output of 
pants test ::
. I see some lines of the form 
X.Ys Scheduling: Task
(examples: 
131.75s Scheduling:  Building 1 requirement for requirements.pex from the pants.lock resolve: pytest
, ``130.75s Scheduling: Building 18 requirements for requirements.pex from the pants.lock resolve: A, B, C...` ). 1. Does these tasks wait in queue to start running? 2. Would you say that the amount of packages on the second example (
A, B, C...
) has a substantial impact on the amount of time needed for the task to complete?
🧵
h
Hmm, subsetting a lockfile (which is what you're seeing here) should be fast
over two minutes for a single requirement is pathological
Tasks are queued, but then run on any available core
And even then, that reported time is supposed to be the actual process active time
So maybe you have huge contention on a small number of cores?
What machine is this running on?
h
It's AWS' 
c6i.12xlarge
, which has 48 cores and 96 GiB of RAM. We have thousands of test files, and about 330 dependencies (counted entries on the lockfile). So when I see "`Scheduling: ...`", does it mean that it is currently in queue or actually running?
Bumping - I'm trying to find the source of slowness we are having in some cases by reading the output of 
pants test ::
. I'm also moving the thread to a Google Doc so I could summarize it as I go. Please comment there.
h
Commented there. PS would you mind moving that doc to a GitHub Discussion, so it can be preserved for posterity?
h
Sure, I've copied it into https://github.com/pantsbuild/pants/discussions/19408. I'll also try investigating it using your comments. TYVM for responding ❤️
2 Views
Open in Slack
PreviousNext
Powered by