I'm working on a new section of our codebase and I'm seeing test targets take a really long time to complete. When I run them individually, they're taking around 5 seconds for pants to do its thing and running concurrently I'm seeing them take upwards of 15 to 20 seconds. The actual tests that run are really simple and only take 0.5 second at most (not accounting for some initial setup that runs in conftest that take around 1.5 seconds). Any ideas why I might be seeing such a large discrepancy?

And this is consistent if you re-run a few times with

--force

?

I wonder if you're experiencing contention. Might be helpful to play with these settings: https://www.pantsbuild.org/docs/using-pants-in-ci#tuning-resource-consumption-advanced

Yeah, seems pretty consistent.

What's the perf like when using

test --debug

, which runs in the foreground as an interactive process?

Okay, actually, the initial numbers were misleading. Here's a better example (I was doing some tricks in

conftest.py

to make the times pytest reports lower than they actually are).

(ops-venv) ~/devel/operations (pants)$ ./pants test --force --output=all ops/utils/wheel_utils_test.py --pytest-args="--durations=3"
20:05:05.79 [INFO] Initializing scheduler...
20:05:05.90 [INFO] Scheduler initialized.
20:05:12.61 [INFO] Completed: Run Pytest - ops/utils/wheel_utils_test.py:tests succeeded.
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /tmp/process-executioneNcGLY, configfile: pytest.ini
plugins: typeguard-2.12.1, timeout-2.1.0, cov-3.0.0
timeout: 4.0s
timeout method: signal
timeout func_only: False
collected 3 items

ops/utils/wheel_utils_test.py ...                                        [100%]

=============================== warnings summary ===============================
../../home/nathanael/.cache/pants/named_caches/pex_root/venvs/s/08108cc5/venv/lib/python3.8/site-packages/marshmallow/__init__.py:17
  /home/nathanael/.cache/pants/named_caches/pex_root/venvs/s/08108cc5/venv/lib/python3.8/site-packages/marshmallow/__init__.py:17: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    __version_info__ = tuple(LooseVersion(__version__).version)

-- Docs: <https://docs.pytest.org/en/stable/warnings.html>
- generated xml file: /tmp/process-executioneNcGLY/ops.utils.wheel_utils_test.py.tests.xml -
============================= slowest 3 durations ==============================
1.61s call     ops/utils/wheel_utils_test.py::CommandSpeedsFromList::test_commanding_speeds_from_list
0.08s call     ops/utils/wheel_utils_test.py::SetSpeedAndStabilizeTests::test_setting_speed_and_stabilize
0.08s call     ops/utils/wheel_utils_test.py::WaitForWheelSpeedStabilized::test_waiting_to_stabilize
========================= 3 passed, 1 warning in 3.74s =========================



✓ ops/utils/wheel_utils_test.py:tests succeeded in 4.51s.

Wrote test XML to `dist/test_results`
(ops-venv) ~/devel/operations (pants)$ pytest ops/utils/wheel_utils_test.py --durations=3
================================================================================================================================================ test session starts ================================================================================================================================================
platform linux -- Python 3.8.10, pytest-5.4.3, py-1.10.0, pluggy-0.13.1
rootdir: /home/nathanael/devel/operations, inifile: pytest.ini
plugins: typeguard-2.12.1, xdist-1.34.0, timeout-2.1.0, forked-1.3.0, cov-3.0.0
timeout: 4.0s
timeout method: signal
timeout func_only: False
collected 3 items                                                                                                                                                                                                                                                                                                   

ops/utils/wheel_utils_test.py ...                                                                                                                                                                                                                                                                             [100%]

============================================================================================================================================= slowest 3 test durations ==============================================================================================================================================
1.49s call     ops/utils/wheel_utils_test.py::CommandSpeedsFromList::test_commanding_speeds_from_list
0.09s call     ops/utils/wheel_utils_test.py::WaitForWheelSpeedStabilized::test_waiting_to_stabilize
0.07s call     ops/utils/wheel_utils_test.py::SetSpeedAndStabilizeTests::test_setting_speed_and_stabilize
================================================================================================================================================= 3 passed in 1.71s =================================================================================================================================================

Are you pulling it in as a third-party req, it sounds like? Any other particularly large third-party deps, like Tensorflow or Numpy? That's actually something we should indeed try to slip into Pants 2.10 @witty-crayon-22786 - we recently gained the infrastructure (thanks to @fast-nail-55400) to safely symlink immutable files like requirements PEXes, rather than copying. It has a non-trivial performance impact for JVM because it makes sandbox creation faster. but we haven't had the time to hook up to Python yet. Stu or Tom, would either of you have time to hook up PEX to immutable caches before the release? Reminder we decided to not block that change on the refactor to use

immutable_paths

API

Even if that's an issue, it should all be cached

It's pulled in directly from VCS in

requirements.txt

.

A ticket with as many details as possible, including whatever you see repeated in

-ldebug

output across multiple runs, would be really appreciated!

Yeah, nothing heavy is running on import so it must be some really inefficient path resolution for each import.

What happens if you nuke all the *.pyc files before running pytest directly (without Pants) ?

No change with

.pyc

files deleted.

Sorry for the wild ride.

Not at all, these cases are always interesting, because it shows an area that can be improved, in terms of insights/feedback/transparency whats going on etc. 🙂 (at least my personal view)

So, I feel pretty confident that Pants isn't really introducing a problem here. Or maybe it is, I'll let y'all weigh in on that. What I've found is that one of our modules in particular is very heavy to import. Not because it's doing a lot, but because it has a lot of transitive dependencies. I think in our venv world, we haven't noticed it because, as was pointed out earlier, things start hitting the pycache and helping us out. Without that caching available in pants because everything is being run independently, we have a very heavy import process for a lot of tests (because they all fundamentally rely on this module with a lot of transitive dependencies). Without knowing more about the system, I'm not sure I know what Pants could do about this.

https://github.com/pantsbuild/pants/issues/11339 is probably not helping. That's why Benjy asked you to delete

.pyc

files. But curious that you said it didn't make a difference. Dang I wish I remembered to put that in the 2022 prioritization survey

The only code that should suffer recompilation is 1st party.

I was deleting the wrong pyc files originally. When I move everything to my monorepo, it's all first party. My problem is that this repo I'm bringing over has a bunch of really small scripts that all use this common, huge library. So executing all their tests is forcing this recompilation to happen every test because each one is getting its own memory space in separate processes.

This would make an interesting blog post. The journey of figuring out what's going on, and how it ultimately led you to this conclusion. Migrating often leads to interesting reveals and ultimately new thinking. Fascinating to hear those lessons.

The smoking gun would be to see how long those imports take outside Pants but in a completely pristine environment with no pycache

Right, that's exactly what I'm seeing. I used tuna to inspect and realized that, without caching, and it's really heavy without the caching.

Hmm, I think that first number was an outlier. It's only 70ms even if I nuke the pycache

That was exactly my observation when I did a dry run of merging the repos. Things are getting rebuilt each time.

So compiling your first-party code has a measurable cost. That is an important use-case to know about and address.

Yeah, I think it's a tough ask. We're starting to get really close to just asking for a compiled form of the library and properly linking it. I think we're getting close to asking python to be more than it is.

If you could open a ticket with all the info that would be excellent! So we can track this

yup, will link here when I get to it. Really appreciate all the thorough look at what turned into a scratchpad of a thread.

One thing that will make things better is, we're working on support for "grouping" processes into a single invocation. Today basically a process such as pytest or pylint either runs on a single source file or all of them. With the obvious tradeoff between cache utilization and concurrency, vs. execution overhead. We're working on supporting intermediate batch sizes.

I like the batching idea. I could see collecting a group of tests into the same process and maintaining speed by running them in parallel internal to the pants process with

pytest-xdist

. You still only need to collect/batch what has changed according to the dependency graph. Feels like "advanced option" territory, but maybe more tractable than handling the general problem. It's definitely fair to suggest that we don't architect code that has such a heavy dependency tree.

yup, will link here when I get to it.

Just an aside: since we're on Slack's free tier, messages disappear from here relatively quickly. So it's best to recap conversation when opening a Pants issue, rather than just linking.

Ah good to know. I linked the thread in the description as additional context so I should probably remove that. The context in the ticket should be self sufficient.

Thanks!

One big missing piece of context is Pants version. Much of what has been theorized about pyc compilation & caching is invalid depending on the Pants version and it's configuration.

Thanks for the reminder, I'll go update.

I'll have to look into things more deeply, but

venv_use_symlinks

here: https://www.pantsbuild.org/docs/reference-pex#advanced-options should make 3rdparty (psychopg?) compilation fully shared leaving only 1st party code dealt with via

PEX_EXTRA_SYS_PATH

internally by Pants pytest setup suspect. IOW your 1st party code compilation time would have to be large assuming it is still added to the

sys.path

for pytest via

PEX_EXTRA_SYS_PATH

.

Yes, sorry, the psycgop2 comment earlier seemed promising, but I actually think it's our first party code (specifically when we hit one of our modules because it ends up having a huge number of transitive dependencies).