Is this the best way I should be doing this?

ansible_pex = await Get(
        Pex,
        PexRequest,
        ansible.to_pex_request(),
    )

This takes about 200 seconds on my macbook, while pip installing ansible is about 45 seconds. I know this is doing more than just installing, but I feel like maybe I'm missing an optimization

You may want to use

VenvPex

. See https://github.com/pantsbuild/pants/blob/8e7aa65c7b3f51f271d6bbb365d22a8fd957dae7/src/python/pants/backend/terraform/dependency_inference.py#L80-L86.

Derives from

PythonToolBase

- I think I basically copied what I did for PyOxidizer

ah TIL

PythonToolBase

derives from

PythonToolRequirementsBase

I'll try VenvPex - and see if that speeds anything up

FWIW pip installing ansible on my macbook takes 140-180 seconds (without Pants involved)

Wow, that feels slow. But, how bad is it through Pants, is the question. 🙂

maybe prebuild the ansible sdists into wheels and store them somewhere? also, why wouldn’t ansible provide wheels?

Do you really need

ansible

? I recommend relying on

ansible-core

for almost everything.

ansible

includes around 100 collections which takes awhile to download and install. It is much better to install

ansible-core

and just the collections you need.

@proud-dentist-22844

ansible

is core + community, correct? Does

core

come with playbooks? The goal here is to streamline end user experience, without re-building too much of what already exists

Rather,

ansible

is core + some Collections from community?

where some = about 100 😉 (quite a bit more than a bunch) There are so many collections, I prefer to get just the collections I depend on on a case by case basis.

Yeah, it's a good call - I was reviewing those 99ish today. Feel like that has creeped substantially over time. We'll need a way to pull in collections anyways, so there definitely isn't any benefit by pulling in

ansible

also, as a general point: PEX uses

pip

under the hood, so in general it should take roughly the same time. some factors that can make it different: 1. cold caches: PEX puts the pip cache in a non-default location (so that we can monitor usage, and more easily clear it) 2. building the PEX for more interpreters than you need it for. for a PEX that you won’t be publishing anywhere, you should ensure that you’re building an “internal_only” PEX: it will only work on a single python interpreter selected by the Pants process, and that avoids building wheels for other interpreters if user’s interpreter constraints are more broad

Thanks @witty-crayon-22786 I was using

internal=True

which is a big reason for my confusion. I didn't know about the cold caches though, that's helpful. Either way, with the switch to

ansible-core

, we're down to like 10 seconds of build time 🚀

I’ll take a look later 🙂