very random question: did pants2 ever run into the pypi total package size problem? (the sum of the size of all uploaded releases has a 10GB limit by default)

Yes, we did, and we had increased our limit. For instance: https://github.com/pypi/support/issues/1376 This limit was one of the motivations for switching to the new GitHub Releases distribution mechanism.

if you don't mind me asking. how is release done these days (I haven't kept up with the latest news in pants)? 😅 our rust-python package is running into a very similar problem recently.

We publish wheels and executable pex files (with the pants code and all dependencies) to GitHub Releases, for instance: https://github.com/pantsbuild/pants/releases/tag/release_2.20.0 The scie-pants launcher knows how to download the pex executables and run them, and we have https://github.com/pantsbuild/wheels.pantsbuild.org that maintains “cheese shop” of links to the wheels for the use cases that need that. Is your package an executable or a library?

we are a library. https://github.com/lancedb/lance

Ah… a dynamic download during pip install might be a bit annoying (eg makes proxying or offline/hermetic installs harder). I know large ML packages like PyTorch will publish to their own index and expect people to pass

--index-url

(https://pytorch.org/get-started/locally/ ). Pants does something slightly simpler than needing to run a whole index with that wheels.pantsbuild.org repo: using GitHub pages to publish a huge page of URLs (pointing to the wheels in GitHub releases) and using

--find-links

nice, that github page removes the need of third party index. totally gonna borrow that idea haha

Btw, tweaking build settings made a big difference to pants’ wheel size: https://github.com/pantsbuild/pants/pull/18838#discussion_r1179486485 But maybe you’re already on top of that 😄