https://pantsbuild.org/ logo
#general
Title
# general
h

hundreds-carpet-28072

02/14/2023, 1:41 PM
Is there a good alternative to targeting
pip.conf
for Pants rather than using
.pants.rc
? I understand that
pip.conf
isn’t supported because of PEX but is there a clean workaround?
e

enough-analyst-54434

02/14/2023, 2:33 PM
What do you want from pip.conf?
h

hundreds-carpet-28072

02/14/2023, 3:30 PM
Looking to use it for specifying extra PyPi indices with relevant creds being stored in a
.netrc
file
e

enough-analyst-54434

02/14/2023, 3:41 PM
Extra indices here: https://www.pantsbuild.org/docs/reference-python-repos#indexes And
~/.netrc
should just work by default.
The syntax is documented but nonetheless trips a lot of folks up. You want:
Copy code
[python-repos]
indexes.add = ["add", "these", "5", "to", "PyPI"]
Assuming you still want PyPI from the default list.
h

hundreds-carpet-28072

02/14/2023, 5:10 PM
That is what we already have, what I’m looking for is getting the indices from pip.conf rather than Pants configs (pants.toml, pants.rc etc.)
f

fresh-cat-90827

02/14/2023, 5:31 PM
hi Danny, I think this deserves an issue on GitHub! 🙂 it is a fairly common request from what I heard. The idea is that you have devs with configuration files that have info on custom PyPI indices (internal ones) which they may use when invoking
pip
outside of Pants repo. Then there's Pants configuration that also needs to be kept up-to-date with the PyPI index URLs. What Danny wants is to be able to ask Pants to load the
pip.conf
and the like into the
pants.toml
instead of duplicating the URLs. This is not currently possible, i.e. we cannot load configuration for
python-repos
from a file. However, we do this in a couple of places such as this one
using the
fromfile
notation with
@
. I guess we could explore doing this for the
python-repos
as well.
e

enough-analyst-54434

02/14/2023, 6:08 PM
Yeah, so Pex could grow an option to not explicitly ignore Pip conf (which it has explicitly ignored since day 1 of switching to Pip as its internal resolver). That would be tricky, but doable - tricky just because Pex needs to control certain Pip options and would need to yell at you with a warning when it needed to ignore some of your settings; so Pex then needs to learn how to find and parse various generations of pip.conf settings. That aside - pretend Pex has that implemented today - relying on any file not controlled by Pants is a problem. With auth probably the least so, but with extra indexes, imagine this: 1. You use remote caching. 2. You add an extra index to your personal pip.conf locally. Unless Pants tracks your personal pip.conf, it will blithely retrieve old (should be invalid now) cache hits. This could confuse the heck out of you.
Really, you don't need 1 even. Local caching will have the same issue. It's just easier to fix: restart pantsd or blow away your local Pants LMDB cache. With remoting you need to turn off the remoting configuration which is probably a non-starter. That remote cache result is effectively there for good without contacting someone who controls the remote cache.
f

fresh-cat-90827

02/15/2023, 6:00 AM
Unless I have misinterpreted the needs of Danny, there's no need to involve Pex here at all. We'd just have to extend how the
python-repos
indices are collected using the
fromfile
syntax just as we do in a couple of other subsystems. Pex would still be given what's coming from the
[python-repos].indexes
h

hundreds-carpet-28072

02/15/2023, 10:01 AM
Unless I have misinterpreted the needs of Danny, there's no need to involve Pex here at all. We'd just have to extend how the
python-repos
indices are collected using the
fromfile
syntax just as we do in a couple of other subsystems. Pex would still be given what's coming from the
[python-repos].indexes
This is correct and what I would be looking to do. Thanks for explaining my case better than I did 😅 I’m happy to create an issue on GitHub for this. @enough-analyst-54434 Would the Pants cache still be an issue for the above? Is so, I’m assuming there isn’t a ignored-by-cache function so that specific files wouldn’t be cached ?
e

enough-analyst-54434

02/15/2023, 3:25 PM
It could be. @fresh-cat-90827 is more current on Pants development than I am at the moment. Last I recalled fromfile just read a file from a known location with no other work done. Your use case would require a variable location (IIRC pip.conf is under ~/Library/Caches/ on Mac and ~/.cache on Linux) and the ability not just to slurp the file but parse it as well to get at the desired index information. To my knowledge we have never done these sorts of gymnastics in the options parsing stage. If so, @fresh-cat-90827 can point these example instances out on the ticket you file (more on that below). If not, this sort of heavy lifting is traditionally done later in @rule code run by the engine after options are parsed. That style code does need to deal with tracking the file contents or at the very least re-reading it every time. Again, to my stale knowledge, we don't do this for files outside the workspace. All this aside, definitely file an issue to track the use case where the details of if and how to implement can be hashed out. It would be good to eliminate this missed expectation and non-DRY it forces on Pants users as Alexey says.
4 Views