:wave: having an issue (i think) passing http_prox...
# general
👋 having an issue (i think) passing http_proxy environment to the pip resolver when pants is bootstrapping / resolving plugins in pants 1.26 on a build machine that requires a proxy to access the internet (full logs with -ltrace and PEX_VERBOSE=9 in thread):
Copy code
10:56:38 15:56:38 [INFO] Resolving new plugins...:
10:56:38   pantsbuild.pants.contrib.avro==1.26.0
10:56:38   pantsbuild.pants.contrib.googlejavaformat==1.26.0
10:56:38   pantsbuild.pants.contrib.python.checks==1.26.0
10:56:38   pantsbuild.pants.contrib.scrooge==1.26.0
10:56:38   pantsbuild.pants.contrib.thrifty==1.26.0
10:56:38 pex: Resolving for:
10:56:39 Starting new HTTPS connection (1): <http://pypi.org:443|pypi.org:443>
10:57:39 Incremented Retry for (url='/simple/pantsbuild-pants-contrib-avro/'): Retry(total=4, connect=None, read=None, redirect=None, status=None)
10:57:39 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fc21492b080>, 'Connection to <http://pypi.org|pypi.org> timed out. (connect timeout=15)')': /simple/pantsbuild-pants-contrib-avro/
Hey John! To confirm, are you all using v1 or v2 for 1.26? If v2, you would have
in your
and the UI would be the new "dynamic UI", where the screen refreshes
this is pants v1 version 1.26
👍 1
Cool, thanks. With v2, you have to tell Pants to propagate your
value because v2 is more hermetic, but that should be irrelevant here. I think we leak
to the underlying subprocesses Is this your first time running it on this machine, or this is a regression?
To confirm, you do intend to pull the wheel for
from pypi.org? That is, you're not hosting it on a custom artifactory/cheeseshop?
correct from pypi.org
this is the first time i'm running this version of pants on this machine that requires the proxy. running the same command on a similar machine that does not require a proxy works fine
and, interestingly, the initial pants boostrap, which i believe also downloads deps from the internet, works just fine, it seems to just be the the plugin resolver
Hm okay. I'm rereading our v1 docs on proxies at https://v1.pantsbuild.org/setup_repo.html, and indeed I'd expect only needing to set
. Does the
part look relevant to you?
i don't believe I'd get the ConnectTimeoutError if that were the issue, but I can try
Also, this is unsatisfying, but in the worst case, you could copy the plugin files and treat them as a first-party plugin rather than resolving via PyPI https://github.com/pantsbuild/pants/tree/1.26.x/contrib/avro/src/python/pants/contrib/avro Lmk if you want to take that shortcut and we can help with how to structure things / tweak
thanks! it's weird, i don't see where http_proxy environment variables get dropped either. i attemped to follow the code from pants -> pex -> pip that gets triggered when resolving plugins. i see code for removing PEX-prefixed env variables, etc, but I don't see where
gets dropped
It is weird, I'm trying to retrace https://github.com/pantsbuild/pants/blob/1.26.x/src/python/pants/init/plugin_resolver.py and can't figure it out either. There, we use Pex as a library. But that should not be dropping any env vars outside of some special ones like
If you want to go with that first-party plugin workaround, check out https://www.pantsbuild.org/docs/plugins-overview#locating-plugin-code In GitHub for pantsbuild/pants, that plugin path is super long so that the source roots work how we want. You can simplify it, e.g. use
, and then in that folder copy everything from https://github.com/pantsbuild/pants/tree/1.26.x/contrib/avro/src/python/pants/contrib/avro Then, set
, and under
, activate
instead of
. Remove the
If you're still using
, you may need to make some tweaks from those docs because it assumes
(added in 1.26)
thanks, i'll have a look into that we may need to anyway give that the avro isn't developed first-party anymore
i believe the plugin download issues are a regression from pants 1.25 though
i believe the plugin download issues are a regression from pants 1.25 though
It's plausible, 1.26 switched to using Pex 2, which vendors pip to do resolves instead of using a custom resolver. Even though that generally made things much more robust, there were some hiccups
give that the avro isn't developed first-party anymore
Indeed. On that note, let us know if you issues upgrading past 1.28 with that plugin! We would be happy to help y'all figure out the changes to get it to 1.30, the last v1 release
(And very happy to help rewrite it to the v2 engine if y'all are interested in upgrading to Pants 2.x. That should make your Python experience much better It's possible to use Pants 2.x for the Python portion of your repo, and Pants 1.x for other languages. "Mixed mode")
ah interesting. its primarily a scala repo, with a few python binaries kicking around. appreciate all this info!
💯 1
i believe the plugin download issues are a regression from pants 1.25 though
Also, it may be worth testing if it's fixed in later Pants versions, as they consume newer versions of Pex. I don't think we had any dedicated bug for this that we fixed..but might have been fixed without us knowing. If you want to try, I'd recommend creating a new minimal repository with nothing more than the
script and a
that loads those plugins. Simply run
./pants --version
to see if resolving the plugins work. This way, you avoid dealing with deprecation warnings
🙇 1
👋 @bored-mechanic-51694 🙂
👋 hey @happy-kitchen-89482!
fwiw, i confirmed it is somehow a proxy issue, and not something else (like a cert issue). using
which is one of those LD_PRELOAD hacks, I was able to intercept all traffic and run it over the proxy, and that worked
👀 1
🙌 1