https://pantsbuild.org/ logo
Join Slack
Powered by
Hi all, I'm testing out pants on a repo at work, b...
# general
s
Hi all, I'm testing out pants on a repo at work, but my workstation doesn't have access to the internet. I'm blocked right now because pants is trying to download pex from github.com/pantsbuild/pex/releases/download and failing. We have an internal pypi proxy, so I can pip install pex, but I can't use curl, etc to download arbitrary files over http(s). Is there a way to force pants to use pex found on 
PATH
? Or can I manually download the pex release (we use an airgapped secure browser w/ download and virus scan capabilities) and place it in a known location for pants to find? We also have a docker proxy setup, so I also looked on dockerhub for a container that would hold all the required deps, but the only container there was 6 years old. For users like myself without internet access it'd be great to bundle everything up in an easy to use package. Thanks!
Ok, I may have found another thread here with some clues on how to do this: https://pantsbuild.slack.com/archives/C046T6T9U/p1636674907169600 I'll try that out and check back. Count this as another vote for either better documentation for how to operate pants offline, or improved offline behavior (e.g. a docker image or apptainer binary)
šŸ‘ 1
p
This is not the first time this question has came up. I think it is worth adding a page to the docs to describe how to work with pants from environment that limit access to the internet. I created an issue: https://github.com/pantsbuild/pants/issues/14343
āž• 1
h
Thanks Asher!
And thanks @stocky-yacht-20557 for the feedback
There are a currently 3 different ways Pants can fetch its own tools from the internet: 1) Standalone binaries are downloaded directly (and verified against an expected sha256) 2) Python tools are represented as a lockfile and resolved (from PyPI by default, but this can be overridden) 3) JVM tools are represented as a lockfile and resolved (from Maven Central by default, but this can be overridden)
So there is a straightforward path for 2) and 3), via custom proxies such as you already have
Pex is special though, because we use it to resolve, so even though it is a Python tool, we download it as a standalone binary
There are other standalone binaries. You may not care about those yet, but we should solve this problem for all of them.
I'll write something up
But meanwhile, 
./pants help-advanced download-pex-bin
(or its web equivalent) gives info on how to point pants at a custom source for that binary, via the 
[download-pex-bin].url_template
option
And that inspires me to make this change: https://github.com/pantsbuild/pants/pull/14344 so that the config section name makes more sense...
Anyway, let us know here if the above worked for you, and what the gotchas were, if any, as this will inform the documentation page I've now committed to writing... šŸ™‚
šŸ™Œ 1
w
sorry for missing this: but note that we do also have a page for this: https://www.pantsbuild.org/docs/proxies#setting-http_proxy-and-https_proxy
šŸ‘ 1
if something is missing (particularly a keyword to help find it!) then we should definitely improve that
āž• 1
h
Was this page not discoverable, or not useful?
s
@happy-kitchen-89482 @witty-crayon-22786 I saw that page, but I missed the part about hosting binaries. Maybe because the main page is called "Proxies" and I knew I couldn't setup a proxy to github, so I just scanned past it. Here are a few things I would recommend to make solutions more discoverable: ā€¢ when the download times out, print some help about how to configure the template or how to get help (
pants help-advanced <tool>
) ā€¢ rename the "Proxies" help page to "Using Pants without Internet" or similar ā€¢ add something to the above docs about hosting binaries locally on a shared server. i.e. using 
file://
I got past my initial problem, so I'm able to do further testing, btw. Thanks for everyone's enthusiastic response to this.
šŸ™Œ 1
šŸš€ 1
w
thanks for the feedback!
h
Glad to hear, and thanks for the advice!
@hundreds-father-404 After you cut the 2.10.x docs version I'll update things there
@stocky-yacht-20557 is hosting the binaries on an internal server an appropriate solution for you? Or is the expectation to be literally networkless?
I ask because I've been thinking of a "./pants offline" goal that eagerly downloads all the tooling and caches it locally, but that does require internet access at some point to do the downloading.
It addresses "working on an airplane" more than "working behind a firewall"
s
yeah, I can't turn the internet on and off at my workplace. I might be able to a CI job that could run such a command, then upload it somewhere internal. why not bundle all this up into a docker container?
I could easily 
docker pull pantsbuild
to pull such a container through our dockerhub proxy.
my 
./pants
script would look like 
docker run -it --rm pants:v2.9.0 -v $(pwd):/repo -W /repo
h
Interesting idea!
Here is a hopefully improved version of that page, with a more obvious title: https://www.pantsbuild.org/v2.10/docs/restricted-internet-access
Thoughts? Feedback?
h
Nice, good change! Maybe mention how to load binaries from the file system? 
url_template
already mentions it, but duplication can be good in docs: https://www.pantsbuild.org/v2.10/docs/reference-protoc#section-url-template
4 Views
Open in Slack
PreviousNext
Powered by