Is there anyway to know the external dependencies ahead of time so we can take care of those requirements w/o Pants trying to access internet?

Yes, run

./pants help tools

. You'll get information about each tool we use and either the Python requirements that we grab the tool via or else the URLs we fetch the tool from.

Ah, at the moment we only have 2.7.1 available internally.

Pants tries very hard to ignore your environment and be fully hermetic so a build that passes on your machine passes on mine. Anything installed in a venv used to run Pants is - hopefully - ignored. Except for the python interpreter and its stdlib.

Gotcha. That explains why even after pip installing, it still went through the entire process again (and forced us to configure it to use internal Artifactory/SSL). So where should I place these binaries it does not fetch from PyPI if the only option is to retrieve them directly myself (not sure how esay it will be for us to host binaries in a downloadable location, unless a network file location would work).

Yeah - you must provide a shared filesystem or an http server if you can't get these from the internet. Each tool will need to be (re)configured to point at that internal location.

Ok this is helpful. I can fetch all of these from my desktop and then place them on an accessible network drive Pants (running on a Linux host) will have access to. When you say configure each tool, is that in pants.toml and which field am I setting (don't have access to the docs rn, but I think it's in the tools or 3rd part deps section?)

So, lets take

download-pex-bin

for example. You'd then ask for

./pants help-advanced

for that tool:

$ ./pants help-advanced download-pex-bin

`download-pex-bin` subsystem options
------------------------------------

The PEX (Python EXecutable) tool (<https://github.com/pantsbuild/pex>).
 
Config section: [download-pex-bin]
 
None available.

`download-pex-bin` subsystem advanced options
---------------------------------------------
 
  --download-pex-bin-version=<str>
  PANTS_DOWNLOAD_PEX_BIN_VERSION
  version
      default: v2.1.54
      current value: v2.1.54
      Use this version of pex.
      
      Supported pex versions: >=2.1.51,<3.0

  --download-pex-bin-known-versions="['<str>', '<str>', ...]"
  PANTS_DOWNLOAD_PEX_BIN_KNOWN_VERSIONS
  known_versions
      default: [
          "v2.1.54|macos_arm64|c892c82961f73e41aaafad3c692cfd51f6013bed2f7b408041be6c1b90d06451|3679238",
          "v2.1.54|macos_x86_64|c892c82961f73e41aaafad3c692cfd51f6013bed2f7b408041be6c1b90d06451|3679238",
          "v2.1.54|linux_x86_64|c892c82961f73e41aaafad3c692cfd51f6013bed2f7b408041be6c1b90d06451|3679238"
      ]
      current value: [
          "v2.1.54|macos_arm64|c892c82961f73e41aaafad3c692cfd51f6013bed2f7b408041be6c1b90d06451|3679238",
          "v2.1.54|macos_x86_64|c892c82961f73e41aaafad3c692cfd51f6013bed2f7b408041be6c1b90d06451|3679238",
          "v2.1.54|linux_x86_64|c892c82961f73e41aaafad3c692cfd51f6013bed2f7b408041be6c1b90d06451|3679238"
      ]
      
      Known versions to verify downloads against.
      
      Each element is a pipe-separated string of `version|platform|sha256|length`, where:
      
        - `version` is the version string
        - `platform` is one of [linux_arm64,linux_x86_64,macos_arm64,macos_x86_64],
        - `sha256` is the 64-character hex representation of the expected sha256
          digest of the download file, as emitted by `shasum -a 256`
        - `length` is the expected length of the download file in bytes, as emitted by
          `wc -c`
      
      E.g., `3.1.2|macos_x86_64|6d0f18cd84b918c7b3edd0203e75569e0c7caecb1367bbbe409b44e28514f5be|42813`.
      
      Values are space-stripped, so pipes can be indented for readability if necessary.

  --download-pex-bin-use-unsupported-version=<UnsupportedVersionUsage>
  PANTS_DOWNLOAD_PEX_BIN_USE_UNSUPPORTED_VERSION
  use_unsupported_version
      one of: [error, warning]
      default: error
      current value: error
      
      What action to take in case the requested version of pex is not supported.
      
      Supported pex versions: >=2.1.51,<3.0

  --download-pex-bin-url-template=<str>
  PANTS_DOWNLOAD_PEX_BIN_URL_TEMPLATE
  url_template
      default: <https://github.com/pantsbuild/pex/releases/download/{version}/pex>
      current value: <https://github.com/pantsbuild/pex/releases/download/{version}/pex>
      URL to download the tool, either as a single binary file or a compressed file (e.g. zip file). You can change this to point to your own hosted file, e.g. to work with proxies or for access via the filesystem through a `file:$abspath` URL (e.g.
      `file:/this/is/absolute`, possibly by [templating the buildroot in a config file](<https://www.pantsbuild.org/v2.9/docs/options#config-file-entries))>.
      
      Use `{version}` to have the value from --version substituted, and `{platform}` to have a value from --url-platform-mapping substituted in, depending on the current platform. For example, <https://github.com/.../protoc-{version}-{platform}.zip>.

  --download-pex-bin-url-platform-mapping="{'key1': val1, 'key2': val2, ...}"
  PANTS_DOWNLOAD_PEX_BIN_URL_PLATFORM_MAPPING
  url_platform_mapping
      default: {}
      current value: {}
      A dictionary mapping platforms to strings to be used when generating the URL to download the tool.
      
      In --url-template, anytime the `{platform}` string is used, Pants will determine the current platform, and substitute `{platform}` with the respective value from your dictionary.
      
      For example, if you define `{"macos_x86_64": "apple-darwin", "linux_x86_64": "unknown-linux"}, and run Pants on Linux with an intel architecture, then `{platform}` will be substituted in the --url-template option with unknown-linux.


`download-pex-bin` subsystem deprecated options
-----------------------------------------------
 
None available.

That gives you the details of which options to set.

well, assuming we move forward w/ pants as our main build automation tool (and get our firm to stop blocking humbug as a dependency), would be happy to!

That would be an awesome feature!

Hrm, the humbug thing is unfortunate since its opt-in.

I tried to implement something like it in Pants v1 years ago, but never got it quite over the finish line (this sort of thing is much easier to do in v2)

yah so the pants wheel is on our artifactory, but when trying to install it gets blocked by humbug dependency not being there because of some tbd issue with the scanner

We can finesse the humbug issue if we must

i'll report back here what the reason for them rejecting humbug is

we have a good relationship with the humbug folks, so if it requires changes on their end I think we can make those happen

Ok. Well, it would definitely make sense for the pants script to use

pantsbuild.pants[telemetry]

by default, but allow you to opt out and just have it use

pantsbuild.pants

. We don't have

humbug

set up as a

telemetry

extra today, but we could structure it that way if this is a (common) problem.

let me see right now if i can find out what our xray blocked it, not sure if it's license, security, unsupported dependency, or what

Just to be clear for other readers, humbug is an unconditional dependency right now, you can't opt out of that yet, but the telemetry it provides is opt-in. We nag you to add a pants.toml config to turn it on or off explicitly, but its off implicitly by defaqult.

I.e., if you don't opt in, humbug code is inert, but still present.

Good news, my request to get humbug cleared was approved because I was able to clean reinstall Pants w/ no issues. And I got it to use a file:// url for Pex. Hung up on another issue now though, will post a different thread.

You're welcome! Check back any time if we can be helpful.

we should probably start a new thread about your original screenshot though… we should definitely never download the same file multiple times in parallel!

I don't think we are - I think that's the dynamic UI not rendering correctly. Look at the timestamps

oh, it’s being smeared downward… got it. @nice-florist-55958: do you commonly see the UI smearing like that? we attempt to disable it in locations without a real TTY, but it looks like we failed in this case

@witty-crayon-22786 So far only when trying to download a binary on the internet like this. This is using PyCharm Professional’s terminal on Linux. Didn’t think to mention it, but it goes wild for a bit spamming the download attempt message and flickering. Also doesn’t gracefully exit after a keyboard interrupt until after several minutes go by. Btw at the office now and was able to successfully deploy pants on production CI servers! Will experiment with some workflows (including multiplatform) and see how it goes.

ah, got it. sorry for the trouble… you can disable the UI with

--no-dynamic-ui

, but i’ll open an issue about that. EDIT: https://github.com/pantsbuild/pants/issues/13595