So, lets take

download-pex-bin

for example. You'd then ask for

./pants help-advanced

for that tool:

$ ./pants help-advanced download-pex-bin

`download-pex-bin` subsystem options
------------------------------------

The PEX (Python EXecutable) tool (<https://github.com/pantsbuild/pex>).
 
Config section: [download-pex-bin]
 
None available.

`download-pex-bin` subsystem advanced options
---------------------------------------------
 
  --download-pex-bin-version=<str>
  PANTS_DOWNLOAD_PEX_BIN_VERSION
  version
      default: v2.1.54
      current value: v2.1.54
      Use this version of pex.
      
      Supported pex versions: >=2.1.51,<3.0

  --download-pex-bin-known-versions="['<str>', '<str>', ...]"
  PANTS_DOWNLOAD_PEX_BIN_KNOWN_VERSIONS
  known_versions
      default: [
          "v2.1.54|macos_arm64|c892c82961f73e41aaafad3c692cfd51f6013bed2f7b408041be6c1b90d06451|3679238",
          "v2.1.54|macos_x86_64|c892c82961f73e41aaafad3c692cfd51f6013bed2f7b408041be6c1b90d06451|3679238",
          "v2.1.54|linux_x86_64|c892c82961f73e41aaafad3c692cfd51f6013bed2f7b408041be6c1b90d06451|3679238"
      ]
      current value: [
          "v2.1.54|macos_arm64|c892c82961f73e41aaafad3c692cfd51f6013bed2f7b408041be6c1b90d06451|3679238",
          "v2.1.54|macos_x86_64|c892c82961f73e41aaafad3c692cfd51f6013bed2f7b408041be6c1b90d06451|3679238",
          "v2.1.54|linux_x86_64|c892c82961f73e41aaafad3c692cfd51f6013bed2f7b408041be6c1b90d06451|3679238"
      ]
      
      Known versions to verify downloads against.
      
      Each element is a pipe-separated string of `version|platform|sha256|length`, where:
      
        - `version` is the version string
        - `platform` is one of [linux_arm64,linux_x86_64,macos_arm64,macos_x86_64],
        - `sha256` is the 64-character hex representation of the expected sha256
          digest of the download file, as emitted by `shasum -a 256`
        - `length` is the expected length of the download file in bytes, as emitted by
          `wc -c`
      
      E.g., `3.1.2|macos_x86_64|6d0f18cd84b918c7b3edd0203e75569e0c7caecb1367bbbe409b44e28514f5be|42813`.
      
      Values are space-stripped, so pipes can be indented for readability if necessary.

  --download-pex-bin-use-unsupported-version=<UnsupportedVersionUsage>
  PANTS_DOWNLOAD_PEX_BIN_USE_UNSUPPORTED_VERSION
  use_unsupported_version
      one of: [error, warning]
      default: error
      current value: error
      
      What action to take in case the requested version of pex is not supported.
      
      Supported pex versions: >=2.1.51,<3.0

  --download-pex-bin-url-template=<str>
  PANTS_DOWNLOAD_PEX_BIN_URL_TEMPLATE
  url_template
      default: <https://github.com/pantsbuild/pex/releases/download/{version}/pex>
      current value: <https://github.com/pantsbuild/pex/releases/download/{version}/pex>
      URL to download the tool, either as a single binary file or a compressed file (e.g. zip file). You can change this to point to your own hosted file, e.g. to work with proxies or for access via the filesystem through a `file:$abspath` URL (e.g.
      `file:/this/is/absolute`, possibly by [templating the buildroot in a config file](<https://www.pantsbuild.org/v2.9/docs/options#config-file-entries))>.
      
      Use `{version}` to have the value from --version substituted, and `{platform}` to have a value from --url-platform-mapping substituted in, depending on the current platform. For example, <https://github.com/.../protoc-{version}-{platform}.zip>.

  --download-pex-bin-url-platform-mapping="{'key1': val1, 'key2': val2, ...}"
  PANTS_DOWNLOAD_PEX_BIN_URL_PLATFORM_MAPPING
  url_platform_mapping
      default: {}
      current value: {}
      A dictionary mapping platforms to strings to be used when generating the URL to download the tool.
      
      In --url-template, anytime the `{platform}` string is used, Pants will determine the current platform, and substitute `{platform}` with the respective value from your dictionary.
      
      For example, if you define `{"macos_x86_64": "apple-darwin", "linux_x86_64": "unknown-linux"}, and run Pants on Linux with an intel architecture, then `{platform}` will be substituted in the --url-template option with unknown-linux.


`download-pex-bin` subsystem deprecated options
-----------------------------------------------
 
None available.

That gives you the details of which options to set.

Yeah - "configure each tool" means setting options in pants.toml.

It would be really useful if we had a `./pants offline tools`goal wouldn't it? That would maybe look like:

./pants offline tools --dir=/download/here/please

and that would both download all tools known and output the configuration you'd need to add to pants.toml to point to those tools.

If you agree, it would mean alot to have that feature request come from a user with your own details about how this would best work for you. If you're willing to file a GitHub issue, that would be great.

well, assuming we move forward w/ pants as our main build automation tool (and get our firm to stop blocking humbug as a dependency), would be happy to!

agree it makes a lot of sense for users in a commercial environment where there's no internet access

That would be an awesome feature!

Hrm, the humbug thing is unfortunate since its opt-in.

I tried to implement something like it in Pants v1 years ago, but never got it quite over the finish line (this sort of thing is much easier to do in v2)

yah so the pants wheel is on our artifactory, but when trying to install it gets blocked by humbug dependency not being there because of some tbd issue with the scanner

i got around it, but it's hard hwen pants wants to bootstrap itself

and impossible in a Jenkins job, so we can only test locally for now

We can finesse the humbug issue if we must

i'll report back here what the reason for them rejecting humbug is

if it's not serious, i don't think it will matterin the long-run

we have a good relationship with the humbug folks, so if it requires changes on their end I think we can make those happen

Ok. Well, it would definitely make sense for the pants script to use

pantsbuild.pants[telemetry]

by default, but allow you to opt out and just have it use

pantsbuild.pants

. We don't have

humbug

set up as a

telemetry

extra today, but we could structure it that way if this is a (common) problem.

let me see right now if i can find out what our xray blocked it, not sure if it's license, security, unsupported dependency, or what

why*

Just to be clear for other readers, humbug is an unconditional dependency right now, you can't opt out of that yet, but the telemetry it provides is opt-in. We nag you to add a pants.toml config to turn it on or off explicitly, but its off implicitly by defaqult.

I.e., if you don't opt in, humbug code is inert, but still present.

Good news, my request to get humbug cleared was approved because I was able to clean reinstall Pants w/ no issues. And I got it to use a file:// url for Pex. Hung up on another issue now though, will post a different thread.

That means we have a much higher chance now of being able to use Pants running from arbitrary build hosts/uids 🙂

And sry the other "issue" is building the black.pex from black_default_lockfile.txt taking awhile, wasn't sure if it was another connection about to timeout

Just ran black on the entire proto repo: "Oh no! 8 files would be reformatted, 99 files would be left unchanged." Feels good!

Thx for all your help guys, I can submit that feature request on Github tomorrow

You're welcome! Check back any time if we can be helpful.

I'm getting a

No module named 'pkg_resources'

error when I include

mongomock

in my

thirdparty/python_test/requirements.txt

and depending on it from another BUILD file like so:

python_tests(
    name="tests",
    sources = ["test/test_*.py", "test/conftest.py"],
    dependencies=[":fixtures", "thirdparty/python:pymongo", "thirdparty/python_test:mongomock"]
)

The usual fix for this error from what I can see is updating setuptools... But I'm not having this error outside of Pants/Pex so not sure how to fix this in Pants. Any ideas?

we should probably start a new thread about your original screenshot though… we should definitely never download the same file multiple times in parallel!

I don't think we are - I think that's the dynamic UI not rendering correctly. Look at the timestamps