#16328 Using Pants in a restricted network access environment is non-trivial and not usefully documented. Issue created by jsirois As the result of a a User asking how to set up Pants to work in their environment which is network-locked-down with Artifactory as the only source for artifacts, I went through a full simulation using the example-python. There are likely further quirks for other backends Pants supports, but in this case the question concerned Python in particular. At the least this needs much more thorough documentation. At the best there is some automation and / or wizard-driven aid to make setup easier. The current docs: • https://www.pantsbuild.org/docs/restricted-internet-access • https://github.com/pantsbuild/pants/blob/main/docs/markdown/Using%20Pants/restricted-internet-access.md --- 1st I black-holed all relevant public network sources of artifacts for the Python backend by adding this line to `/etc/hosts`:

127.0.0.1	<http://files.pythonhosted.org|files.pythonhosted.org> <http://pypi.org|pypi.org> <http://github.com|github.com>

My aim was to get

./pants fmt lint check test ::

working. That failed of course and I went through alot of trial and error with insider knowledge and wound up with these changes: On the simulated corporate user side I had to modify Artifactory by making a generic repo and copying over the relevant Pex PEX release from https://github.com/pantsbuild/pex/releases/download/v2.1.90/pex. I just used the default Artifactory setup for proxying / mirroring PyPI. On the Pants side, I made these edits (The user was attempting with Pants 2.12.0 and the example-python repo was on 2.8.0 at the time): $ git diff pants pants.toml diff --git a/pants b/pants index 2e9a10c..759c580 100755 --- a/pants +++ b/pants @@ -34,9 +34,9 @@ fi PANTS_BOOTSTRAP="${PANTS_SETUP_CACHE}/bootstrap-$(uname -s)-$(uname -m)" -PEX_VERSION=2.1.42 -PEX_URL="[https://github.com/pantsbuild/pex/releases/download/v${PEX_VERSION}/pex](https://github.com/pantsbuild/pex/releases/download/v$%7BPEX_VERSION%7D/pex)" -PEX_EXPECTED_SHA256="69d6b1b1009b00dd14a3a9f19b72cff818a713ca44b3186c9b12074b2a31e51f" +PEX_VERSION=2.1.90 +PEX_URL="[https://pextest.jfrog.io/artifactory/default-generic//pantsbuild/pex/releases/download/v${PEX_VERSION}/pex](https://pextest.jfrog.io/artifactory/default-generic//pantsbuild/pex/releases/download/v$%7BPEX_VERSION%7D/pex)" +PEX_EXPECTED_SHA256="2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66" VIRTUALENV_VERSION=20.4.7 VIRTUALENV_REQUIREMENTS=$( @@ -244,7 +244,7 @@ function bootstrap_pex { local staging_dir staging_dir=$(tempdir "${PANTS_BOOTSTRAP}") cd "${staging_dir}" - curl -LO "${PEX_URL}" + curl -n -LO "${PEX_URL}" fingerprint="$(compute_sha256 "${python}" "pex")" if [[ "${PEX_EXPECTED_SHA256}" != "${fingerprint}" ]]; then die "SHA256 of ${PEX_URL} is not as expected. Aborting." @@ -270,7 +270,9 @@ function bootstrap_virtualenv { staging_dir=$(tempdir "${PANTS_BOOTSTRAP}") cd "${staging_dir}" echo "${VIRTUALENV_REQUIREMENTS}" > requirements.txt - "${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex + "${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex \ + --no-pypi \ + --index https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/ mkdir -p "$(dirname "${bootstrapped}")" mv -f "${staging_dir}/virtualenv.pex" "${bootstrapped}" rm -rf "${staging_dir}" diff --git a/pants.toml b/pants.toml index c62c9d4..8cc2ee7 100644 --- a/pants.toml +++ b/pants.toml @@ -2,7 +2,7 @@ # Licensed under the Apache License, Version 2.0 (see LICENSE). [GLOBAL] -pants_version = "2.8.0" +pants_version = "2.12.0" backend_packages.add = [ "pants.backend.python", "pants.backend.python.lint.docformatter", @@ -27,8 +27,36 @@ root_patterns = ["/"] interpreter_constraints = [">=3.7"] # Use a constraints file. See https://www.pantsbuild.org/docs/python-third-party-dependencies. requirement_constraints = "constraints.txt" + +[python-bootstrap] # We search for interpreters on both on the $PATH and in the

$(pyenv root)/versions

folder. # If you're using macOS, you may want to leave off the <PATH> entry to avoid using the # problematic system Pythons. See # https://www.pantsbuild.org/docs/python-interpreter-compatibility#changing-the-interpreter-search-path. -interpreter_search_paths = ["<PATH>", "<PYENV>"] +search_path = ["<PATH>", "<PYENV>"] + +[pex-cli] +url_template = "https://%(env.ARTIFACTORY_USER)s:%(env.ARTIFACTORY_PASS)s@pextest.jfrog.io/artifactory/default-generic/pantsbuild/pex/releases/download/{version}/pex" + +[python-repos] +indexes = [ + "https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/", +] + +[black] +lockfile = "tools/black.lock" + +# Work around https://github.com/psf/black/issues/2964 +extra_requirements = "click==8.0.4" + +[pytest] +lockfile = "tools/pytest.lock" + +[isort] +lockfile = "tools/isort.lock" + +[flake8] +lockfile = "tools/flake8.lock" + +[mypy] +lockfile = "tools/mypy.lock" With all that set up, bootstrapping Pants required a 1-time:

$ PIP_INDEX_URL=<https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/> ./pants

And then I could re-gen lockfiles to pick up Artifactory urls:

$ ./pants generate-lockfiles
09:17:52.68 [INFO] Completed: Generate lockfile for isort
09:17:55.46 [INFO] Completed: Generate lockfile for flake8
09:17:56.13 [INFO] Completed: Generate lockfile for mypy
09:17:56.86 [INFO] Completed: Generate lockfile for black
09:18:09.91 [INFO] Completed: Generate lockfile for pytest
09:18:09.91 [INFO] Wrote lockfile for the resolve `pytest` to tools/pytest.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `black` to tools/black.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `flake8` to tools/flake8.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `isort` to tools/isort.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `mypy` to tools/mypy.lock

And, finally, I could achieve the initial goal:

$ ./pants fmt lint check test ::
09:18:32.32 [INFO] Completed: Format with docformatter - docformatter made no changes.
09:18:32.32 [INFO] Completed: Format with Black - black made no changes.
09:18:32.32 [INFO] Completed: Format with isort - isort made no changes.

✓ black made no changes.
✓ docformatter made no changes.
✓ isort made no changes.
09:18:32.32 [INFO] Completed: Lint with Flake8 - flake8 succeeded.

✓ black succeeded.
✓ docformatter succeeded.
✓ flake8 succeeded.
✓ isort succeeded.
09:18:32.33 [INFO] Completed: Typecheck using MyPy - mypy succeeded.
Success: no issues found in 8 source files


✓ mypy succeeded.
09:18:32.33 [INFO] Completed: Run Pytest - helloworld/greet/greeting_test.py:tests succeeded.
09:18:32.33 [INFO] Completed: Run Pytest - helloworld/translator/translator_test.py:tests succeeded.

✓ helloworld/greet/greeting_test.py:tests succeeded in 0.21s (memoized).
✓ helloworld/translator/translator_test.py:tests succeeded in 0.16s (memoized).

pantsbuild/pants