I have another question. I can't access pypi or any other public repo. Instead, we have artifactory which acts as a "repo proxy" + hosts private repos. I added a [python-repos] sections and a indexes entry with a list of URLs in pants.toml but I'm not sure pex honors it. When trying to run pytest, I get the following messages:

pex.result.ResultError: There were 10 errors downloading required artifacts:
1. pytest-cov 3 from <https://files.pythonhosted.org/packages/20/49/b3e0edec68d81846f519c602ac38af9db86e1e71275528b3e814ae236063/pytest_cov-3.0.0-py3-none-any.whl>
    <urlopen error [Errno 104] Connection reset by peer>
2. pytest 7.0.1 from <https://files.pythonhosted.org/packages/38/93/c7c0bd1e932b287fb948eb9ce5a3d6307c9fc619db1e199f8c8bc5dad95f/pytest-7.0.1-py3-none-any.whl>
    <urlopen error [Errno 104] Connection reset by peer>
3. coverage 6.3.2 from <https://files.pythonhosted.org/packages/60/a8/0f951285dff76cbedbbba89c1969586020150a3b53267b5d8674b2ead15a/coverage-6.3.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl>
    <urlopen error [Errno 104] Connection reset by peer>

Is there another option to set?

There's some relevant documentation here: https://www.pantsbuild.org/docs/restricted-internet-access

I followed that doc (I think), and set the python-repos:indexes option. I don't use lockfiles but I tried your command and retry, I got the same error. this is when doing a

./pants test ::

@ripe-cpu-85141 can you provide your

pants.toml

again with redactions as needed?

Sure

[GLOBAL]
pants_version = "2.12.0+git5d31cdb5"
backend_packages = [
    "pants.backend.python",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.flake8",
    "pants.backend.python.typecheck.mypy",
    "pants.backend.awslambda.python"
]
ca_certs_path = "/etc/pki/tls/certs/ca-bundle.crt"


[anonymous-telemetry]
enabled = false

[source]
#marker_filenames = ["pyproject.toml"]
#root_patterns = ["/"]

[python]
interpreter_constraints = ["CPython==3.9.*"]
tailor_ignore_solitary_init_files = false

[python-repos]
indexes = [
    "<https://artifactory.mycompany.net/artifactory/api/pypi/pypi/simple/>",
    "<https://artifactory.mycompany.net/artifactory/api/pypi/pd-its-pypi-local-csp/simple/>",
    "<https://artifactory.mycompany.net/artifactory/api/pypi/pd-its-pypi-local-linux/simple/>",
    "<https://artifactory.mycompany.net/artifactory/api/pypi/dv-its-pypi-local-tooling/simple/>",
]
repos = []

[test]
use_coverage = true

[pytest]
args = ["--import-mode=importlib"]

@ripe-cpu-85141 can you reflect exactly what you see? Full command line plus full logs would be very useful.

@enough-analyst-54434 So for the pants_version, I use the pex from the release page of github as I don't have full access to internet and the bash script has some limitations (it doesn't honors the Pypi repos on a call to pex). If I just put

pants_version = "2.12.0", pants complains that the version is not the right one

Aha, ok. Thanks for the logs, that's always very helpful. So the failure is resolving a default tool lockfile that ships with Pants, in this case, for pytest. For each Python tool Pants executes to achieve some goal (pytest for test, mypy for check, etc.) we ship a lockfile for the tool and its requirements. We generate that lockfile using a default setup, and so PyPI. So, unfortunately, I think this is going to be a bit of a pain and whac-a-mole. As you hit errors like this, you'll need to re-generate lock files for the internal tool in question. In this case, you'll need to configure

[pytest] lockfile = "some/path/of/your/choice.lock"

and then run `./pants generate-lockfiles --resolve=pytest`: https://www.pantsbuild.org/docs/reference-pytest#section-lockfile

ok, I'm not up-to-date about how to deal with lockfiles yet. That's on my list. Meanwhile, how should I deal with the lockfiles? Is it more a temporary file in my context or something I should add in my git repo?

Pants will generate these and you should commit them. It should be a one time operation per tool lockfile.

Ok, I just tested and it works. I now need to figure out some dependencies issue I see 🙂 Thank you both for your help!

Ah yes, I forgot about that part ☹️ - you need to tell Pants not to use the internal lockfile for each tool, as John has explained

You're welcome. I just finished this exercise for myself, and it is not pretty. I simulated a locked down network with this

/etc/hosts

entry:

127.0.0.1	<http://files.pythonhosted.org|files.pythonhosted.org> <http://pypi.org|pypi.org> <http://github.com|github.com>

I already have a custom Artifactory set up for testing. Then, over in example-python I had to:

$ git diff pants pants.toml
diff --git a/pants b/pants
index 2e9a10c..759c580 100755
--- a/pants
+++ b/pants
@@ -34,9 +34,9 @@ fi
 
 PANTS_BOOTSTRAP="${PANTS_SETUP_CACHE}/bootstrap-$(uname -s)-$(uname -m)"
 
-PEX_VERSION=2.1.42
-PEX_URL="<https://github.com/pantsbuild/pex/releases/download/v${PEX_VERSION}/pex>"
-PEX_EXPECTED_SHA256="69d6b1b1009b00dd14a3a9f19b72cff818a713ca44b3186c9b12074b2a31e51f"
+PEX_VERSION=2.1.90
+PEX_URL="<https://pextest.jfrog.io/artifactory/default-generic//pantsbuild/pex/releases/download/v${PEX_VERSION}/pex>"
+PEX_EXPECTED_SHA256="2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66"
 
 VIRTUALENV_VERSION=20.4.7
 VIRTUALENV_REQUIREMENTS=$(
@@ -244,7 +244,7 @@ function bootstrap_pex {
       local staging_dir
       staging_dir=$(tempdir "${PANTS_BOOTSTRAP}")
       cd "${staging_dir}"
-      curl -LO "${PEX_URL}"
+      curl -n -LO "${PEX_URL}"
       fingerprint="$(compute_sha256 "${python}" "pex")"
       if [[ "${PEX_EXPECTED_SHA256}" != "${fingerprint}" ]]; then
         die "SHA256 of ${PEX_URL} is not as expected. Aborting."
@@ -270,7 +270,9 @@ function bootstrap_virtualenv {
       staging_dir=$(tempdir "${PANTS_BOOTSTRAP}")
       cd "${staging_dir}"
       echo "${VIRTUALENV_REQUIREMENTS}" > requirements.txt
-      "${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex
+      "${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex \
+        --no-pypi \
+        --index <https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/>
       mkdir -p "$(dirname "${bootstrapped}")"
       mv -f "${staging_dir}/virtualenv.pex" "${bootstrapped}"
       rm -rf "${staging_dir}"
diff --git a/pants.toml b/pants.toml
index c62c9d4..8cc2ee7 100644
--- a/pants.toml
+++ b/pants.toml
@@ -2,7 +2,7 @@
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
 
 [GLOBAL]
-pants_version = "2.8.0"
+pants_version = "2.12.0"
 backend_packages.add = [
   "pants.backend.python",
   "pants.backend.python.lint.docformatter",
@@ -27,8 +27,36 @@ root_patterns = ["/"]
 interpreter_constraints = [">=3.7"]
 # Use a constraints file. See <https://www.pantsbuild.org/docs/python-third-party-dependencies>.
 requirement_constraints = "constraints.txt"
+
+[python-bootstrap]
 # We search for interpreters on both on the $PATH and in the `$(pyenv root)/versions` folder.
 #  If you're using macOS, you may want to leave off the <PATH> entry to avoid using the
 #  problematic system Pythons. See
 #  <https://www.pantsbuild.org/docs/python-interpreter-compatibility#changing-the-interpreter-search-path>.
-interpreter_search_paths = ["<PATH>", "<PYENV>"]
+search_path = ["<PATH>", "<PYENV>"]
+
+[pex-cli]
+url_template = "https://%(env.ARTIFACTORY_USER)s:%(env.ARTIFACTORY_PASS)s@pextest.jfrog.io/artifactory/default-generic/pantsbuild/pex/releases/download/{version}/pex"
+
+[python-repos]
+indexes = [
+  "<https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/>",
+]
+
+[black]
+lockfile = "tools/black.lock"
+
+# Work around <https://github.com/psf/black/issues/2964>
+extra_requirements = "click==8.0.4"
+
+[pytest]
+lockfile = "tools/pytest.lock"
+
+[isort]
+lockfile = "tools/isort.lock"
+
+[flake8]
+lockfile = "tools/flake8.lock"
+
+[mypy]
+lockfile = "tools/mypy.lock"

And bootstrap Pants:

PIP_INDEX_URL=<https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/> ./pants

And re-gen lockfiles:

$ ./pants generate-lockfiles
09:17:52.68 [INFO] Completed: Generate lockfile for isort
09:17:55.46 [INFO] Completed: Generate lockfile for flake8
09:17:56.13 [INFO] Completed: Generate lockfile for mypy
09:17:56.86 [INFO] Completed: Generate lockfile for black
09:18:09.91 [INFO] Completed: Generate lockfile for pytest
09:18:09.91 [INFO] Wrote lockfile for the resolve `pytest` to tools/pytest.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `black` to tools/black.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `flake8` to tools/flake8.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `isort` to tools/isort.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `mypy` to tools/mypy.lock

And finally things worked:

$ ./pants fmt lint check test ::
09:18:32.32 [INFO] Completed: Format with docformatter - docformatter made no changes.
09:18:32.32 [INFO] Completed: Format with Black - black made no changes.
09:18:32.32 [INFO] Completed: Format with isort - isort made no changes.

✓ black made no changes.
✓ docformatter made no changes.
✓ isort made no changes.
09:18:32.32 [INFO] Completed: Lint with Flake8 - flake8 succeeded.

✓ black succeeded.
✓ docformatter succeeded.
✓ flake8 succeeded.
✓ isort succeeded.
09:18:32.33 [INFO] Completed: Typecheck using MyPy - mypy succeeded.
Success: no issues found in 8 source files



✓ mypy succeeded.
09:18:32.33 [INFO] Completed: Run Pytest - helloworld/greet/greeting_test.py:tests succeeded.
09:18:32.33 [INFO] Completed: Run Pytest - helloworld/translator/translator_test.py:tests succeeded.

✓ helloworld/greet/greeting_test.py:tests succeeded in 0.21s (memoized).
✓ helloworld/translator/translator_test.py:tests succeeded in 0.16s (memoized).

Ah yes, your changes to add the --no-pypi and --index options is what I saw when I tried to use the script before switching to downloading the pex file from the github release page.

Yeah, this is a real mess right now and quite a thicket of fixes to work through to get set up in a restricted internet access environment. I plan to circle back to these notes and file some issues.