https://pantsbuild.org/ logo
#general
Title
# general
r

ripe-cpu-85141

07/25/2022, 4:06 AM
I have another question. I can't access pypi or any other public repo. Instead, we have artifactory which acts as a "repo proxy" + hosts private repos. I added a [python-repos] sections and a indexes entry with a list of URLs in pants.toml but I'm not sure pex honors it. When trying to run pytest, I get the following messages:
Copy code
pex.result.ResultError: There were 10 errors downloading required artifacts:
1. pytest-cov 3 from <https://files.pythonhosted.org/packages/20/49/b3e0edec68d81846f519c602ac38af9db86e1e71275528b3e814ae236063/pytest_cov-3.0.0-py3-none-any.whl>
    <urlopen error [Errno 104] Connection reset by peer>
2. pytest 7.0.1 from <https://files.pythonhosted.org/packages/38/93/c7c0bd1e932b287fb948eb9ce5a3d6307c9fc619db1e199f8c8bc5dad95f/pytest-7.0.1-py3-none-any.whl>
    <urlopen error [Errno 104] Connection reset by peer>
3. coverage 6.3.2 from <https://files.pythonhosted.org/packages/60/a8/0f951285dff76cbedbbba89c1969586020150a3b53267b5d8674b2ead15a/coverage-6.3.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl>
    <urlopen error [Errno 104] Connection reset by peer>
Is there another option to set?
h

happy-kitchen-89482

07/25/2022, 4:13 AM
There's some relevant documentation here: https://www.pantsbuild.org/docs/restricted-internet-access
I'm guessing you'll have to regenerate all the built-in tool lockfiles to resolve from your private artifactory
Try
./pants generate-lockfiles
r

ripe-cpu-85141

07/25/2022, 4:19 AM
I followed that doc (I think), and set the python-repos:indexes option. I don't use lockfiles but I tried your command and retry, I got the same error. this is when doing a
./pants test ::
e

enough-analyst-54434

07/25/2022, 4:28 AM
@ripe-cpu-85141 can you provide your
pants.toml
again with redactions as needed?
r

ripe-cpu-85141

07/25/2022, 4:36 AM
Sure
Copy code
[GLOBAL]
pants_version = "2.12.0+git5d31cdb5"
backend_packages = [
    "pants.backend.python",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.flake8",
    "pants.backend.python.typecheck.mypy",
    "pants.backend.awslambda.python"
]
ca_certs_path = "/etc/pki/tls/certs/ca-bundle.crt"


[anonymous-telemetry]
enabled = false

[source]
#marker_filenames = ["pyproject.toml"]
#root_patterns = ["/"]

[python]
interpreter_constraints = ["CPython==3.9.*"]
tailor_ignore_solitary_init_files = false

[python-repos]
indexes = [
    "<https://artifactory.mycompany.net/artifactory/api/pypi/pypi/simple/>",
    "<https://artifactory.mycompany.net/artifactory/api/pypi/pd-its-pypi-local-csp/simple/>",
    "<https://artifactory.mycompany.net/artifactory/api/pypi/pd-its-pypi-local-linux/simple/>",
    "<https://artifactory.mycompany.net/artifactory/api/pypi/dv-its-pypi-local-tooling/simple/>",
]
repos = []

[test]
use_coverage = true

[pytest]
args = ["--import-mode=importlib"]
I can see the index URLs on the call to ./pex when I run
./pants -ldebug test ::
e

enough-analyst-54434

07/25/2022, 5:06 AM
@ripe-cpu-85141 can you reflect exactly what you see? Full command line plus full logs would be very useful.
Also, can you explain the
pants_version = "2.12.0+git5d31cdb5"
? Why not
pants_version = "2.12.0"
?
r

ripe-cpu-85141

07/25/2022, 1:22 PM
@enough-analyst-54434 So for the pants_version, I use the pex from the release page of github as I don't have full access to internet and the bash script has some limitations (it doesn't honors the Pypi repos on a call to pex). If I just put
pants_version = "2.12.0", pants complains that the version is not the right one
for the command + logs: https://pastebin.com/V1Hr5Yt3
e

enough-analyst-54434

07/25/2022, 2:00 PM
Aha, ok. Thanks for the logs, that's always very helpful. So the failure is resolving a default tool lockfile that ships with Pants, in this case, for pytest. For each Python tool Pants executes to achieve some goal (pytest for test, mypy for check, etc.) we ship a lockfile for the tool and its requirements. We generate that lockfile using a default setup, and so PyPI. So, unfortunately, I think this is going to be a bit of a pain and whac-a-mole. As you hit errors like this, you'll need to re-generate lock files for the internal tool in question. In this case, you'll need to configure
[pytest] lockfile = "some/path/of/your/choice.lock"
and then run `./pants generate-lockfiles --resolve=pytest`: https://www.pantsbuild.org/docs/reference-pytest#section-lockfile
Eventually you'll get an Artifactory based lockfile for each tool you use and Pants will run normally.
r

ripe-cpu-85141

07/25/2022, 2:04 PM
ok, I'm not up-to-date about how to deal with lockfiles yet. That's on my list. Meanwhile, how should I deal with the lockfiles? Is it more a temporary file in my context or something I should add in my git repo?
e

enough-analyst-54434

07/25/2022, 2:05 PM
Pants will generate these and you should commit them. It should be a one time operation per tool lockfile.
So, just choose any path you'd like with the configuration option above, since that path will need to be committed.
To be clear, your code doesn't need to use a lock yet, this is just for each Python tool Pants uses internally to satisfy goals you ask it to achieve.
I've re-read the thread and wanted to point out Benjy had this right. The only other detail was needing to set a
lockfile
path for each Python tool before being able to
./pants generate-lockfiles
- I think.
r

ripe-cpu-85141

07/25/2022, 3:03 PM
Ok, I just tested and it works. I now need to figure out some dependencies issue I see šŸ™‚ Thank you both for your help!
h

happy-kitchen-89482

07/25/2022, 3:37 PM
Ah yes, I forgot about that part ā˜¹ļø - you need to tell Pants not to use the internal lockfile for each tool, as John has explained
e

enough-analyst-54434

07/25/2022, 3:52 PM
You're welcome. I just finished this exercise for myself, and it is not pretty. I simulated a locked down network with this
/etc/hosts
entry:
Copy code
127.0.0.1	<http://files.pythonhosted.org|files.pythonhosted.org> <http://pypi.org|pypi.org> <http://github.com|github.com>
I already have a custom Artifactory set up for testing. Then, over in example-python I had to:
Copy code
$ git diff pants pants.toml
diff --git a/pants b/pants
index 2e9a10c..759c580 100755
--- a/pants
+++ b/pants
@@ -34,9 +34,9 @@ fi
 
 PANTS_BOOTSTRAP="${PANTS_SETUP_CACHE}/bootstrap-$(uname -s)-$(uname -m)"
 
-PEX_VERSION=2.1.42
-PEX_URL="<https://github.com/pantsbuild/pex/releases/download/v${PEX_VERSION}/pex>"
-PEX_EXPECTED_SHA256="69d6b1b1009b00dd14a3a9f19b72cff818a713ca44b3186c9b12074b2a31e51f"
+PEX_VERSION=2.1.90
+PEX_URL="<https://pextest.jfrog.io/artifactory/default-generic//pantsbuild/pex/releases/download/v${PEX_VERSION}/pex>"
+PEX_EXPECTED_SHA256="2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66"
 
 VIRTUALENV_VERSION=20.4.7
 VIRTUALENV_REQUIREMENTS=$(
@@ -244,7 +244,7 @@ function bootstrap_pex {
       local staging_dir
       staging_dir=$(tempdir "${PANTS_BOOTSTRAP}")
       cd "${staging_dir}"
-      curl -LO "${PEX_URL}"
+      curl -n -LO "${PEX_URL}"
       fingerprint="$(compute_sha256 "${python}" "pex")"
       if [[ "${PEX_EXPECTED_SHA256}" != "${fingerprint}" ]]; then
         die "SHA256 of ${PEX_URL} is not as expected. Aborting."
@@ -270,7 +270,9 @@ function bootstrap_virtualenv {
       staging_dir=$(tempdir "${PANTS_BOOTSTRAP}")
       cd "${staging_dir}"
       echo "${VIRTUALENV_REQUIREMENTS}" > requirements.txt
-      "${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex
+      "${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex \
+        --no-pypi \
+        --index <https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/>
       mkdir -p "$(dirname "${bootstrapped}")"
       mv -f "${staging_dir}/virtualenv.pex" "${bootstrapped}"
       rm -rf "${staging_dir}"
diff --git a/pants.toml b/pants.toml
index c62c9d4..8cc2ee7 100644
--- a/pants.toml
+++ b/pants.toml
@@ -2,7 +2,7 @@
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
 
 [GLOBAL]
-pants_version = "2.8.0"
+pants_version = "2.12.0"
 backend_packages.add = [
   "pants.backend.python",
   "pants.backend.python.lint.docformatter",
@@ -27,8 +27,36 @@ root_patterns = ["/"]
 interpreter_constraints = [">=3.7"]
 # Use a constraints file. See <https://www.pantsbuild.org/docs/python-third-party-dependencies>.
 requirement_constraints = "constraints.txt"
+
+[python-bootstrap]
 # We search for interpreters on both on the $PATH and in the `$(pyenv root)/versions` folder.
 #  If you're using macOS, you may want to leave off the <PATH> entry to avoid using the
 #  problematic system Pythons. See
 #  <https://www.pantsbuild.org/docs/python-interpreter-compatibility#changing-the-interpreter-search-path>.
-interpreter_search_paths = ["<PATH>", "<PYENV>"]
+search_path = ["<PATH>", "<PYENV>"]
+
+[pex-cli]
+url_template = "https://%(env.ARTIFACTORY_USER)s:%(env.ARTIFACTORY_PASS)s@pextest.jfrog.io/artifactory/default-generic/pantsbuild/pex/releases/download/{version}/pex"
+
+[python-repos]
+indexes = [
+  "<https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/>",
+]
+
+[black]
+lockfile = "tools/black.lock"
+
+# Work around <https://github.com/psf/black/issues/2964>
+extra_requirements = "click==8.0.4"
+
+[pytest]
+lockfile = "tools/pytest.lock"
+
+[isort]
+lockfile = "tools/isort.lock"
+
+[flake8]
+lockfile = "tools/flake8.lock"
+
+[mypy]
+lockfile = "tools/mypy.lock"
And bootstrap Pants:
Copy code
PIP_INDEX_URL=<https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/> ./pants
And re-gen lockfiles:
Copy code
$ ./pants generate-lockfiles
09:17:52.68 [INFO] Completed: Generate lockfile for isort
09:17:55.46 [INFO] Completed: Generate lockfile for flake8
09:17:56.13 [INFO] Completed: Generate lockfile for mypy
09:17:56.86 [INFO] Completed: Generate lockfile for black
09:18:09.91 [INFO] Completed: Generate lockfile for pytest
09:18:09.91 [INFO] Wrote lockfile for the resolve `pytest` to tools/pytest.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `black` to tools/black.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `flake8` to tools/flake8.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `isort` to tools/isort.lock
09:18:09.91 [INFO] Wrote lockfile for the resolve `mypy` to tools/mypy.lock
And finally things worked:
Copy code
$ ./pants fmt lint check test ::
09:18:32.32 [INFO] Completed: Format with docformatter - docformatter made no changes.
09:18:32.32 [INFO] Completed: Format with Black - black made no changes.
09:18:32.32 [INFO] Completed: Format with isort - isort made no changes.

āœ“ black made no changes.
āœ“ docformatter made no changes.
āœ“ isort made no changes.
09:18:32.32 [INFO] Completed: Lint with Flake8 - flake8 succeeded.

āœ“ black succeeded.
āœ“ docformatter succeeded.
āœ“ flake8 succeeded.
āœ“ isort succeeded.
09:18:32.33 [INFO] Completed: Typecheck using MyPy - mypy succeeded.
Success: no issues found in 8 source files



āœ“ mypy succeeded.
09:18:32.33 [INFO] Completed: Run Pytest - helloworld/greet/greeting_test.py:tests succeeded.
09:18:32.33 [INFO] Completed: Run Pytest - helloworld/translator/translator_test.py:tests succeeded.

āœ“ helloworld/greet/greeting_test.py:tests succeeded in 0.21s (memoized).
āœ“ helloworld/translator/translator_test.py:tests succeeded in 0.16s (memoized).
r

ripe-cpu-85141

07/25/2022, 8:36 PM
Ah yes, your changes to add the --no-pypi and --index options is what I saw when I tried to use the script before switching to downloading the pex file from the github release page.
e

enough-analyst-54434

07/25/2022, 8:39 PM
Yeah, this is a real mess right now and quite a thicket of fixes to work through to get set up in a restricted internet access environment. I plan to circle back to these notes and file some issues.
šŸ™ 1
Ok, I documented the pain here: https://github.com/pantsbuild/pants/issues/16328 More needs to be done - docs, maybe wizards - but at least the scope of the effort needed to get Pants running (for Python) behind a corp firewall is mainly detailed now.
šŸ™ 1
4 Views