https://pantsbuild.org/ logo
#general
Title
# general
p

proud-appointment-36730

06/23/2022, 9:59 PM
It seems that running
./pants lint src/python::
is failing when trying to pull in the dependencies for the linters I've configured from my private PyPI repo. I have the
PANTS_PYTHON_REPOS_INDEXES
and
PANTS_PYTHON_REPOS_REPOS
env variables to strings containing the correct credentials embedded in the URL. Is there any reason pants wouldn't be using these credentials?
It looks like the
-i
flag to the
./pex
invocation in
__run.sh
in the process sandbox dir isn't getting set at all
h

hundreds-father-404

06/23/2022, 10:04 PM
Hello! One thing you can try is using
-ldebug
to see the argv we pass to Pex, which will ensure it is properly setting the value: https://www.pantsbuild.org/docs/troubleshooting#debug-tip-enable-stack-traces-and-increase-logging Then, we can figure out why Pex/pip aren't doing the right thing
ah cool, you already are looking at
__run.sh
. Great, that's the other way to check the argv
Try
./pants help-advanced python-repos
with those env vars set - it should show the current value for the options
p

proud-appointment-36730

06/23/2022, 10:06 PM
Output of the help command looks correct. Pex seems to be looking at the correct repo, but for some reason isn't using the correct credentials and is hitting 401 error
w

witty-crayon-22786

06/23/2022, 10:09 PM
which version of pants is this? are you using PEX for lockfile generation?
p

proud-appointment-36730

06/23/2022, 10:11 PM
Yes, using pex for lockfile generation:
Copy code
[python]

enable_resolves = true
default_resolve = "kode"
lockfile_generator = "pex"
# interpreter_constraints = ["CPython==3.5.*"]
interpreter_constraints = ["CPython==3.8.*"]
And using version
2.11.0
It also looks like the lockfile for some of the linting tools is pulling dependencies from the public pypi, which is not something I want to happen. I determined this by preserving sandbox directories and checking out the lockfile for one of them. Here is an example:
Copy code
{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "064e6d81f04ac96bc0d176cbaae953a0332482b22d3ad70d47c8a7f2732eef6f",
              "url": "<https://files.pythonhosted.org/packages/8a/7b/b32b3952ee4e9fd76c7a9b18d4bafb70ed65ac6426d1802103c2eaf1d0de/docformatter-1.4.tar.gz>"
            }
          ],
          "project_name": "docformatter",
          "requires_dists": [
            "untokenize"
          ],
          "requires_python": null,
          "version": "1.4"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "3865dbbbb8efb4bb5eaa72f1be7f3e0be00ea8b7f125c69cbd1f5fda926f37a2",
              "url": "<https://files.pythonhosted.org/packages/f7/46/e7cea8159199096e1df52da20a57a6665da80c37fb8aeb848a3e47442c32/untokenize-0.1.1.tar.gz>"
            }
          ],
          "project_name": "untokenize",
          "requires_dists": [],
          "requires_python": null,
          "version": "0.1.1"
        }
      ],
      "platform_tag": [
        "cp39",
        "cp39",
        "macosx_11_0_arm64"
      ]
    }
  ],
  "pex_version": "2.1.72",
  "prefer_older_binary": false,
  "requirements": [
    "docformatter<1.5,>=1.4"
  ],
  "requires_python": [
    "<4,>=3.7"
  ],
  "resolver_version": "pip-2020-resolver",
  "style": "universal",
  "transitive": true,
  "use_pep517": null
}
w

witty-crayon-22786

06/23/2022, 10:16 PM
so, iirc, there were some bugs related to using private repositories with PEX lockfiles in those versions. you might be able to either upgrade PEX to
2.1.93
, or use a custom-built lockfile
…and
pex --lock
now properly handles
authentication.
https://github.com/pantsbuild/pex/releases/tag/v2.1.93
main
is just upgrading now: https://github.com/pantsbuild/pants/pull/15911, and we’ll likely cherry-pick this to
2.12
p

proud-appointment-36730

06/23/2022, 10:22 PM
How can I upgrade the version of pex that pants uses?
w

witty-crayon-22786

06/23/2022, 10:24 PM
by setting
Copy code
[pex-cli]
version = "v2.1.93"
known_versions = [
    "v2.1.93|macos_arm64|80fc6b94f5db253a71061974cb8d8ce520932aef44d989e9057917cc33a30fd6|3802280",
  "v2.1.93|macos_x86_64|80fc6b94f5db253a71061974cb8d8ce520932aef44d989e9057917cc33a30fd6|3802280",
  "v2.1.93|linux_x86_64|80fc6b94f5db253a71061974cb8d8ce520932aef44d989e9057917cc33a30fd6|3802280",
]
p

proud-appointment-36730

06/23/2022, 10:34 PM
Ok. I guess I'm just sort of confused. The
package
goal works just fine. The
publish
goal works just fine. The
lint
goal does not. I vaguely understand that it has something to do with how pants is managed the installation of the linters it needs to run, and it's doing that through pex somehow, but that's about it
I can paste the entire traceback here if that's helpful
Or rather, the relevant portion of the traceback
I'm even more confused because lockfile generation via pex worked just fine
I temporarily got around this by just disabling resolves and not generating lockfiles, but I would love to work with someone on getting this piece sorted out. Everything seems to be correctly pulling from my private PyPI repo without resolves. If I enable them and generate a lockfile for my default resolve, a bunch of goals (
test
,
lint
etc) start breaking again for python targets because building
requirements.pex
from my pinned 3rd party deps in my lockfile fails when trying to authenticate to my private PyPI repo. What I have for now is good enough to pitch to my managers tomorrow, but when y'all get back online please do reach out. I'd be happy to interactively debug this with folks and get it all sorted out
I'll file a bug report tomorrow morning
w

witty-crayon-22786

06/24/2022, 4:23 PM
did you try upgrading PEX as mentioned above?
there was a fix for this issue (afaict) in the release i linked
p

proud-appointment-36730

06/24/2022, 5:58 PM
I tried but got stumped and had to move on to get a POC working for a meeting today. I did try running pants from sources on both the
main
branch and the
2.12.x
branch, but had the same issues there. When I get some time, I'll really dive into this and try to pull together an actionable, descriptive issue on Github for you all
This could honestly be user error in some way, I've been moving sort of fast and loose to get to a working prototype that I could demo for folks on my team
5 Views