https://pantsbuild.org/ logo
Join Slack
Powered by
It seems that running `./pants lint src/python::` ...
# general
p
It seems that running 
./pants lint src/python::
is failing when trying to pull in the dependencies for the linters I've configured from my private PyPI repo. I have the 
PANTS_PYTHON_REPOS_INDEXES
and 
PANTS_PYTHON_REPOS_REPOS
env variables to strings containing the correct credentials embedded in the URL. Is there any reason pants wouldn't be using these credentials?
It looks like the 
-i
flag to the 
./pex
invocation in 
__run.sh
in the process sandbox dir isn't getting set at all
h
Hello! One thing you can try is using 
-ldebug
to see the argv we pass to Pex, which will ensure it is properly setting the value: https://www.pantsbuild.org/docs/troubleshooting#debug-tip-enable-stack-traces-and-increase-logging Then, we can figure out why Pex/pip aren't doing the right thing
ah cool, you already are looking at 
__run.sh
. Great, that's the other way to check the argv
Try 
./pants help-advanced python-repos
with those env vars set - it should show the current value for the options
p
Output of the help command looks correct. Pex seems to be looking at the correct repo, but for some reason isn't using the correct credentials and is hitting 401 error
w
which version of pants is this? are you using PEX for lockfile generation?
p
Yes, using pex for lockfile generation:
Copy code
[python]

enable_resolves = true
default_resolve = "kode"
lockfile_generator = "pex"
# interpreter_constraints = ["CPython==3.5.*"]
interpreter_constraints = ["CPython==3.8.*"]
And using version 
2.11.0
It also looks like the lockfile for some of the linting tools is pulling dependencies from the public pypi, which is not something I want to happen. I determined this by preserving sandbox directories and checking out the lockfile for one of them. Here is an example:
Copy code
{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "064e6d81f04ac96bc0d176cbaae953a0332482b22d3ad70d47c8a7f2732eef6f",
              "url": "<https://files.pythonhosted.org/packages/8a/7b/b32b3952ee4e9fd76c7a9b18d4bafb70ed65ac6426d1802103c2eaf1d0de/docformatter-1.4.tar.gz>"
            }
          ],
          "project_name": "docformatter",
          "requires_dists": [
            "untokenize"
          ],
          "requires_python": null,
          "version": "1.4"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "3865dbbbb8efb4bb5eaa72f1be7f3e0be00ea8b7f125c69cbd1f5fda926f37a2",
              "url": "<https://files.pythonhosted.org/packages/f7/46/e7cea8159199096e1df52da20a57a6665da80c37fb8aeb848a3e47442c32/untokenize-0.1.1.tar.gz>"
            }
          ],
          "project_name": "untokenize",
          "requires_dists": [],
          "requires_python": null,
          "version": "0.1.1"
        }
      ],
      "platform_tag": [
        "cp39",
        "cp39",
        "macosx_11_0_arm64"
      ]
    }
  ],
  "pex_version": "2.1.72",
  "prefer_older_binary": false,
  "requirements": [
    "docformatter<1.5,>=1.4"
  ],
  "requires_python": [
    "<4,>=3.7"
  ],
  "resolver_version": "pip-2020-resolver",
  "style": "universal",
  "transitive": true,
  "use_pep517": null
}
w
so, iirc, there were some bugs related to using private repositories with PEX lockfiles in those versions. you might be able to either upgrade PEX to 
2.1.93
, or use a custom-built lockfile
…and 
pex --lock
now properly handles
authentication.
https://github.com/pantsbuild/pex/releases/tag/v2.1.93
main
is just upgrading now: https://github.com/pantsbuild/pants/pull/15911, and we’ll likely cherry-pick this to 
2.12
p
How can I upgrade the version of pex that pants uses?
w
by setting
Copy code
[pex-cli]
version = "v2.1.93"
known_versions = [
    "v2.1.93|macos_arm64|80fc6b94f5db253a71061974cb8d8ce520932aef44d989e9057917cc33a30fd6|3802280",
  "v2.1.93|macos_x86_64|80fc6b94f5db253a71061974cb8d8ce520932aef44d989e9057917cc33a30fd6|3802280",
  "v2.1.93|linux_x86_64|80fc6b94f5db253a71061974cb8d8ce520932aef44d989e9057917cc33a30fd6|3802280",
]
p
Ok. I guess I'm just sort of confused. The 
package
goal works just fine. The 
publish
goal works just fine. The 
lint
goal does not. I vaguely understand that it has something to do with how pants is managed the installation of the linters it needs to run, and it's doing that through pex somehow, but that's about it
I can paste the entire traceback here if that's helpful
Or rather, the relevant portion of the traceback
I'm even more confused because lockfile generation via pex worked just fine
I temporarily got around this by just disabling resolves and not generating lockfiles, but I would love to work with someone on getting this piece sorted out. Everything seems to be correctly pulling from my private PyPI repo without resolves. If I enable them and generate a lockfile for my default resolve, a bunch of goals (
test
, 
lint
etc) start breaking again for python targets because building 
requirements.pex
from my pinned 3rd party deps in my lockfile fails when trying to authenticate to my private PyPI repo. What I have for now is good enough to pitch to my managers tomorrow, but when y'all get back online please do reach out. I'd be happy to interactively debug this with folks and get it all sorted out
I'll file a bug report tomorrow morning
w
did you try upgrading PEX as mentioned above?
there was a fix for this issue (afaict) in the release i linked
p
I tried but got stumped and had to move on to get a POC working for a meeting today. I did try running pants from sources on both the 
main
branch and the 
2.12.x
branch, but had the same issues there. When I get some time, I'll really dive into this and try to pull together an actionable, descriptive issue on Github for you all
This could honestly be user error in some way, I've been moving sort of fast and loose to get to a working prototype that I could demo for folks on my team
5 Views
Open in Slack
PreviousNext
Powered by