https://pantsbuild.org/ logo
#general
Title
# general
c

calm-ambulance-65371

08/24/2022, 9:42 PM
Hello, we suddenly had packages in private repositories start 404ing overnight with no changes on our end and it's unlikely GitLab suddenly changed a broad policy. Our best guess is Pants stopped using the authentication portion of the
[python-repo].indexes.add
stuff. Any thoughts?
Our initial thought was the tokens expired, but that's not the case and new tokens don't work either.
w

witty-crayon-22786

08/24/2022, 9:44 PM
do they 404 outside of Pants, e.g. via
curl
?
c

calm-ambulance-65371

08/24/2022, 9:44 PM
no
the tokens are good, we've verified half a dozen different ways
w

witty-crayon-22786

08/24/2022, 9:47 PM
are you using lockfiles?
c

calm-ambulance-65371

08/24/2022, 9:48 PM
yes
and a remote cache, so it's very complicated...
w

witty-crayon-22786

08/24/2022, 9:48 PM
even “old” builds/SHAs fail, i assume?
c

calm-ambulance-65371

08/24/2022, 9:49 PM
as in checkout older versions of the repo and try?
w

witty-crayon-22786

08/24/2022, 9:49 PM
yes
just to be quite sure it was not a code change.
once you’ve confirmed that, next step might be to: 1. run with
-ldebug
, and capture the full arguments of the failing resolve and see if you see anything suspicious 2. run with
--no-process-cleanup
and see if you can reproduce the failure via
e

enough-analyst-54434

08/24/2022, 10:26 PM
@calm-ambulance-65371 no changes on your end include the Pants version stayed the same? What Pants version is that? Also how did 404 make you think auth. It's 404 and not 403? What is the private index implementation? JFrog? Something else?
Maybe if my line of questioning turns out good?: https://www.jfrog.com/jira/browse/RTFACT-17764
c

calm-ambulance-65371

08/24/2022, 10:37 PM
It's GitLab's pip registry. We're using pants 2.12.0, however we've tried 2.11.0. It appears the behavior changes depending on the version. We get a 404 in 2.12, but a 401 in 2.11
further, the URL from the stacktrace works
(I have not confirmed the 401/404 behavior myself, a member of our devops team conveyed that to me)
e

enough-analyst-54434

08/24/2022, 10:57 PM
So, the only thing I can think of on the Pants - really Pex - side is this fix in Pex 2.1.93: https://github.com/pantsbuild/pex/issues/1803 That fix applies when Pants needs to read a Pex lock file / use it to download artifacts and the fix is only needed for registries that use Bearer auth, Digest and Basic worked fine. That fix is not in 2.12.x, but it is in 2.13.x. Maybe gitlab changed auth schemes?
c

calm-ambulance-65371

08/25/2022, 2:11 PM
I find it hard to believe they'd drop HTTP basic auth, and as far as I can tell it still works with that scheme. We're also hard-coding the credentials in the pants.toml
e

enough-analyst-54434

08/25/2022, 2:21 PM
Agreed, its just the only thing I can think of - that or 401 no longer being issued to initiate a challenge. Some servers in the past have had that sort of bug, That said, that would be a huge issue that would get fixed super fast - so also almost certainly not it.
c

calm-ambulance-65371

08/25/2022, 2:40 PM
also 404 for an unauthed request is extremely common to throw off scrapers and leak information
the fact that pants 2.12 and 2.11 appear to be doing different things in this regard is extremely concerning
we suspect it's an issue of pants not passing the credentials to pex, but it's difficult to debug
to reiterate: we are using HTTP Basic auth, gitlab has no problem with this, all other attempts to download the package in question work just fine
pip works fine
h

hundreds-father-404

08/25/2022, 2:44 PM
one debugging idea is to use 2.13 but with the Pex version from 2.12: https://www.pantsbuild.org/docs/reference-pex-cli#section-version
e

enough-analyst-54434

08/25/2022, 3:00 PM
@calm-ambulance-65371 stepping back, the key bit of information you gave was you made no changes. So Pants 2.12 worked and then it didn't suddenly was my take. Can you confirm this is true?
c

calm-ambulance-65371

08/25/2022, 3:03 PM
that appears to be the case
though I will say, we're using a remote cache and that can easily hide issues
on a possibly related, but likely not note:
Copy code
ProcessExecutionFailure: Process 'Generate lockfile for plugins' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants.testutil<2.13,>=2.12.0a0
ERROR: No matching distribution found for pantsbuild.pants.testutil<2.13,>=2.12.0a0
the only requirements in this resolve are
Cython
and
pants_requirements()
this seems very broken
stepping back, the key bit of information you gave was you made no changes. So Pants 2.12 worked and then it didn't suddenly was my take. Can you confirm this is true?
To add as much information as I can: Our CI broke early yesterday morning (before any merges had gone in) with the above error in our early on
package
step. Our initial thought was the access token for our single private dependency had expired, but this turned out to not be the case.
We're using multiple resolves,
lockfile_generator='pex'
, and hardcoding creds using the
https:<username>:<password>@url
format in
indexes.add
we think it could be an interaction with the pex-based lockfiles. When we hack the credentials into the URLs stored in there, we get passes
was there an update to pex or any of it's dependencies recently?
e

enough-analyst-54434

08/25/2022, 3:24 PM
Pants doesn't work that way - it uses frozen things. So, plenty of Pex releases and Pants updates to use it, but not on 2.12.0 - which is frozen on Pex 2.1.90 unless you configure for new Pex.
1
h

hundreds-father-404

08/25/2022, 3:24 PM
was there an update to pex or any of it's dependencies recently?
But bump that you can experiment with different versions of Pex by changing the
[pex-cli]
options
e

enough-analyst-54434

08/25/2022, 3:25 PM
Ok, let me let go of the we didn;t change anything bit. Focusing on creds in lockfiles, that is known broken in Pex 2.1.90 and fixed in 2.1.93 as I mentioned above,
If you'd like to try 2.1.93 but not upgrade to Pants 2.13.x I can give you some instructions.
c

calm-ambulance-65371

08/25/2022, 3:25 PM
I've very confused as to how it worked until now then. I'll try bumping the pex version
1
I think we've been on 2.12 for a month or so
e

enough-analyst-54434

08/25/2022, 3:25 PM
I am also very confused.
The bump is non-trivial. Have you done this before?
c

calm-ambulance-65371

08/25/2022, 3:25 PM
remote caching is a blessing/curse probably
nope
e

enough-analyst-54434

08/25/2022, 3:26 PM
Ok, Pex is a "binary" so this is a bit different.
Give me a sec to get the right strings...
👍 1
c

calm-ambulance-65371

08/25/2022, 3:30 PM
I'm trying pants 2.13.0rc1 in the meantime
e

enough-analyst-54434

08/25/2022, 3:30 PM
Oh - that would be way better! The basic structure is the following - these are the defaults for Pants 2.12.0:
Copy code
[pex-cli]
version = "v2.1.90"
known_versions = [  
    "v2.1.90|macos_arm64|2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66|3755345",
    "v2.1.90|macos_x86_64|2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66|3755345",
    "v2.1.90|linux_x86_64|2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66|3755345"
]
I'll get you some updated hashes and sizes...
c

calm-ambulance-65371

08/25/2022, 3:32 PM
we're not afraid of using a RC, but historically upgrades have been very painful
though 2.11 -> 2.12 was easy, no real changes
e

enough-analyst-54434

08/25/2022, 3:33 PM
Yeah - I think the community has spoken up about that and we're listening better now. We have some longer deprecation policies coming up.
Ok, If you stay on Pants 2.12.0 you can try Pex latest:
Copy code
[pex-cli]
version = "v2.1.103"
known_versions = [  
    "v2.1.103|macos_arm64|4d45336511484100ae4e2bab24542a8b86b12c8cb89230463593c60d08c4b8d3|3814407",
    "v2.1.103|macos_x86_64|4d45336511484100ae4e2bab24542a8b86b12c8cb89230463593c60d08c4b8d3|3814407",
    "v2.1.103|linux_x86_64|4d45336511484100ae4e2bab24542a8b86b12c8cb89230463593c60d08c4b8d3|3814407"
]
Or Just the immediate fix for lock auth in 2.1.93:
Copy code
[pex-cli]
version = "v2.1.93"
known_versions = [  
    "v2.1.93|macos_arm64|80fc6b94f5db253a71061974cb8d8ce520932aef44d989e9057917cc33a30fd6|3802280",
    "v2.1.93|macos_x86_64|80fc6b94f5db253a71061974cb8d8ce520932aef44d989e9057917cc33a30fd6|3802280",
    "v2.1.93|linux_x86_64|80fc6b94f5db253a71061974cb8d8ce520932aef44d989e9057917cc33a30fd6|3802280"
]
2
c

calm-ambulance-65371

08/25/2022, 3:38 PM
I will say, y'all have been awesome in providing support. I just don't want to have to waste anyones time
❤️ 1
ok, bumping the version appeared to fix it. I'm still seeing errors when trying to rebuild our plugins lockfile
h

hundreds-father-404

08/25/2022, 4:04 PM
ok, bumping the version appeared to fix it.
Great!
I'm still seeing errors when trying to rebuild our plugins lockfile
Which errors?
c

calm-ambulance-65371

08/25/2022, 4:08 PM
Copy code
11:07:53.04 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for plugins' failed with exit code 1.
stdout:

stderr:
pid 149055 -> /home/noah/.cache/pants/named_caches/pex_root/venvs/1d025be2b3d2024670ccbfbce824218209fcd325/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /home/noah/.cache/pants/named_caches/pex_root/venvs/1d025be2b3d2024670ccbfbce824218209fcd325/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/noah/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-dVr3ve/.tmp/pex-pip-log.8_l9nknb/pip.log download --dest /tmp/pants-sandbox-dVr3ve/.tmp/tmp_qufi1q_/usr.bin.python3.8 --requirement __pip_args.txt pantsbuild.pants.testutil<2.14,>=2.13.0a0 pantsbuild.pants<2.14,>=2.13.0a0 --index-url <https://pypi.org/simple/> --extra-index-url <redacted> --extra-index-url <redacted> --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants.testutil<2.14,>=2.13.0a0
ERROR: No matching distribution found for pantsbuild.pants.testutil<2.14,>=2.13.0a0
plugins is the resolve that's only used for our internal plugins
The BUILD looks exactly like:
Copy code
pants_requirements(
    resolve="plugins"
)

python_sources(
    name="pants-plugins0",
    resolve="plugins"
)
👍 1
h

hundreds-father-404

08/25/2022, 4:12 PM
Hm and it's using Python 3.8 it looks like, which is good. The wheel is on PyPI 🤔 https://pypi.org/project/pantsbuild.pants.testutil/#history For the sake of debugging, could you please try setting
testutil=False
on the
pants_requirement
target?
c

calm-ambulance-65371

08/25/2022, 4:20 PM
Copy code
11:20:11.45 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for plugins' failed with exit code 1.
stdout:

stderr:
pid 159395 -> /home/noah/.cache/pants/named_caches/pex_root/venvs/1d025be2b3d2024670ccbfbce824218209fcd325/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /home/noah/.cache/pants/named_caches/pex_root/venvs/1d025be2b3d2024670ccbfbce824218209fcd325/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/noah/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-uyyliU/.tmp/pex-pip-log.pgz9g8ks/pip.log download --dest /tmp/pants-sandbox-uyyliU/.tmp/tmp5wqgsfe6/usr.bin.python3.8 --requirement __pip_args.txt pantsbuild.pants<2.14,>=2.13.0a0 --index-url <https://pypi.org/simple/> --extra-index-url <redacted> --extra-index-url <redacted> --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants<2.14,>=2.13.0a0
ERROR: No matching distribution found for pantsbuild.pants<2.14,>=2.13.0a0




Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
this is fun
h

hundreds-father-404

08/25/2022, 4:23 PM
fun fun fun under the sun 😭 what OS and arch are you using? what is
[python].interpreter_constraints
, and if you set it,
[python].resolves_to_interpreter_constraints
?
c

calm-ambulance-65371

08/25/2022, 4:25 PM
Copy code
[python]
interpreter_constraints = [">=3.8.*,<3.11"]
enable_resolves = true
default_resolve = '<redacted>'
lockfile_generator = 'pex'
that's the whole section
e

enough-analyst-54434

08/25/2022, 4:26 PM
You might also ditch the
pants_requirements
gizmo and use a plain old pinned requirement that matches what you need. A confounding angle here may be:
Copy code
$ pip download --help
...
--pre                       Include pre-release and development versions. By default, pip only finds stable versions.
It is supposed to be the case that if you directly require a pre-release, Pip switches into
--pre
mode automatically. The ranged requirement may be foiling it?
c

calm-ambulance-65371

08/25/2022, 4:27 PM
it was doing the same thing in
2.12.0
so maybe, but I sort of doubt it?
e

enough-analyst-54434

08/25/2022, 4:27 PM
Ok, that's key info. Definitely doubt it then.
c

calm-ambulance-65371

08/25/2022, 4:27 PM
this resolve is used for our internal plugins, so I think that target is necessary too?
h

hundreds-father-404

08/25/2022, 4:28 PM
Ahhh Noah, try setting this:
Copy code
[python]
resolves_to_interpreter_constraints = { plugin = [">=3.8,<3.10"]
e

enough-analyst-54434

08/25/2022, 4:28 PM
It is but you can always say pantsbuild.pants.testutil==2.13.0a0 in a reqular requirement instead.
h

hundreds-father-404

08/25/2022, 4:29 PM
I suspect what's happening is that when generating the lockfile, if
[python].resolves_to_interpreter_constraints
is not set, then we default to
[python].interpreter_constraints
, so here
">=3.8.*,<3.11"
. Pants isn't released for 3.10, so maybe Pex is failing to lock for that reason Now, unclear why this was working before. That would be the next question, if this fixes it
e

enough-analyst-54434

08/25/2022, 4:30 PM
Maybe -> definitely.
👍 2
There is no Pants for Python 3.10 - lock fail
c

calm-ambulance-65371

08/25/2022, 4:31 PM
Oh, I think I have 3.10 on my system, that could definitely be it (we use a mix of Ubuntu 20.04 and 22.04)
all AMD64
e

enough-analyst-54434

08/25/2022, 4:31 PM
On your system does not matter here - just the constraint allowing 3.10 is all that matters
c

calm-ambulance-65371

08/25/2022, 4:31 PM
that fixed it!
h

hundreds-father-404

08/25/2022, 4:31 PM
Great! You will also need to update
interpreter_constraints
field for any
python_source
and
python_tests
targets that belong to
resolve=plugin
to use
>=3.8,<3.10
. See https://www.pantsbuild.org/docs/plugins-overview#building-in-repo-plugins-with-pants
e

enough-analyst-54434

08/25/2022, 4:32 PM
Excellent. This was all crazy. Thanks for sticking with it.
💯 2
❤️ 2
c

calm-ambulance-65371

08/25/2022, 4:32 PM
it would be really great if pex told us why it wasn't resolving...
there's no updates needed, nothing there had particular constraints
e

enough-analyst-54434

08/25/2022, 4:32 PM
@calm-ambulance-65371 I will say - the only fix in Pex 2.1.93+ re auth is handling Bearer auth - Digest worked prio and had integration tests.
h

hundreds-father-404

08/25/2022, 4:33 PM
there's no updates needed, nothing there had particular constraints
They will thus default to
[python].interpreter_constraints
, which includes 3.10. Invalid. So you need to override the default by manually setting the field
e

enough-analyst-54434

08/25/2022, 4:33 PM
Agreed on Pex. It's super-hard though since it calls out to Pip.
👍 1
c

calm-ambulance-65371

08/25/2022, 4:39 PM
is there a reason pants doesn't support 3.10?
e

enough-analyst-54434

08/25/2022, 4:39 PM
No. Time / money.
Its got a tracking issue out there on GH somewhere.
c

calm-ambulance-65371

08/25/2022, 4:49 PM
Is Twitter still supporting Pants?
e

enough-analyst-54434

08/25/2022, 4:59 PM
2 nos: No in general and if they did it would be Pants v1, which is what they use(d) - and that's a totally different beast.
👍 1
And if by support you mean tech-support, that was in large part Toolchain employees, but the community support is growing stronger and stronger. Certain backends - like docker - are well supported and the support mainly comes from non TC employees. iManage & IBM employees for example.
That 3.10 tracking link is here: https://github.com/pantsbuild/pants/issues/14111 A confounding where-to-apply-effort element is an effort to completely change how Pants ships, which, instead of adding CI shards for 3.10, would make it such that Pants includes the Python it needs - which would happen to be 3.10 - so you need not have Python installed at all on your machine. Chris is working on that in https://github.com/pantsbuild/pants/issues/7369 We're banking on the latter and so work on the former has lagged / been deferred.
👍 1
Right now were in the top ... I forget 20? ... storage abusers on PyPI because of shipping 7 native wheels.
c

calm-ambulance-65371

08/25/2022, 5:16 PM
ouch
7 Views