https://pantsbuild.org/ logo
Join Slack
Powered by
Hello, we suddenly had packages in private reposit...
# general
c
Hello, we suddenly had packages in private repositories start 404ing overnight with no changes on our end and it's unlikely GitLab suddenly changed a broad policy. Our best guess is Pants stopped using the authentication portion of the 
[python-repo].indexes.add
stuff. Any thoughts?
Our initial thought was the tokens expired, but that's not the case and new tokens don't work either.
w
do they 404 outside of Pants, e.g. via 
curl
?
c
no
the tokens are good, we've verified half a dozen different ways
w
are you using lockfiles?
c
yes
and a remote cache, so it's very complicated...
w
even “old” builds/SHAs fail, i assume?
c
as in checkout older versions of the repo and try?
w
yes
just to be quite sure it was not a code change.
once you’ve confirmed that, next step might be to: 1. run with 
-ldebug
, and capture the full arguments of the failing resolve and see if you see anything suspicious 2. run with 
--no-process-cleanup
and see if you can reproduce the failure via
e
@calm-ambulance-65371 no changes on your end include the Pants version stayed the same? What Pants version is that? Also how did 404 make you think auth. It's 404 and not 403? What is the private index implementation? JFrog? Something else?
Maybe if my line of questioning turns out good?: https://www.jfrog.com/jira/browse/RTFACT-17764
c
It's GitLab's pip registry. We're using pants 2.12.0, however we've tried 2.11.0. It appears the behavior changes depending on the version. We get a 404 in 2.12, but a 401 in 2.11
further, the URL from the stacktrace works
(I have not confirmed the 401/404 behavior myself, a member of our devops team conveyed that to me)
e
So, the only thing I can think of on the Pants - really Pex - side is this fix in Pex 2.1.93: https://github.com/pantsbuild/pex/issues/1803 That fix applies when Pants needs to read a Pex lock file / use it to download artifacts and the fix is only needed for registries that use Bearer auth, Digest and Basic worked fine. That fix is not in 2.12.x, but it is in 2.13.x. Maybe gitlab changed auth schemes?
c
I find it hard to believe they'd drop HTTP basic auth, and as far as I can tell it still works with that scheme. We're also hard-coding the credentials in the pants.toml
e
Agreed, its just the only thing I can think of - that or 401 no longer being issued to initiate a challenge. Some servers in the past have had that sort of bug, That said, that would be a huge issue that would get fixed super fast - so also almost certainly not it.
c
also 404 for an unauthed request is extremely common to throw off scrapers and leak information
the fact that pants 2.12 and 2.11 appear to be doing different things in this regard is extremely concerning
we suspect it's an issue of pants not passing the credentials to pex, but it's difficult to debug
to reiterate: we are using HTTP Basic auth, gitlab has no problem with this, all other attempts to download the package in question work just fine
pip works fine
h
one debugging idea is to use 2.13 but with the Pex version from 2.12: https://www.pantsbuild.org/docs/reference-pex-cli#section-version
e
@calm-ambulance-65371 stepping back, the key bit of information you gave was you made no changes. So Pants 2.12 worked and then it didn't suddenly was my take. Can you confirm this is true?
c
that appears to be the case
though I will say, we're using a remote cache and that can easily hide issues
on a possibly related, but likely not note:
Copy code
ProcessExecutionFailure: Process 'Generate lockfile for plugins' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants.testutil<2.13,>=2.12.0a0
ERROR: No matching distribution found for pantsbuild.pants.testutil<2.13,>=2.12.0a0
the only requirements in this resolve are 
Cython
and 
pants_requirements()
this seems very broken
stepping back, the key bit of information you gave was you made no changes. So Pants 2.12 worked and then it didn't suddenly was my take. Can you confirm this is true?
To add as much information as I can: Our CI broke early yesterday morning (before any merges had gone in) with the above error in our early on 
package
step. Our initial thought was the access token for our single private dependency had expired, but this turned out to not be the case.
We're using multiple resolves, 
lockfile_generator='pex'
, and hardcoding creds using the 
https:<username>:<password>@url
format in 
indexes.add
we think it could be an interaction with the pex-based lockfiles. When we hack the credentials into the URLs stored in there, we get passes
was there an update to pex or any of it's dependencies recently?
e
Pants doesn't work that way - it uses frozen things. So, plenty of Pex releases and Pants updates to use it, but not on 2.12.0 - which is frozen on Pex 2.1.90 unless you configure for new Pex.
➕ 1
h
was there an update to pex or any of it's dependencies recently?
But bump that you can experiment with different versions of Pex by changing the 
[pex-cli]
options
e
Ok, let me let go of the we didn;t change anything bit. Focusing on creds in lockfiles, that is known broken in Pex 2.1.90 and fixed in 2.1.93 as I mentioned above,
If you'd like to try 2.1.93 but not upgrade to Pants 2.13.x I can give you some instructions.
c
I've very confused as to how it worked until now then. I'll try bumping the pex version
➕ 1
I think we've been on 2.12 for a month or so
e
I am also very confused.
The bump is non-trivial. Have you done this before?
c
remote caching is a blessing/curse probably
nope
e
Ok, Pex is a "binary" so this is a bit different.
Give me a sec to get the right strings...
👍 1
c
I'm trying pants 2.13.0rc1 in the meantime
e
Oh - that would be way better! The basic structure is the following - these are the defaults for Pants 2.12.0:
Copy code
[pex-cli]
version = "v2.1.90"
known_versions = [  
    "v2.1.90|macos_arm64|2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66|3755345",
    "v2.1.90|macos_x86_64|2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66|3755345",
    "v2.1.90|linux_x86_64|2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66|3755345"
]
I'll get you some updated hashes and sizes...
c
we're not afraid of using a RC, but historically upgrades have been very painful
though 2.11 -> 2.12 was easy, no real changes
e
Yeah - I think the community has spoken up about that and we're listening better now. We have some longer deprecation policies coming up.
Ok, If you stay on Pants 2.12.0 you can try Pex latest:
Copy code
[pex-cli]
version = "v2.1.103"
known_versions = [  
    "v2.1.103|macos_arm64|4d45336511484100ae4e2bab24542a8b86b12c8cb89230463593c60d08c4b8d3|3814407",
    "v2.1.103|macos_x86_64|4d45336511484100ae4e2bab24542a8b86b12c8cb89230463593c60d08c4b8d3|3814407",
    "v2.1.103|linux_x86_64|4d45336511484100ae4e2bab24542a8b86b12c8cb89230463593c60d08c4b8d3|3814407"
]
Or Just the immediate fix for lock auth in 2.1.93:
Copy code
[pex-cli]
version = "v2.1.93"
known_versions = [  
    "v2.1.93|macos_arm64|80fc6b94f5db253a71061974cb8d8ce520932aef44d989e9057917cc33a30fd6|3802280",
    "v2.1.93|macos_x86_64|80fc6b94f5db253a71061974cb8d8ce520932aef44d989e9057917cc33a30fd6|3802280",
    "v2.1.93|linux_x86_64|80fc6b94f5db253a71061974cb8d8ce520932aef44d989e9057917cc33a30fd6|3802280"
]
➕ 2
c
I will say, y'all have been awesome in providing support. I just don't want to have to waste anyones time
❤️ 1
ok, bumping the version appeared to fix it. I'm still seeing errors when trying to rebuild our plugins lockfile
h
ok, bumping the version appeared to fix it.
Great!
I'm still seeing errors when trying to rebuild our plugins lockfile
Which errors?
c
Copy code
11:07:53.04 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for plugins' failed with exit code 1.
stdout:

stderr:
pid 149055 -> /home/noah/.cache/pants/named_caches/pex_root/venvs/1d025be2b3d2024670ccbfbce824218209fcd325/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /home/noah/.cache/pants/named_caches/pex_root/venvs/1d025be2b3d2024670ccbfbce824218209fcd325/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/noah/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-dVr3ve/.tmp/pex-pip-log.8_l9nknb/pip.log download --dest /tmp/pants-sandbox-dVr3ve/.tmp/tmp_qufi1q_/usr.bin.python3.8 --requirement __pip_args.txt pantsbuild.pants.testutil<2.14,>=2.13.0a0 pantsbuild.pants<2.14,>=2.13.0a0 --index-url <https://pypi.org/simple/> --extra-index-url <redacted> --extra-index-url <redacted> --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants.testutil<2.14,>=2.13.0a0
ERROR: No matching distribution found for pantsbuild.pants.testutil<2.14,>=2.13.0a0
plugins is the resolve that's only used for our internal plugins
The BUILD looks exactly like:
Copy code
pants_requirements(
    resolve="plugins"
)

python_sources(
    name="pants-plugins0",
    resolve="plugins"
)
👍 1
h
Hm and it's using Python 3.8 it looks like, which is good. The wheel is on PyPI 🤔 https://pypi.org/project/pantsbuild.pants.testutil/#history For the sake of debugging, could you please try setting 
testutil=False
on the 
pants_requirement
target?
c
Copy code
11:20:11.45 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for plugins' failed with exit code 1.
stdout:

stderr:
pid 159395 -> /home/noah/.cache/pants/named_caches/pex_root/venvs/1d025be2b3d2024670ccbfbce824218209fcd325/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /home/noah/.cache/pants/named_caches/pex_root/venvs/1d025be2b3d2024670ccbfbce824218209fcd325/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/noah/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-uyyliU/.tmp/pex-pip-log.pgz9g8ks/pip.log download --dest /tmp/pants-sandbox-uyyliU/.tmp/tmp5wqgsfe6/usr.bin.python3.8 --requirement __pip_args.txt pantsbuild.pants<2.14,>=2.13.0a0 --index-url <https://pypi.org/simple/> --extra-index-url <redacted> --extra-index-url <redacted> --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants<2.14,>=2.13.0a0
ERROR: No matching distribution found for pantsbuild.pants<2.14,>=2.13.0a0




Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
this is fun
h
fun fun fun under the sun 😭 what OS and arch are you using? what is 
[python].interpreter_constraints
, and if you set it, 
[python].resolves_to_interpreter_constraints
?
c
Copy code
[python]
interpreter_constraints = [">=3.8.*,<3.11"]
enable_resolves = true
default_resolve = '<redacted>'
lockfile_generator = 'pex'
that's the whole section
e
You might also ditch the 
pants_requirements
gizmo and use a plain old pinned requirement that matches what you need. A confounding angle here may be:
Copy code
$ pip download --help
...
--pre                       Include pre-release and development versions. By default, pip only finds stable versions.
It is supposed to be the case that if you directly require a pre-release, Pip switches into 
--pre
mode automatically. The ranged requirement may be foiling it?
c
it was doing the same thing in 
2.12.0
so maybe, but I sort of doubt it?
e
Ok, that's key info. Definitely doubt it then.
c
this resolve is used for our internal plugins, so I think that target is necessary too?
h
Ahhh Noah, try setting this:
Copy code
[python]
resolves_to_interpreter_constraints = { plugin = [">=3.8,<3.10"]
e
It is but you can always say pantsbuild.pants.testutil==2.13.0a0 in a reqular requirement instead.
h
I suspect what's happening is that when generating the lockfile, if 
[python].resolves_to_interpreter_constraints
is not set, then we default to 
[python].interpreter_constraints
, so here 
">=3.8.*,<3.11"
. Pants isn't released for 3.10, so maybe Pex is failing to lock for that reason Now, unclear why this was working before. That would be the next question, if this fixes it
e
Maybe -> definitely.
👍 2
There is no Pants for Python 3.10 - lock fail
c
Oh, I think I have 3.10 on my system, that could definitely be it (we use a mix of Ubuntu 20.04 and 22.04)
all AMD64
e
On your system does not matter here - just the constraint allowing 3.10 is all that matters
c
that fixed it!
h
Great! You will also need to update 
interpreter_constraints
field for any 
python_source
and 
python_tests
targets that belong to 
resolve=plugin
to use 
>=3.8,<3.10
. See https://www.pantsbuild.org/docs/plugins-overview#building-in-repo-plugins-with-pants
e
Excellent. This was all crazy. Thanks for sticking with it.
💯 2
❤️ 2
c
it would be really great if pex told us why it wasn't resolving...
there's no updates needed, nothing there had particular constraints
e
@calm-ambulance-65371 I will say - the only fix in Pex 2.1.93+ re auth is handling Bearer auth - Digest worked prio and had integration tests.
h
there's no updates needed, nothing there had particular constraints
They will thus default to 
[python].interpreter_constraints
, which includes 3.10. Invalid. So you need to override the default by manually setting the field
e
Agreed on Pex. It's super-hard though since it calls out to Pip.
👍 1
c
is there a reason pants doesn't support 3.10?
e
No. Time / money.
Its got a tracking issue out there on GH somewhere.
c
Is Twitter still supporting Pants?
e
2 nos: No in general and if they did it would be Pants v1, which is what they use(d) - and that's a totally different beast.
👍 1
And if by support you mean tech-support, that was in large part Toolchain employees, but the community support is growing stronger and stronger. Certain backends - like docker - are well supported and the support mainly comes from non TC employees. iManage & IBM employees for example.
That 3.10 tracking link is here: https://github.com/pantsbuild/pants/issues/14111 A confounding where-to-apply-effort element is an effort to completely change how Pants ships, which, instead of adding CI shards for 3.10, would make it such that Pants includes the Python it needs - which would happen to be 3.10 - so you need not have Python installed at all on your machine. Chris is working on that in https://github.com/pantsbuild/pants/issues/7369 We're banking on the latter and so work on the former has lagged / been deferred.
👍 1
Right now were in the top ... I forget 20? ... storage abusers on PyPI because of shipping 7 native wheels.
c
ouch
7 Views
Open in Slack
PreviousNext
Powered by