Copy code

ERROR: Failed to build '<https://aws>:****@<redacted>.<http://d.codeartifact.us-east-1.amazonaws.com/pypi/altana/simple/psycopg2-binary/2.9.12/psycopg2_binary-2.9.12.tar.gz|d.codeartifact.us-east-1.amazonaws.com/pypi/altana/simple/psycopg2-binary/2.9.12/psycopg2_binary-2.9.12.tar.gz>' when getting requirements to build wheel

2.9.9

2.9.12

Now the picture is complete and definitive:

- PyPI sandbox → experiment-psycopg2.lock was written (4861 bytes) = SUCCESS
- CodeArtifact sandbox → no lock file = FAILURE

…yet both pex-pip-download.log files end in Would install psycopg2-binary-2.9.12 with zero errors. So the failure happens in the pex universal-lock step that runs after pip download — which this log doesn't capture. Let me grab the actual pex error by re-running the CodeArtifact invocation (its token may still be valid — the run was earlier today).

The two pex-pip-download.log files are essentially identical — that's the trap

I diffed them every way that matters. For psycopg2-binary, both logs succeed the same way:

┌─────────────────────────┬──────────────────────────────────────┬──────────────────────────────────────┐
│                         │        CodeArtifact (j7VyQZ)         │            PyPI (JhtdIw)             │
├─────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────┤
│ Versions offered        │ 2.7.x–2.9.12 (identical)             │ 2.7.x–2.9.12 (identical)             │
├─────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────┤
│ cp312 wheels for 2.9.12 │ identical 11 wheels                  │ identical 11 wheels                  │
├─────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────┤
│ Resolved                │ psycopg2-binary 2.9.12               │ psycopg2-binary 2.9.12               │
├─────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────┤
│ Last line               │ Would install psycopg2-binary-2.9.12 │ Would install psycopg2-binary-2.9.12 │
├─────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────┤
│ error/warning markers   │ 0                                    │ 0                                    │
└─────────────────────────┴──────────────────────────────────────┴──────────────────────────────────────┘

The one real difference in the logs: PyPI serves PEP 658 metadata sidecars, so pip fetches a 4.9 kB ...whl.metadata file (Obtaining dependency information ... 5 .metadata refs). CodeArtifact advertises no PEP 658 metadata (0 .metadata refs), so pip downloads the full 3.4 MB wheel to read its metadata. That's a symptom, not the cause.

The actual failure isn't in this log at all

pex-pip-download.log only captures pip's resolve sub-step. The lock failure happens in the next pex step, which your full -ldebug run shows plainly:

- PyPI → Wrote lockfile ... experiment-psycopg2.lock ✅ (I confirmed the file exists in that sandbox; j7VyQZ has no lock)
- CodeArtifact → ProcessExecutionFailure: 'Generate lockfile for experiment-psycopg2' failed with exit code 1, at pex's fingerprint_artifact step:

pip download --no-deps  .../psycopg2-binary/2.9.12/psycopg2_binary-2.9.12.tar.gz   ← the SDIST
  × python setup.py egg_info did not run successfully.
    Error: pg_config executable not found.
    pg_config is required to build psycopg2 from source.
  error: metadata-generation-failed

I re-ran the CodeArtifact __run.sh and reproduced this exactly — so it's not a stale-token fluke.

Why the sdist gets built against CodeArtifact but not PyPI

A --style=universal lock records every artifact of the locked version — including the .tar.gz sdist — each with a sha256. The successful PyPI lock proves it (the sdist is recorded with hash 5ac9444… next to all the wheel hashes).

- PyPI hands pex the per-file hashes + PEP 658 core-metadata inline in the Simple API, so pex records the sdist's hash without ever downloading it.
- CodeArtifact exposes neither (the same deficiency you see as "no .metadata" in the download log), so pex is forced to download the sdist to fingerprint it — and pip download of an sdist runs its setup.py, which needs pg_config (libpq dev headers), absent on the lock host → the whole lock dies.

So: "besides the index URL" the download logs genuinely differ in nothing that fails — the divergence is that CodeArtifact's index is not hash/PEP‑658‑complete, and that gap only bites downstream, on the sdist fingerprint.

Fix

Stop pex from fingerprinting/building the sdist — same knob that resolved the DX‑2101/DX‑2128 lockfile-gen outage:

[python]
resolves_to_only_binary = { experiment-psycopg2 = [":all:"] }   # or = ["psycopg2-binary"]

With only-binary, the sdist isn't a lock candidate, nothing gets built, and it locks against CodeArtifact. (Installing pg_config on the lock host would also "work" but is the wrong fix for a binary-wheel package.) I saw you're already in pants.toml — that's the line.

Confirmed — and I verified it against the actual version, not the clone's HEAD.

First, the version check: your clone was on v2.97.3, but the pex that Pants 2.30.0 actually ran is 2.66.0 — from the sandbox ./pex itself:
- PEX-INFO → "build_properties":{"pex_version":"2.66.0"}
- .deps/pex-2.66.0-py2.py3-none-any.whl/pex/version.py → __version__ = "2.66.0"

I checked out v2.66.0 in ~/pex and re-read the code (it's verbatim identical to 2.97.3, same line numbers). Your suspicion is exactly right. The chain:

1. Fingerprint comes from the index link's #sha256= fragment — pex/artifact_url.py:208-223:
fragment_parameters = parse_qs(url_info.fragment)
if fragment_parameters:
    # Artifact URLs from indexes may contain pre-computed hashes. We isolate those here...
    # See: <https://peps.python.org/pep-0503/#specification>
    for alg in RANKED_ALGORITHMS:
        hashes = fragment_parameters.pop(alg, None)
        ...
        fingerprints.append(Fingerprint(algorithm=alg, hash=hashes[0]))
…and pex/artifact_url.py:268-270:
def fingerprint(self):
    return self.fingerprints[0] if self.fingerprints else None   # no #sha256= → None

2. That flows straight into the artifact — pex/resolve/locker.py:288:
partial_artifact = PartialArtifact(artifact_url, fingerprint=artifact_url.fingerprint)

3. The decision to download — pex/resolve/downloads.py:141-149:
def _to_file_artifact(self, artifact):
    fingerprint = artifact.fingerprint
    if fingerprint:                                # index gave a hash → done, no download
        return SpawnedJob.completed(self._create_file_artifact(url, fingerprint, ...))
    return self._download_and_fingerprint(url)     # no hash → must download to hash it

4. Download-to-fingerprint runs pip on the artifact — downloads.py:117-139 → _download (109-115):
download_dir = safe_mkdtemp(prefix="fingerprint_artifact.", ...)   # ← your temp dir
...
return self.pip.spawn_download_distributions(requirements=[download_url], transitive=False, ...)
For a .tar.gz sdist, pip download prepares metadata (setup.py egg_info) → psycopg2 needs pg_config → metadata-generation-failed.

So it's precisely as suspected: PyPI puts #sha256= on every link, so fingerprint is set and pex never touches the sdist. CodeArtifact's Simple API serves those links without a usable hash fragment, so fingerprint is None, pex hits the else branch, downloads the sdist to hash it, and that download triggers the source build that dies on pg_config.

The extra confirmation the source gives: under --style=universal the sdist is always recorded as a fallback artifact even though a wheel was selected — which is why it gets fingerprinted at all, and why any package with a build-requiring sdist breaks against CodeArtifact (matching adbc/ddtrace from the earlier outage). resolves_to_only_binary drops the sdist from the artifact set, so there's nothing to fingerprint.

So it's precisely as suspected: PyPI puts #sha256= on every link, so fingerprint is set and pex never touches the sdist. CodeArtifact's Simple API serves those links without a usable hash fragment, so fingerprint is None, pex hits the else branch, downloads the sdist to hash it, and that download triggers the source build that dies on pg_config.

[python.resolves_to_only_binary]

@brief-scientist-13682 — proxy experiment done, theory confirmed, and I now have the full mechanism. Everything below is verifiable against pex source; line numbers are v2.66.0 (the relevant files are unchanged through 2.98.2 — 2.93.3/.4 only added FIPS guards — and I reproduced on 2.98.2 too).

1. The trigger, proven by A/B. I put a ~40-line localhost proxy in front of our CodeArtifact repo that rewrites pip's

Accept

header to

text/html

on simple-index pages and passes everything else through. Same pex 2.66.0, same pip 24.2, same repo,

--style universal

, `psycopg2-binary==2.9.12`:

• direct (pip negotiates PEP 691 JSON since the rollout) →

metadata-generation-failed: pg_config executable not found

• through the proxy (forced PEP 503 HTML) → lock succeeds, sdist recorded with sha256, and pex's

downloads/

dir is empty — zero fingerprint downloads

So nothing was nuked and no hashes were dropped: CodeArtifact's HTML still carries

#sha256=

fragments on every link (why it worked for 2 years, and why forcing HTML works today). What changed is which format pip receives — pip ≥22.2 always sends

Accept: application/vnd.pypi.simple.v1+json, …+html;q=0.1, text/html;q=0.01

, and CodeArtifact only just started honoring it. There's no pip flag to turn that off; the header is hardcoded in

_get_simple_response

.

2. Why JSON flips pex into download-everything mode. Per PEP 691 the JSON

url

carries no hash fragment — hashes move to the per-file

hashes

object. So

ArtifactURL.fingerprint

is None for every artifact and

_to_file_artifact

falls into

_download_and_fingerprint

, exactly as you suspected. (This also reconciles the "identical pip logs" confusion: the resolve step picks the wheel in both eras and never touches the sdist — the sdist build happens in the separate per-artifact

pip download --no-deps <sdist-url>

fingerprint invocations afterwards.)

But pex has machinery for this case:

Locker

records PEP-691-capable endpoints by scraping pip's

Fetched page <url> as application/vnd.pypi.simple.v1+json

log lines (locker.py:531-537), then

FingerprintService.fingerprint(endpoints=…)

(locker.py:641) re-queries the JSON API for the

hashes

dicts before any download fallback.

3. The actual pex bug: that FingerprintService can't authenticate to CodeArtifact — two independent ways.

• (a) It inherits pip's log redaction. Our index URL carries basic auth (

<https://aws|aws>:<token>@<domain>-<acct>.d.codeartifact.<region>.<http://amazonaws.com/pypi/|amazonaws.com/pypi/><repo>/simple

). pip logs page URLs through

redact_auth_from_url

, so the endpoint pex scrapes literally contains

aws:****

. With `PEX_VERBOSE=3`:

pex: Fetching PEP-691 index metadata from <https://aws|aws>:****@<host>/pypi/<repo>/simple/<project>/ for application/vnd.pypi.simple.v1+json

→ it authenticates with the literal password

****

→ 401.

• (b) Even with a clean endpoint URL and creds available (netrc; and create.py:304-308 does wire

password_entries

into the service's URLFetcher), urllib can't complete the request: CodeArtifact answers 401 with

WWW-Authenticate: Bearer

, and urllib raises

ValueError: AbstractDigestAuthHandler does not support the following scheme: 'Bearer'

Repro is one line:

URLFetcher().get_body_stream("https://<host>/pypi/<repo>/simple/<project>/")

with creds in

~/.netrc

. Looks like the same family as #1803/#1811, still alive on this path.

• Every failure is swallowed by

catch(self._api.request, endpoint)

(fingerprint_service.py:132-134), so the service degrades silently to download+fingerprint for all artifacts. That silence is what made this so hard to diagnose.

4. Why the blast radius was "everything, at once". Universal locks record the sdist as a fallback artifact for ~every project. Pre-rollout, all of them were fingerprinted for free off HTML fragments; post-rollout, every one gets downloaded, and pip ≥22 runs PEP-517 metadata prep on each — so any sdist that can't build on the lock host kills the whole lock: pg_config for psycopg2-binary, missing rust/native toolchains for others, and old `setup.py`s doing

import pkg_resources

, which is freshly fatal because build isolation installs latest setuptools and ≥81 removed pkg_resources. Two-year-old pins "suddenly compiling" was never about the pins.

5. Asks. Would you take an issue (I'll file it with full token-redacted PEX_VERBOSE traces) for:

1. re-attaching known credentials from the PasswordDatabase to scraped PEP-691 endpoints instead of the redacted

****

userinfo,

2. tolerating

WWW-Authenticate: Bearer

on 401 (fall back to Basic when creds are known),

3. a warning instead of a silent download fallback when every PEP-691 fingerprint request fails?

Fixing those would make pex lock creation zero-download against CodeArtifact JSON again. Happy to test patches against our repo. Meanwhile we're unblocked via the Accept-rewrite proxy for regens,

PIP_CONSTRAINT=setuptools<81

threaded through

_PEX_USE_PIP_CONFIG=true

, and

resolves_to_only_binary

for the never-buildable sdists — and credit where due: only_binary as a debugging tool is what isolated the sdist path.