https://pantsbuild.org/ logo
Join Slack
Powered by
When using a separate `[python-repo].repos` for a ...
# general
r
When using a separate 
[python-repo].repos
for a custom find-links option to fetch custom prebuilt 
grpcio
wheels (built with linux-aarch64 and macos-arm64) and I run 
./pants generate-lockfiles
in macOS, it tries to fetch the linux-aarch64 whl file and says:
Copy code
OSError: pid 67995 -> /Users/joongi/.cache/pants/named_caches/pex_root/venvs/3865e68cbdb1d654d6cca639c59b5dfbc8be0dda/f57e61ad0b603cc5d644c45312fc4081d6790c14/bin/python -sE /Users/joongi/.cache/pants/named_caches/pex_root/venvs/3865e68cbdb1d654d6cca639c59b5dfbc8be0dda/f57e61ad0b603cc5d644c45312fc4081d6790c14/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /Users/joongi/.cache/pants/named_caches/pex_root/pip_cache --log /Users/joongi/workspace/bai-grpcio/.tmp/process-executionSNsh5K/.tmp/pex-pip-log.h1stom78/pip.log download --dest /Users/joongi/.cache/pants/named_caches/pex_root/downloads/fingerprint_artifact.8wtp11jr --no-deps <http://localhost:8000/grpcio-1.47.0-cp310-cp310-manylinux_2_17_aarch64.whl> --index-url <https://pypi.org/simple/> --find-links <http://localhost:8000> --trusted-host localhost:8000 --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: grpcio-1.47.0-cp310-cp310-manylinux_2_17_aarch64.whl is not a supported wheel on this platform.
The find-links target is served from 
python -m http.server
from a directory that contains:
Copy code
-rw-r--r--@ 1 joongi  staff   2981258 Jul 13 23:10 grpcio-1.47.0-cp310-cp310-macosx_12_0_arm64.whl
-rw-r--r--  1 joongi  staff  62209358 Jul 14 01:39 grpcio-1.47.0-cp310-cp310-manylinux_2_17_aarch64.whl
-rw-r--r--@ 1 joongi  staff   1941172 Jul 13 23:10 grpcio_tools-1.47.0-cp310-cp310-macosx_12_0_arm64.whl
-rw-r--r--  1 joongi  staff  31484142 Jul 14 01:39 grpcio_tools-1.47.0-cp310-cp310-manylinux_2_17_aarch64.whl
How could I let it resolve the correct wheel?
e
@rhythmic-morning-87313 what version of Pants is this?
r
2.12.0
tried with both pex 2.1.94 and 2.1.97, but the results were same
e
Ok, yeah 2.12.0 is Pex 2.1.90 by default which uses the older download code which does not have this issue I don't think. This issue is specifically with using 
pip download
to do the downloading to enable auth working for custom indexes that use the `Bearer`authentication scheme. I've lost track - are you using a custom Pex version for that exact reason?
r
no, i'm using it to avoid alternating platform tag problem when regenerating lockfiles from linux or macos
e
Ah. Well, that's the price you'll pay for now. Let me repro this issue and file a Pex bug.
👀 1
Thanks for finding this, it affects 2.13.x and 2.14.x default Pex which is 2.1.96 at this point.
👍 1
Ah, right. I don't have the repro yet, but this all makes sense. Its only an issue if you use a custom repo that does not serve up hashes in the URLs like PyPI does. In that case Pex needs to download each artifact to hash it in order to create the lock. You are exactly in this narrow case with your custom find-links repo.
OK, fully sussed here: https://github.com/pantsbuild/pex/issues/1849 I'm going to try to get out a fix for this in 2.1.98 this afternoon if possible.
Alright, a fix is here: https://github.com/pantsbuild/pex/pull/1851 Thanks again for finding that @rhythmic-morning-87313
And Pex 2.1.98 is now released with the fix. If you can give that a try @rhythmic-morning-87313 tomorrow and report back I'll be grateful.
🙌 1
r
@enough-analyst-54434 confirmed the new release works as expected for us!
👍
eh.....
i'm seeing a new error..
(which seems not directly related to pex#1849 / pex#1851)
Copy code
Error: 1.35 [ERROR] 1 Exception encountered:
  ProcessExecutionFailure: Process 'Building black.pex from black_default.lock' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/runner/.cache/pants/named_caches/pex_root/unzipped_pexes/eb0f832b1cd25529fca948fec79afb4cf08e01ce/__main__.py", line 103, in <module>
    bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
  File "/home/runner/.cache/pants/named_caches/pex_root/unzipped_pexes/eb0f832b1cd25529fca948fec79afb4cf08e01ce/.bootstrap/pex/pex_bootstrapper.py", line 601, in bootstrap_pex
    pex.PEX(entry_point).execute()
  File "/home/runner/.cache/pants/named_caches/pex_root/unzipped_pexes/eb0f832b1cd25529fca948fec79afb4cf08e01ce/.bootstrap/pex/pex.py", line 528, in execute
    sys.exit(self._wrap_coverage(self._wrap_profiling, self._execute))
  File "/home/runner/.cache/pants/named_caches/pex_root/unzipped_pexes/eb0f832b1cd25529fca948fec79afb4cf08e01ce/.bootstrap/pex/pex.py", line 435, in _wrap_coverage
    return runner(*args)
  File "/home/runner/.cache/pants/named_caches/pex_root/unzipped_pexes/eb0f832b1cd25529fca948fec79afb4cf08e01ce/.bootstrap/pex/pex.py", line 466, in _wrap_profiling
    return runner(*args)
  File "/home/runner/.cache/pants/named_caches/pex_root/unzipped_pexes/eb0f832b1cd25529fca948fec79afb4cf08e01ce/.bootstrap/pex/pex.py", line 555, in _execute
    return self.execute_entry(
  File "/home/runner/.cache/pants/named_caches/pex_root/unzipped_pexes/eb0f832b1cd25529fca948fec79afb4cf08e01ce/.bootstrap/pex/pex.py", line 737, in execute_entry
    return self.execute_entry_point(entry_point)
  File "/home/runner/.cache/pants/named_caches/pex_root/unzipped_pexes/eb0f832b1cd25529fca948fec79afb4cf08e01ce/.bootstrap/pex/pex.py", line 755, in execute_entry_point
    return runner()
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/bin/pex.py", line 775, in main
    catch(
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/result.py", line 103, in catch
    return func(*args, **kwargs)
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/bin/pex.py", line 798, in do_main
    pex_builder = build_pex(
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/bin/pex.py", line 641, in build_pex
    resolve_from_lock(
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 318, in resolve_from_lock
    pool.map(
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 196, in download_artifact
    return catch(
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/result.py", line 103, in catch
    return func(*args, **kwargs)
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/resolve/lockfile/download_manager.py", line 154, in store
    self.save(
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 66, in save
    return self._downloader.download(artifact=artifact, dest_dir=dest_dir, digest=digest)
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/resolve/downloads.py", line 169, in download
    self._download(url=artifact.url, download_dir=dest_dir).wait()
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/resolve/downloads.py", line 101, in _download
    return get_pip().spawn_download_distributions(
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/pip/tool.py", line 755, in get_pip
    pip = Pip.create(path=os.path.join(ENV.PEX_ROOT, "pip.pex"), interpreter=interpreter)
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/pip/tool.py", line 278, in create
    return cls(ensure_venv(PEX(pip_pex_path, interpreter=pip_interpreter)))
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/pex_bootstrapper.py", line 532, in ensure_venv
    shebang = populate_venv(
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/venv/pex.py", line 120, in populate_venv
    record_provenance(_populate_deps(venv, pex, venv_python, symlink))
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/venv/pex.py", line 116, in record_provenance
    for src, dst in src_to_dst:
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/venv/pex.py", line 275, in _populate_deps
    for script in venv.rewrite_scripts(python=venv_python, python_args="-sE"):
  File "/home/runner/.cache/pants/named_caches/pex_root/installed_wheels/2e490f9090701724ab89e69f23d9779b74a2e8cc69c92c59c2149e620c28c30f/pex-2.1.98-py2.py3-none-any.whl/pex/venv/virtualenv.py", line 310, in rewrite_scripts
    fileinput.input(files=sorted(python_scripts), inplace=True, mode="rb")
  File "/usr/lib/python3.8/fileinput.py", line 92, in input
    raise RuntimeError("input() already active")
RuntimeError: input() already active
ok, this happens when i run:
Copy code
./pants tailor --check update-build-files --check
but does NOT happen when i run:
Copy code
./pants tailor --check
./pants update-build-files --check
i'm not sure this is a side effect of having pex 2.1.98 on pants 2.12.0, or something else......
strangely, in the PR's action runs, it did not happen but only happened after merge..;;
4 Views
Open in Slack
PreviousNext
Powered by