https://pantsbuild.org/ logo
Join Slack
Powered by
Is there an "easy" way of temporarily overriding a...
# general
g
Is there an "easy" way of temporarily overriding a resolved dependency to use a local version of a package? For example, we might have found a bug in foo-pkg from PyPi, and would want to fix that bug and also test it with our full stack.
I can sort of do this with multiple resolves + building the dep into a wheel; but it's a bit onerous.
e
The easiest way is to replace the requirement you have with a direct reference local file URL requirement. 2nd easiest is to build that local package with a local version identifier (so that the version is unique and will not be found on PyPI) and configure Pants to find it in a local directory by adding that directory to this list: https://www.pantsbuild.org/docs/reference-python-repos#find_links
g
Yeah. Neither option is super-smooth. I thought I had a good idea with "vendoring it" but then extras become a hassle.
e
Those both seem pretty smooth to me, but I have a distortion field working on all this. What were you envisioning? Maybe we could make it so.
Fork the repo if there is one public to fork
g
Yeah. In the cases where I expect this to be a common thing is for our own packages, so we could skip pypi and depend on git-tags. Then changing to a branch + re-resolve isn't too onerous. Though I guess you'll have to re-resolve per-edit. But for local wheels that'd be build + resolve I guess? So one step less.
e
So, I still don't understand what you're envisioning as optimal. Something like editable installs?
g
Yeah; I'm not quite sure either. Editable installs; or being able to override the resolve to point at a local dir (extra src-root that's external?). But then that wouldn't have any BUILD files etc so it would maybe become problematic. And then there's edge-cases with dependencies too.
e
Option 1 I gave can do this I'm pretty sure: "thing-project @ file:///this/directory/here/with/setup.py/or/pyproject.toml/in/it"
Worth a try. Pex definitely can do it. Pants may need help with artifical url parameters to un-cache, like 
#v1
or something like that at end.
g
I think that failed if I just targeted the repository instead of a .whl or .sdist. Will try again to make sure.
Yeah, fails.
Copy code
The distribution at path '/home/ts/Repositories/foo/' does not have a file name matching known sdist or wheel file name formats
Copy code
foo-pkg @ file:///home/ts/Repositories/foo/
where 
/home/ts/Repositories/foo/
contains a 
pyproject.toml
e
I just cloned ansicolors: https://github.com/jonathaneunice/colors/
Copy code
$ pex "ansicolors @ file:///tmp/colors" -- -c 'import colors; print(colors.green("OK"))'
OK
So this is a Pants issue. Looking...
Pants often gets in the way in these areas unfortunately still.
Aha, Ok. This is Pex (me!) getting in your way. 
pex
works, 
pex3 lock create
does not. That's where that error comes from. I think this: https://github.com/pantsbuild/pex/issues/2057, which I happen to be working on, is the same problem and that fix will fix this. I'll add this as a test case in that PR. So - I do think this is the most natural way, and Pants may still get in the way, but we'll see after that Pex fix is released later today.
g
Nice; I see! I still haven't used Pex at all as a tool of its own. Out of curiosity, are Pex's updatable? One thing I've done with container images in similar situations before is to build it with all lockfiles etc but then forcibly override the one package. So could I do something like 
pants package ... && pex dist/cmd.pex 'pkg @ file://...' out.pex
?
e
PEX files are most definitely not updateable. They are intended to be totally hermetic. That said, that's not how you should be building images with PEXes. See: + Pex specific: https://pex.readthedocs.io/en/v2.1.122/recipes.html#pex-app-in-a-container + Extra options when using Pants to build PEXes: https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/
g
Cool; thank you! Yeah; it's a very pre-pants workflow 🙂 Will definitely investigate those links. Already switching buildah + PIP for pex + my own OCI builder I've cut build times by an order of magnitude for a clean build. (Admittedly with a lot less legacy cruft and dependencies too due to the dependency inference!)
e
Ok, great. Yeah, I'm on the podman stack myself.
Ok, Pex 2.1.123 now supports locking local requirements with direct reference URL syntax; so you should be able to say 
foo @ file:///this/local/python/project
. You can try it out by adding this to your `pants.toml`:
Copy code
[pex-cli]
version = "v2.1.123"
known_versions = [
    "v2.1.123|linux_arm64|3a2cba02946eb8859393906673bb56ecf6ebee72961bc8f3ca1ae754493733c6|4076395",
    "v2.1.123|linux_x86_64|3a2cba02946eb8859393906673bb56ecf6ebee72961bc8f3ca1ae754493733c6|4076395",
    "v2.1.123|macos_arm64|3a2cba02946eb8859393906673bb56ecf6ebee72961bc8f3ca1ae754493733c6|4076395",
    "v2.1.123|macos_x86_64|3a2cba02946eb8859393906673bb56ecf6ebee72961bc8f3ca1ae754493733c6|4076395",
]
g
Awesome! Thank you. I'm going to guess this might mess a bit with caching at the Pants layer now, even if it works.
Is it intentional that I should have to specify 
#egg=pkg-name
? I've got 
emote-rl @ file:///path/to-emote-rl
which fails but 
emote-rl @ file:///path/to-emote-rl#egg=emote-rl
which works.
e
Not near enough context, a full command line and error message would be good, but those look like invalid URLs. Try 
file:///...
.
g
Dropped a 
/
. It's about putting a local path in a requirements file. It works to lock, but I can't run it - then it fails with
Copy code
pex: Building pex :: Resolving distributions (atomicwrites==1.4.0 bottle~=0.12.0 coloredlogs~=15.0 emote-rl@ file:///home/ts/Repositories/emote grpcio-health-checking>=1.40.0 grpcio-reflection>=1.40.0 grpcio>=1.40.0 gym[atari,box2d,classic_control]~=0.23.0 netifaces==0.11.0 numpy~=1.20 protobuf~=3.20.1 ruamel.yaml~=0.16.0 torch==1.10.2+cpu) :: Resolving requirements from lock file locks/pants.lock :: Parsing requirements
Traceback (most recent call last):
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/bin/pex.py", line 844, in do_main
    pex_builder = build_pex(
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/bin/pex.py", line 677, in build_pex
    resolve_from_lock(
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 244, in resolve_from_lock
    subset(
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/resolve/lockfile/subset.py", line 66, in subset
    requirements_to_resolve = OrderedSet(
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/orderedset.py", line 29, in __init__
    self.update(iterable)
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/orderedset.py", line 45, in update
    for key in iterable:
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/resolve/lockfile/subset.py", line 67, in <genexpr>
    lock.local_project_requirement_mapping[os.path.abspath(parsed_requirement.path)]
KeyError: '/home/ts/Repositories/emote'
'/home/ts/Repositories/emote'
e
Can you run Pants with -ldebug and provide full output?
g
Copy code
16:20:00.95 [DEBUG] spawned local process as Some(5814) for Process { argv: ["/usr/bin/python3.9", "./pex", "--tmpdir", ".tmp", "--jobs", "13", "-vvvvvvvvv", "--python-path", "/home/ts/.local/bin:/home/ts/bin:/home/ts/.yarn/bin:/home/ts/.config/yarn/global/node_modules/.bin:/home/ts/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/Git/mingw64/bin:/mnt/c/Program Files/Git/usr/bin:/mnt/c/Users/tom.solberg/bin:/mnt/c/Python38/Scripts:/mnt/c/Python38:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0:/mnt/c/ProgramData/chocolatey/bin:/mnt/c/Program Files/Git/cmd:/mnt/c/Program Files/WireGuard:/mnt/c/windows/System32/OpenSSH:/mnt/c/Program Files/Perforce:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0:/mnt/c/WINDOWS/System32/OpenSSH:/mnt/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/mnt/c/Program Files/Microsoft VS Code/bin:/mnt/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin:/mnt/c/Program Files/nodejs:/mnt/c/Program Files/Go/bin:/mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/mnt/c/Program Files (x86)/Bluetooth Command Line Tools/bin:/mnt/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/mnt/c/Program Files/dotnet:/mnt/c/Program Files/WezTerm:/mnt/c/Users/tom.solberg/scoop/apps/llvm/current/bin:/mnt/c/Users/tom.solberg/.cargo/bin:/mnt/c/Users/tom.solberg/scoop/shims:/mnt/c/Users/tom.solberg/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/tom.solberg/.dotnet/tools:/mnt/c/Users/tom.solberg/bin:/mnt/c/Users/tom.solberg/AppData/Roaming/npm:/mnt/c/Users/tom.solberg/go/bin:/mnt/c/Users/tom.solberg/AppData/Local/Programs/Ark:/snap/bin:/home/ts/.krew/bin:/home/ts/.go/bin", "--output-file", "server.pex", "--no-emit-warnings", "--no-strip-pex-env", "--requirements-pex", "local_dists.pex", "--venv", "--seed", "verbose", "--venv-site-packages-copies", "--python", "/usr/bin/python3.9", "--entry-point", "server", "--sources-directory=source_files", "atomicwrites==1.4.0", "bottle~=0.12.0", "coloredlogs~=15.0", "emote-rl@ file:///home/ts/Repositories/emote", "grpcio-health-checking>=1.40.0", "grpcio-reflection>=1.40.0", "grpcio>=1.40.0", "gym[atari,box2d,classic_control]~=0.23.0", "netifaces==0.11.0", "numpy~=1.20", "protobuf~=3.20.1", "ruamel.yaml~=0.16.0", "torch==1.10.2+cpu", "--lock", "locks/pants.lock", "--no-pypi", "--index=<https://pypi.org/simple/>", "--index=<https://download.pytorch.org/whl/cpu/>", "--manylinux", "manylinux2014", "--layout", "packed"], env: {"CPPFLAGS": "", "LANG": "C.UTF-8", "LDFLAGS": "", "PATH": "/home/ts/.local/bin:/home/ts/bin:/home/ts/.yarn/bin:/home/ts/.config/yarn/global/node_modules/.bin:/home/ts/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/Git/mingw64/bin:/mnt/c/Program Files/Git/usr/bin:/mnt/c/Users/tom.solberg/bin:/mnt/c/Python38/Scripts:/mnt/c/Python38:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0:/mnt/c/ProgramData/chocolatey/bin:/mnt/c/Program Files/Git/cmd:/mnt/c/Program Files/WireGuard:/mnt/c/windows/System32/OpenSSH:/mnt/c/Program Files/Perforce:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0:/mnt/c/WINDOWS/System32/OpenSSH:/mnt/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/mnt/c/Program Files/Microsoft VS Code/bin:/mnt/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin:/mnt/c/Program Files/nodejs:/mnt/c/Program Files/Go/bin:/mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/mnt/c/Program Files (x86)/Bluetooth Command Line Tools/bin:/mnt/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/mnt/c/Program Files/dotnet:/mnt/c/Program Files/WezTerm:/mnt/c/Users/tom.solberg/scoop/apps/llvm/current/bin:/mnt/c/Users/tom.solberg/.cargo/bin:/mnt/c/Users/tom.solberg/scoop/shims:/mnt/c/Users/tom.solberg/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/tom.solberg/.dotnet/tools:/mnt/c/Users/tom.solberg/AppData/Roaming/npm:/mnt/c/Users/tom.solberg/go/bin:/mnt/c/Users/tom.solberg/AppData/Local/Programs/Ark:/snap/bin:/home/ts/.krew/bin:/home/ts/.go/bin", "PEX_IGNORE_RCFILES": "true", "PEX_ROOT": ".cache/pex_root"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<25136ed8b0c16d712fa2d39c6dae8bff9730ebf5dca29c314d1d4872185308ce>, size_bytes: 411 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<25136ed8b0c16d712fa2d39c6dae8bff9730ebf5dca29c314d1d4872185308ce>, size_bytes: 411 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: {} }, output_files: {}, output_directories: {RelativePath("server.pex")}, timeout: None, execution_slot_variable: None, concurrency_available: 13, description: "Building 13 requirements for server.pex from the locks/pants.lock resolve: atomicwrites==1.4.0, bottle~=0.12.0, coloredlogs~=15.0, emote-rl@file:///home/ts/Repositories/emote, grpcio-health-checking>=1.40.0, grpcio-reflection>=1.40.0, grpcio>=1.40.0, gym[atari,box2d,classic_control]~=0.23.0, netifaces==0.11.0, numpy~=1.20, protobuf~=3.20.1, ruamel.yaml~=0.16.0, torch==1.10.2+cpu", level: Info, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root")}, jdk_home: None, platform: Linux_x86_64, cache_scope: Successful, execution_strategy: Local, remote_cache_speculation_delay: 0ns }
16:20:01.88 [INFO] Completed: Building 13 requirements for server.pex from the locks/pants.lock resolve: atomicwrites==1.4.0, bottle~=0.12.0, coloredlogs~=15.0, emote-rl@ file:///home/ts/Repositories/emote, grpcio-health-checking>... (182 characters truncated)
16:20:01.88 [DEBUG] Completed: Scheduling: Building 13 requirements for server.pex from the locks/pants.lock resolve: atomicwrites==1.4.0, bottle~=0.12.0, coloredlogs~=15.0, emote-rl@ file:///home/ts/Repositories/emote, grpcio-heal... (194 characters truncated)
16:20:01.88 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
16:20:01.88 [DEBUG] Completed: pants.backend.python.goals.run_python_source.create_python_source_run_request
16:20:01.88 [DEBUG] Completed: `run` goal
16:20:01.88 [DEBUG] computed 1 nodes in 4.770677 seconds. there are 3194 total nodes.
16:20:01.88 [ERROR] 1 Exception encountered:

Engine traceback:
  in `run` goal

ProcessExecutionFailure: Process 'Building 13 requirements for server.pex from the locks/pants.lock resolve: atomicwrites==1.4.0, bottle~=0.12.0, coloredlogs~=15.0, emote-rl@ file:///home/ts/Repositories/emote, grpcio-health-checking>=1.40.0, grpcio-reflection>=1.40.0, grpcio>=1.40.0, gym[atari,box2d,classic_control]~=0.23.0, netifaces==0.11.0, numpy~=1.20, protobuf~=3.20.1, ruamel.yaml~=0.16.0, torch==1.10.2+cpu' failed with exit code 1.
stdout:

stderr:
pex: Resolving interpreters
pex: Resolving interpreters: 0.1ms
pex: Parsing lock locks/pants.lock
pex: Parsing lock locks/pants.lock: 200.8ms
pex: Building pex
pex: Building pex :: Laying out Spread PEX directory local_dists.pex
pex: Building pex :: Searching dependency cache: /home/ts/.cache/pants/named_caches/pex_root/unzipped_pexes/1a8ef61dcd71222b70c7538d5cce995f77812b71/.deps
pex: Building pex :: Resolving distributions (atomicwrites==1.4.0 bottle~=0.12.0 coloredlogs~=15.0 emote-rl@ file:///home/ts/Repositories/emote grpcio-health-checking>=1.40.0 grpcio-reflection>=1.40.0 grpcio>=1.40.0 gym[atari,box2d,classic_control]~=0.23.0 netifaces==0.11.0 numpy~=1.20 protobuf~=3.20.1 ruamel.yaml~=0.16.0 torch==1.10.2+cpu)
pex: Building pex :: Resolving distributions (atomicwrites==1.4.0 bottle~=0.12.0 coloredlogs~=15.0 emote-rl@ file:///home/ts/Repositories/emote grpcio-health-checking>=1.40.0 grpcio-reflection>=1.40.0 grpcio>=1.40.0 gym[atari,box2d,classic_control]~=0.23.0 netifaces==0.11.0 numpy~=1.20 protobuf~=3.20.1 ruamel.yaml~=0.16.0 torch==1.10.2+cpu) :: Resolving requirements from lock file locks/pants.lock
pex: Building pex :: Resolving distributions (atomicwrites==1.4.0 bottle~=0.12.0 coloredlogs~=15.0 emote-rl@ file:///home/ts/Repositories/emote grpcio-health-checking>=1.40.0 grpcio-reflection>=1.40.0 grpcio>=1.40.0 gym[atari,box2d,classic_control]~=0.23.0 netifaces==0.11.0 numpy~=1.20 protobuf~=3.20.1 ruamel.yaml~=0.16.0 torch==1.10.2+cpu) :: Resolving requirements from lock file locks/pants.lock :: Parsing requirements
Traceback (most recent call last):
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/bin/pex.py", line 844, in do_main
    pex_builder = build_pex(
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/bin/pex.py", line 677, in build_pex
    resolve_from_lock(
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 244, in resolve_from_lock
    subset(
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/resolve/lockfile/subset.py", line 66, in subset
    requirements_to_resolve = OrderedSet(
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/orderedset.py", line 29, in __init__
    self.update(iterable)
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/orderedset.py", line 45, in update
    for key in iterable:
  File "/home/ts/.cache/pants/named_caches/pex_root/installed_wheels/ac5601bb1fbcf211ba841ae9ee25afc952bbd0d71cff2922dd0671d291464b89/pex-2.1.123-py2.py3-none-any.whl/pex/resolve/lockfile/subset.py", line 67, in <genexpr>
    lock.local_project_requirement_mapping[os.path.abspath(parsed_requirement.path)]
KeyError: '/home/ts/Repositories/emote'
'/home/ts/Repositories/emote'
e
Ok. Thanks, I'll take a look.
g
Thanks! 🙂 I can also confirm (probably to no surprise, really) that this interacts quite badly with caching. It's hard to reliably get the pex to update from the local package when editing it. The most reliable iteration right now is to 
generate-lockfiles
, which I guess dirties all downstream packages. Which is fine, but takes 90 seconds to do in our tiny workspace.
e
Yeah, that bit I could have told you up front. The lock is considered valid by Pants until you change requirement string text and that stays constant with your local file:// URL.
Hrm, yeah. Bad software. The test I added in Pex 2.1.123 tests creating a lock with a "foo @file:///bar" requirements and then using that lock, but it uses the whole lock (specifies no requirements) when building a PEX from it. Add a requirement to engage lock subsetting when building the PEX and the test fails just as you are experiencing:
Copy code
diff --git a/tests/integration/cli/commands/test_issue_2057.py b/tests/integration/cli/commands/test_issue_2057.py
index a2b0e52..bb875f9 100644
--- a/tests/integration/cli/commands/test_issue_2057.py
+++ b/tests/integration/cli/commands/test_issue_2057.py
@@ -119,6 +119,7 @@ def test_lock_create_local_project_direct_reference(tmpdir):
                 pex_root,
                 "--lock",
                 lock,
+                "ansicolors @ file://{}".format(clone_dir),
                 "--",
                 "-c",
                 "import colors; print(colors.yellow('Vogon Constructor Fleet!'))",

$ tox -epy311-integration -- -vvsk test_issue_2057
...
E       pex: Building pex :: Resolving distributions (ansicolors @ file:///tmp/pytest-of-jsirois/pytest-2/popen-gw2/test_lock_create_local_project0/ansicolors) :: Resolving requirements from lock file /tmp/pytest-of-jsirois/pytest-2/popen-gw2/test_lock_create_local_project0/lock.json :: Parsing requirements
E       Traceback (most recent call last):
E         File "/home/jsirois/dev/pantsbuild/jsirois-pex/pex/result.py", line 105, in catch
E           return func(*args, **kwargs)
E                  ^^^^^^^^^^^^^^^^^^^^^
E         File "/home/jsirois/dev/pantsbuild/jsirois-pex/pex/bin/pex.py", line 856, in do_main
E           pex_builder = build_pex(
E                         ^^^^^^^^^^
E         File "/home/jsirois/dev/pantsbuild/jsirois-pex/pex/bin/pex.py", line 689, in build_pex
E           resolve_from_lock(
E         File "/home/jsirois/dev/pantsbuild/jsirois-pex/pex/resolve/lock_resolver.py", line 244, in resolve_from_lock
E           subset(
E         File "/home/jsirois/dev/pantsbuild/jsirois-pex/pex/resolve/lockfile/subset.py", line 66, in subset
E           requirements_to_resolve = OrderedSet(
E                                     ^^^^^^^^^^^
E         File "/home/jsirois/dev/pantsbuild/jsirois-pex/pex/orderedset.py", line 29, in __init__
E           self.update(iterable)
E         File "/home/jsirois/dev/pantsbuild/jsirois-pex/pex/orderedset.py", line 45, in update
E           for key in iterable:
E         File "/home/jsirois/dev/pantsbuild/jsirois-pex/pex/resolve/lockfile/subset.py", line 67, in <genexpr>
E           lock.local_project_requirement_mapping[os.path.abspath(parsed_requirement.path)]
E           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       KeyError: '/tmp/pytest-of-jsirois/pytest-2/popen-gw2/test_lock_create_local_project0/ansicolors'
E       '/tmp/pytest-of-jsirois/pytest-2/popen-gw2/test_lock_create_local_project0/ansicolors'
I'll get an issue filed here momentarily and see what I can do.
g
I have a workaround so low prio for my side fwiw, though obviously unexpected. And yeah; I know why the caching is bad - I did expect it, after all. I'd obviously be happier if I'd been wrong 😉
e
Alright, this should fix:
Copy code
[pex-cli]
version = "v2.1.127"
known_versions = [
    "v2.1.127|linux_arm64|0e8e7cd9abb5f9e7d88fb5b0de023c67eaeefe502e804b01ed775705b4c8ca87|4081107",
    "v2.1.127|linux_x86_64|0e8e7cd9abb5f9e7d88fb5b0de023c67eaeefe502e804b01ed775705b4c8ca87|4081107",
    "v2.1.127|macos_arm64|0e8e7cd9abb5f9e7d88fb5b0de023c67eaeefe502e804b01ed775705b4c8ca87|4081107",
    "v2.1.127|macos_x86_64|0e8e7cd9abb5f9e7d88fb5b0de023c67eaeefe502e804b01ed775705b4c8ca87|4081107",
]
g
Awesome, thank you! 🥳
Open in Slack
PreviousNext
Powered by