Thoughts on handling forked dependencies? Our mon...
# general
s
Thoughts on handling forked dependencies? Our monorepo workflow sometimes requires us to (hopefully temporarily) fork dependencies. When that happens, our pre-pants workflow was as follows: 1. fork the dep to our own separate git repo, not the monorepo (ex github:alextremblay/dependency-name 2. update pyproject.toml dependency from
dependency-name >= ...
to
dependency-name @ git+<https://github>....
3. add this fork as a git submodule to the monorepo 4.
pip install -e custom-forks/dependency-name
so we can iterate and test our change to the fork live within our activated venv 5. commit / push changes to the forked repo so that our published packages which depend on this forked dep can pull / install the updated code from github when deployed in the wild 6. work on getting our changes merged upstream (can sometimes take months or longer 😞 ) I'm trying to achieve the same with pants, but it's proving to be difficult. setting the dependency to point to a git repo works fine (unless you try to pin lockfile deps with a constraints file, at which point it just entirely blows up But I can't figure out how to tell pants to override that dep with a local editable install when exporting the venv or running any tasks which include that dep Thoughts?
👍 1
b
Re the parenthetical re constraints, did I hallucinate this?: https://github.com/pex-tool/pex/issues/3089 Have you upgraded Pex and still hit the constraints issue @stale-twilight-79248?
s
you did not. U[pgrading pex DID solve that issue, but i then immediately ran into another constraints related issue. at that point i got so frustrated i just gave up on the whole uv lock -> constraints -> pants lock idea. I will probably eventually get back to it, repro the error and open an issue
b
That would be great. It's slightly cruel to bait me with a bug I can't see. I don't like Pex bugs!
h
It's not 100% straightforward, but you can build the submodule via its own setup.py using Pants's "local dists" concept, and then consume that on local runs. See here for example: https://github.com/benjyw/example-python/compare/main...forked_thirdparty_dep
there are a couple of caveats
s
that's awesome! Thank you guys! lol sorry @brief-scientist-13682, i can see how that would suck. I'll carve out some time today to repro and open an issue. @happy-kitchen-89482 have you any idea how to get pants to
pip install -e
the local fork when exporting a venv?
b
So, Pex can lock editable (although its not easy right now, a requirements.txt as input is required):
Copy code
:; git clone <https://github.com/VaasuDevanS/cowsay-python>
:; cat requirements.txt 
-e cowsay-python/
:; pex3 lock create -r requirements.txt --indent 2 -o lock.json
# Note `"editable": true,` in the artifacts list below:
:; jq .locked_resolves[0].locked_requirements lock.json 
[
  {
    "artifacts": [
      {
        "algorithm": "sha256",
        "editable": true,
        "hash": "111eb24cead5476d6384dfe692e61f73a897275b48373bcc5385c0b4bfbf7dc2",
        "url": "file:///home/jsirois/support/pants/slack/3-4-2026/cowsay-python"
      }
    ],
    "project_name": "cowsay",
    "requires_dists": [
      "coverage; extra == \"test\"",
      "pytest; extra == \"test\""
    ],
    "requires_python": ">=3.8",
    "version": "6.1"
  }
]
But Pex cannot install as editable:
Copy code
:; pex3 venv create --lock lock.json -d editable-venv

# Edit local clone and show this breaks it:
:; echo "INVALID" >> cowsay-python/cowsay/__main__.py 
:; PYTHONPATH=cowsay-python/ python -mcowsay -t 'Moo!'
  ____
| Moo! |
  ====
    \
     \
       ^__^
       (oo)\_______
       (__)\       )\/\
           ||----w |
           ||     ||
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/jsirois/support/pants/slack/3-4-2026/cowsay-python/cowsay/__main__.py", line 34, in <module>
    INVALID
NameError: name 'INVALID' is not defined

# But the venv has pristinely installed copies and no break:
:; editable-venv/bin/python -mcowsay -t 'Moo!'
  ____
| Moo! |
  ====
    \
     \
       ^__^
       (oo)\_______
       (__)\       )\/\
           ||----w |
           ||     ||
That said @stale-twilight-79248 re: > But I can't figure out how to tell pants to override that dep with a local editable install when exporting the venv or running any tasks which include that dep Why are you trying to do things differently than item 4 in your initial list? Can't you just export a venv using the existing Pants means and then, exactly as in 4. above:
pip install -e custom-forks/dependency-name
? If not, what is the error?
Also, @happy-kitchen-89482's example includes https://www.pantsbuild.org/dev/reference/goals/export#py_editable_in_resolve no-where; so I'm not sure how that adresses an exported venv with editable installs. That said, the
py_editable_in_resolve
looks promising, but I'll leave that dig in to you all. I'm just sniffing out Pex bugs I can fix to help with any needed workarounds outside Pants.
@stale-twilight-79248 I'm still waiting on a repro / bug report, but your case did get me working on https://github.com/pex-tool/pex/releases/tag/v2.91.0 With that you can now
pex3 venv create --lock lock.json --override='-e foo @ this/local/foo' --override='-e bar @ this/local/bar' -d /venv/dir
to create a venv with overrides of the locked
foo
and
bar
projects using local editable installs. It looks like Pants does not create venvs directly from the lock like this for pants export and instead it creates a PEX 1st, and then runs
PEX_TOOLS=1 the/PEX venv /venv/dir
; so this new capability likely won't help you unless Pants changes how it creates venvs and allows you to pass through `--override`s in some manner. I still think
pants export && dist/the/venv/bin/pip install --force-reinstall -e this/local/foo -e this/local/bar
is likely the right path to be taking here; so I'm still curious to know why you don't want to or cannot take that path.
s
Hi @brief-scientist-13682 thanks for this! Apologies for not getting back to you with a repro. I had to step away from my pants adoption journey for quite a while due to a number of factors, but I'm back at it now. I've reproduced the error that i stumbled into after updating pants to use pex 2.86.1, and i've reopened the original pants issue with a new comment outlining it here: https://github.com/pantsbuild/pants/issues/23040#issuecomment-4200637668 if you wish to drill into it more, the exact git commit in my momorepo which will produce this error is https://github.com/uoft-networking/tools/tree/3f68310b5a2e53c00075c3d016671e51be724724
b
Thanks - I'll take a look.
Alex - the issue is this and it is from Pip - you cannot do what you're trying to do: https://github.com/uoft-networking/tools/blob/3f68310b5a2e53c00075c3d016671e51be724724/uv.lock.txt#L865-L872 You can tie the error you see:
Copy code
pip: ERROR: Can't verify hashes for these requirements because we don't have a way to hash version control repositories:
pip:     nautobot@ git+<https://github.com/utsc-networking/nautobot@utsc-custom> from git+<https://github.com/utsc-networking/nautobot@utsc-custom>
To their doc: https://pip.pypa.io/en/stable/topics/secure-installs/#hash-checking-mode Back before I deleted my PyPA discourse account, I was involved in the PEP-751 pylock.toml discussions and brought up directory hashing & VCS hashing. As with ~everything PyPA, this went nowhere, but this was the thread that spun out and died: https://discuss.python.org/t/how-to-hash-a-directory-in-lockfiles/70487 I think Pip would require some sort of standard for hashing a project source directory like that thread was trying to move towards to adopt hashes for VCS. That said, that ship has sailed I think and imagine they would only support pylock.toml at this point. Now Pex fully supports pylock.toml and uv as well, so maybe you're missing a hack along those lines ... but I've linked another Pants user's attempt at that angle below.
@stale-twilight-79248 you need to stop - gather Pants people - and yell it out. The Pants state of uv adoption and workarounds is an insane mess.
@happy-kitchen-89482 @curved-manchester-66006 - this is bad.
I'll disengage at this point, because I've done this 10s of times now, but the latest attempt: https://github.com/pantsbuild/pants/pull/23227
If you follow all links and issues and spider out and spend a few hours reading, you'll see the full extent of workarounds, half-done attempts, etc.
s
thanks john, i'm going through the links now, there's.... a lot
and thanks again for all the work you've done making pex as amazing as it is 😄