I'm attempting to add a requirement to a requirement.txt file. This requirement is a raw github URL of the form git+https://github.com/...@v0.x.xxx When I add this to the requirement file, it breaks. mbp293@MBP293-Z zero-ai % pants check :: 151840.09 [ERROR] 1 Exception encountered: Engine traceback: in

check

goal in Find targets from input specs ValueError: Invalid requirement 'git+https://github.com/...@v0.x.xxx in third_party/zero-api_requirements.txt at line 9: Expected end or semicolon (after name and no valid version specifier) git+https://github.com/...@v0.x.xxx One thing I found recommended that I put this into a

python_requirement

block and it seems to grab and build this repo fine. The problem is I want to include this third_party in my default resolve so that it's a part of the lockfile I use with my IDE. Perhaps there's some simple way to do this, but I'm having a hard time solving this.

If I am understanding things correctly, you have a

requirements.txt

file that you are using with

python_requirements(source="requirements.txt")

and now adding your new requirement as a separate

python_requirement(requirements=["something"])

target. Is that right? Good news, this should do exactly what you want already. If you'll notice, both python_requirement and python_requirements have a

resolve

key that can be used to declare which resolve the requirement(s) belong to (defaulting to your default resolve). The resolve doesn't care where they came from in the first place, they all get converted to individual

python_requirement

targets behind the scenes and the resolve will use them all.

@brainy-airline-59624 you're being bit by https://github.com/pantsbuild/pants/issues/22239. For a requirement of the form

git+https://...#egg=<project-name>...

, the

python_requirement

code gets things right (https://github.com/pantsbuild/pants/blob/8e7f71e5ff858c9d46fe9cdd81cddaf95616817a/src/python/pants/util/pip_requirement.py#L33), but when the same requirement string is in a requirement file loaded via

python_requirements

, faulty comment stripping (Comments in a requirement file begin with

#

too) cuts the

egg=<project-name>

off your otherwise completely valid requirement string (https://github.com/pantsbuild/pants/blob/8e7f71e5ff858c9d46fe9cdd81cddaf95616817a/src/python/pants/util/requirements.py#L17). You can maintain using a requirements file if you switch from

git+https://...#egg=<project-name>...

form to the more standard `project-name @ git+https://..`form. That said, if you use the

#subdirectory=<some subdir in the clone>

form, there is no workaround except to follow @elegant-florist-94385's lead and stick to

python_requirement

until https://github.com/pantsbuild/pants/issues/22239 gets fixed.

Yeah, that still failed and I ended up switching everything to a Poetry requirements which has a bit more clear specification and worked normally. Will try to do that more later for the project. So, I've mentioned before this code base I inherited is in a horrible state. This github repo I'm trying to use, was used previously in our codebase. Even though the GIt repo had a proper set-up and could always be called with pip, the previous owners imported it as a submodule, manually generated the protobuf wrappers in the project, and then checked that into OUR git repo. And so, right now, Pants is confused, it keeps thinking the manually generated code is where these imports are coming from. As, I slowly clean this code base up, how do I incrementally specify to projects to use the Github version of the code imported in a Poetry file? I am not installing this outside repo using the Poetry file, and this poetry file is associated with the default resolve. Alongside this, I'm also trying to make a default resolve that everyone uses, as the previous owners of it just copy and pasted the same requirements file for every single project to make new resolve files. It's kind of a mess that I'm still trying to untangle when I have time.

how do I incrementally specify to projects to use the Github version of the code imported in a Poetry file?

I would try to enforce the idea (during the length of the refactor, that is), that the dependency should never exist in the same resolve twice. So you would end up with the following process (loosely): • Start with a number of (copy-pasted) resolves • Create the standard default resolve (which should include the git project under discussion) ◦ At the moment, no first party code is using dependencies from this resolve (They are still using their multitude of specific resolves) ◦ If one of your original resolves was considered a default, force the old default into a specific named resolve and let your new default be on its own • One by one, remove the individual specific resolves. For each one, you will need to: ◦ Ensure all first party code is compatible with the versions of the 3rd party packages in your default resolve (lint, check, test, functionality, etc) ◦ Set first party code to point at default resolve ◦ Remove the resolve (requirements, lockfile, declaration of resolve in

pants.toml

) This enables you to work on one project at a time, and to have a (relatively) clean divide of "If its using the default resolve, it is fixed, else it is considered legacy". There will potentially be some ordering dependencies where (apart from the git repo as 3rd party dependency) you might have some common library type code that needs to be ported over before some projects that are using it, but then it still needs to know about the resolves for those projects. If you can do them all at once, the above process would work, otherwise those libs would need to follow a multi-resolve process in the meantime (Hopefully you can avoid needing to do this).