Question about GRPC support — I realize that generating the source isn’t something that’s supported right now but I’m mostly curious why the decision was made to only include it in the PEX? I’m asking mostly because I have a mental model of Pants where codegen isn’t limited to the distribution/publish process and generating client/server code should make it easier to develop against, model service dependencies and build integration tests. Is there a workflow where not having the client/server generated code in the repo is more desirable?

Generally, generated code doesn’t get checked into version control. The files get generated into locations in your actual project, like src/protos/p_pb2.py or project/p_pb2.py, and it would be annoying to generate into those locations because you have to set up version control to ignore Probably more importantly, writing the files to the build root would mess with the concurrency model - there’s a lock on writing to the build root, and it can only happen in some places If we added support for saving the generated files, they would go into dist/ folder, and would be mostly for you to inspect. But we haven’t added a

gen

goal because we don’t want people to think they must manually run that goal for codegen to work. We can add it, only wanted to wait for user need Also, to clarify, the generated files get used everywhere, including running tests. That tip in the docs about building a Pex to inspect the generated files is solely a workaround to be able to see them, because we don’t have that

gen

goal

That makes sense to have a consistent and reproducible build but it also makes IDE supported static analysis harder, unless I’m missing a capability or overlooked some config. In the absence of the generated code within the same layout that would be deployed how would you prevent jedi or (in my case) pycharm from complaining about imports that don’t exist. By the way: this may simply be a case where my current workflow is incongruent with the intent of the GRPC plugin. I tend rely heavily on my IDE and tend to run debuggers and REPLs directly through it. This includes launching servers or clients and attaching debuggers within pycharm. I’m not actually suggesting codegen is the solution but I’m mostly curious about other potential workflows.

Ah, makes sense why you want it for your IDE. I’m OOO next two days, but can send you a gist with the code to create a manual gen goal on Wednesday :) Also, we’d be willing to upstream this if we can think of a better goal name than

gen

. Maybe

write-codegen

? We want to make it abundantly clear that you never need to run this goal for things to work, it’s only if you want to save the files to your build root

@hundreds-father-404: i think that this might be closer to an explicit IDE support request than a codegen issue

Maybe, I can imagine other reasons you’d want to see the files saved to disk. Such as debugging codegen code

Yeah, a lot of the engineers here prefer to debug using

import pdb; pdb.set_trace()

. I suppose they can do that (probably?) using the

repl

command but it seems like it adds a layer of indirection.

Yea, can pdb using either

repl

,

run

, or

test --debug

@witty-crayon-22786 so I’ve been thinking about this strictly from the perspective of how we write code here. All the engineers use their preferred editors so we have folks that use emacs, vim, code, pycharm etc… but we have one way of bootstrapping the dev environment (basically when you call

workon

we manipulate the python path and do a bunch of other things to load various scripts etc…). My impression is that virtualenvs are the closest thing to a lowest common denominator across all the editors we’re using and in our project setup so making it easy to, for example, automatically load all the source roots into the python path is probably something I would end up adding to our environment bootstrap script so that imports could cleanly map to how they would be structured in the generated pexs. I am not sure if thats a universal way people want to interact with their dev environment though so I’ll be more than willing to upstream any changes but it may not solve a problem for everyone. For me personally, a pycharm integration be nice but not certainly not a deal breaker. Getting it to work with a pex seems like it could be challenge…I don’t know if pycharm supports a concept of sourcemapping or anything like that.

@stocky-engine-79940: great, thank you: that’s valuable feedback.

Hm, so I wrote a

write-codegen

goal, which writes the files to the same place they end up actually being generated when used by Pants. I could see one thing being a big issue and being confusing: if those Python files are generated in the same folder where you have a

python_library()

with the default sources field, then Pants will think that those generated files are your own and like normal Python files. If dep inference is used, it would result in those checked-in files being used, but Pants still attempting to do codegen; this could result in a merge conflict if the checked-in conflicts are stale, which results in an exception. Two solutions, I think: 1. Keep writing to the build root, but detect somehow if those files aren’t covered by

pants_ignore

/

gitignore

. If they aren’t ignored, then log a warning and strongly encourage ignoring them. 2. Always write to the

dist/

folder instead, which should never have a

BUILD

file and is always ignored by Pants. I think #2 is simpler. Omar, would that be fine if we write the files like this, with the

dist/

prefix:

dist/src/python/project/util/my_file_pb2.py

? Keep in mind that that folder is usually gitignored - not sure if that would be an issue for IDEs

@stocky-engine-79940: hey! we’ve opened up the feedback gathering if you’re interested in submitting some ideas: https://groups.google.com/g/pants-devel/c/XMjiqAGITtI/m/6wjMvU_LAQAJ … if not, i’d be happy to summarize this thread

Nit: I think it should be

dist/codegen/src/python/project/util/my_file_pb2.py

@hundreds-father-404 I think @happy-kitchen-89482 offers the cleanest solution since it leaves end users with the option of either copying those files (if they want), adding the dist folder to the virtualenv path (which is their local dev env anyway) or just not using the files.

New

write-codegen

goal. Feedback welcomed on the two questions posed in comments: 1) if that’s a good goal name 2) whether we should only activate if you’re using Protobuf https://github.com/pantsbuild/pants/pull/11107

this is maybe a good intermediate step on the way to better IDE integration… it would allow you to “roll your own” IDE integration by 1) exporting the requirements for the relevant targets, 2) exporting generated code. i expect that we’ll want something unified fairly soon though.

Sounds good. I think even with that unification, we’ll still want

export-codegen

as a more light weight option. Totally feasible for it to be

./pants export --codegen-only

for example

Totally feasible for it to be 

./pants export --codegen-only

 for example

yea, that might be a reasonable direction

2.1.0rc0 is now out with the new

export-codegen

rule. See https://www.pantsbuild.org/v2.1/docs/upgrade-tips for tips with upgrading, and https://www.pantsbuild.org/v2.1/docs/release-notes-2-1 for a changelog