One thing I'm evaluating is related to developer e...
# general
One thing I'm evaluating is related to developer ergonomics. There's a few things that we've done in some other projects, as well as some tooling thoughts. Going to make a few posts here to allow for proper threading, sorry for any spammyness 🙂 🧵 First is that we're used to having the ability to specify different dependencies in development vs. our release environments. Specifically, this is useful for dependencies such as
which have significant burdens for local development (especially on some OSes) while realizing possible wins when the code is deployed. What I've done in my poetry projects is to have
setup as a dev dependency and then have a specific extras section for a production only dependency after flagging it as optional.
Copy code
psycopg2 = { version = "~2.9.3", optional = true }

pg8000 = "~1.29.1"

production-only = ["psycopg2"]
This has reduced the burden on our developers while maintaining api compatibility and the code is largely unaware of the distinction. (I only have one place where I have to do multiple imports and that's for a try/except block). From what I can see there's not an obvious way to do a similar dichotomy when using pants and this is a considerable regression that I'm having to weigh. Is there anything I'm missing? Could a plugin enable such a pattern?
Hmm, musing aloud here: can this be achieved by having two resolves/lockfiles, one for dev and one for prod, I wonder?
The dev one can be the default, and you can override that on your pex_binary or python_distribution targets, which are your production targets.
I don't think multiple lockfiles/resolves is necessary. You can easily use transitive exclude on the
+ the "prod" dep
Additionally, a plugin could easily accomplish this, but I think a macro is sufficient/ideal for simplicity
That too, yes
a macro can help keep the boilerplate down
coke 2
@important-psychiatrist-39230 see
You can use the prefix
to transitively exclude a dependency, meaning that even if a target's dependencies include the bad dependency, the final result will not include the value.
is relevant (and is also relevant for your below thread about
. Macros are here:
I'l reread this in the morning, but I'm not really tracking how I would use the transitive exclusions to switch between the two.
The idea is not switching between the two, the idea is instead excluding dev dependencies from production artifacts. You'd add excludes to
targets, for example, that depended on `python_sources`targets that had the dev-only deps baked in.
That said, I don't like that default - everything gets dev-only dependencies and you must exclude from prod artifacts. I'd rather have that inverted, because prod is all important (you don't want to ship a slow postgres DBI to prod silently by mistake!). I don't think its as easy to do it this other way around though with Pants today. You'd need, I think, more macro / plugin code. Even if you're OK with defaulting to everything having dev deps added and then subtracting from prod, this is still definitely more awkward than what you're used to. The basic mismatch is that dev-dependencies in Poetry et. al go into making a venv to run tools in / repl with, but they do not go in your built distribution. In Pants, we have ~dev-dependencies, but they are per-tool (for example, for pytest you add requirements to this list: - and similar options exist for all other Python tools Pants can run). You'd need, basically, to add these deps to every tool that needed to see them + the run goal + the repl goal. We do not have options like this for the two latter goals and even if we did, this all would be quite unwieldy. Clearly we have work to do here.
I don’t think you’d want to add those deps as if they were tool requirements though? That would emulate what poetry et al do, but it’s not very idiomatic for Pants.
Agreed - I'm just trying to point out how we model dev dependencies, which is tool-by-tool wherease things like Poetry model it as 1 big bag of all the tools.
To figure this out more precisely, I’m curious about how your code switches between the two? AFAICT to use pg8000 you import from
and to use psycopg2 you import from
? What does the code that disambiguates the two look like?
We're sorely missing someone adept at both worlds here to play Rosetta Stone
Yeah, most python tooling works by throwing the tool requirements and your code’s requirements into one venv and assuming they don’t collide. “dev” vs “prod” is implied by some external info like an env var.
Presumably on the `pg8000`thing, django-ish? Db hidden behind pluggable driver.
👍 1
With pants whether something is “dev” or “prod” is implied by the target type
Yeah, and making ~duplicate targets is getting easiser all the time but still a crappy way to do things afaict.
It's like you want to be able to run
./pants enter-role dev
and have that stick until you clear the role or enter a new one. Then role aware goals do the right things.
This is vaguely like lots of mechanisms we're making have made (environments come to mind).
Not sure we need duplicate targets though
e.g., a test is always “dev”
and a pex/dist is ~always “prod”
run is both as is repl
It depends what you want
In Poetry you run and repl in a venv and you can control what the venv has in it via groups
All this to say, @important-psychiatrist-39230 if you check out macros and find it's too brittle, we can help you write a very simple plugin which should do this across the board (exclude/include the dev/prod dep based on target type)
Hey guys catching up. 🙂
re: How they're used. For the most part my code doesn't know what's up. SQLAlchemy allows me to switch between them with the postgres DSN being in different formats.
This is the only code I have that imports from either driver directly
Copy code
    from psycopg2 import NotSupportedError as EngineNotSupportedError
except ImportError:
    from pg8000 import NotSupportedError as EngineNotSupportedError
pants run
I've been doign that against a pex_binary target
Copy code
typing out
pants run services/swizzler
is a lot better than me having to type
pants run services/swizzler/src/swizzler/main:service
or whatever the source target would be
What I commonly see in other languages, and what I setup with poetry/poethepoet, is two different commands for dev versus production.
So in dev I'd do
poe dev
and then in prod I'd do
poe start
maybe a more idiomatic approach here would be do have a dev goal that could also target the pex binary and then have a
argument when defining your
Then I could do
pants dev services/swizzler
hrm, not sure if that'd be the ideal though for sub deps that a library comes in on. It'd almost be nice if there was a
target that you could do that configuring on and then the pex_binary would read it's properties if it depends on that library.
BTW in dev you can
./pants run services/swizzler/
(or whatever the entry point file is) and that is a lot faster than running the target (running the target is the same as building the pex and then running it)
So I think you can get this with two resolves, and parametrizing the pex_binary on them?
There does seem to be an impedance mismatch between how you’re used to doing things and how Pants idiomatically does things, but I think we can find modeling that makes sense
@happy-kitchen-89482 we're also having a parallel related convo here: I had an interesting idea where someone could model "dev_dependencies" in a plugin: (and the subsequent message)
as odd as it is, I'm not sure I want to tell devs to run the entry point directly. especially given the inherent verbosity of the command that'd create. I think it'd need to be something like this:
Copy code
pants run services/swizzler/src/swizzler/
no dev on my side is going to think that's great 😕
You used to be able to use a `target`target like an alias to shorten up paths but this appears to no longer work. You can add global aliases in pants.toml by configuring `[cli.alias]`as described here and dog-fooded here I think those are the only historical and present means of dealing with typing too much.