https://pantsbuild.org/ logo
#general
Title
# general
i

important-psychiatrist-39230

10/03/2022, 11:08 PM
One thing I'm evaluating is related to developer ergonomics. There's a few things that we've done in some other projects, as well as some tooling thoughts. Going to make a few posts here to allow for proper threading, sorry for any spammyness 🙂 🧵 First is that we're used to having the ability to specify different dependencies in development vs. our release environments. Specifically, this is useful for dependencies such as
psycopg2
which have significant burdens for local development (especially on some OSes) while realizing possible wins when the code is deployed. What I've done in my poetry projects is to have
pg8000
setup as a dev dependency and then have a specific extras section for a production only dependency after flagging it as optional.
Copy code
[tool.poetry.dependencies]
psycopg2 = { version = "~2.9.3", optional = true }

[tool.poetry.dev-dependencies]
pg8000 = "~1.29.1"

[tool.poetry.extras]
production-only = ["psycopg2"]
This has reduced the burden on our developers while maintaining api compatibility and the code is largely unaware of the distinction. (I only have one place where I have to do multiple imports and that's for a try/except block). From what I can see there's not an obvious way to do a similar dichotomy when using pants and this is a considerable regression that I'm having to weigh. Is there anything I'm missing? Could a plugin enable such a pattern?
h

happy-kitchen-89482

10/04/2022, 12:48 AM
Hmm, musing aloud here: can this be achieved by having two resolves/lockfiles, one for dev and one for prod, I wonder?
The dev one can be the default, and you can override that on your pex_binary or python_distribution targets, which are your production targets.
b

bitter-ability-32190

10/04/2022, 12:49 AM
I don't think multiple lockfiles/resolves is necessary. You can easily use transitive exclude on the
pex_binary
+ the "prod" dep
Additionally, a plugin could easily accomplish this, but I think a macro is sufficient/ideal for simplicity
h

happy-kitchen-89482

10/04/2022, 12:49 AM
That too, yes
a macro can help keep the boilerplate down
coke 2
b

bitter-ability-32190

10/04/2022, 12:51 AM
@important-psychiatrist-39230 see https://www.pantsbuild.org/docs/targets#dependencies-field
You can use the prefix
!!
to transitively exclude a dependency, meaning that even if a target's dependencies include the bad dependency, the final result will not include the value.
is relevant (and is also relevant for your below thread about
rich
. Macros are here: https://www.pantsbuild.org/docs/macros
i

important-psychiatrist-39230

10/04/2022, 3:01 AM
I'l reread this in the morning, but I'm not really tracking how I would use the transitive exclusions to switch between the two.
e

enough-analyst-54434

10/04/2022, 3:29 AM
The idea is not switching between the two, the idea is instead excluding dev dependencies from production artifacts. You'd add excludes to
pex_binary
targets, for example, that depended on `python_sources`targets that had the dev-only deps baked in.
That said, I don't like that default - everything gets dev-only dependencies and you must exclude from prod artifacts. I'd rather have that inverted, because prod is all important (you don't want to ship a slow postgres DBI to prod silently by mistake!). I don't think its as easy to do it this other way around though with Pants today. You'd need, I think, more macro / plugin code. Even if you're OK with defaulting to everything having dev deps added and then subtracting from prod, this is still definitely more awkward than what you're used to. The basic mismatch is that dev-dependencies in Poetry et. al go into making a venv to run tools in / repl with, but they do not go in your built distribution. In Pants, we have ~dev-dependencies, but they are per-tool (for example, for pytest you add requirements to this list: https://www.pantsbuild.org/docs/reference-pytest#extra_requirements - and similar options exist for all other Python tools Pants can run). You'd need, basically, to add these deps to every tool that needed to see them + the run goal + the repl goal. We do not have options like this for the two latter goals and even if we did, this all would be quite unwieldy. Clearly we have work to do here.
h

happy-kitchen-89482

10/04/2022, 5:32 AM
I don’t think you’d want to add those deps as if they were tool requirements though? That would emulate what poetry et al do, but it’s not very idiomatic for Pants.
e

enough-analyst-54434

10/04/2022, 5:35 AM
Agreed - I'm just trying to point out how we model dev dependencies, which is tool-by-tool wherease things like Poetry model it as 1 big bag of all the tools.
h

happy-kitchen-89482

10/04/2022, 5:35 AM
To figure this out more precisely, I’m curious about how your code switches between the two? AFAICT to use pg8000 you import from
pg8000
and to use psycopg2 you import from
psycopg2
? What does the code that disambiguates the two look like?
e

enough-analyst-54434

10/04/2022, 5:35 AM
We're sorely missing someone adept at both worlds here to play Rosetta Stone
h

happy-kitchen-89482

10/04/2022, 5:38 AM
Yeah, most python tooling works by throwing the tool requirements and your code’s requirements into one venv and assuming they don’t collide. “dev” vs “prod” is implied by some external info like an env var.
e

enough-analyst-54434

10/04/2022, 5:39 AM
Presumably on the `pg8000`thing, django-ish? Db hidden behind pluggable driver.
👍 1
h

happy-kitchen-89482

10/04/2022, 5:39 AM
With pants whether something is “dev” or “prod” is implied by the target type
ish
e

enough-analyst-54434

10/04/2022, 5:39 AM
Yeah, and making ~duplicate targets is getting easiser all the time but still a crappy way to do things afaict.
It's like you want to be able to run
./pants enter-role dev
and have that stick until you clear the role or enter a new one. Then role aware goals do the right things.
This is vaguely like lots of mechanisms we're making have made (environments come to mind).
h

happy-kitchen-89482

10/04/2022, 5:42 AM
Not sure we need duplicate targets though
e.g., a test is always “dev”
and a pex/dist is ~always “prod”
e

enough-analyst-54434

10/04/2022, 5:43 AM
run is both as is repl
It depends what you want
In Poetry you run and repl in a venv and you can control what the venv has in it via groups
b

bitter-ability-32190

10/04/2022, 10:28 AM
All this to say, @important-psychiatrist-39230 if you check out macros and find it's too brittle, we can help you write a very simple plugin which should do this across the board (exclude/include the dev/prod dep based on target type)
i

important-psychiatrist-39230

10/04/2022, 3:58 PM
Hey guys catching up. 🙂
re: How they're used. For the most part my code doesn't know what's up. SQLAlchemy allows me to switch between them with the postgres DSN being in different formats.
This is the only code I have that imports from either driver directly
Copy code
try:
    from psycopg2 import NotSupportedError as EngineNotSupportedError
except ImportError:
    from pg8000 import NotSupportedError as EngineNotSupportedError
for
pants run
I've been doign that against a pex_binary target
Copy code
pex_binary(
    entry_point="swizzler.main:service",
    restartable=True,
)
typing out
pants run services/swizzler
is a lot better than me having to type
pants run services/swizzler/src/swizzler/main:service
or whatever the source target would be
What I commonly see in other languages, and what I setup with poetry/poethepoet, is two different commands for dev versus production.
So in dev I'd do
poe dev
and then in prod I'd do
poe start
maybe a more idiomatic approach here would be do have a dev goal that could also target the pex binary and then have a
dev_dependencies
argument when defining your
pex_binary
Then I could do
pants dev services/swizzler
?
hrm, not sure if that'd be the ideal though for sub deps that a library comes in on. It'd almost be nice if there was a
library
target that you could do that configuring on and then the pex_binary would read it's properties if it depends on that library.
h

happy-kitchen-89482

10/04/2022, 5:49 PM
BTW in dev you can
./pants run services/swizzler/main.py
(or whatever the entry point file is) and that is a lot faster than running the target (running the target is the same as building the pex and then running it)
So I think you can get this with two resolves, and parametrizing the pex_binary on them?
There does seem to be an impedance mismatch between how you’re used to doing things and how Pants idiomatically does things, but I think we can find modeling that makes sense
b

bitter-ability-32190

10/04/2022, 5:51 PM
@happy-kitchen-89482 we're also having a parallel related convo here: https://pantsbuild.slack.com/archives/C046T6T9U/p1664838855556019 I had an interesting idea where someone could model "dev_dependencies" in a plugin: https://pantsbuild.slack.com/archives/C046T6T9U/p1664904603042689?thread_ts=1664838855.556019&cid=C046T6T9U (and the subsequent message)
i

important-psychiatrist-39230

10/04/2022, 8:05 PM
as odd as it is, I'm not sure I want to tell devs to run the entry point directly. especially given the inherent verbosity of the command that'd create. I think it'd need to be something like this:
Copy code
pants run services/swizzler/src/swizzler/main.py:app
no dev on my side is going to think that's great 😕
e

enough-analyst-54434

10/04/2022, 8:22 PM
You used to be able to use a `target`target like an alias to shorten up paths but this appears to no longer work. You can add global aliases in pants.toml by configuring `[cli.alias]`as described here https://www.pantsbuild.org/docs/reference-cli#alias and dog-fooded here https://github.com/pantsbuild/pants/blob/7acbcda329db6fb9a2ebcd0a547c3e0ab123ba0c/pants.toml#L80-L82 I think those are the only historical and present means of dealing with typing too much.
4 Views