https://pantsbuild.org/ logo
#general
Title
# general
l

loud-laptop-89838

12/09/2022, 6:03 PM
I'm trying to come up with a good way to manage local environment variables. I asked a question a couple days ago about using a
.env
file, but hardcoding the file into a
BUILD
file doesn't work well since
.env
isn't tracked in version control, so CI tests throw warnings. I'm wondering whether a better approach is a plugin that explicitly loads the
.env
variables as environment variables for the test, then run that plugin as a command line argument to test? That way I could run it locally, but not include it in CI. Two things I'm wondering about: 1. would this approach work? I know tests are "hermetic", so would building a plugin that loads the variables work? 2. Can you "stack" targets; e.g. call the
.env
plugin prior to test so that they work together?
e

enough-analyst-54434

12/09/2022, 6:14 PM
So
.pants.bootstrap
is a hidden thing. That can be arbitrary bash, but should really just stick to setting env vars.
I'm in the middle of writing a new Pants launcher to replace
./pants
and it supports that but also loads
.env
Hopefully
.pants.bootstrap
tides you over?
You still need to expose those loaded env vars to rules / targets with pants.toml and BUILD edits of course using the appropriate config, e.g.:
[subprocess-environment] env_vars
,
python_tests.extra_env_vars
etc.
b

busy-vase-39202

12/09/2022, 6:42 PM
Is there an intention for it to be undocumented, or is that just an accidental omission?
l

loud-laptop-89838

12/09/2022, 7:16 PM
ok so to be clear, I would create a file named
pants.bootstrap
and in it I put, for example
HOSTNAME=localhost
, then in pants.toml I have
Copy code
[test]
extra_env_vars = ["HOSTNAME"]
Now my tests should have
$HOSTNAME
available as an environment variable?
e

enough-analyst-54434

12/09/2022, 7:20 PM
Yes, except
.pants.bootstrap
- leading dot.
@busy-vase-39202 I'm not sure, but I'd like it to die eventually! It was an awkward bit transitioning from a
pants
bash script to a `scie-pants`native binary to support. The
.env
support should cover most cases and that should be documented once it lands but I'd hope we can wean away from `.pants.bootstrap`once the GIT_COMMIT case is solved.
l

loud-laptop-89838

12/09/2022, 7:24 PM
hmm still missing something. I can tell it's working because I can cause errors in
.pants.bootstrap
and I have the variable defined in the pants.toml. But when I run the test in python
os.environ["HOSTNAME"]
throws a key error because it can't find the environment variable
e

enough-analyst-54434

12/09/2022, 7:25 PM
@loud-laptop-89838 please try poking the hole using BUILD instead of `pants.toml`https://www.pantsbuild.org/docs/reference-python_tests#codeextra_env_varscode I just had to use this yesterday for my own project and it worked.
It looks like the way you did it should work, but we can revisit that with maybe a bug report if the target config works to get you moving along.
l

loud-laptop-89838

12/09/2022, 7:29 PM
yeah target isn't working either. Really weird. Like I said, I know bootstrap is loading because I can make it throw errors. Then under the directory BUILD file in the python_tests target I added
extra_env_vars
as just an array of strings, which match the values set in the bootstrap file
e

enough-analyst-54434

12/09/2022, 7:30 PM
Hrm, OK. I'm on 2.14.0. I'll have a review out with my repo later today so I can point at that as an example, but makes no sense to me how this isn't working for you.
l

loud-laptop-89838

12/09/2022, 7:31 PM
same. And just to confirm, if I set
extra_env_vars=["HOSTNAME=123"]
that should add the env var to the test right?
e

enough-analyst-54434

12/09/2022, 7:32 PM
Yeah, that's a way to narrow the debugging here. The test should see 123.
My example, have Pants build a
pex_binary
and then use it in integration tests:
Copy code
and$ cat tools/tests/BUILD
python_test_utils(
    name="test_utils",
)

python_tests(
    runtime_package_dependencies=[
        "//tools"
    ],
    extra_env_vars=[
        "TOOLS_PEX=tools/tools.pex",
    ],
)
l

loud-laptop-89838

12/09/2022, 7:34 PM
Ok so that worked. I got my test to print out
os.environ
when it fails and it shows HOSTNAME=123
e

enough-analyst-54434

12/09/2022, 7:35 PM
Ok, so its just the pass-through somehow not working.
l

loud-laptop-89838

12/09/2022, 7:35 PM
Seems to be.
e

enough-analyst-54434

12/09/2022, 7:35 PM
Do you have a pants script with the lines I linked?
Oh, yeah
Your error test
l

loud-laptop-89838

12/09/2022, 7:35 PM
Yup.
e

enough-analyst-54434

12/09/2022, 7:35 PM
No clue. You'll need to dig a bit.
l

loud-laptop-89838

12/09/2022, 7:36 PM
yeah I guess so.
e

enough-analyst-54434

12/09/2022, 7:37 PM
So, another unlocker but still a bug needs to be fixed, go back to your pants.toml approach, but edit like so:
Copy code
[test]
extra_env_vars = ["HOSTNAME=%(env.HOSTNAME)s"]
That funky interpolation syntax is not widely known I think.
l

loud-laptop-89838

12/09/2022, 7:38 PM
ahh stupid mistake. I didn't have
export
at the start of the lines in bootstrap. I just copied from .env.
so it wasn't setting the variables
e

enough-analyst-54434

12/09/2022, 7:38 PM
Aha. Ok great. Works then?
l

loud-laptop-89838

12/09/2022, 7:39 PM
I'm getting some other weirdness, but the key error is gone now, so the variables seem to be set
e

enough-analyst-54434

12/09/2022, 7:39 PM
Cool.
l

loud-laptop-89838

12/09/2022, 7:39 PM
appreciate the help!
When is that new update coming out @enough-analyst-54434 just curious now?
e

enough-analyst-54434

12/09/2022, 7:51 PM
The
./pants
replacement thing? That'll be a new pantsbuild/scie-pants repo later today. I'd just like to finish the new update capability ~
SCIE_BOOT=update-scie-pants pants
to self-update the new "pants" script, which will just be a
scie-pants
binary anywhere on your
PATH
. I'm afraid without the update capability folks will start using it and then not get fixes for this very new thing easily if there are issues.
l

loud-laptop-89838

12/09/2022, 8:07 PM
you're probably right. I look forward to seeing it
e

enough-analyst-54434

12/09/2022, 8:52 PM
CI setup, a few PRs to add self-update and pants-from-sources support later today along with releases: https://github.com/pantsbuild/scie-pants Once there are releases those will be safe to download and use, static binaries so just download, name to your desired name and put on PATH.
👏 1
Alrighty, the
scie-pants
./pants
replacement experiment is ready for testing. If you're interested in trying it out and providing feedback, please give this a read: https://github.com/pantsbuild/scie-pants/blob/main/README.md In short, download
scie-pants
, mark it executable, put it on your PATH and use
scie-pants
instead of
./pants
. Thanks in advance for your testing and feedback.
❤️ 8
w

wide-midnight-78598

12/20/2022, 1:31 AM
I've been using it today. So far, so dope!
If you run
scie-pants
in a directory where Pants is not already set up, it will prompt you, and you can let it set up the latest Pants stable version for your project.
Such a "small" thing, but SUCH a huge quality of life improvement
e

enough-analyst-54434

12/20/2022, 2:12 AM
I think the big quality of life improvement is you can write all this slow-path work in Python now.
You could get fancy and be tailor like and enable or offer to enable the right seeming set of backends, etc.
That sort of thing stresses Pants stability though; so maybe better that is just a goal in Pants that could be run. As things were I already needed to handle `s/--python-repos-repos/--python-repos-find-links/`on the 2.13 boundary. It will not be good if that sort of thing has to be done in too many places. It does put a nice spotlight on stability of some core names and options though.
l

loud-laptop-89838

12/20/2022, 1:38 PM
The readme mentions that python is self contained in
scie-pants
. Does this mean for all python projects as well? Or just the pants internals? Ie how does this interact with, eg
pyenv
?
w

wide-midnight-78598

12/20/2022, 1:44 PM
One of the ideas here is to distribute an embedded Python interpreter with Pants, thereby removing the requirement of a specific system Python just to act as a Pants runner. So, pants becomes more of a "standalone binary", rather than "a collection of python files that are run by the user" So, this requirement goes away: https://www.pantsbuild.org/docs/prerequisites
Python 3.7, 3.8, or 3.9 discoverable on your
PATH
Here's a long-running tracking issue: https://github.com/pantsbuild/pants/issues/7369
l

loud-laptop-89838

12/20/2022, 1:47 PM
Totally makes sense since not all projects are going to be on python backends. But just so I understand, this bundled python interpreter would not be used if I have a python project. Eg I couldn't just install pants (no python on the system) and then turn on the python backend and then run
test
for example on a python project? I would still need a python interpreter for my project. It's just that pants has one for it's own internals.
w

wide-midnight-78598

12/20/2022, 1:54 PM
That's a good question. Today, you can specify a subset of the Python interpreters available on your machine to be used by your application code (e.g. I use Python 3.10 for my code, even though it can't be used to run Pants - so I have a separate Python 3.9 for Pants). In a blank system though, I wonder if you could (or even would want to) expose that embedded interpreter to act as a system python to run your application 🤔 For some more context, these are embedded: https://github.com/indygreg/python-build-standalone and they're placed in your system's cache directory, so they could be theoretically wiped out from beneath you (but
scie-pants
doesn't care, as it'll just grab it again).
Personally, I would/do package my python applications using PyOxidizer, but will be switching to using `scie`s once I land a packaging PR
l

loud-laptop-89838

12/20/2022, 1:56 PM
Packaging pr? Are you creating a build system in pants?
w

wide-midnight-78598

12/20/2022, 1:56 PM
Oh, sorry...
goal
, like
./pants package xyz
Just a
scie
plugin for packaging. https://github.com/pantsbuild/pants/compare/main...sureshjoshi:pants:scie-jump However, in thinking about it, on this hypothetical "blank" system, you still need an interpreter to run your
./pants tests ::
So, yeah, in summary - I'm guessing you'll need the correct system python (or
pyenv
or whatever), unless we start to download standalone interpreters to run user's
interpreter_constraints
🤯
l

loud-laptop-89838

12/20/2022, 2:08 PM
Well what about bundling
pyenv
as a backend. Then you could just get whatever the user specifies as a constraint and easily manage any number of different version constraints in the monorepo. Not saying I want to do it 😆 but maybe sometime in the future. In the meantime I'm just looking forward to testing this!
w

wide-midnight-78598

12/20/2022, 2:10 PM
You can play around with
scie-pants
today - I've been using it since yesterday
e

enough-analyst-54434

12/20/2022, 3:04 PM
Right now the Python distribution used to bootstrap and run Pants is added to the PATH last here: https://github.com/pantsbuild/scie-pants/blob/main/package/scie-pants.lift.json#L93 That's probably a bad idea though. These interpreters are not full featured. Any attempt to use them to build a wheel with native extensions will fail (that is the ~only known limitation though). The question of helping you use pyenv or not is a Pants question and not a scie-pants question. The role of scie-pants should stay tightly scoped to launching Pants (and installing it as needed). It should be very fast in the former role and constrained in the latter. Pants itself and Pants goals and rules are the right tools for almost all other features.
1
l

loud-laptop-89838

12/20/2022, 5:15 PM
That's helpful to understand the scope actually.
e

enough-analyst-54434

12/20/2022, 5:20 PM
Right, so you could still have
scie-pants init
and the
scie-pants
part of that would see you don't have a
pants.toml
and it would set that up, which would launch pants, which would interpret the
init
goal and run that to further initialize the setup. Etc. So scie-pants got out of the way quick there and delegated everything fancy to the Pants
init
goal - a full fledged goal with rules and all the fancy complicated logic.
@loud-laptop-89838 I realized I never actually shipped
.env
support. Now available in 0.2.0: https://github.com/pantsbuild/scie-pants/releases/tag/v0.2.0
👏 1