Hey folks; when I run `pants package src/lambdas/w...
# general
m
Hey folks; when I run
pants package src/lambdas/whatever
I get a little bit of metadata:
Copy code
14:37:05.54 [INFO] Wrote dist/src.lambdas.whatever/whatever.zip
    Runtime: python3.12
    Architecture: x86_64
    Handler: lambda_function.handler
Is there any way to have that metadata output to a file instead? (My bash is weaker than I thought it was, I don’t seem to be able to pipe it to a file from stdout or stderr either!) I’d like to be able to configure my terraform to set up my lambda with the runtime, arch, and handler as configured in pants, so I don’t have duplicate info — if I can get this into a JSON/similar file, I can configure Terraform to read it and use those values 💪
s
Do you have this information in the target? If so, you could use
pants peek
m
Oh nice! Thanks! I can get the
handler
, and
architecture
, but the Python version is set at the monorepo level, so I presume it’s not pulled up to being able to peek at the target?
Copy code
[python]
interpreter_constraints = ["==3.12.*"]
If it’s useful, here’s the
jq
command for extracting the handler & architecture:
Copy code
pants peek src/lambdas/whatever | jq '.[] | select(.target_type == "python_aws_lambda_function") | {handler: (.handler | sub("\\.py:"; ":")), architecture}'
I’ll keep hunting for ways to pull the runtime version out — I don’t really want to have to specify it in the
BUILD
for every lambda (as the whole point here is to minimize duplicate config)
Ahh; or it may be that I’m specifying my actual python version in my
pyproject.tml
via poetry… I’ll keep going 😊
s
You can set this in your root BUILD:
Copy code
__defaults__(all=dict(interpreter_constrains=["==3.12.*"]))
Then it should show up in peek output
👌 1
m
Thanks @square-psychiatrist-19087 — does setting
interpreter_constraints
in the
pants.toml
not do the same? (I don’t have a root
BUILD
at the moment, but I could make one for defaults — it just seems odd to not have the interpreter constraints in the pants.toml not be used here?)
s
It does get used, but pants.toml is a separate mechanism for defaults. Basically in the pants backend there is code something like this:
Copy code
if target.interpreter_constraints is not None: 
    return target.interpreter_constraints
else:
    return config.interpreter_constraints
So if the target doesn't have interpreter constrains it will look it up in pants.toml. However,
pants peek
can't do that and only looks at target values. This
__defaults__
trick sets the value on the target, so it becomes visible to
pants peek
👍 1
m
I can’t seem to get any info on what Python version is being used with
pants peek
, regardless of where I define my contraints: • as a default in my root
BUILD
(as you describe above) • with
python_sources(interpreter_constraints=["==3.12.*"])
in my lambda’s
BUILD
• in the pants.toml in the
[python]
section as an
interpreter_constraints
parameter • in the lambda’s
pyproject.toml
(as a poetry dependency, used with a
poetry_requirements
in the lambda’s
BUILD
) I can get most of what’s output with
pants package src/lambdas::
with
jq
and
pants peek
, but not everything:
Copy code
$ pants package src/lambdas::
09:50:12.32 [INFO] Wrote dist/src.lambdas.whatever/whatever.zip
    Runtime: python3.12
    Architecture: x86_64
    Handler: lambda_function.handler

$ pants peek src/lambdas:: | jq '.[] | select(.target_type == "python_aws_lambda_function") | {path: (.address | sub(":.*$"; "")), handler: (.handler | sub("\\.py:"; ".")), architecture}'
{
  "address": "src/lambdas/whatever",
  "handler": "lambda_function.handler",
  "architecture": "x86_64"
}
(I also have to do extra bash to split the records & convert
address
into the output path
dist/src.lambdas.whatever/whatever.json
for each of the results
pants peek
returns)
I still think it’d be immensely useful to be able to configure pants to output the data shown on screen when packaging a lambda (runtime, architecture, handler) to a JSON file adjacent to the zipped lambda, so that the terraform which deploys the lambda could derrive its config from pants:
Copy code
locals {
  lambda_base = "../dist/src.lambdas.whatever/whatever"
  lambda_config = jsondecode(file("${lambda_base}.json"))
}

resource "aws_lambda_function" "whatever_lambda" {
  function_name = "whatever_lambda"

  handler = local.lambda_config.handler
  runtime = local.lambda_config.runtime
  architectures = [local.lambda_config.architecture]

  filename         = "${lambda_base}.zip"
  source_code_hash = filebase64sha256("${lambda_base}.zip")
}
Is there a way of doing this I’m missing? Or a good place for me to make this request of the pants team?
s
Or you infer it based on python sources?
If you set
runtime
, it should be visible in pants peek
m
Ahh super, thankyou! That does work! 🥳 The only problem is, I’m now setting my python runtime in two places; once for the local python interpreter (in my
pyproject.toml
, which is calculated by inference by
pants package
and output to the terminal), and once in my
BUILD
for in
python_aws_lambda_function(runtime="python3.12")
. Any ideas on how to get
peek
to output the inferred python runtime?
s
How do you set your python dependencies? via
dependencies
or
handler
?
If you don't care about multiple python versions you could set it all in a single place
Copy code
python_version = "3.12"
__defaults__(all=dict(
    interpreter_constraints=[f"=={python_version}.*"],
    runtime=f"python{python_version}",
))
I don't think there is an easy way to get it from inside pyproject.toml If you really want to get it from there, you could try some magic with `.pants.bootstrap`:
Copy code
#!/bin/bash

export PYTHON_REQUIRES=$(grep python_requires pyproject.toml)
Then in BUILD:
Copy code
python_requires = env('PYTHON_REQUIRES')
# then parse it and use it
m
This might work, but I’m going to have multiple pythons in my monorepo 😔 Thank you so much for your help here, you’ve offered so many tips for some stranger who has a rather specific request 😅
❤️ 1
s
No, that's actually a natural request, pants is mostly a metadata layer on top of your existing code, so it's whole purpose is to provide ways to inspect metadata I think there was a feature request to read external files directly from BUILD, but I don't think it was implemented
🙇‍♂️ 1
but I’m going to have multiple pythons in my monorepo 😔
You could try inferring this from dependencies then, you could do pants peek on your lambda and get
dependencies
from it. Then inside dependencies find
python_source
it depends on and do pants peek on
python_source
, it should have interpreter_constraints in it
m
So such luck; possibly because I’m using
poetry
to manage my python version? I’ve put all the relevant bits in this gist (note that I’ve switched to python 3.13 since we started this conversation). I’ve got to duck out for the rest of the day — so I won’t be able to reply for a while, but thanks for your help & thoughts!
h
Separately from this workaround, it seems like a reasonable feature to have, and it would not be difficult to add.
In fact it could be added generically for all packaged artifacts
Well, there are a couple of ways of doing it
For example, we gather those “extra log lines” and log them here
👀 1
So it would be easy to write them to
dist
as well just above that
Alternatively, we can have a separate metadata pathway that is more likely to be machine-readable and unchanging
since “log lines” implies “human readable, and subject to change”
1
The other place we use these extra log lines is for docker images, but there the artifact is already a machine-readable metadata file
so there is no extra value in logging those lines
So I’m leaning towards just adding it as a little bit of JSON in the FAAS case
❤️ 1
which would also be fairly easy
I can try and hack it together in a bit
m
That sounds superb! Let me know if I can be of any help!
h
Best help would be to test this out, once it’s merged