https://pantsbuild.org/ logo
Join Slack
Powered by
Hey folks; when I run `pants package src/lambdas/w...
# general
m
Hey folks; when I run 
pants package src/lambdas/whatever
I get a little bit of metadata:
Copy code
14:37:05.54 [INFO] Wrote dist/src.lambdas.whatever/whatever.zip
    Runtime: python3.12
    Architecture: x86_64
    Handler: lambda_function.handler
Is there any way to have that metadata output to a file instead? (My bash is weaker than I thought it was, I don’t seem to be able to pipe it to a file from stdout or stderr either!) I’d like to be able to configure my terraform to set up my lambda with the runtime, arch, and handler as configured in pants, so I don’t have duplicate info — if I can get this into a JSON/similar file, I can configure Terraform to read it and use those values 💪
✅ 1
s
Do you have this information in the target? If so, you could use 
pants peek
m
Oh nice! Thanks! I can get the 
handler
, and 
architecture
, but the Python version is set at the monorepo level, so I presume it’s not pulled up to being able to peek at the target?
Copy code
[python]
interpreter_constraints = ["==3.12.*"]
If it’s useful, here’s the 
jq
command for extracting the handler & architecture:
Copy code
pants peek src/lambdas/whatever | jq '.[] | select(.target_type == "python_aws_lambda_function") | {handler: (.handler | sub("\\.py:"; ":")), architecture}'
I’ll keep hunting for ways to pull the runtime version out — I don’t really want to have to specify it in the 
BUILD
for every lambda (as the whole point here is to minimize duplicate config)
Ahh; or it may be that I’m specifying my actual python version in my 
pyproject.tml
via poetry… I’ll keep going 😊
s
You can set this in your root BUILD:
Copy code
__defaults__(all=dict(interpreter_constrains=["==3.12.*"]))
Then it should show up in peek output
👌 1
m
Thanks @square-psychiatrist-19087 — does setting 
interpreter_constraints
in the 
pants.toml
not do the same? (I don’t have a root 
BUILD
at the moment, but I could make one for defaults — it just seems odd to not have the interpreter constraints in the pants.toml not be used here?)
s
It does get used, but pants.toml is a separate mechanism for defaults. Basically in the pants backend there is code something like this:
Copy code
if target.interpreter_constraints is not None: 
    return target.interpreter_constraints
else:
    return config.interpreter_constraints
So if the target doesn't have interpreter constrains it will look it up in pants.toml. However, 
pants peek
can't do that and only looks at target values. This 
__defaults__
trick sets the value on the target, so it becomes visible to 
pants peek
👍 1
m
I can’t seem to get any info on what Python version is being used with 
pants peek
, regardless of where I define my contraints: • as a default in my root 
BUILD
(as you describe above) • with 
python_sources(interpreter_constraints=["==3.12.*"])
in my lambda’s 
BUILD
• in the pants.toml in the 
[python]
section as an 
interpreter_constraints
parameter • in the lambda’s 
pyproject.toml
(as a poetry dependency, used with a 
poetry_requirements
in the lambda’s 
BUILD
) I can get most of what’s output with 
pants package src/lambdas::
with 
jq
and 
pants peek
, but not everything:
Copy code
$ pants package src/lambdas::
09:50:12.32 [INFO] Wrote dist/src.lambdas.whatever/whatever.zip
    Runtime: python3.12
    Architecture: x86_64
    Handler: lambda_function.handler

$ pants peek src/lambdas:: | jq '.[] | select(.target_type == "python_aws_lambda_function") | {path: (.address | sub(":.*$"; "")), handler: (.handler | sub("\\.py:"; ".")), architecture}'
{
  "address": "src/lambdas/whatever",
  "handler": "lambda_function.handler",
  "architecture": "x86_64"
}
(I also have to do extra bash to split the records & convert 
address
into the output path 
dist/src.lambdas.whatever/whatever.json
for each of the results 
pants peek
returns)
I still think it’d be immensely useful to be able to configure pants to output the data shown on screen when packaging a lambda (runtime, architecture, handler) to a JSON file adjacent to the zipped lambda, so that the terraform which deploys the lambda could derrive its config from pants:
Copy code
locals {
  lambda_base = "../dist/src.lambdas.whatever/whatever"
  lambda_config = jsondecode(file("${lambda_base}.json"))
}

resource "aws_lambda_function" "whatever_lambda" {
  function_name = "whatever_lambda"

  handler = local.lambda_config.handler
  runtime = local.lambda_config.runtime
  architectures = [local.lambda_config.architecture]

  filename         = "${lambda_base}.zip"
  source_code_hash = filebase64sha256("${lambda_base}.zip")
}
Is there a way of doing this I’m missing? Or a good place for me to make this request of the pants team?
✅ 1
s
Or you infer it based on python sources?
If you set 
runtime
, it should be visible in pants peek
m
Ahh super, thankyou! That does work! 🥳 The only problem is, I’m now setting my python runtime in two places; once for the local python interpreter (in my 
pyproject.toml
, which is calculated by inference by 
pants package
and output to the terminal), and once in my 
BUILD
for in 
python_aws_lambda_function(runtime="python3.12")
. Any ideas on how to get 
peek
to output the inferred python runtime?
s
How do you set your python dependencies? via 
dependencies
or 
handler
?
If you don't care about multiple python versions you could set it all in a single place
Copy code
python_version = "3.12"
__defaults__(all=dict(
    interpreter_constraints=[f"=={python_version}.*"],
    runtime=f"python{python_version}",
))
I don't think there is an easy way to get it from inside pyproject.toml If you really want to get it from there, you could try some magic with `.pants.bootstrap`:
Copy code
#!/bin/bash

export PYTHON_REQUIRES=$(grep python_requires pyproject.toml)
Then in BUILD:
Copy code
python_requires = env('PYTHON_REQUIRES')
# then parse it and use it
m
This might work, but I’m going to have multiple pythons in my monorepo 😔 Thank you so much for your help here, you’ve offered so many tips for some stranger who has a rather specific request 😅
❤️ 1
s
No, that's actually a natural request, pants is mostly a metadata layer on top of your existing code, so it's whole purpose is to provide ways to inspect metadata I think there was a feature request to read external files directly from BUILD, but I don't think it was implemented
🙇‍♂️ 1
but I’m going to have multiple pythons in my monorepo 😔
You could try inferring this from dependencies then, you could do pants peek on your lambda and get 
dependencies
from it. Then inside dependencies find 
python_source
it depends on and do pants peek on 
python_source
, it should have interpreter_constraints in it
m
So such luck; possibly because I’m using 
poetry
to manage my python version? I’ve put all the relevant bits in this gist (note that I’ve switched to python 3.13 since we started this conversation). I’ve got to duck out for the rest of the day — so I won’t be able to reply for a while, but thanks for your help & thoughts!
h
Separately from this workaround, it seems like a reasonable feature to have, and it would not be difficult to add.
In fact it could be added generically for all packaged artifacts
Well, there are a couple of ways of doing it
For example, we gather those “extra log lines” and log them here
👀 1
So it would be easy to write them to 
dist
as well just above that
Alternatively, we can have a separate metadata pathway that is more likely to be machine-readable and unchanging
since “log lines” implies “human readable, and subject to change”
✅ 1
The other place we use these extra log lines is for docker images, but there the artifact is already a machine-readable metadata file
so there is no extra value in logging those lines
So I’m leaning towards just adding it as a little bit of JSON in the FAAS case
❤️ 1
which would also be fairly easy
I can try and hack it together in a bit
m
That sounds superb! Let me know if I can be of any help!
h
Best help would be to test this out, once it’s merged
@mammoth-dawn-85816 Is this fit for purpose?
m
Thanks for the nudge — it’s been a busy week for me! This looks like exactly what I was seeking! I have pants 2.27.0 installed (via Homebrew), but I’ll see if I can get v2.29 (presumably a dev version) installed to test it out — thank you so much, I really appreciate all the thought & work you put into understanding my issue then building this new functionality!
(As an aside, it doesn’t look like anyone has run the bump scie-pants workflow to release v2.28 via homebrew yet!)
h
That is for upgrading 
scie-pants
itself. The current scie-pants version should be able to upgrade Pants to 2.28
m
Perhaps I’m not following how to upgrade properly, but I installed via homebrew, and have just run:
Copy code
SCIE_BOOT=update pants
pants help
This created a new virtual environment (apparently I upgraded from 
scie-pants
0.12.3 to 0.12.5), but 
pants help
still shows:
Copy code
Pants 2.27.0
When I think 2.28.0 was released a week or so back. To be honest, I’m a little confused by scie-pants, I’ve only just clocked that the way to give it arguments is with 
SCIE_BOOT=xxxx pants
, rather than (
pants upgrade
or similar) — so the chances are very hight this is a pebkac! 😅 I’m a fool. It turns out if you specify a pants version in yout 
pants.toml
then pants very correctly provides that version 😂
h
Yes, exactly
🙇‍♂️ 1
15 Views
Open in Slack
PreviousNext
Powered by