https://pantsbuild.org/ logo
Join Slack
Powered by
Hello all, Has anybody messed up with the OTel Lam...
# general
s
Hello all, Has anybody messed up with the OTel Lambda Python Collector + SDK layer and Pants? Any example to offer that we can reuse? There is lots of magic in e.g. https://github.com/open-telemetry/opentelemetry-lambda/blob/main/python/src/otel/otel_sdk/otel_wrapper.py that is preventing from making the 2 compatibles apparently.
f
For use in Pants rules?
s
was about to edit We use Serverless
f
My question is: Are you asking about using OTEL with your user services which are built by Pants or do you want trace the execution of Pants itself using OTEL?
s
We want to trace what happens inside lambda runtimes
We would like to auto instrument using the standards https://aws-otel.github.io/docs/getting-started/python-sdk/auto-instr This library contains the code linked above, and at a first glance looks an hard task to make it compatible with Pants.
f
How so? I would think you just need to: (1) Add those packages to the Pants "resolve" you are using to manage the lambda's third party dependencies. And (2) in your lambda just initialize the OTEL library as required.
What magic is 
opentelemetry-instrument
actually providing to the lambda?
For example, I read:
Here, use the 
opentelemetry-bootstrap
command to automatically detect and install OpenTelemetry python packages.
So just add those packages to your resolve and get them to the lockfile.
The automatic installation here seems like a way to save time but the actions can be done manually, right?
And just manage the dependencies directly.
That page links to the "manual" setup instructions: https://aws-otel.github.io/docs/getting-started/python-sdk/manual-instr
And that "manual setup" page lists the exact PyPi packages to install (via the Pants resolve / lockfile mechanism) and the code snippets you need in your user code to setup the OTEL instrumentation.
s
Sorry IDK why that page got linked but context here is Lambdas https://aws-otel.github.io/docs/getting-started/lambda/lambda-python
https://github.com/open-telemetry/opentelemetry-lambda/blob/main/python/src/otel/otel_sdk/otel_wrapper.py There is lots of magic that happens in this file to be able to intercept invocations and instrument the Lambda only when invoked
There is a whole lot more on the actual instrumentation code: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumen[…]lambda/src/opentelemetry/instrumentation/aws_lambda/__init__.py We are actually going down the manual instrumentation route now, but having second thoughts because of the obvious disadvantages of not leveraging the community efforts.
f
Why? Seems to me you would just add an invocation of 
AwsLambdaInstrumentor().instrument()
to your lambda. Is that not a fair assessment?
The "magic" of the automated setup code seems to me to complicate understanding of configuring OTEL instead of just running the standard setup directly in your lambda.
s
Why? Seems to me you would just add an invocation of 
AwsLambdaInstrumentor().instrument()
to your lambda. Is that not a fair assessment?
IDK might be not Just looking at this line https://github.com/open-telemetry/opentelemetry-python-contrib/blob/main/instrumen[…]lambda/src/opentelemetry/instrumentation/aws_lambda/__init__.py (but there are more examples) we thought there are lots of expectations to use the provided layer/sdk
f
Maybe try it in a staging environment and see what happens?
s
yeah we done that m8 🙂
f
as I would expect 🙂 and so what did you expect to occur which did not occur?
(and to clarify, I am not familiar with OTEL and AWS Lambdas, so I can only ask the same generic questions which I would ask for any troubleshooting issue)
This does not seem like a Pants issue at all, but a lambda setup issue.
s
I agree Not even a lambda one, prob would have been nicer to offer a "DIY auto instrumentation" from the OTel community, say a decorator on the lambda handler, before having to buy into the full fledged wrapperised solution. It is nice though to have a simple oneliner to instrument your lambda
f
It is nice though to have a simple oneliner to instrument your lambda
I agree. But I am not sure what Pants can do per se there. This OTEL setup problem would exist regardless of the choice of build system.
s
Yeah good point Prob worth for me to come back later with an open source repo and some logs
f
That seems like a good next step. Given this is the Pants Slack, we are better able to help with issues where you expected Pants to take an action and it did not take that action. 🙂
On the other hand, there are probably people knowledgeable here on AWS and Lambda but that advice depends on who you happen to talk to ...
👍 1
e
f
Hey! We’ve had a similar problem to this by instrumenting datadog. My understanding is that if you are looking to instrument it through the layer my understanding from reading this page is that there are three things needed here: 1. The layer is added when the lambda is created, alongside of the code 2. Active tracing on the lambda is enabled 3. A magic environment variable is set to have opentelemetry wrap your lambda handler All 3 of those are settings for the lambda deployment and shouldn’t require any changes to the 
.zip
made through pants. If that’s all true, why not leave the instrumentation to the tool you use to deploy the lambda instead of the tool that builds the lambda?
It’s been a while since I’ve used serverless, but it should have options for all of those things
s
Hey @faint-businessperson-86903 , thanks! Re #2 I think you want active tracing only when using XRay. There are some weird things going on with OTel, e.g. when doing open-telemetry manual instrumentation we had to add those vars (don't remember exactly if both or which one in particular) otherwise Python was ending up in the REPL:
Copy code
PYTHONPATH: /var/task
PEX_EXTRA_SYS_PATH: /opt/python
We are now halfway in going the manual instrumentation route and decided this morning to just stick to it, to give us much more control the software we are running.
b
@strong-xylophone-41556 just wondering, did you ever resolve this? My suspicion is I’m stuck at getting 
AWS_LAMBDA_EXEC_WRAPPER
env var to do anything currently
e
@breezy-bear-13848 are you in a position to try Pants 2.17+ and https://www.pantsbuild.org/v2.17/docs/reference-lambdex#layout (use "zip"). That should get Pex totally out of the picture and net you a vanilla zip with exactly the layout AWS recommends.
b
If it resolves issues with tracing, I could justify that yeah. Using the zip method, would the exec wrapper env var affect execution automatically?
e
Using "zip" your problems should reduce to problems you would have had / had to work through anyhow is the shortest answer.
It gets Pants and Pex totally out of the picture and hands you a zip exactly in the layout and content that AWS reccomends.
I don't use AWS lambda functions; so that's the best I can provide quickly.
b
All good, I’ll try that out anyway and see how that goes. Thanks 🙂
I get a completely unrelated issue when I tried this:
Copy code
A distribution for bcrypt could not be resolved for cp39-cp39-linux_x86_64.
Found 1 distribution for bcrypt that do not apply:
1.) The wheel tags for bcrypt 4.0.1 are cp36-abi3-manylinux_2_24_x86_64 which do not match the supported tags of cp39-cp39-linux_x86_64:
cp39-cp39-manylinux2014_x86_64
Might just try PowerTools instead of opentelemetry lol
s
Hey @breezy-bear-13848 sorry just noticed We ended up creating our own layer with only the OTel collector, avoid the Python wrapping done by the aws otel python distro, and doing manual instrumentation Is not terribly complicated and turned out to be a better solution e.g. we created an API to instrument a lambda with attributes that get automatically propagated to every telemetry from that lambda
8 Views
Open in Slack
PreviousNext
Powered by