https://pantsbuild.org/ logo
Join Slack
Powered by
Hey people! Has anyone used opentelemetry autoinst...
# general
f
Hey people! Has anyone used opentelemetry autoinstrumentation + Pex? Works?
Does not seem to work at all unfortunately. Would be amazing if it did! I will dive into this a bit today to see if there is any possibility that it might work.
My setup is that I declare my auto-instrumentation using a Kubernetes operator like so: https://gist.github.com/Peder2911/5f27bdae8b5b6df17fdac58f1d9b84f9
And I run Docker images built by Pants (pex in Docker).
e
f
I am using defaults, so I don't think so. Sorry, I can't tell you too much about how opentelemetry auto instrumentation works to help solve this, but this looks promising! I'll try it./
Quoting the eminent Aaron Abbot from the Otel slack (I am cross posting):
@famous-xylophone-36532
not super familiar with pex internals. Auto instrumentation works
through setting PYTHONPATH to include a sitecustomize.py file which does
the monkey patching at startup. If pex is overriding the PYTHONPATH
before it starts the interpreter, that might be the issue. You can try
logging the sys.path in your code to check?
e
See: https://www.pantsbuild.org/docs/reference-pex_binary#codeinherit_pathcode A good thing to keep in the forefront of your mind when using Pants & Pex - they both strive very hard for reproducibility and this heavily involves hermeticity. They both try hard to isolate from local system happenstance so that results are repeatable anywhere. One very common happenstance that can foil reproducibility is an env var I have set but you do not. As such, both Pants & Pex tend to scrub relevant env vars from the environment unless you tell them not to. The link above is to a setting that tells Pex not to scrub 
PYTHONPATH
.
šŸ™Œ 1
f
Ah, this seems key to let opentelemetry do it's (dark) magic to the path. I'll try this option as well.
Hmm, no, still does not seem to work..
I am trying 
opentelemetry-instrument --traces_exporter console --metrics_exporter console --service_name your-service-name python ./bin.pex
(FastAPI application) built with 
execution_mode="venv"
and 
inherit_path="prefer"
and it does not emit any logs.
So, Opentelemetry auto instrumentation injects stuff into the Pythonpath in order for it to work, but printing 
sys.path
from inside a pex, even running in venv mode and with inherit_path, seems to indicate that that isn't working:
Copy code
['', '/root/.pex/venvs/e3d3f05fa9939d70cefbb33a63372359b532d899/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055', '/usr/local/lib/python39.zip', '/usr/local/lib/python3.9', '/usr/local/lib/python3.9/lib-dynload', '/root/.pex/venvs/e3d3f05fa9939d70cefbb33a63372359b532d899/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055/lib/python3.9/site-packages']
See how the path is injected when I just run a Python script the "regular" way: 
['/mnt/wd', '/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/auto_instrumentation', '/mnt/wd', '/usr/local/lib/python39.zip', '/usr/local/lib/python3.9', '/usr/local/lib/python3.9/lib-dynload', '/usr/local/lib/python3.9/site-packages']
Get how auto-instrumentation might seem contrary to what Pex is built to do, but it would be super nice to pull off autoinstrumentation of Pex files regardless of that šŸ˜„
So does Pex strip environment variables or what? Setting PYTHONPATH seems to have no effect on 
sys.path
.
Hmmm
e
I gave you lots of links, you gave me ~none. Are you using this?: https://pypi.org/project/opentelemetry-instrumentation/
šŸ‘ 1
f
Ah yes sorry. Indeed I am!
e
Ok, I'll do some homework and get back to you within the hour.
šŸ™Œ 1
f
In the end I would like to just use this Operator to have the instrumentation happen automatically, but that should be equivalent to 
opentelemetry-instrument
(https://github.com/open-telemetry/opentelemetry-operator)
No worries John, thank you for your help so far!
Here are some more useful links: The 
opentelemetry-autoinstrument
cli: https://opentelemetry.io/docs/instrumentation/python/automatic/ How opentelemetry injects in Kubernetes with the above-posted operator: https://github.com/open-telemetry/opentelemetry-operator/blob/0067c3547b1032c0a1c9f7d45391e18d79b68289/pkg/instrumentation/python.go#L54-L59
e
Copy code
$ tree
.
ā””ā”€ā”€ src
    ā””ā”€ā”€ app.py

1 directory, 1 file

$ cat src/app.py
import os
import sys

from random import randint
from flask import Flask, request

app = Flask(__name__)

@app.route("/rolldice")
def roll_dice():
    return str(do_roll())

def do_roll():
    return randint(1, 6)

print(f"sys.path:\n{os.linesep.join(sys.path)}", file=sys.stderr)

# The PEX must contain all needed opentelemetry requirements:
$ pex opentelemetry-distro opentelemetry-instrumentation-flask flask requests -D src -c opentelemetry-instrument -o app.pex --include-tools

# The PEX must then be installed in a venv and its console scripts both ppre-pended to the PATH and made non-hermetic so PYTHONPATH can leak through and influence them.
$ PEX_TOOLS=1 ./app.pex venv --non-hermetic-scripts --bin-path prepend app/venv

$ app/venv/pex --traces_exporter console --metrics_exporter console flask --app app run
sys.path:
/home/jsirois/support/OpenTelemetryPex
/home/jsirois/support/OpenTelemetryPex/app/venv/bin
/home/jsirois/support/OpenTelemetryPex/app/venv/lib/python3.10/site-packages/opentelemetry/instrumentation/auto_instrumentation
/home/jsirois/support/OpenTelemetryPex
/usr/lib/python310.zip
/usr/lib/python3.10
/usr/lib/python3.10/lib-dynload
/home/jsirois/support/OpenTelemetryPex/app/venv/lib/python3.10/site-packages
 * Serving Flask app 'app'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on <http://127.0.0.1:5000>
Press CTRL+C to quit
^Z
[1]+  Stopped                 app/venv/pex --traces_exporter console --metrics_exporter console flask --app app run
$ bg %1
[1]+ app/venv/pex --traces_exporter console --metrics_exporter console flask --app app run &

# Test it:
$ curl <http://127.0.0.1:5000/rolldice>
127.0.0.1 - - [23/Feb/2023 09:17:18] "GET /rolldice HTTP/1.1" 200 -
5jsirois@Gill-Windows:~/support/OpenTelemetryPex $ {
    "name": "/rolldice",
    "context": {
        "trace_id": "0x2ebd6f04e90dc521d927d4009b16fb98",
        "span_id": "0x8cb36c0be87b2f65",
        "trace_state": "[]"
    },
    "kind": "SpanKind.SERVER",
    "parent_id": null,
    "start_time": "2023-02-23T17:17:18.281424Z",
    "end_time": "2023-02-23T17:17:18.284099Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "http.method": "GET",
        "http.server_name": "127.0.0.1",
        "http.scheme": "http",
        "net.host.port": 5000,
        "http.host": "127.0.0.1:5000",
        "http.target": "/rolldice",
        "net.peer.ip": "127.0.0.1",
        "http.user_agent": "curl/7.81.0",
        "net.peer.port": 48440,
        "http.flavor": "1.1",
        "http.route": "/rolldice",
        "http.status_code": 200
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.16.0",
            "telemetry.auto.version": "0.37b0",
            "service.name": "unknown_service"
        },
        "schema_url": ""
    }
}
^C
$ kill %1
[1]+  Terminated              app/venv/pex --traces_exporter console --metrics_exporter console flask --app app run
šŸ™Œ 1
So the keys are: 1. Pex must use venv created by 
pex-tool
(
PEX_TOOLS=1
) and that venv needs, critically, both of: 
--non-hermetic-scripts --bin-path prepend
2. The PEX must contain all the opentelemetry packages needed.
šŸ™Œ 1
Luckily for you, the reccomended Docker setup for PEXes has you installing them im the image using 
PEX_TOOLS=1
already.
šŸ™Œ 1
Hopefully that gets you down the road @famous-xylophone-36532, but please speak up if you have trouble.
šŸ™Œ 1
f
Amazing! Will look at how to make this happen in my Pants project. You are a hero šŸ’«
I am having no luck with getting this to work with my current setup unfortunately. Will try to set up a minimal reprex repo so we can figure this out.
Here I tried using the same procedure that you demonstrated in your script, using opentelemetry-instrument as the entrypoint, and then pointing to the "actual" entrypoint in the arguments. As you can see in the repo, this is kind of clunky since I'd ideally like to let developers define the entrypoint in the BUILD file and not have them worry about instrumentation at all (the ideal would be 
opentelemetry-instrument ./my-pex.pex
). Anyways, I cannot get any telemetry from the current implementation either: https://github.com/Peder2911/otel_pants_reprex Not sure if I did this correctly? Also, would be interesting to look into whether the above snippet could work with modifications to Pex?
e
@famous-xylophone-36532 the Pex issue you filed does not perform the 
PEX_TOOLS...
step I indicated above was critical. Is there a reason you don't wish to add a 
RUN ...
line to your Dockerfile to do that required step?
šŸ™Œ 1
f
Hey sorry, I mentioned your fix below. The issue is more about clarifying whether it would be possible to keep the configuration and execution of opentelemetry auto-instrumentation wholly outside of the Pex.
I tried using your suggested fix in the context of my repo, but I could not get it to work. Probably missed something. You can see that I am using the fix here: https://github.com/Peder2911/otel_pants_reprex/blob/main/apps/dockerfiles/app.dockerfile
e
It's extremely confusing that the Pex issue completely elides acknowledgement of the missing PEX_TOOLS step. Moving on from that, though, if you could add a top level README to your example repo with the commands you run to end up in failure, that would be great.
f
If you are thinking about the two code-blocks in the top post in the issue, I am just trying to show what the ideal method for instrumenting would look like (simply 
opentelemetry-instrument
ing the pex). I do mention the fix below, and I'd be happy to use your working fix in our repo if I can get it to work. I'll add the README right away.
Maybe I could write some kind of tool that would be able to augment an existing Pex, injecting the Opentelemetry magic. Hmmmm....
That would be much preferable to changing the codebase to fit my one use case I guess? Do you know if anyone has implemented anything like that?
e
So, your repo used PEX_TOOLS=1 to install the PEX file in a venv ... and then proceeded to not actually use the venv. This fixes:
Copy code
diff --git a/apps/dockerfiles/app.dockerfile b/apps/dockerfiles/app.dockerfile
index 5b47ef0..7138bdb 100644
--- a/apps/dockerfiles/app.dockerfile
+++ b/apps/dockerfiles/app.dockerfile
@@ -3,4 +3,4 @@ FROM python:3.9
 ARG PEX_FILE
 COPY $PEX_FILE /app.pex
 RUN PEX_TOOLS=1 /app.pex venv --non-hermetic-scripts --bin-path prepend /app/venv
-ENTRYPOINT ./app.pex --metrics_exporter console --traces_exporter console uvicorn <http://asciinator.app:app|asciinator.app:app> --host 0.0.0.0
+ENTRYPOINT /app/venv/pex --metrics_exporter console --traces_exporter console uvicorn <http://asciinator.app:app|asciinator.app:app> --host 0.0.0.0
šŸ˜… 1
Small miss, but does that make sense @famous-xylophone-36532?
f
Indeed, let me try.
What do you think about my frankenstein idea ^
e
I think you're moving too fast. Try that 1st and then we can talk PEX, you're basing alot off 1 example I gave you.
f
Good call!
e
Great - so if you can report back quick here on success / failure with that edit, we can move on to ideal workflow / setup.
šŸ™Œ 1
f
We have telemetry! Awesome! Thanks John.
e
Ok, so back to ideal flow. 1st - how opentelemetry works - the key bit is not the details of the instrumentation code, its the high level launch procedure: 1. set PYTHONPATH 2. execv
So, on #1:
Copy code
'/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/auto_instrumentation',
'/usr/local/lib/python3.9/site-packages'
That's a snip from a plain-old venv use - you provided that snip. A key point here is the 2nd entry.
The opentelemtry assumes it is in the same venv as your app. It adds the 1st line, but not the second. The second comes along only because opentelemtry is installed in the venv with your app.
So, to get opentelemetry installed in the same venv as your PEX, it needs to be included in the PEX. What is not required is that `opentelemetry-instrument`be the default entrypoint of your PEX as I used in the example.
You can re-direct to an alternate entrypoint only when desired via env var: https://pex.readthedocs.io/en/v2.1.123/api/vars.html#PEX_SCRIPT
I'll stop there @famous-xylophone-36532 and let you page that in / ask further questions.
f
Right, this makes sense! So if I could allow my developers to set whatever entrypoint they need to set, and then tease out what entrypoint with some kind of inspection, then I could set 
PEX_SCRIPT
to 
opentelemetry-instrument $EXISTING_ENTRYPOINT
and have my instrumented app?
e
Correct. I think we can eliminate the tease out part actually though and $EXISTING_ENTRYPOINT can just be the pex.
f
Yeah, that's a good point. I'll try that!
e
So, for Docker use you want to do the extra PEX_TOOLS install in a normal venv step. That - opentelemetry aside - gives you best performance at image runtime. But, for development, that is not ideal. Ideally you could just run the PEX file itself.
The missing bit for the latter is exposing 
--non-hermetic-scripts
as a Pex build option like 
--venv
. Right now you can only get `--non-hermetic-scripts`when using 
PEX_TOOLS
to install a venv by hand.
I'll run a quick experiment with that and send up a PR attached to the issue you filed if that works - I think it will.
f
Awesome! Well, since I don't really need working telemetry in dev I can just keep my existing workflow. So if I understand you correctly, the idea would be to set the 
PEX_SCRIPT
env variable in the Docker image before executing the PEX itself to be something like 
opentelemetry-instrument app/venv/pex
?
e
You want exactly 
PEX_SCRIPT=opentelemetry-instrument
- the rest of the existing (fixed) Dockerfile 
ENTRYPOINT
remains the same.
When you build the PEX, the 
-c opentelemetry-instrument
says, make the 
opentelemetry-instrument
console script the entrypoint. Setting the 
PEX_SCRIPT=opentelemetry-instrument
env var does the same thing, but at runtime.
f
Aha! Ok, well, I was hoping that I would somehow be able to keep the "included" entrypoint from the Pex around and not have to specify it in the Dockerfile. Would that be possible?
(using a single generic Dockerfile for multiple applications)
e
It looks like that requires 1 more leak:
Copy code
$ head -1 app/venv/pex
#!/home/jsirois/support/OpenTelemetryPex/app/venv/bin/python3.10 -sE
That -E tells python to ignore 
PYTHON*
env vars.
šŸ™Œ 1
Pex is very very hermetic!
I'll look at folding leaking there too when using 
--no-hermetic-scripts
.
šŸ™Œ 1
I think I have enough info about your use case to continue this all in the Pex issue you filed. Thanks for all the details and the repo repro case.
ā­ 1
f
Yeah seems like this case is breaking more than a couple idioms with Pex, but it would be pretty cool to pull it off.
Thank you for your excellent work here. Weekend time in Norway, have a good one!
e
@famous-xylophone-36532 support for `pex_binary`with `venv_hermetic_scripts=False`lands in Pants 2.16.x here: https://github.com/pantsbuild/pants/pull/18376/files#diff-ec50dab75eb23cdaee86ba8014950269b46859d2a21fcc826a2765297e4086b4R653 Afaict that fully solves you use case. I'll add to the ticket a description & example of how to even eliminate 
opentelemetry-instrument
from the use PEX by creating a single 
opentelemetry-instrument.pex
that can then be combined with user PEXes only in container images. I think this works anyhow and I'll describe success or failure on that ticket.
šŸ™Œ 1
f
Amazing. Thank you! I had fun looking at the code for Pex / Pants while investigating this. Great work!
11 Views
Open in Slack
PreviousNext
Powered by