Hi, I'm running into some issues with pex packaging not being compatible with opentelemtry auto-instrumentation (https://github.com/open-telemetry/opentelemetry-python-contrib/issues/1994) It think the issue is that the autoinstrumentation uses python's sitecuztomize infra to inject hooks (https://kinvolk.io/blog/2020/06/opentelemetry-python-implementation-deep-dive/), and I think the pex boilerplate does something to mess with this with the whole venv thing, but I'm not fully clear on what it does or if this can be resolved

Does https://pantsbuild.slack.com/archives/C046T6T9U/p1677161127907199 or https://github.com/pantsbuild/pex/issues/2065 give useful guidance? (I found it by doing a slack/web search for "pex opentelemetry", I haven't checked if they actually resolve the problem)

It seems like there is some solution but the thread is very long so I am not clear on what it is yet 😂 thanks for the pointer, I couldn't find it on Google!

Hey Alex! I previously tried to solve this, but I didn't manage to get a solid solution for this up and running, so I just decided to do instrumentation calls in my code manually instead. Would be very cool if we managed to find a better way!

I filed a bug for this in otel, but I did find a workaround by importing the sitecuztomize package directly: https://github.com/open-telemetry/opentelemetry-python-contrib/issues/1994

@famous-xylophone-36532 that is extremely frustrating to hear. I did a lot of work to prove auto instrumentation works with Pex to the point of learning the ecosystem, reading their code and distilling down the mechanism and principles.

Indeed, I remember that! I would be happy to circle back to this and give you a better report on what caused me to rever to manual instrumentation once I finish up some stuff.

FWIW I followed the PEX unpacking Gist steps and was able to successfully send OTEL metrics to DataDog for EMR Serverless jobs, without any issues.