:wave: I’m new to Pants and trying to address a se...
# general
g
👋 I’m new to Pants and trying to address a security issue that was first detected in a PEX binary we use to run a Python web application. The PEX binary was built with Pants
2.7
. I think that our security scanner detected a version of the Python
wheel
in package in the PEX application binary with the security vulnerability… apparently all versions of
wheel
below
0.38.1
have this vulnerability. When I built the PEX binary with Pants
2.7
:
wheel
version
0.36.2
was detected. When I built the PEX binary with Pants
2.14
:
wheel
version
0.37.1
was detected. I changed as little as possible in the Python application and configuration when upgrading the Pants versions (but I did have to make some small changes as a result of the API / CLI changes between the two Pants versions). I’m reasonably certain that the increase in
wheel
version was due to the Pants upgrade, having gone to some pains to rule out other factors. My question is: is there any way I can ask Pants to embed a newer version of the
wheel
package that replaces these older versions in the
PEX
binary? Is there any other information I could give that would help shed any more light on this issue? Thanks in advance for taking a gander!
e
The CVE is for uses of the wheel CLI, Pex does not use the wheel CLI. Is that sort of fact considered by your security audit process, or is it more basic name matching - Pex includes CVE package, so blacklist despite details?
g
That is a fantastic question @enough-analyst-54434. I am not 100% sure, The detection seems to me to be the sort of naive name matching that you’re describing.
h
TBH the naive name matching stance is not unreasonable. It would be hard to maintain a security posture that required digging into the specific uses of potentially vulnerable packages and accepting the assurances of the consuming code that it wasn't using the bad bits... For one thing, there's no easy way to automate that check. So we should probably figure out a way to solve this.
g
Can I provide any context or color that would help us better understand what’s going on here? Or does my description above give you a sense of what may be going on @happy-kitchen-89482?
h
I think it's clear! Pex vendors wheel and bundles it into .pex files it creates. Recent versions of Pex vendor wheel==0.37.1, and earlier ones vendored 0.36.2.
So Pex would need to vendor a version of wheel untainted by CVEs, I think
I don't see a strong reason not to upgrade its wheel version, will take a look
g
I don’t like to use the word “hero” a lot… but it seems like it might be fitting in this case.
e
This is all very strange though. Pex vendors wheel, PEXes do not.
g
I think [target of the scan is] a Docker image where the PEX is [being] built, if that helps add any context.
But I will definitely defer to the experts here.
It did seem fairly clear from my investigation that the move from Pants 2.7 -> 2.14 caused the version of
wheel
caught by the scan (whatever it was scanning) to to go from
wheel
0.36.2
->
wheel
0.37.1
.
e
Well, Pex is capped as it currently stands for Python 2.7 support. If you use --pip-version 22.3 you get modern wheel downloaded and used dynamically, but 37.1 is still vendored.
Presumably that distinction is also too fine for coarse grained CVE mitigation.
h
Right, my bad, wheel doesn't get bundled into a pex file, the tool uses it when generating the pex file.
Is that Docker image invoking Pants? Which is in turn invoking pex?
g
Yes, I think that is what is happening.
h
So when you upgrade to Pants 2.16 (when that is released, currently it has dev releases) the
[python].pip_version
option will be available, and you can set that to use a more recent pip version, which will in turn use a more recent wheel version.
But that presumably won't silence the security scan, since the older wheel version will still exist, vendored inside the pex tool, it just won't be used.
g
The security warning is getting trigged by the following
RUN
invocation inside a Docker container that builds our application’s
.pex
binary. It seems that the security warning is raised as a result of packaging the PEX…
Copy code
RUN ./pants --level=debug package /path/to/our/app:app-bin # (pex_binary target)
It seems that we use the same Docker container to both build and run the Python web application. Perhaps one potential work-around for the security issue is building the PEX binary outside of the docker container that we use to run the Python web application?
h
Is the security warning based on static analysis of the container image, or is triggered when that invocation is actually run? I have no idea how these things work in practice...
Generally, though, I would recommend deploying/running in a separate container from the build-time one, for multiple reasons.
Including, I guess, this reason 🙂
For one thing, why deploy your build-time repo state?
The idiomatic thing would be to run
package
in CI or in a build-time container, capture the .pex file out of it and embed that in a production Docker image
(Pants has Docker support and can do the "build the production image and embed the .pex in it" for you, but one thing at a time...)
e
Say you do manage to separate build from run. Reality is still the same. If there is a real security issue at build time (there are 0 security issues here, but we're checking boxes), is it actually ok / secure to have the CVE present at build time?
g
Is the security warning … triggered when that invocation is actually run
Yes. The security warning is being triggered when
./pants … package
invocation is actually run. The scan points to that invocation line and not from the subsequent
ENTRYPOINT
line in the Dockerfile which goes on to run the Python web app.
Say you do manage to separate build from run. Reality is still the same. If there is a real security issue at build time (there are 0 security issues here, but we’re checking boxes), is it actually ok / secure to have the CVE present at build time?
@enough-analyst-54434 we are absolutely checking boxes here. I’ll raise the issue, but I’m guessing that the answer will be “our policy is to not worry about build-time, only run-time.” As for whether that’s a reasonable security posture or not, I will let The Security Professionals™ determine that. I just work here.
And y’all I have to say a very heartfelt thank you for helping me better understand this problem. You really didn’t have to do all the analysis but it was so appreciated.
FWIW this is what I heard back from my question about why we care about CVEs at run time vs. build time, it sounded reasonable enough to me 🤷:
With all the reported CVEs we’re mainly concerned about someone’s ability to exploit them and gain access to our infra or customer data. Typically we expect this would be achieved by interacting maliciously with running services. When a container is just used for building, and is then thrown away, there isn’t much opportunity for an attacker to insert themselves into that process.
e
Ok, well wisdom / opinions aside - if that's the stance and separation of the steps works for you it sounds like you have a path forward.
g
Following up here: everything has been resolved successfully. We’ve eliminated the security warning and all is well. Thanks again for everything you all did to support us here.
🎉 1
h
You're welcome! So the solution was to separate build time and runtime?
g
So the solution was to separate build time and runtime?
That’s correct.