https://pantsbuild.org/ logo
Join Slack
Powered by
Hey there! We're using Pants in the CI in gitlab. ...
# general
f
Hey there! We're using Pants in the CI in gitlab. One of the things we do is build and publish a docker image with 
pants publish
. Under the hood, this uses Docker - which in a CI environment requires 
dind
, which is highly discouraged because it requires root privileges in the runners. Our DevOps team will soon forbid this - and gitlab recommends using tools like 
buildah
and 
kaniko
that don't require privileges. We've tried to find a way to get Pants to use them, but we haven't been very lucky. If anyone has any tips it'd be much appreciated!
➕ 1
l
Looking at past history this appears to be an existing issue. https://github.com/pantsbuild/pants/issues/14395
Santiago, could you describe what you have done so far to attempt to use kaniko from pants?
There are related threads in slack which you can find by searching for 
kaniko
, but none seem to come to a clear general-purpose solution. Kaniko's strong recommendation to run from their own image seem tricky to mesh with the way one usually runs pants in CI, but I am quite new to this.
h
cc. @curved-television-6568
c
yea, I’ve looked into supporting something not relying on the docker daemon in the past, but haven’t made any real head way with that.
my main hurdle I think was that those tools are linux friendly, while I’m on a Mac.. which makes the development of the feature less streamlined 😉 things may have changed since then, tho..?
e
One thing to note to the Mac folks out there - unlike Mac, it's legal to run Linux in a vm (IIUC that's exactly what Docker for mac does); so you could do integration work in a Linux vm (as opposed using DinD).
c
yes, true.. that was what the “less streamlined” alluded to. I wasn’t motivated enough at the time to go down that road for that feature alone.
fwiw I’m not an Mac evangelist, but as long as it is the tool my $-provider puts in my lap, it’s what I have to work with. Would very much enjoy having a pure linux setup as well/instead.
e
I've been using Windows for >1 year now to try to get in Windows support in the background but 95% of my time has been in the WSL vm - honestly not bad working in that vm anyhow. Can't wait to get back.
Does your $-provider let you wipe? That's what I did. Linux on the Mac they gave me. Asahi I think is pretty decent now.
c
I just tried out the WSL on my private Win box last week and it was a pleasant surprise. There’s some fidgy edges, but overall an OK experience.
e
Yeah - its fine from what I've found. I hate the Windows ads - ads in an OS is insane, but the WSL works fine.
c
ADs?!.. not sure I’ve seen any of those (yet?) — I shy windows for most parts, mainly have it for games 😛
e
Yeah - you get ad notifications in a side bar!
Not from Windows, from 3rd parties.
😲 1
Horrific.
c
Does your $-provider let you wipe?
Unlikely (as in I’m pretty sure it’s a no)
e
Ok. I actually didn't ask when I did it 😉 but that's a bit bold.,
One thing totally insane in Windows / WSL is WSL Pythons are at least 10% faster than the Windows ones.
Python is pretty damn laggy - even for Python - on Windows.
c
heh, interesting
e
A stupid comparison, but indicative:
Copy code
jsirois@Gill-Windows:~ $ hyperfine 'echo import this | python3.12'
Benchmark 1: echo import this | python3.12
  Time (mean ± σ):      22.8 ms ±   0.7 ms    [User: 19.6 ms, System: 3.4 ms]
  Range (min … max):    21.5 ms …  25.2 ms    117 runs
Copy code
PS C:\Users\John Sirois> hyperfine 'echo import this | py -3.12'
Benchmark 1: echo import this | py -3.12
  Time (mean ± σ):      34.4 ms ±   0.9 ms    [User: 3.7 ms, System: 4.5 ms]
  Range (min … max):    32.9 ms …  37.0 ms    72 runs
👀 1
c
huh. seems 10% is a conservative number, at least for this dummy scenario.
e
Yeah - it's actually worse. That was using the Windows 
py
launcher and 
pyenv
shims. If I use the 
python.exe
directly and the pyenv interpreter executable directly its more like 20ms Windows, 8ms Linux.
I think its well known though that Linux is king of low process spawn overhead; so the test is really not generally great here.
h
TIL that there are ads in the OS on Windows. I find that shocking but I guess not surprising.
I have a spare windows laptop I never use, I should spend some time in it.
f
hey, Gautham, I just tried to find something in Pants' docs (and actually that last comment by djuarez in the issue is from someone on our team 😅), didn't really attempt to implement anything
but it's seeming to me like there isn't much to do right now short of forking pants ourselves
c
you never need to fork pants for stuff like this, it would “only” be to write a custom plugin.
l
@fancy-policeman-6755 as Andreas points out, pants is designed so that users can write and incorporate plugins to get around problems like what you are encountering. All the built-in functionality that comes with pants is actually built from the same kind of plugins. What this will probably look like for you is adding some python code to your monorepo (an in-repo plugin). There is a learning curve to plugins, but we can help here with that, and there is good documentation.
If you are able to get around the shortcoming in the pants builtin docker support by running a script, there is a good chance that you'll be able to mold that into a plugin.
f
hey, sorry for bumping an old post - I just realized I never thanked you for the answer
fwiw for now we're rolling with adhoc runners to keep using docker, as isolated as possible - the overhead of writing and maintaining a plugin seems too much here (even if it's better than forking 😅)
but thanks in any case for the explanations!
Open in Slack
PreviousNext
Powered by