https://pantsbuild.org/ logo
Join Slack
Powered by
Hey all — curious what targets or patterns exist f...
# general
g
Hey all — curious what targets or patterns exist for pulling external artifacts that require auth? I have a use case where I want to pull a static .tar.gz (HTML + assets) from a private S3 bucket and include it in a docker_image target. I could maybe hack this with an adhoc_tool and file targets, then add the file as a dep, but this breaks down when folks build locally and don’t have S3 access. Has anyone solved something similar or have suggestions? Thanks! p.s. it would be fine to have a single empty index.html for local builds.
f
and a 
file
target
✅ 1
g
What version of pants was the s3-url-handler released in? I'm on 2.24.
It looks like 2.27?
f
I don't recall. You can just change the docs version in the drop-down at the top.
g
2.25
cool
thanks
👍 1
f
It exists prior to 2.25, but I added support for KMS encrypted buckets around 2.25 iirc
✅ 1
g
@fresh-mechanic-68429 what is the pattern for this to work with auth? I have AWS_PROFILE set and auth is failing. Not sure what the magic sauce is.
f
2.26 has some fixes for that, can you try that version or later? https://github.com/pantsbuild/pants/pull/22008
g
I upgraded to 2.26.0 last night so that's the version I'm using.
f
you also need to enable the handler in the pants.toml
Copy code
backend_packages = [
    ...
    "pants.backend.url_handlers.s3",
]
g
oh son of a... you're right 😉
dur
@fresh-mechanic-68429
Copy code
File "/Users/jasonbarnett/Library/Caches/nce/8fb836cadd1e1f50180fea66b6acab1f23108e9a74f158744e706c6da41a534b/bindings/venvs/2.26.0/lib/python3.11/site-packages/pants/engine/internals/selectors.py", line 113, in __await__
    result = yield self
             ^^^^^^^^^^
  File "/Users/jasonbarnett/Library/Caches/nce/8fb836cadd1e1f50180fea66b6acab1f23108e9a74f158744e706c6da41a534b/bindings/venvs/2.26.0/lib/python3.11/site-packages/pants/backend/url_handlers/s3/register.py", line 38, in access_aws_credentials
    from botocore import credentials
ModuleNotFoundError: No module named 'botocore'
@fresh-mechanic-68429 Is it necessary to add botocore somewhere or is this a bug?
f
You can add it to 
[GLOBAL].plugins
✅ 1
Less of a bug and more of a point of ongoing technical discussion. When should a dependency of an included backend be in the core Pants lockfile?
Probably a bug because it should be included in a 
requirements.txt
in the backend itself so Pants installs the dependency when the backend is enabled.
And we should document (but I recall don't currently) that 
botocore
needs to be added to 
[GLOBAL].plugins
.
g
as in...?:
Copy code
# pants.toml
[GLOBAL]
plugins = ["botocore"]
f
yeah
✅ 1
g
That worked. Thank you folks.
f
hilarious:
# In src/python/pants/backend/url_handlers/s3/register.py we advise adding
# 
botocore
to 
[GLOBAL].plugins
, but 
botocore
is particular about the
# version of urllib3 used. To make the s3 handler easier to use, constrain
# urlib3 here. Per the voluminous thread at
# https://github.com/boto/botocore/issues/2926 this can likely be relaxed when
# Pants itself is on a newer version of Python
urllib3<2
so we documented in a source comment ...
😄 1
g
Presumably now that pants is on python 3.11 this is possible?
f
Probably? Someone would need to eventually look into it.
👍 1
4 Views
Open in Slack
PreviousNext
Powered by