https://pantsbuild.org/ logo
#general
Title
# general
m

many-agent-62725

03/08/2021, 5:20 PM
Hello! I am having some troubles with the
files
target in Pants v2.X I have the following files (see screenshot), as well as the following BUILD file:
Copy code
python_library(
    name='test-handler',
    sources=[
        'lambda_handler/test.py',
        ':test-queries'
    ],
    dependencies=[],
)

python_awslambda(
    name='test-lambda',
    dependencies=[
        ':test-handler',
    ],
    runtime='python3.8',
    handler='test.lambda_handler.test:handler',
    output_path='lambda/test/test.zip'
)

pex_binary(
    name='test-pex',
    dependencies=[
        ':test-handler',
    ],
    entry_point='test.lambda_handler.test:handler',
    output_path='lambda/test/test-pex.zip'
)

files(
    name='test-queries',
    sources=[
        'queries/test.sql'
    ]
)
The objective is to include some SQL files in an AWS lambda target. However, when I unzip the final artefacts, the SQL files are not included (as shown by the screenshot). We started having this issue since the migration from 1.30 to 2.0. How can I include files in an AWS lambda target?
h

hundreds-father-404

03/08/2021, 5:28 PM
Hi @many-agent-62725! You're hitting a problem that another user experienced last month and we haven't figured out a great answer for.
files()
targets currently are not included when creating a
pex_binary
and
python_awslambda
because loading the files usually does not work how you would expect: https://github.com/pantsbuild/pants/pull/11551 It's not totally clear what we should do, but at a minimum, I think we should warn in this case For your specific use case, could you use
resources
instead of
files
? Those do work great with
pex_binary
and
python_awslambda
https://www.pantsbuild.org/docs/resources
m

many-agent-62725

03/09/2021, 12:08 PM
Ohhhh, I see! Yup, if I use resources it starts working again, thanks 🙏
❤️ 1
In that case, I have some questions: in what cases should
file
be used? When using
archive
?
loading the files usually does not work how you would expect
Could you elaborate a little bit more about this, please? Because previously we were using
files
, and the
open
call was working as expected 😅
Also, is this behaviour documented? I’m going to be honest, this was driving me crazy as I was not finding anything in the docs regarding this 😅
h

hundreds-father-404

03/09/2021, 8:45 PM
Could you elaborate a little bit more about this, please?
See https://github.com/pantsbuild/pants/pull/11551#issuecomment-777926798. Because we strip source roots (like
src/py
) when packaging a PEX/AWS Lambda, using
__file__
behaves inconsistently depending on
./pants run
vs.
./pants package
(were we to allow files to be included in
./pants package
)
in what cases should file be used? When using archive?
Yeah, I think
files()
should only be for the
test
goal and the
archive
target type as far as I can tell
is this behaviour documented?
The part about not including
files
is not documented, which is our bad. I started to address it last month with that PR #11511 to allow using
files
with
pex_binary
, but it stalled because there are lots of gotchas and it's not clear we should actually do it But, for all Pants releases before the current dev release of 2.4, we're not going to backport this change, so I'll go ahead and tighten up the documentation now
using file behaves inconsistently depending
And you need to use
__file__
because
open()
opens relative to the cwd. The actual Pex process runs in some folder like
~/.cache/pex
iirc, and you need to do something like
open(os.path.join(__file__,  '../file.json')
to look in the correct place But then that gets into the above issues with
__file__
changing it's meaning between
./pants run
and
./pants package
🤷‍♂️
@happy-kitchen-89482 can you please double check these new docs warnings I've made about
files()
not working with
pex_binary
? * https://www.pantsbuild.org/v2.0/docs/resources#files * https://www.pantsbuild.org/v2.0/docs/python-package-goal#creating-a-pex-file-from-a-pex_binary-target * https://www.pantsbuild.org/v2.0/docs/awslambda-python#step-2-define-a-python_awslambda-target I may revive https://github.com/pantsbuild/pants/pull/11551 to add a runtime warning when depending on
files
@many-agent-62725 in addition to the docs changes, https://github.com/pantsbuild/pants/pull/11659 adds a runtime warning. Feedback welcomed on the docs + the wording of that warning! Hopefully that avoids future users having the same confusion/frustration you had this week
❤️ 1
m

many-agent-62725

03/10/2021, 5:05 PM
Because we strip source roots (like 
src/py
) when packaging a PEX/AWS Lambda
Ahhh, OK, now I get it 😄
And thank you so much for updating the docs and adding the warning 🙏
❤️ 1
4 Views