https://pantsbuild.org/ logo
Join Slack
Powered by
Hi all! Does anyone have recommendations for best ...
# general
f
Hi all! Does anyone have recommendations for best practices using pants to handle importing first party internal packages within Google Cloud Functions? I assume most people build the internal packages, push the build file(s) to Google Artifact Registry, and import the file from 
requirements.txt
in Cloud Function? Update: • Packaging: The answer is that if the packages are located in the same repository, then best practice is to let Pants handle the dependencies and skip Artifact Registry. This works best if all packages are in a single 
resolve
. • Local development: The only thing that worked for me was creating a custom shell script to: ◦ Update 
PYTHONPATH
to enable first party imports; and ◦ Create venvs using 
requirements.txt
for each 
resolve
▪︎ Note: Had to use standard python and pip to create venvs as had issues with the virtualenvs exported using Pants
b
Hello! Are the packages within the repository that publishes the GCFs too (i.e. managed by the same pants)? Or are they in a separate repository?
f
Thanks so much for the reply! Yes same repository. This is rough project structure if it helps.
Copy code
pants.toml
libs/
    lib1/
    lib2/
projects/
    cloud_function/
        src/
            main.py
    another_cloud_function/
        src/
            main.py
And 
cloud_function/src/main.py
has an import like
Copy code
from lib2 import function_2
And the main 
lib2
python file has an import like
Copy code
from lib1 import function_1
Assuming based on your question that we can skip Artifact Registry if in same repo?
b
Yeah, exactly. Pants can package all the relevant dependencies into the package automatically
Is it not working as you'd expect?
f
Wonderful, thanks! I found this article which described publishing packages to remote repos so wasn't sure what the best practice was. We started out using separate venvs for each project / lib which has made it harder to get up and running with Pants, but now that I know best practice is to let Pants handle all the dependencies will keep going on that route and try to consolidate our dependencies. Thank you!
b
ah yes, this works best if all the libs etc. are in a single "resolve" as it sounds like you've discovered. For instance for my work code base (which publishes a bunch of AWS Lambdas), we have one main resolve for all of our app code & tests, and then a few other small ones to manage some tools (i.e. helper scripts and CLI tools we run, that we don't ever import from our real Python code).
f
Excellent, really appreciate the info and advice!
Thanks again for the advice! I switched to using 2 resolves, and have been able to use pants to package the cloud function. Unfortunately I am still having problems testing and running a cloud function locally using pants. Would you be able to help me with the following questions? Question #1: How do you usually run a cloud function locally (for development purposes) using pants? Normally I would do something like
Copy code
$ functions-framework --source=src/main.py --target=example_endpoint --debug
but cannot find a good way to do this with pants What I tried: • Changing directory structure of cloud function • 
pants run
• 
run_shell_command
• Activating virtual env exported using pants, which works for 3rd party deps but not first party deps (e.g. 
lib1
) • Using 
pants package
to create the zip file, then extracting the files, cd'ing to directory, activating virtual env exported using pants, and running 
functions-framework
◦ This technically 'works' but would be a pain to do on a regular basis while developing Question #2: The only way I could get tests working for cloud function was to place 
test_main.py
next to 
main.py
. Is this necessary? We generally place tests in a separate tests directory like below
Copy code
cloud_function/
    src/
        main.py
    tests/
        unit/
            test_main.py
when using pants I tried adding this line to 
test_main.py
Copy code
from cloud_function.src.main import get_data
This is the error when using the tests/ directory structure: Command: 
pants test projects/cloud_function/tests/unit/test_main.py
Error:
Copy code
ModuleNotFoundError: No module named 'cloud_function'
b
1. I don't know anything about GCF and how to run things locally... but either of the last two sound along the right lines: a. for the venv, I think setting 
PYTHONPATH
based on the 
pants roots
output will allow Python to find the first-party modules: https://www.pantsbuild.org/2.20/docs/using-pants/setting-up-an-ide#first-party-sources b. for the package, there's a new 
layout="flat"
field coming in 2.22 (not yet released) that packages to just loose files, not a zip file, see the notes https://github.com/pantsbuild/pants/blob/main/docs/notes/2.22.x.md#python or docs https://www.pantsbuild.org/2.22/reference/targets/python_google_cloud_function#layout. This would slightly simplify 2. this one depends on your source root configuration, see 
pants roots
. A source root defines a location that contains libraries (i.e. what can be imported). If 
cloud_function/src
is a root, then 
main.py
would be expected to be loaded as just 
import main
. If you want the import path to be 
cloud_function.src.main
, you'll need to have the parent directory of 
cloud_function
be the source root instead. https://www.pantsbuild.org/2.20/docs/using-pants/key-concepts/source-roots
🙏🏻 1
f
Ok got it - thanks very much for your help! Will try those tips. Appreciate the quick response!
Thanks again for the tips! For anyone reading this thread, here are the fixes that allowed me to run and test cloud function locally: 1. Running locally a. Setting up 
.env
file with 
pants roots
does allow VS Code to trace references, but didn't help with running locally using 
functions-framework
i. Tried updating VS Code and reinstalling extensions etc, also found this thread which was somewhat helpful re: VS Code settings b. Ended up creating custom shell script to update 
PYTHONPATH
variable to include 
pants roots
so you can access first-party deps within any subdirectory of repo c. Tried at first using the virtualenvs created using 
pants export
but then couldn't import first-party deps using 
PYTHONPATH
config above d. Ended up creating custom shell script to create separate venvs and install packages from requirements.txt files, then in any directory you can just activate the exported venv 2. Testing a. Using 
import main
works!
3 Views
Open in Slack
PreviousNext
Powered by