curved-manchester-66006
11/04/2022, 3:00 PMrequirements.txt
file. I think I can gather my python files together by having pants make a tarball or sdist. But I'm not sure how to generate a requirements.txt
style file to go along with them. (I know about /pants dependencies
but now how to make it output in requirements.txt style nor include it with the tarball/sdist)happy-kitchen-89482
11/04/2022, 3:10 PMcurved-manchester-66006
11/04/2022, 3:35 PMhappy-kitchen-89482
11/04/2022, 3:36 PMcurved-manchester-66006
11/04/2022, 3:37 PMaws s3 cp dag_def.py <s3://YOUR_S3_BUCKET_NAME/dags/>
If I had a tarball, unpackng it and uploading the .py files is trivial.happy-kitchen-89482
11/04/2022, 3:37 PMcurved-manchester-66006
11/04/2022, 3:38 PM- aws s3 cp modules/ s3://$S3_BUCKET/dags/modules/ --recursive
- aws s3 cp custom_dags/ s3://$S3_BUCKET/dags/custom_dags/ --recursive
and that dag_def.py references code in requirements.txt?Correct, the requirements file has
pandas==1.2.4
and then in the example dag_def.py code can import pandas
happy-kitchen-89482
11/04/2022, 3:41 PMs3://$S3_BUCKET/requirements.txt
it looks likecurved-manchester-66006
11/04/2022, 3:49 PMnumpy
listed. It doesn't install *
from the zipfile. That would be reasonable to glue together with a script.
I don't suppose there is a pants goal that will give me a directory filled with wheels? ;-)bitter-ability-32190
11/04/2022, 4:30 PMpex
also has directory layouts, so not just tarballs)curved-manchester-66006
11/04/2022, 6:09 PMLooking at the requirements a little more, a followup is if there a pants equivalent to pip's constraints filehttps://www.pantsbuild.org/docs/python-third-party-dependencies#constraints-files I think I found the answer for that one.
happy-kitchen-89482
11/04/2022, 6:11 PMcurved-manchester-66006
11/04/2022, 6:18 PMI think that lets you point to a constraints file, but you want Pants to generate one?I think in this case the constraints are more a feature of the deploy environment -- kind of like tomcat in java-land -- and not something Pants need to infer or stitch together from multiple sources. Just having pants respect them would be enough.
Pants has all the ingredients, just not the exact recipe for putting them togetherYeah this feels really close! With either bash or a plugin. At least to start with bash, What is the least hacky goal for getting "all the transitive wheels needed by target X"? Is unzipping the PEX like I suggested actually the right path?
happy-kitchen-89482
11/05/2022, 3:55 PMlayout="zipapp"
(the default) or layout="loose"
PEX contains the wheels in their installed, loose-file statelayout="packed"
PEX contains the installed-state wheels zipped back up as .whl
files, which you may be able to use for this, even though they are not identical to the pypi wheels (e.g., they contain an INSTALLED file)curved-manchester-66006
12/20/2022, 2:33 PMhappy-kitchen-89482
12/20/2022, 3:22 PM