Hi! I have not used pants in a while and not sure how to formulate the vocabulary exactly. I'm working with a managed service (AWS Managed Airflow) that expects me to upload some python files (conceptually a tarball) and a

requirements.txt

file. I think I can gather my python files together by having pants make a tarball or sdist. But I'm not sure how to generate a

requirements.txt

style file to go along with them. (I know about

/pants dependencies

but now how to make it output in requirements.txt style nor include it with the tarball/sdist)

Hi, interesting!

Yeah its a little quirky: https://docs.aws.amazon.com/mwaa/latest/userguide/working-dags-dependencies.html Looking at the requirements a little more, a followup is if there a pants equivalent to pip's constraints file (the docs site returns many different results for "constraints"): https://pip.pypa.io/en/stable/user_guide/#constraints-files With the managed service, we are deploying into a system that looks like this already: https://raw.githubusercontent.com/apache/airflow/constraints-2.2.2/constraints-3.7.txt

I was reading that guide, it talks about uploading a requirements.txt to S3 and then pointing to it, but I couldn’t see any description of where your own python code goes, i.e., the tarball you mentioned

Sorry I was trying to simply a bit, you upload the python files as individual S3 objects. Something like:

aws s3 cp dag_def.py <s3://YOUR_S3_BUCKET_NAME/dags/>

If I had a tarball, unpackng it and uploading the .py files is trivial.

“Uploading DAG code to Amazon S3” ?

CI today looks something like:

- aws s3 cp modules/ s3://$S3_BUCKET/dags/modules/ --recursive
    - aws s3 cp custom_dags/ s3://$S3_BUCKET/dags/custom_dags/ --recursive

And the requirements.txt goes to

s3://$S3_BUCKET/requirements.txt

it looks like

I think if I'm reading that section of the docs right it can install from a wheel in a zip file, but you still have to have a requirements.txt file to tell it what to install. eg in the example they still have

numpy

listed. It doesn't install

*

from the zipfile. That would be reasonable to glue together with a script. I don't suppose there is a pants goal that will give me a directory filled with wheels? ;-)

pex

also has directory layouts, so not just tarballs)

Looking at the requirements a little more, a followup is if there a pants equivalent to pip's constraints file

https://www.pantsbuild.org/docs/python-third-party-dependencies#constraints-files I think I found the answer for that one.

I think that lets you point to a constraints file, but you want Pants to generate one?

I think that lets you point to a constraints file, but you want Pants to generate one?

I think in this case the constraints are more a feature of the deploy environment -- kind of like tomcat in java-land -- and not something Pants need to infer or stitch together from multiple sources. Just having pants respect them would be enough.

Pants has all the ingredients, just not the exact recipe for putting them together

Yeah this feels really close! With either bash or a plugin. At least to start with bash, What is the least hacky goal for getting "all the transitive wheels needed by target X"? Is unzipping the PEX like I suggested actually the right path?

a

layout="zipapp"

(the default) or

layout="loose"

PEX contains the wheels in their installed, loose-file state

I'm not sure I'm going to be able to get to this before the slack expiring history cliff. Would it be okay to excerpt some of this to the github discussions page?

Absolutely! That is a great idea