Hi there, I need help with a dependency mapping pr...
# general
Hi there, I need help with a dependency mapping problem. I am building a PEX that requires a dependency "airflow-common" which is listed in requirements.txt. "airflow_common" imports "airflow" as its dependency. Since the module name for airflow is "apache-airflow", I added a mapping "module_mapping={"apache-airflow": ["airflow"]," in the BUILD script at the source root. However, when I build my PEX file and runs it, I still see the error " File "/Users/mingshiwang/code/pynest/dist/libs.src.python.airflow_cli/airflow_cli.pex/airflow_cli/main.py", line 15, in <module> File "/Users/mingshiwang/code/pynest/dist/libs.src.python.airflow_cli/airflow_cli.pex/airflow_cli/main.py", line 12, in main File "/Users/mingshiwang/.pex/installed_wheels/4edcadb09e1093c6ef2fedd4cfc4d40e68f835e2/airflow_common-35.10.3-py3-none-any.whl/airflow_common/cli.py", line 55, in main from airflow.bin.cli import CLIFactory ModuleNotFoundError: No module named 'airflow'". Does anyone know how to resolve this? Thanks!
Does airflow_common express it's dependency on airflow? Sometimes deps don't do this properly. You'd check it's setup.py or pyproject.toml
Yeah that's the problem:
Copy code
$ unzip -qc airflow_commons-0.0.17-py3-none-any.whl airflow_commons-0.0.17.dist-info/METADATA | grep Requires
Requires-Dist: pytz (>=2018.4)
Requires-Dist: datetime
Requires-Dist: google.cloud
Requires-Dist: pandas
Requires-Dist: sqlalchemy
Requires-Dist: boto3
Requires-Dist: botocore
Requires-Dist: pyyaml
Requires-Dist: s3fs
Requires-Dist: s3transfer
Requires-Dist: pyarrow
@powerful-florist-1807 since airflow-commons does not declare a dependency on apache-airflow, you'll need to add a BUILD file dependencies entry manually.
Even better, do this thread's recommendation :) https://pantsbuild.slack.com/archives/C046T6T9U/p1620893638017200
Thanks @enough-analyst-54434 @hundreds-father-404! Sorry that I haven't been very clear. "airflow-common" is an in-house module built by my company (not the public one) in a very old repo that doesn't use pant. It is basically a wrapper of "apache-airflow".
👍 1
Aha. Can you share that distribution's METADATA file or PKG-INFO file?
Sure. this is the metadata
As Eric pointed out - this is likely a transitive dependency issue
My PEX -> (depends on) -> airflow-common -> (depends on) ->apache-airflow (import name is "airflow")
Problem is that there is no module mapping inside "airflow-common" because it is not published by pants
Ah, ok. To be clear Pants is not at all needed to publish a distribution with properly declared dependencies. That internal project just needs to update its
to include its dependencies depending on how it publishes itself.
👍 2