Hi there, I need help with a dependency mapping pr...
# generalp
powerful-florist-1807
05/15/2021, 2:55 AMHi there, I need help with a dependency mapping problem. I am building a PEX that requires a dependency "airflow-common" which is listed in requirements.txt. "airflow_common" imports "airflow" as its dependency. Since the module name for airflow is "apache-airflow", I added a mapping "module_mapping={"apache-airflow": ["airflow"]," in the BUILD script at the source root. However, when I build my PEX file and runs it, I still see the error " File "/Users/mingshiwang/code/pynest/dist/libs.src.python.airflow_cli/airflow_cli.pex/airflow_cli/main.py", line 15, in <module>
File "/Users/mingshiwang/code/pynest/dist/libs.src.python.airflow_cli/airflow_cli.pex/airflow_cli/main.py", line 12, in main
File "/Users/mingshiwang/.pex/installed_wheels/4edcadb09e1093c6ef2fedd4cfc4d40e68f835e2/airflow_common-35.10.3-py3-none-any.whl/airflow_common/cli.py", line 55, in main
from airflow.bin.cli import CLIFactory
ModuleNotFoundError: No module named 'airflow'". Does anyone know how to resolve this? Thanks!
h
hundreds-father-404
05/15/2021, 2:59 AMDoes airflow_common express it's dependency on airflow? Sometimes deps don't do this properly. You'd check it's setup.py or pyproject.toml
e
enough-analyst-54434
05/15/2021, 3:46 AMYeah that's the problem:
Copy code
$ unzip -qc airflow_commons-0.0.17-py3-none-any.whl airflow_commons-0.0.17.dist-info/METADATA | grep Requires
Requires-Dist: pytz (>=2018.4)
Requires-Dist: datetime
Requires-Dist: google.cloud
Requires-Dist: pandas
Requires-Dist: sqlalchemy
Requires-Dist: boto3
Requires-Dist: botocore
Requires-Dist: pyyaml
Requires-Dist: s3fs
Requires-Dist: s3transfer
Requires-Dist: pyarrowenough-analyst-54434
05/15/2021, 3:47 AM@powerful-florist-1807 since airflow-commons does not declare a dependency on apache-airflow, you'll need to add a BUILD file dependencies entry manually.
h
hundreds-father-404
05/15/2021, 3:50 AMEven better, do this thread's recommendation :)
https://pantsbuild.slack.com/archives/C046T6T9U/p1620893638017200
p
powerful-florist-1807
05/15/2021, 4:22 AMThanks @enough-analyst-54434 @hundreds-father-404! Sorry that I haven't been very clear. "airflow-common" is an in-house module built by my company (not the public one) in a very old repo that doesn't use pant. It is basically a wrapper of "apache-airflow".
👍 1
e
enough-analyst-54434
05/15/2021, 4:28 AMAha. Can you share that distribution's METADATA file or PKG-INFO file?
p
powerful-florist-1807
05/15/2021, 4:35 AMSure. this is the metadata
powerful-florist-1807
05/15/2021, 4:46 AMAs Eric pointed out - this is likely a transitive dependency issue
powerful-florist-1807
05/15/2021, 4:47 AMMy PEX -> (depends on) -> airflow-common -> (depends on) ->apache-airflow (import name is "airflow")
powerful-florist-1807
05/15/2021, 4:49 AMProblem is that there is no module mapping inside "airflow-common" because it is not published by pants
e
enough-analyst-54434
05/15/2021, 3:34 PMAh, ok. To be clear Pants is not at all needed to publish a distribution with properly declared dependencies. That internal project just needs to update its
setup.pysetup.cfgpytproject.toml👍 2
4 Views