I ve been beating my head against the wall for some time now Pants #general

I’ve been beating my head against the wall for som...

thankful-stone-5860

06/21/2023, 5:21 PM

I’ve been beating my head against the wall for some time now trying to understand how to get PEX to use my first-party

python_distribution

as a dependency. Currently, this is the part of my

BUILD

file that matters:

Copy code

python_distribution(
    name="agent_pool_tf_cdk",
    provides=python_artifact(
        name="agent_pool_tf_cdk",
        version="0.0.0"
    ),
    dependencies=[":src"]
)

pex_binary(
    name="main",
    dependencies=[":agent_pool_tf_cdk", ":requirements"],
    entry_point="src/agent_pool_tf_cdk/main.py",
    include_sources=False,
)

After generating

main.pex

I run

tar -tvf dist/main.pex

and see it listed in `./deps`:

Copy code

RIEBEC2-C02G537QMD6P:AgentPoolsTFCDK riebec2$ tar -tvf dist/main.pex | grep agent_pool_tf_cdk
drwxr-xr-x  0 0      0           0 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/
-rw-r--r--  0 0      0          86 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/.layout.json
-rw-r--r--  0 0      0         217 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/__init__.py
-rw-r--r--  0 0      0         749 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/backend_shim.py
-rw-r--r--  0 0      0        6153 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/main.py
drwxr-xr-x  0 0      0           0 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/clients/
-rw-r--r--  0 0      0           0 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/clients/__init__.py
-rw-r--r--  0 0      0       12654 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/clients/aws.py
-rw-r--r--  0 0      0        2638 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/clients/azure.py
drwxr-xr-x  0 0      0           0 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/stacks/
-rw-r--r--  0 0      0           0 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/stacks/__init__.py
-rw-r--r--  0 0      0       11614 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/stacks/pool.py
drwxr-xr-x  0 0      0           0 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/stacks/resources/
-rw-r--r--  0 0      0           0 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/stacks/resources/__init__.py
-rw-r--r--  0 0      0        2379 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/stacks/resources/base.py
-rw-r--r--  0 0      0        4542 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/stacks/resources/config.py
-rw-r--r--  0 0      0        6043 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/stacks/resources/iam.py
-rw-r--r--  0 0      0        6950 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/stacks/resources/kms.py
-rw-r--r--  0 0      0        7653 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/stacks/resources/security_groups.py
drwxr-xr-x  0 0      0           0 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/models/
-rw-r--r--  0 0      0           0 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/models/__init__.py
-rw-r--r--  0 0      0        5355 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/models/aws.py
-rw-r--r--  0 0      0         696 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/models/azure.py
-rw-r--r--  0 0      0        1156 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/models/terraform.py
drwxr-xr-x  0 0      0           0 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/policies/
-rw-r--r--  0 0      0         830 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/policies/__init__.py
-rw-r--r--  0 0      0        5503 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/policies/factories.py
-rw-r--r--  0 0      0        2274 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/policies/managed.py
drwxr-xr-x  0 0      0           0 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/agent_pool_tf_cdk-0.0.0.dist-info/
-rw-r--r--  0 0      0           4 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/agent_pool_tf_cdk-0.0.0.dist-info/INSTALLER
-rw-r--r--  0 0      0         390 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/agent_pool_tf_cdk-0.0.0.dist-info/METADATA
-rw-r--r--  0 0      0           0 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/agent_pool_tf_cdk-0.0.0.dist-info/REQUESTED
-rw-r--r--  0 0      0          92 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/agent_pool_tf_cdk-0.0.0.dist-info/WHEEL
-rw-r--r--  0 0      0           1 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/agent_pool_tf_cdk-0.0.0.dist-info/namespace_packages.txt
-rw-r--r--  0 0      0          32 Jan  1  1980 .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/agent_pool_tf_cdk-0.0.0.dist-info/top_level.txt

But when I run

main.pex

, it acts like it’s not in the dependencies:

Copy code

RIEBEC2-C02G537QMD6P:AgentPoolsTFCDK riebec2$ ./dist/main.pex 
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/riebec2/.pex/unzipped_pexes/9dd5f19bb7d778f51087f46602c05f335c96a59e/__main__.py", line 106, in <module>
    bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
  File "/Users/riebec2/.pex/unzipped_pexes/9dd5f19bb7d778f51087f46602c05f335c96a59e/.bootstrap/pex/pex_bootstrapper.py", line 618, in bootstrap_pex
    pex.PEX(entry_point).execute()
  File "/Users/riebec2/.pex/unzipped_pexes/9dd5f19bb7d778f51087f46602c05f335c96a59e/.bootstrap/pex/pex.py", line 560, in execute
    sys.exit(self._wrap_coverage(self._wrap_profiling, self._execute))
  File "/Users/riebec2/.pex/unzipped_pexes/9dd5f19bb7d778f51087f46602c05f335c96a59e/.bootstrap/pex/pex.py", line 467, in _wrap_coverage
    return runner(*args)
  File "/Users/riebec2/.pex/unzipped_pexes/9dd5f19bb7d778f51087f46602c05f335c96a59e/.bootstrap/pex/pex.py", line 498, in _wrap_profiling
    return runner(*args)
  File "/Users/riebec2/.pex/unzipped_pexes/9dd5f19bb7d778f51087f46602c05f335c96a59e/.bootstrap/pex/pex.py", line 603, in _execute
    return self.execute_entry(
  File "/Users/riebec2/.pex/unzipped_pexes/9dd5f19bb7d778f51087f46602c05f335c96a59e/.bootstrap/pex/pex.py", line 800, in execute_entry
    return self.execute_module(entry_point.module)
  File "/Users/riebec2/.pex/unzipped_pexes/9dd5f19bb7d778f51087f46602c05f335c96a59e/.bootstrap/pex/pex.py", line 808, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/riebec2/.pex/installed_wheels/a7921f9a7c339548325ec2d4b4d4b68b3a2f7b3e1db0ebd6e000a348a87697f0/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/main.py", line 13, in <module>
    from agent_pool_tf_cdk.clients.aws import TFCDKAutoscalingClient, TFCDKKMSClient
ModuleNotFoundError: No module named 'agent_pool_tf_cdk'

Any help would be GREATLY appreciates.

✅ 1

bitter-ability-32190

06/21/2023, 5:27 PM

Is there any chance you're hitting https://github.com/pantsbuild/pex/issues/2160?

thankful-stone-5860

06/21/2023, 6:16 PM

The output of

tar -Oxf dist/main.pex .deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/agent_pool_tf_cdk-0.0.0.dist-info/top_level.txt

is:

Copy code

clients
models
policies
stacks

Which are submodules of

agent_pool_tf_cdk

bitter-ability-32190

06/21/2023, 6:17 PM

I'd say see if wiping the pex cache fixes it.

rm -rf ~/.pex

then run the pex.

bitter-ability-32190

06/21/2023, 6:17 PM

If it does, may be the same issue

thankful-stone-5860

06/21/2023, 6:18 PM

I’ll give it a try

thankful-stone-5860

06/21/2023, 6:20 PM

Still the same issue. I would just expect that PEX would be able to resolve the top-level package of

agent_pool_tf_cdk

, considering it exists in

.deps/

with all other dependencies and it can import the 3rd party dependencies just fine.

refined-addition-53644

06/21/2023, 6:20 PM

On a different note, why dependency on python distribution? You just want to run the PEX with a given entry point, right? Then just providing the entry point should be enough without any python distribution

bitter-ability-32190

06/21/2023, 6:21 PM

The linked issue had it so everything looked right in the PEX, but the cache was poisoned. So just looking at the PEX it wasn't clear why it'd fail. If wiping the cache exhibits the same issue, you likely aren't hitting this then

👍 1

thankful-stone-5860

06/21/2023, 6:23 PM

I want to stop with the ugly:

Copy code

try:
    from first_party_package.submodule import MyClass
except (ImportError, ModuleNotFoundError): 
    from submodule import MyClass

👍 1

thankful-stone-5860

06/21/2023, 6:31 PM

In the

PEX-INFO

file, my first-party package isn’t listed despite being included:

Copy code

{
  "bootstrap_hash": "5be3b46f7d6634cba8393be7fbfd89c2df7b49f8",
  "build_properties": {
    "pex_version": "2.1.134"
  },
  "code_hash": "fd6c27fae463474293c3e6256617d6bf04bff741",
  "distributions": {
    "agent_pool_tf_cdk-0.0.0-py3-none-any.whl": "aa0887862af2e963c2cba880838b1123cabc7ebca52d1cc211a4167a926c7ad1",
    "attrs-23.1.0-py3-none-any.whl": "6c3861638ec4fd267653f203e53754b70438c2d0696a8955f9eb6345b1c3bb48",
    "aws_cdk.asset_awscli_v1-2.2.198-py3-none-any.whl": "ad7a930584dadce6e47d52dab90dc8e559252397ec6bd07798513748cc1d19d8",
    "aws_cdk.asset_kubectl_v20-2.1.2-py3-none-any.whl": "9dc28fea723b4c9750537e9042144d923542f7a3d8a85aa1d2d3c51747bd27e3",
    "aws_cdk.asset_node_proxy_agent_v5-2.0.165-py3-none-any.whl": "bf9d5b25b396b0d61cdc8a7959a94caf939bafb07ae5dc11dae9af58b8f67696",
    "aws_cdk_lib-2.72.1-py3-none-any.whl": "1c5ad7e3aed57be9bb9c4ad0f65e8bf394313700e77855ab48341074629ca8c5",
    "boto3-1.26.104-py3-none-any.whl": "bb7f7d5a909c0f3617b91bde9ee9d92429eb64e4b688aebec0be79008fae8421",
    "botocore-1.29.104-py3-none-any.whl": "08cafdfe0e7f5c7d44303a05ac2f47abf5e27b4f0b82d2b5077f51492b3914f5",
    "cattrs-23.1.2-py3-none-any.whl": "1f38676b2c7026e18e5ce5b6b93f2767a7dbe07e304afebf18a82f0422e246f4",
    "cdktf-0.15.5-py3-none-any.whl": "61baed89b8cc5374cb3588d3cb670c4d8e2c0e97e605d0b77219dda6fdacc3b1",
    "cdktf_cdktf_provider_aws-12.0.12-py3-none-any.whl": "f566af73388052b499d5b0f626d5431f0402dbc551b8208f9217bb03e0d719a1",
    "certifi-2023.5.7-py3-none-any.whl": "1def630d7f0bfeca0309eaaaeaf407d9344290c437cbf34d3d2aa65a4f1ebadb",
    "charset_normalizer-3.1.0-cp310-cp310-macosx_10_9_x86_64.whl": "10661217bbb5878a2050ce57c47146307e41dc3f18e3dae996d2c79ca63b38b2",
    "charset_normalizer-3.1.0-cp39-cp39-macosx_10_9_x86_64.whl": "0dfe9cc3dd42f43f24fee1e582c1a08c061aa8baf95b30cad0e818b1aebf0195",
    "constructs-10.1.300-py3-none-any.whl": "e922917413a5a98f8348f2a0a8d60bcab0c1651173cad1449ad30e5b02274db4",
    "exceptiongroup-1.1.1-py3-none-any.whl": "730cde1f0b1c13bf7d7ff13eddda044ace6bc766763fe1f717a82c1e4a00dbe2",
    "idna-3.4-py3-none-any.whl": "3baea0ffbdea9f349783bacbdb82dbd50eadfec75339388aaf1597754786e3ed",
    "importlib_resources-5.12.0-py3-none-any.whl": "610b1dd139a7ea480762911aff3ee36099e5479579d03ed6032cbea2f3f26562",
    "jmespath-1.0.1-py3-none-any.whl": "299c3a18595a39d4d54252e98a3c39343899b9bd1997e3eebfee63ea5757588b",
    "jsii-1.84.0-py3-none-any.whl": "3be29f27594af1877fb555b0321cc8e480489e06a4aeb70fe8fe191bbda4c571",
    "publication-0.0.3-py2.py3-none-any.whl": "c7afebed04446b72c25fc07f16467fd3d27c4de557d6389c8e88105b3cc2bb0b",
    "pydantic-1.10.7-cp310-cp310-macosx_10_9_x86_64.whl": "558e3c84f1fbb0bedba6d105df1b61fd86723493dc07c5d1c52792eb9dbe975f",
    "pydantic-1.10.7-cp39-cp39-macosx_10_9_x86_64.whl": "720a43f71d726f28313682b65cf66be49d7268403e02ce448cbd756deb67fee6",
    "python_dateutil-2.8.2-py2.py3-none-any.whl": "202c27a293331dd8fa9d41d1fcdd5ba4f4d6d2de0a2f00fa8547adc7c1aac629",
    "requests-2.28.2-py3-none-any.whl": "bd5fbf4ad4e1d61ce4de27aeb31e0ba74111cd5869ffb0ad8f6e0caa99719d4d",
    "s3transfer-0.6.1-py3-none-any.whl": "c788ced49dbf90640e1bf2504c9760f7cc920d7ac3b980869c9c21c08da824e4",
    "six-1.16.0-py2.py3-none-any.whl": "3e1c439c88d2e7681372427bab751b3fc99969891e95a714fed9604bf7710213",
    "typeguard-2.13.3-py3-none-any.whl": "0fac1bf1ab1a9b1275399faf6b21f93ae0be4a98980529a60cec9e412160f9bd",
    "typing_extensions-4.6.3-py3-none-any.whl": "5cef856ffc1ffec55ad71470364dfb634f305ce233d10ea0713152ec2c777d46",
    "urllib3-1.26.16-py2.py3-none-any.whl": "55ed1adea278529a458f440b5ed6ce3b7be6dac8041ee06ba2c06e83821a33d9",
    "zipp-3.15.0-py3-none-any.whl": "80fb844a41c3fd9bc4ba67f765b6340a84aeca5e699f681d01e66a94bfaddbed"
  },
  "emit_warnings": false,
  "entry_point": "main",
  "ignore_errors": false,
  "includes_tools": false,
  "inherit_path": "false",
  "inject_args": [],
  "inject_env": {},
  "interpreter_constraints": [
    "CPython<4,>=3.8"
  ],
  "pex_hash": "14c6269d5d6fc15db9c69093142692104efd04ed",
  "pex_path": "",
  "pex_paths": [],
  "requirements": [
    "agent-pool-tf-cdk==0.0.0",
    "aws-cdk-lib==2.72.1",
    "boto3==1.26.104",
    "botocore==1.29.104",
    "cdktf-cdktf-provider-aws==12.0.12",
    "cdktf==0.15.5",
    "constructs==10.1.300",
    "pydantic==1.10.7",
    "requests==2.28.2"
  ],
  "strip_pex_env": true,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false,
  "venv_hermetic_scripts": true,
  "venv_site_packages_copies": false
}

bitter-ability-32190

06/21/2023, 6:32 PM

Oh duh, that isn't a valid module name in the installed venv

bitter-ability-32190

06/21/2023, 6:33 PM

agent_pool_tf_cdk

is the package name and the wheel name, but looks like the code inside the wheel starts at the top-level

bitter-ability-32190

06/21/2023, 6:33 PM

For this to work, you'd see something like:

.deps/agent_pool_tf_cdk-0.0.0-py3-none-any.whl/agent_pool_tf_cdk/__init__.py

in the unpacked pex

thankful-stone-5860

06/21/2023, 7:07 PM

So, it appears that

pants

is stripping the

src/agent_pool_tf_cdk/

from the source files. Is there any way to add a prefix so it can import correctly?

thankful-stone-5860

06/21/2023, 7:08 PM

Normally, this would be done in the

packages=

part of the distribution, but you can’t specify that when using pants.

bitter-ability-32190

06/21/2023, 7:08 PM

I couldn't say. I only use

python_distribution

for extension modules

thankful-stone-5860

06/21/2023, 7:27 PM

I found the problem!!!

thankful-stone-5860

06/21/2023, 7:28 PM

The issue was how I configured my

root_patterns

pants.toml

. I had

/src/*

and changing it to

src

fixed it.

Open in Slack

Previous Next