Hello 👋 I am trying to install https://pypi.org/project/PyICU/ for use in my project, we are using pants and when the pex environment builds on my arm64 MacOS machine and tries to run tests, I get the following error:

___________ ERROR collecting src/lumos_expression/_evaluator_test.py ___________
ImportError while importing test module 'backend/python/libs/lumos_expression/src/lumos_expression/_evaluator_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/florian/.pyenv/versions/3.10.6/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
/Users/florian/.cache/pants/named_caches/pex_root/venvs/1/s/1af2c70d/venv/lib/python3.10/site-packages/ddtrace/internal/module.py:309: in _exec_module
    self.loader.exec_module(module)
backend/python/libs/lumos_expression/src/lumos_expression/__init__.py:2: in <module>
    from ._evaluator import ExpressionEvaluation, ExpressionInput, LumosExpressionClient
/Users/florian/.cache/pants/named_caches/pex_root/venvs/1/s/1af2c70d/venv/lib/python3.10/site-packages/ddtrace/internal/module.py:309: in _exec_module
    self.loader.exec_module(module)
backend/python/libs/lumos_expression/src/lumos_expression/_evaluator.py:10: in <module>
    from ._environment import DEFAULT_TRANSLITERATOR_IDENTIFIER, ENVIRONMENT_SINGLETON, transliterate
/Users/florian/.cache/pants/named_caches/pex_root/venvs/1/s/1af2c70d/venv/lib/python3.10/site-packages/ddtrace/internal/module.py:309: in _exec_module
    self.loader.exec_module(module)
backend/python/libs/lumos_expression/src/lumos_expression/_environment.py:1: in <module>
    import icu
/Users/florian/.cache/pants/named_caches/pex_root/venvs/1/s/1af2c70d/venv/lib/python3.10/site-packages/ddtrace/internal/module.py:309: in _exec_module
    self.loader.exec_module(module)
/Users/florian/.cache/pants/named_caches/pex_root/venvs/1/s/1af2c70d/venv/lib/python3.10/site-packages/icu/__init__.py:37: in <module>
    from ._icu_ import *
/Users/florian/.cache/pants/named_caches/pex_root/venvs/1/s/1af2c70d/venv/lib/python3.10/site-packages/ddtrace/internal/module.py:248: in _create_module
    return self.loader.create_module(spec)
E   ImportError: dlopen(/Users/florian/.cache/pants/named_caches/pex_root/venvs/1/s/1af2c70d/venv/lib/python3.10/site-packages/icu/_icu_.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '__ZN6icu_7718AnnualTimeZoneRule8MAX_YEARE'

When I compile this package locally with

pip

directly I am able to use it by doing

ICU_VERSION=77 LDFLAGS=$(brew --prefix icu4c)/lib CPPFLAGS=$(brew --prefix icu4c)/include pip install pyicu

and I am achieving the same ability to build with

pex

when running

pants test

by passing these variables through

subprocess-environment.env_vars

:

[subprocess-environment]
env_vars = [
    "ICU_VERSION",
    "LDFLAGS",
    "CPPFLAGS",
    "PKG_CONFIG_PATH",
]

I am a bit stuck on debugging at this point, I can't figure out how to get

pex

to output detailed logs so I can see exactly how the icu lib is being compiled. I tried

pants -ltrace test

but it did not show me the underlying logs when pex was building the environment. Does anyone have ideas on where to go next? For reference when I pip install with

-vvv

flag I see the following output:

Building PyICU 2.15 for ICU 77 (max ICU major version supported: 77)

  running bdist_wheel
  running build
  running build_py
  creating build/lib.macosx-15.1-arm64-cpython-310/icu
  copying py/icu/__init__.py -> build/lib.macosx-15.1-arm64-cpython-310/icu
  running build_ext
  building 'icu._icu_' extension
  creating build/temp.macosx-15.1-arm64-cpython-310
  clang++ -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/opt/homebrew/opt/icu4c@77/include -I/Users/florian/code/lumos/.direnv/python-3.10/include -I/Users/florian/.pyenv/versions/3.10.6/include/python3.10 -c _icu_.cpp -o build/temp.macosx-15.1-arm64-cpython-310/_icu_.o -std=c++17 -DPYICU_VER=\"2.15\" -DPYICU_ICU_MAX_VER=\"77\"

And I was hoping to compare the output from pex if I could get any visibility into it

run your pants build like

pants --keep-sandboxes=always package ::

and look for an output saying something about a temp directory, probably something like

/tmp/pants-sandbox-asdhkh

In this sandbox, there will be a

__run.sh

file which is the exact command run by pants. (and the rest of the sandbox is the dependencies used in the job). You can inspect this file, tweak it (eg. to add verbosity, test options, etc.) and run it yourself to experiment. If you can narrow down what changes need to be made there, you can work backward to translate them to pants options

Ok after doing some more investigation into this, pex is building the wheel in a hermetic way using the

--manylinux

flag which is creating an isolated build context that strips headers and prevents ICU from dynamically linking. Given that, what are my options? Can I change the flags passed to pex when it builds? Is there some other way to link the wheel I built locally instead of pex building it?

I believe complete_platforms is what you need, though I haven't worked with it myself, so my info is a bit limited.

I’m a little confused about what your use-case is vs the one that works with

subprocess-environment.env_vars

?

@happy-kitchen-89482 I am able to build pyicu when passing

subprocess-environment.env_vars

to provide the ICU_VERSION and CPPFLAGS / LDFLAGS, but once built, I cannot run

pants test

on the built environment. The error I get is:

E   ImportError: dlopen(/Users/florian/.cache/pants/named_caches/pex_root/venvs/1/s/1af2c70d/venv/lib/python3.10/site-packages/icu/_icu_.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '__ZN6icu_7718AnnualTimeZoneRule8MAX_YEARE'

My two hypotheses were: • Pex is building with the --manylinux flag which strips out headers that are necessary • Pants is building a standalone python version in a hermetic way which is preventing it from linking out to shared libraries for ICU I am not sure if they are right, but that is what I am operating on. Given that this is what I have tried: • get pants to not use a hermetic standalone python build and use the one on my path instead (pyenv) but I was not able to figure out how to do this • remove the --manylinux flag from pex when building the deps, but I was not able to figure out hgow to do this • prebuild the wheel and link it so it does not need to build it, this was successful but I got the same error, this led me to believe it wasn't an issue with building the wheel, but with the python version I am running against • provide a static version of pyicu where all the shared libraries are statically linked and embedded so it does not need to rely on shared libraries -- haven't tried this yet

(Presumably there are missing

-L

and

-I

in the variable settings above?)

Thanks for trying this out! I appreciate it. I wonder what part of your setup made the difference, was it having

PKG_CONFIG_PATH

, was it the

--no-binary

or was it the

DYLD_LIBRARY_PATH

? Or some combination. Anyways -- interestingly today I was able to get it working by symlinking icu (

brew "icu4c@77", link: :force

) and then pants and pex were able to build it without issue and run tests without the symbol error. I think symlinking did something similar to the

DYLD_LIBRARY_PATH

by making sure that it knew where to look for the shared objects?

I’m assuming it was

PKG_CONFIG_PATH

at pex build time and

DYLD_LIBRARY_PATH

at test runtime (so both of those at Pants runtime)