Ok, guys, sorry that my question rate is high but ...
# general
f
Ok, guys, sorry that my question rate is high but it seems I'm again stuck somehow - can someone push me into the right direction about dependencies which have wheels? Is seems pip tries to build a wheel from source but fails to do so, I could figure out that in the sandbox the CPPFLAGS and LDFLAGS aren't set - here openssl isn't found. There is maybe a wheel available already (or pip cache) - if I manually install the dependency in a venv it works just fine. I also tried to find things in docs or with google without luck so far...
for completeness here is the error:
Copy code
INFO:add_license_files:adding license file 'LICENCE'
      INFO:execute:writing manifest file 'src/M2Crypto.egg-info/SOURCES.txt'
      INFO:run_command:running build_ext
      INFO:build_extension:building 'M2Crypto._m2crypto' extension
      INFO:swig_sources:swigging src/SWIG/_m2crypto.i to src/SWIG/_m2crypto_wrap.c
      INFO:spawn:swig -python -py3 -I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/14.0.3/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include "-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)" -I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/14.0.3/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include "-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)" -I/Users/elmcrest/.cache/pants/named_caches/pex_root/venvs/839dd85d778df5927a872ace6e92e15b4f301d3d/db8266b13362d394ecbbe9b1d67bbc2c832a080c/include -I/opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11/include/python3.11 -I/usr/include/openssl -includeall -modern -builtin -outdir /Users/elmcrest/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-_xs4lqvc/src/M2Crypto -o src/SWIG/_m2crypto_wrap.c src/SWIG/_m2crypto.i
      Deprecated command line option: -py3. Ignored, this option is no longer supported.
      Deprecated command line option: -modern. Ignored, this option is now always on.
      src/SWIG/_m2crypto.i:62: Error: Unable to find 'openssl/opensslv.h'
      src/SWIG/_m2crypto.i:68: Error: Unable to find 'openssl/safestack.h'
      src/SWIG/_evp.i:12: Error: Unable to find 'openssl/opensslconf.h'
      src/SWIG/_rc4.i:5: Error: Unable to find 'openssl/opensslconf.h'
      src/SWIG/_ec.i:7: Error: Unable to find 'openssl/opensslconf.h'
      error: command '/opt/homebrew/bin/swig' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for M2Crypto
ERROR: Failed to build one or more wheels
pants --python-native-code-cpp-flags="-I$(brew --prefix openssl)/include" --python-native-code-ld-flags="-L$(brew --prefix openssl)/lib" test foo/bar/asset_management/tests/tests_asset.py
doesn't work for me, too
same with test after pants executable
same for:
pants test --python-native-code-cpp-flags="['-I/opt/homebrew/opt/openssl@3/include']" --python-native-code-ld-flags="['-L/opt/homebrew/opt/openssl@3/lib']" foo/bar/asset_management/tests/tests_asset.py
or
pants test --python-native-code-cpp-flags="['-I /opt/homebrew/opt/openssl@3/include']" --python-native-code-ld-flags="['-L /opt/homebrew/opt/openssl@3/lib']" foo/bar/asset_management/tests/tests_asset.py
I wasn't sure if
-L /
or
-L/
is correct, both don't work anyways
ok, running into same error when trying with:
pip install M2Crypto --no-cache-dir
ok, turns out it works with:
CFLAGS="-I$(brew --prefix openssl)/include" LDFLAGS="-L$(brew --prefix openssl)/lib" SWIG_FEATURES="-I$(brew --prefix openssl)/include" pip install M2Crypto --no-cache-dir
e
It should, that's what's in their INSTALL docs, right?
f
trying now versions of:
pants test --extra-env-vars="['CFLAGS="-I$(brew --prefix openssl)/include"','LDFLAGS="-L$(brew --prefix openssl)/lib"','SWIG_FEATURES="-I$(brew --prefix openssl)/include"']" foo/bar/asset_management/tests/tests_asset.py
so far no luck though 😕
yeah, I lack a talent of reading the right things in right order I guess - or in general having a hard time with navigating through the unlimited resources... but you can't accuse me of being lazy, can you? 🙂 @enough-analyst-54434
so, with that command
pants test --python-native-code-cpp-flags="['-I/opt/homebrew/opt/openssl@3/include']" --python-native-code-ld-flags="['-L/opt/homebrew/opt/openssl@3/lib']" --extra-env-vars="['SWIG_FEATURES=-I/opt/homebrew/opt/openssl@3/include']" foo/bar/asset_management/tests/tests_asset.py --keep-sandboxes=always
I get the cpp and ld flags into the sandbox - at least I can see something in the
__run.sh
file but the SWIG_FEATURES is missing
@enough-analyst-54434 pleeeeease haalp!?
I could offer to send a german beer if that helps 😆
e
@freezing-fall-2672 you're only lazy in giving up so fast. I think you should expend at least 4 hours alone on this problem keeping track of what you tried and why.
f
ok, I'm used to being fast, that's true - and the downside of it is exactly as you describe, a little lack of patience
FYI tried to include the vars via pants.toml but also no effect
e
Ok, I'll step away. I don't want to enable this style. Good luck.
f
it would be helpful to suggest a better style, so far it's not really clear what you'd promote (at least to me)
c
he promotes a systematic step by step approach where you check each step thoroughly making sure you understand what is going on in each one. it takes time and patience, and you’ll get better and faster at it the more you practice it, but it’ll be painful in the beginning, like with most things you’re new to.
👆 1
I’ll feed you what I found out, tho… here’s what’s been working for me (not this particular dep, but may push you in a certain direction..?)
Copy code
[subprocess-environment]
env_vars.add = [
  "GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1",
  "GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1",
]

[python-native-code]
cpp_flags = "-I/opt/homebrew/opt/openssl@3/include"
ld_flags = "-L/opt/homebrew/opt/openssl@3/lib"
f
Ok, i’ll give it a try asap, thx. Ok, so that’s what I’m doing minus documenting every step…
But sure, I could add that
@curved-television-6568 works perfect, thx
👍 1
Good morning. Ok, to maybe fulfill the requirement of understanding what I've done here, some words about it. the
[subprocess-environment]
section in the pant.toml config file for pants gives the opportunity to configure things for the subprocess env (as the name suggests) it uses to execute target language specific functionality, f.e. python tests or linters - and I guess the same if you'd use JS or some other supported language. the
env_vars.add
variable is a list of env vars which will be added to the subprocess env. here I had to use it to provided the needed
SWIG_FEATURES
flag. the
[python-native-code]
section in the pants.toml config file for pants gives the opportunity to configure the cpp_flag and ld_flag if python pip needs to build a package from source. The
-L
flag points the C/C++ compiler to 'to shared libraries, directory containing *.so, *.dll or *.dlyb depending on the current operating system.' while the
-I
flag points the compiler to header files. Both is implemented in a
PATH
fashion, so the compiler knows where to look for things. Why this solved my issue here: pants was building the sandbox with my 3rd party requirements and while doing so it hit
M2Crypto
which doesn't provide a wheel (precompiled binary) which could just be downloaded instead of compiling locally from source - so it had no other chance than compiling it. since pants is executing things in a subprocess with it's own shell and inline env vars aren't passed along so there needs to be different options to do so. I hope that's fulfilling your requirements @enough-analyst-54434 - let me know if something is harshly wrong and of course if parts or not correct enough so to speak. Besides that I would expect that the
--test-extra-env-vars
or effectively the same
--extra-env-vars
would still work here for all three environment variables. Considering that pip seems to create an isolated build environment since version 10 (with PEP 517, right?) where env vars from the main pip process are passed along, I would assume that this, for me, unexpected behavior is related to how pants internals are working and specifically the rust side of things - which is also the side I have the least knowledge about so that's an easy blame for me 🙂 happy friday and until my next question, I hope 😆
👍 1
c
regarding the test extra env vars, at the point where these come into play (invoking
pytest
) any requirements needed to run your tests have already been installed, so it’s too late to affect any third party building using that option.
f
ah ok, that makes sense of course
c
another note to make while here, is that
subprocess-environment
sounds very generic, but it really only applies for a subset of invoked subprocesses (it’s a case by case, as they need to respect this option per invocation site) which is slightly unfortunate situation, but good to be aware of I think.. 😉
thanks, happy friday to you too 🙂
f
ah ok, so
subprocess-environment
isn't implemented for everything - that's what you mean with case by case, right?
c
yes, it applies to most (all?) python/pex subprocesses but not much else..
f
ok, thx