One more hopefully simple question. I am trying to resolve constraints and build the

cryptography

library. This requires rust and openssl. It fails to compile because it cannot find headers for openssl.

build/temp.macosx-12.1-arm64-3.9/_openssl.c:575:10: fatal error: 'openssl/opensslv.h' file not found
  #include <openssl/opensslv.h>

I am surprised by this, because I explicitly set some env variables to avoid this.

$ echo $CPPFLAGS $LDFLAGS $PKG_CONFIG_PATH 
-I/opt/homebrew/opt/openssl@3/include -L/opt/homebrew/opt/openssl@3/lib /opt/homebrew/opt/openssl@3/lib/pkgconfig

$ ls /opt/homebrew/opt/openssl@3/include/openssl/opensslv.h 
/opt/homebrew/opt/openssl@3/include/openssl/opensslv.h

What should I change to get cargo to look in the right spot for that file?

Check out https://www.pantsbuild.org/docs/reference-python-native-code#advanced-options. Looks like the docs are lying about the default, the default is actually

os.environ.get("CPPFLAGS", "")

Also we have a bug in that code that we are not invalidating Pantsd, OOF. This was code written a long time ago. Try using

--no-pantsd

or explicitly setting the options

--python-native-code-cpp-flags

and

--python-native-code-ld-flags

Eric isn't this just a normal resolve though? Thus: https://www.pantsbuild.org/docs/reference-subprocess-environment#section-env-vars

this is a dumb question but if I have

CPPFLAGS

set in my shell, shouldn't that get passed on to any child process spawned by me in that shell, i.e. pants -> cargo -> clang?

Nope

oh lol

We still mix in

python-native-code

.

pex_cli.py

line 129. I've tried a few times to get rid of

python-native-code

in favor of something like

subprocess-environment

, but I couldn't figure out how to not break our deprecation policy. That subsystem is so random, it's a holdover from v1

Ah, yeah. Eric is right. The env var business is super confusing.

I think I'd be fine with just breaking, so we can get rid of that complexity

If we do that, we should keep the options around but turn them off. That violates the policy:

Deprecated options must behave the same as before.

But at least better than Pants complaining the option does not exist, and the deprecation message will point you to how to fix things

wait but i'm not crazy though right, if i just call

fork()

in c, the child process and the parent have the same env, right? 🙂

Yeah, that's correct! What I suspect is happening is that Pants needs to pass the correct env vars to the initial child process. Pants tries doing things hermetically and strips all env vars, except for a few blessed ones like the

[python-native-code]

options +

[subprocess-environment]

trying...

ah, yeah if it was already built, then you'll be hitting the cache on-disk. You could use

--no-local-cache --no-pantsd

to be extra extra confident, but I think you're in the clear. FYI the cache would not be used if

LDFLAGS

and

CPPFLAGS

was set to a different value because the input to building the PEX would have changed. You can convince yourself of this by trying another run with

--python-native-code-cpp-flags=nonsense

, and Pants should try rebuilding the PEX.

running with

--no-local-cache --no-pantsd

was also successful. I couldn't force it to fail with

--python-native-code-cpp-flags=nonsense

which is a little 👻 to me. But I am also still not sure why it even tried to build cryptography from source, given that my constraint was

cryptography==36.0.1

and a wheel for that exists

cryptography-36.0.1-cp36-abi3-macosx_10_10_universal2.whl

https://pypi.org/project/cryptography/#files so maybe it just used the wheel?

That's lovely to hear. @acoustic-librarian-3937 would it be okay for us to quote you? (With or without attribution, whichever you prefer.)

Go for it! I think I prefer to be anonymous for now if that is cool with you

ah Ryan what Pants version is this?

2.9.0 still

Hm, I thought that we had updated Pex to understand

universal2

and backported that to 2.9, but need to check the changelog if that got cherry-picked. (logging off for the night, will check tomorrow if someone doesn't beat me)

pretty sure you did, because it was using the wheel earlier today

Thanks Ryan, and good luck. Keep on asking questions! 🙂

I'll echo what @acoustic-librarian-3937 said about the very high level of support here; I come from Java-land and did a lot with Gradle, and I know that build tools are not simple and have to do a ton of little things well, and so the Pants folks are getting complex questions from all angles. I'm making steady progress with shifting our current setup (which has some warts that make the transition to Pants a bit more difficult) and have been thinking about whether a blog/article/etc about our transition would be helpful for other Pants users (don't hold me to that, I may not want to divulge the warts of our current setup 🙂 )

Re

universal2

I dropped the ball. The PR is here and still needs cherry-pick: https://github.com/pantsbuild/pants/pull/14256 It looks like that landed on the 26th and I had gone on OOO for a few weeks the 25th (technically). I'll get the cherry-pick up here presently.

@lively-exabyte-12840 that absolutely would be valuable and much appreciated, both for insights it gives to users and to the maintainers as well. So I hope you do go forward with it. Feel free to ping me (head of devrel) if you need any help, feedback, etc.