pantsbuild #general

Channels

welcome

github-notifications

announce

ancient-stone-50795

07/22/2015, 7:02 PM

@mammoth-autumn-2551: A ssl certificate issue is one we’ve been dealing with for a while. What platform are you on?

ancient-stone-50795

07/22/2015, 7:02 PM

Copy code

#!/bin/bash
#
# Echos the name of a pem file of CA certs. Creates the
# file if needed.
#
# Linux: There's already a file in /etc
# Mac: Extract the SSL CA certificates from the keychain

set -e
PANTS_BOOTSTRAPDIR=~/.cache/pants

UNAME_S=$(uname -s)

case ${UNAME_S} in
  Darwin)
  mkdir -p ${PANTS_BOOTSTRAPDIR}
  CACERTS_PEM=${PANTS_BOOTSTRAPDIR}/cacerts.pem
  security find-certificate -a -p /Library/Keychains/System.keychain > ${CACERTS_PEM}
  security find-certificate -a -p /System/Library/Keychains/SystemRootCertificates.keychain >> ${CACERTS_PEM}
  ;;

  Linux)
  CACERTS_PEM="/etc/pki/tls/certs/ca-bundle.crt"
  ;;

  *)
    echo "*** ${UNAME_S} not supported by this script!" 1>&2
    exit 1
esac

echo ${CACERTS_PEM}

ancient-stone-50795

07/22/2015, 7:02 PM

That file in /etc/ on Linux may have been something created by our team at Square.

ancient-stone-50795

07/22/2015, 7:03 PM

Then, what we do is set the environment variable

REQUESTS_CA_BUNDLE

from the output of that script:

Copy code

export REQUESTS_CA_BUNDLE=$(squarepants/bin/pants_cacerts_bootstrap.sh)

jolly-chef-92794

07/22/2015, 7:04 PM

So, I'm noticing that jmake doesn't seem to recompile .java sources that haven't changed, even when using different arguments like

-C-source

and

-C-target

that really should trigger a recompile.

jolly-chef-92794

07/22/2015, 7:05 PM

(I know jmake is actually running from the debug output, so it's not just a pants fingerprinting/caching problem)

enough-analyst-54434

07/22/2015, 7:09 PM

This doesn't sound particularly surprising. If you need a fix on the jmake side in the interim until you flip over to zinc, we maintain/release/use this fork: https://github.com/pantsbuild/jmake

mammoth-autumn-2551

07/22/2015, 7:10 PM

@ancient-stone-50795: we’re seeing it on Ubuntu 12.04 (fairly vanilla with security updates) and osx

ancient-stone-50795

07/22/2015, 7:11 PM

@jolly-chef-92794: we should be able to flip back over to zinc for java soon - You can enable it and see if we have the same problem with zinc.

mammoth-autumn-2551

07/22/2015, 7:11 PM

From what I understand, requests supports bundling updated roots certificates via the project

certifi

, so that my solve the issue, but doesn’t explain why it showed up in this pants upgrade

ancient-stone-50795

07/22/2015, 7:12 PM

@jolly-chef-92794: The only problem we are having with zinc at the moment is error output is garbled.

ancient-stone-50795

07/22/2015, 7:12 PM

@mammoth-autumn-2551: What URL are you trying to hit that is giving you the error. Is it a self-signed certificate from some internal server (that was what we ran into)

mammoth-autumn-2551

07/22/2015, 7:13 PM

actually, no its the mailgun validate apihttps://documentation.mailgun.com/api-email-validation.html

ancient-stone-50795

07/22/2015, 7:14 PM

@mammoth-autumn-2551: you probably just updated to a more recent version of the requests library. I did bump the version some time ago...

ancient-stone-50795

07/22/2015, 7:14 PM

Copy code

commit d8a8c311085bcc987d3be0b34f5bd73c2e5b2a41
Author: Eric Ayers <zundel@squareup.com>
Date:   Sun Mar 8 08:58:11 2015 -0400

    Update our requests library to something more recent
    
    I noticed that this library is way behind the current release which is 2.5.3.  See <http://docs.python-requests.org/en/latest/community/updates/>

mammoth-autumn-2551

07/22/2015, 7:45 PM

So mystery solved @ancient-stone-50795 and @enough-analyst-54434 : These issues were only happening for us during a high-level release script, which is invoked via

pants run

. One of the first things that script does is clean up pants state, which I think blows away the now cached chroot. So downstream code that would load resources would fail (in the case of requests, it would be loading a bundled copy of the root certificate)

enough-analyst-54434

07/22/2015, 7:47 PM

so .

/pants run ...

-> python code that knows about $CWD/.pants.d and kills it in the middle of pants run?

enough-analyst-54434

07/22/2015, 7:49 PM

If so that sounds like the winner for sure

enough-analyst-54434

07/22/2015, 7:51 PM

I guess the proof of that would be to s/run/binary/ and run the pex plopped in dist instead of

./pants run

- that should still work

witty-crayon-22786

07/22/2015, 8:29 PM

@ancient-stone-50795, @bored-art-40741 : do you guys use the mid-graph form of ivy excludes?

witty-crayon-22786

07/22/2015, 8:29 PM

bored-art-40741

07/22/2015, 8:29 PM

We do not use ivy excludes at all. We only use ivy for tool resolution

witty-crayon-22786

07/22/2015, 8:30 PM

Copy code

java_library(name='blah', dependencies=[..], excludes=[exclude(x, y)])

bored-art-40741

07/22/2015, 8:30 PM

We definitely do not do that

witty-crayon-22786

07/22/2015, 8:30 PM

kk. @ancient-stone-50795 ?

ancient-stone-50795

07/22/2015, 8:57 PM

@witty-crayon-22786: we don’t use that. We use deploy_excludes on

jvm_binary

targets though:

Copy code

jvm_binary(name='server',
  main = 'com.squareup.gns.server.GnsServerApp',
  basename= 'gns-server',
  dependencies = [
    ':lib',
    ':server-signed-jars'
  ],
  manifest_entries = square_manifest({
    'Class-Path': 'gns-server-signed-jars/bcprov-jdk15on.jar gns-server-signed-jars/ncipherkm.jar',
  }),
  deploy_excludes = [
    exclude(org='com.squareup.nonmaven', name='ncipherkm'),
    exclude(org='org.bouncycastle', name='bcprov-jdk15on')
  ],
)

witty-crayon-22786

07/22/2015, 8:57 PM

yea, that's the way to go.

witty-crayon-22786

07/22/2015, 8:58 PM

ok. based on your responses, i'm going to assume that we have some freedom to change the default behaviour of those mid-graph excludes

ancient-stone-50795

07/22/2015, 8:58 PM

SGTM

enough-analyst-54434

07/22/2015, 8:59 PM

Or kill them? They are an ivy - only feature - does not exist for mvn.