https://pantsbuild.org/ logo
Join Slack
Powered by
I've looked into this jar ( <https://mvnrepository...
# general
b
I've looked into this jar ( https://mvnrepository.com/artifact/com.thesamet/play-pants-tool/0.0.6 ) and it's not a fat jar containing all the dependencies. How does pants resolve this classpath ?
e
Publishing java artifacts for consumption by a resolver (Pants via ivy or coursier, maven, sbt, etc.) requires publishing at least two files: the jar and a pom. The pom is an xml file containing metadata, part of which includes a declaration of dependencies. It's the metadata that supports the recursive construction of a full classpath of jars.
Presumably you hand-published vs using 
./pants publish ...
?
For example, from inside the https://github.com/pantsbuild/jarjar repo:
Copy code
$ ./pants publish.jar --local=/tmp/repo --no-prompt --no-dryrun src/main/java::
...
$ $ tree /tmp/repo/
/tmp/repo/
└── org
    └── pantsbuild
        └── jarjar
            └── 1.6.7-SNAPSHOT
                ├── ivy-1.6.7-SNAPSHOT.xml
                ├── ivy-1.6.7-SNAPSHOT.xml.md5
                ├── ivy-1.6.7-SNAPSHOT.xml.sha1
                ├── jarjar-1.6.7-SNAPSHOT-CHANGELOG.txt
                ├── jarjar-1.6.7-SNAPSHOT-CHANGELOG.txt.md5
                ├── jarjar-1.6.7-SNAPSHOT-CHANGELOG.txt.sha1
                ├── jarjar-1.6.7-SNAPSHOT.jar
                ├── jarjar-1.6.7-SNAPSHOT.jar.md5
                ├── jarjar-1.6.7-SNAPSHOT.jar.sha1
                ├── jarjar-1.6.7-SNAPSHOT-javadoc.jar
                ├── jarjar-1.6.7-SNAPSHOT-javadoc.jar.md5
                ├── jarjar-1.6.7-SNAPSHOT-javadoc.jar.sha1
                ├── jarjar-1.6.7-SNAPSHOT.pom
                ├── jarjar-1.6.7-SNAPSHOT.pom.md5
                ├── jarjar-1.6.7-SNAPSHOT.pom.sha1
                ├── jarjar-1.6.7-SNAPSHOT-sources.jar
                ├── jarjar-1.6.7-SNAPSHOT-sources.jar.md5
                └── jarjar-1.6.7-SNAPSHOT-sources.jar.sha1

4 directories, 18 files
Pants knows hot generate the pom file from the information you fed it in your BUILD files.
b
I used ./pants publish.jar thanks to autocompletion XD
e
OK - if that's how you published and your repos configuration is good, then:
However I'm surprised as I have to specify all its dependency in the classpath argument unlike the code in the github
That is surprising.
b
yeah and I've just retry it still does not work
I use this to publish 
./pants publish pants-plugins/scala/play-gen-tools --publish-jar-local=~/.ivy2/local --no-publish-jar-dryrun
and I checked, I have a correct pom generated
but to invalidate the cache I'm not sure I'm doing it rigth
it this case 
./pants clean-all
is useless
I have to 
rm ~/.ivy2/pants/com.mediarithmics.pants* -rf
e
./pants clean-all should be enough to invalidate if that's what's needed
b
and 
./pants ng-killall
e
You would not have to mess with the ~/.ivy2/pants cache unless you earlier published a bad pom or no pom
b
well I've only published using the command above
e
OK - then the remaining issue is configuration of coursier repos.
b
even if I'm using coursier, it seems that in this case it still uses ivy to resolve the dependency
e
That smells like a bug - what is your proof?
Check: $ ./pants options --scope=resolve --name=resolver
Should answer 
coursier
b
coursier is not configured to solve the local repo ?
e
You configure pants to use one resolver or the other - it will not split resolves
b
and I had to edit the ivysettings.xml to add a resolver toward ~/.ivy2/local ?
e
What is the answer you find for the question above?
b
resolver.resolver = coursier (from CONFIG in pants.ini)
e
OK, and what about this?: 
./pants options --scope=coursier --name=repos
b
e
Is that what you expect? Does that include where you published the item not resolving as expected?
b
nope, I publish in 
~/.ivy2/local
e
You'll nee to add that to repos then for your test
b
so here I have an unsolved question
e
I don't think there is a portable way to add a homedir, but you can try and see if coursier understands 
~
b
I wish to have both 'https://mediarithmics.nexus' and the local repo in my configured repos
however 'https://mediarithmics.nexus' is a private nexus witch require credentials
hence I'm configuring it using an env var
e
$ ./pants options --scope=coursier --name=repos --coursier-repos=bob --coursier-repos=jane coursier.repos = ['https://repo1.maven.org/maven2', 'bob', 'jane'] (from FLAG)
b
yeah ideally I'd like it to be configured for the whole team without additional args …
e
If you just want to append and you're just testing - you can do as I did above and append --coursier-repos elements.
OK - so pausing there
How will the whole team get the jar in 
~/.ivy2/local
on their machine?
b
well of course it won't
but if they have to do a publish local, compile cycle, I would like it to be already configured
e
Gotcha
b
so don't know if it's possible
worst case scenario the flag is viable solution, but if I can have better …
e
Well - 2 options I can think of: 1. Introduce a script they use that adds flags or exports vars 2. check in an alternate pants.ini
Not sure if you knew you could add pants.inis?
b
I've seen it but it's in the "add options scenario"
well we could always use an alias
e
Yes - it is
b
I'v tried my 
PANTS_COURSIER_REPOS
env var and use 
repos: +[
in pants.ini but it's not working
neither using a bash array for the 
PANTS_COURSIER_REPOS
var
anyway, I'll settle for the "temporary" solutions
e
$ PANTS_COURSIER_REPOS="+['env']" ./pants options --scope=coursier --name=repos coursier.repos = ['https://repo1.maven.org/maven2', 'nancy', 'env'] (from ENVIRONMENT in ../../../.pants.rc, from env var PANTS_COURSIER_REPOS)
Thats shows off adding in two ways, 'env' is from the env var, nancy is in ~/.pants.rc:
Copy code
$ tail -3 ~/.pants.rc 

[coursier]
repos: nancy
b
there is a .pants.rc ?!
e
Hopefully one of those two helps. I expect you just got the env var syntax wrong - it is titchy.
Yes - pants supports user-customization with ~/.pants.rc
Its just another pants.ini file
b
ok I'm trying it
e
Its a bit dangerous though since you can add repos there, have things working locally, forget you have ~/.pants.rc - then they fail elsewhere mysteriously
b
ok so I have my coursier repo correctly configured
thank you very much !
I do agree with your warning but we need a solution to isolate the credentials
e
You're welcome! Until pants can support directly building from source and consuming in the same run be very afraid of -SNAPSHOT and be careful!
b
so I have the correct value for 
coursier.repos
but it does not solve my initial problem 😞
It does not fetch the transitive dependencies
e
does 
-ldebug
give more information about the resolve?
I have not used coursier - but for ivy this will show what it's doing
b
not sure what to do with the output
Executing tasks in goals: bootstrap -> imports -> unpack-jars -> deferred-sources -> gen -> jvm-platform-validate -> resolve -> compile
Copy code
18:37:36 00:00   [gen]
18:37:36 00:00     [antlr-java]
18:37:36 00:00     [antlr-py]
18:37:36 00:00     [jaxb]
18:37:36 00:00     [protoc]
18:37:36 00:00     [ragel]
18:37:36 00:00     [thrift-java]
18:37:36 00:00     [thrift-py]
18:37:36 00:00     [wire]
18:37:36 00:00     [twirl]
                   play_tasks_twirl_gen_TwirlGen will read from local artifact cache at /home/lorilan/dev/mediarithmics-platform-2/.cache/38cc49ee3f6d74794dc9d5c801197dfcbbacf699
18:37:36 00:00       [cache]
                   No cached artifacts for 1 target.
                   Invalidated 1 target.DEBUG] Using previous fetch.
e
OK, my testing is in the pants repo: 1. publish local
Copy code
$ ./pants publish.jar --no-dryrun --no-prompt --local=/tmp/coursier src/java/org/pantsbuild::
3. create target to test with
Copy code
$ cat BUILD.coursier-test 
jar_library(
  name='coursier-test',
  jars=[
    jar(org='org.pantsbuild', name='args4j', rev='0.0.20-SNAPSHOT')
  ],
)
3. try coursier resolve
Copy code
$ ./pants --resolver-resolver=coursier --coursier-repos=/tmp/coursier -ldebug resolve //:coursier-test
...
10:45:50 00:01   [resolve]
10:45:50 00:01     [ivy]
10:45:50 00:01     [coursier]
                   Invalidated 1 target.
10:45:50 00:01       [bootstrap-coursier]DEBUG] Using previous fetch.

10:45:50 00:01       [coursier]DEBUG] Nailgun ng_CoursierResolve_resolve_coursier state: updated=True running=False fingerprint=None new_fingerprint=6a2227e1706eb48f3a776a9a60844077732ae2a3 distribution=None new_distribution=/usr/bin/java
DEBUG] Spawning nailgun server ng_CoursierResolve_resolve_coursier with fingerprint=6a2227e1706eb48f3a776a9a60844077732ae2a3, jvm_options=['-Xmx1g', u'-Dpants.buildroot=/home/jsirois/dev/pantsbuild/pants', u'-Dpants.nailgun.owner=/home/jsirois/dev/pantsbuild/pants/.pants.d/ng/CoursierResolve_resolve_coursier', u'-Dpants.nailgun.fingerprint=6a2227e1706eb48f3a776a9a60844077732ae2a3'], classpath=[u'/home/jsirois/dev/pantsbuild/pants/.pants.d/ivy/jars/com.martiansoftware/nailgun-server/jars/nailgun-server-0.9.1.jar', u'/home/jsirois/.cache/pants/tools/jvm/coursier/1.1.0.cf365ea27a710d5f09db1f0a6feee129aa1fc417/coursier.jar']
DEBUG] purging metadata directory: /home/jsirois/dev/pantsbuild/pants/.pids/ng_coursierresolve_resolve_coursier
DEBUG] Executing: /usr/bin/java -Xmx1g -Dpants.buildroot=/home/jsirois/dev/pantsbuild/pants -Dpants.nailgun.owner=/home/jsirois/dev/pantsbuild/pants/.pants.d/ng/CoursierResolve_resolve_coursier -Dpants.nailgun.fingerprint=6a2227e1706eb48f3a776a9a60844077732ae2a3 -cp .pants.d/ivy/jars/com.martiansoftware/nailgun-server/jars/nailgun-server-0.9.1.jar:../../../.cache/pants/tools/jvm/coursier/1.1.0.cf365ea27a710d5f09db1f0a6feee129aa1fc417/coursier.jar com.martiansoftware.nailgun.NGServer :0 args={'close_fds': True} at cwd=/home/jsirois/dev/pantsbuild/pants
DEBUG] Spawned nailgun server ng_CoursierResolve_resolve_coursier with fingerprint=6a2227e1706eb48f3a776a9a60844077732ae2a3, pid=3085 port=39913
DEBUG] Verified new ng server is connectable at ('127.0.0.1', 39913)
DEBUG] Executing via NailgunClient(host=u'127.0.0.1', port=39913, workdir=u'/home/jsirois/dev/pantsbuild/pants'): /usr/bin/java -Xmx1g -cp /home/jsirois/.cache/pants/tools/jvm/coursier/1.1.0.cf365ea27a710d5f09db1f0a6feee129aa1fc417/coursier.jar coursier.cli.Coursier fetch -t --cache /home/jsirois/.cache/pants/coursier -r <https://repo1.maven.org/maven2> -r /tmp/coursier -A jar,bundle,test-jar,maven-plugin,src,doc,aar -q --no-default -n 8 --json-output-file /home/jsirois/dev/pantsbuild/pants/.pants.d/tmp/tmpc934QA org.pantsbuild:args4j:0.0.20-SNAPSHOT

                     ==== stderr ====
                     Error with repositories:
                       Error parsing URL Right(/tmp/coursier) (no protocol: /tmp/coursier)
                     
                     ==== stdout ====
                     DEBUG] releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x7f33640b4710>

FAILURE: The coursier process exited non-zero: 255


               Waiting for background workers to finish.
10:45:51 00:02   [complete]
               FAILURE
So trying a file:// url for the local repo now...
Which works:
Copy code
$ ./pants --resolver-resolver=coursier --coursier-repos=file:/tmp/coursier -ldebug resolve //:coursier-test
...
10:49:00 00:01   [resolve]
10:49:00 00:01     [ivy]
10:49:00 00:01     [coursier]
                   Invalidated 1 target.
10:49:00 00:01       [bootstrap-coursier]DEBUG] Using previous fetch.

10:49:00 00:01       [coursier]DEBUG] Nailgun ng_CoursierResolve_resolve_coursier state: updated=False running=True fingerprint=6a2227e1706eb48f3a776a9a60844077732ae2a3 new_fingerprint=6a2227e1706eb48f3a776a9a60844077732ae2a3 distribution=/usr/bin/java new_distribution=/usr/bin/java
DEBUG] Executing via NailgunClient(host=u'127.0.0.1', port=39913, workdir=u'/home/jsirois/dev/pantsbuild/pants'): /usr/bin/java -Xmx1g -cp /home/jsirois/.cache/pants/tools/jvm/coursier/1.1.0.cf365ea27a710d5f09db1f0a6feee129aa1fc417/coursier.jar coursier.cli.Coursier fetch -t --cache /home/jsirois/.cache/pants/coursier -r <https://repo1.maven.org/maven2> -r file:/tmp/coursier -A jar,bundle,test-jar,maven-plugin,src,doc,aar -q --no-default -n 8 --json-output-file /home/jsirois/dev/pantsbuild/pants/.pants.d/tmp/tmpCHAgxX org.pantsbuild:args4j:0.0.20-SNAPSHOT

10:49:04 00:05     [go]
10:49:04 00:05     [scala-js-compile]
10:49:04 00:05     [scala-js-link]
10:49:04 00:05     [node]DEBUG] releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x7fe1bc0427d0>

               Waiting for background workers to finish.
10:49:04 00:05   [complete]
               SUCCESS

$ $ jq . /home/jsirois/dev/pantsbuild/pants/.pants.d/tmp/tmpCHAgxX
{
  "conflict_resolution": {},
  "dependencies": [
    {
      "coord": "args4j:args4j:2.33",
      "file": "/home/jsirois/.cache/pants/coursier/https/repo1.maven.org/maven2/args4j/args4j/2.33/args4j-2.33.jar",
      "dependencies": []
    },
    {
      "coord": "com.google.code.findbugs:jsr305:3.0.2",
      "file": "/home/jsirois/.cache/pants/coursier/https/repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar",
      "dependencies": []
    },
    {
      "coord": "com.google.guava:guava:18.0",
      "file": "/home/jsirois/.cache/pants/coursier/https/repo1.maven.org/maven2/com/google/guava/guava/18.0/guava-18.0.jar",
      "dependencies": []
    },
    {
      "coord": "org.pantsbuild:args4j:0.0.20-SNAPSHOT",
      "file": "/tmp/coursier/org/pantsbuild/args4j/0.0.20-SNAPSHOT/args4j-0.0.20-SNAPSHOT.jar",
      "dependencies": [
        "com.google.guava:guava:18.0",
        "args4j:args4j:2.33",
        "com.google.code.findbugs:jsr305:3.0.2"
      ]
    }
  ],
  "version": "0.1.0"
}
If you could do something similar after 
./pants clean-all
perhaps we can spot the differences. My test just confirms coursier handles additional local repos fine as long as you use 
file:/path/to/local/repo
b
ok I'm trying it. already have an error a step 1 : 
Error parsing URL Right(/home/lorilan/.ivy2/local) (no protocol: /home/lorilan/.ivy2/local)
. This is kind of a good sign I'm adding 
file://
.
anyway step 2 raise a question. How does 
BUILD dot someting
works ?
e
Its a little known afordance that may go away. ALL BUILD* files in a directory are loically concatenated before parsing
b
I have a 
BUILD.tools
at root because I saw this is how to configure a custom scala version but I've seen it again once or twice
e
Its a quick way to add a hack target for testing without mucking with checked in files
b
ok ! thank you
e
Yeah - the .tools extension is convention only. A plain BUILD or BUILD.bob would work fine
The initial use case was for a twitter repo that published parts as OSS but not all. BUILD.twitter vs BUILD allowed way to figure out which bits went OSS, which stayed inside.
b
so it seems to work as in your test
e
OK - great.
b
well
I don't have the 
--json-output-file /home/jsirois/dev/pantsbuild/pants/.pants.d/tmp/tmpCHAgxX
part
e
I just got that location from the -ldebug output - it was the last argument on the coursier command line
You have no such similar output? What version of pants?
b
the only log I have with coursier is 
[coursier]DEBUG] releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x7f3e9007cad0>
1.9.0
e
With ./pants -ldebug ... ?
b
yep I used the same line as you 
./pants --resolver-resolver=coursier --coursier-repos=file:/tmp/coursier -ldebug resolve //:coursier-test
e
OK - well, we only need that bit to prove transitive resolves, which my test already proved - for master anyhow.
After all this setup, is your original problem solved?
b
no 😞
this is what I've done
./pants --resolver-resolver=coursier --coursier-repos=file:/tmp/coursier -ldebug resolve //:coursier-test
Copy code
cat BUILD.coursier-test
jar_library(
  name='coursier-test',
  jars=[
    jar(org='com.mediarithmics.pants', name='play-gen-tools', rev='0.0.1-SNAPSHOT')
  ],
)
./pants --resolver-resolver=coursier --coursier-repos=/tmp/coursier -ldebug resolve //:coursier-test
fails
./pants --resolver-resolver=coursier --coursier-repos=file:/tmp/coursier -ldebug resolve //:coursier-test
is a sucess
but then 
./pants --resolver-resolver=coursier --coursier-repos=file:/tmp/coursier -ldebug compile mytarget
e
Yeah - those were just debug steps to confirm coursier + Pants worked with local repos
b
still ends up with a 
java.lang.NoClassDefFoundError: scala/collection/Seq
e
OK - I found it
The short answer is you can't use coursier or you have to dual configure ivy with the local repo - because you were right, pants is split internally with a hardcoded tool fetching system that always uses ivy 😕
I'll look for an existing issue here presently and file one if I don't fine one/ Not fun. Sorry about this!
b
no problem !
but I have ivy configured :
Copy code
<ivysettings>
    <credentials host="mediarithmics.nexus"
                 realm="Artifactory Realm"
                 username="${login}"
                 passwd="${password}"/>

    <property name="local.ivy.pattern"
              value="[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"
              override="false" />
    <property name="local.artifact.pattern"
              value="[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"
              override="false" />


    <property name="web.ivy.pattern"
              value="[organisation]/[module]/[revision]/[type]s/[artifact].[ext]"
              override="false" />
    <property name="web.artifact.pattern"
              value="[organisation]/[module]/[revision]/[type]s/[artifact].[ext]"
              override="false" />
    <settings defaultResolver="chain"/>
    <resolvers>
        <chain name="chain" returnFirst="true">

            <url name="web">
                <ivy pattern="<https://mediarithmics.nexus/${web.ivy.pattern}>" />
                <artifact pattern="<https://mediarithmics.nexus/${web.artifact.pattern}>" />
            </url>
            <url name="local" m2compatible="true" >
                <ivy pattern="file:///home/lorilan/.ivy2/local/${local.ivy.pattern}" />
                <artifact pattern="file:///home/lorilan/.ivy2/local/${local.artifact.pattern}" />
            </url>
            <ibiblio name="public" m2compatible="true" />
            <!--<ivyrep name="mics-nexus" ivyroot="<https://mediarithmics.nexus>"/>-->
        </chain>

    </resolvers>
</ivysettings>
e
That looks ike a bad config for the local bit. Just a sec.
b
and in my pants.ini
Copy code
[ivy]
ivy_settings: %(buildroot)s/build-support/ivysettings.xml
e
OK - trying to keep my stack from getting too deep. 1st, there was no issue tracking this bug, now tracked here: https://github.com/pantsbuild/pants/issues/6675
See here, the "local.m2" <filesystem/> repo. It is configured correctly: https://github.com/twitter/commons/blob/master/build-support/ivy/ivysettings.xml
b
great ! trying it, thank you
still not working 😕
anyway thanks a lot for your help I'll keep diging
hey maybe a last question. can I completely clean the cache ? I'm doing this POC to see if we migrate from a nix/sbt build to pants. I want to build my top target from scratch to see the time it takes
e
Two methods: 1. rm -rf ~/.cache/pants && ./pants clean-all 2. ./pants --cache-ignore ...<rest>...
You probably want 2
1 will re-download and install pants itself
b
yes thanks !
6 Views
Open in Slack
PreviousNext
Powered by