https://pantsbuild.org/ logo
Join Slack
Powered by
More specifically my use case is the following. I ...
# general
b
More specifically my use case is the following. I have this directory structure in my repo.
Copy code
pants-plugin
      python
          foo
              targets
                  foo_library.py
              tasks
                  foo_gen.py
              register.py
      scala
          foo-cli
              Foo.scala
              BUILD
In 
foo_gen.py
, there is
Copy code
cls.register_jvm_tool(register, 'foo-cli',
                              classpath=[
                                  JarDependency(org='bar',
                                                name='foo-cli',
                                                rev='0.0.1-SNAPSHOT'),
                              ],
                              classpath_spec='//pants-plugins/scala/foo-cli'
                              )
and in 
scala/foo-cli/BUILD
Copy code
java_library(
    sources = rglobs('*.scala'),
    dependencies = [
        '3rdparty/jvm/some/dependency'
        ],
    provides = artifact(org='bar',
                        name='foo-cli',
                        repo=snapshots)
    )
When I run my task I still get a 
ClassNotFoundException
from my third party dependency. (I'm sure that 
foo-cli
is correctly published with a pom listing the 3rd party dependencies and that the foo-cli jar used is fetched from the nexus repository). Do you see how to fix this ?
e
classpath_spec is a BUILD file address that can be used to over-ride the classpath for a jvm tool. It is assumed the address will point to resolvable pre-built jars. This is more of the same pants not currently being able to run itself in arbitrary loops issue you ran into earlier. So... the classpath_spec should not point to 
java_library
, but 
jar_library
or nothing. I think if you left 
classpath_spec
out and accepted the default things would work.
b
well my first try was in this setup but since it didn't worked I tried to use the classpath_spec
I even tried to define
Copy code
jar_library(
    name= 'foo-cli',
    jars = [
        jar(
        org='bar',
        name='foo-cli,
        rev='0.0.1-SNAPSHOT'
        ),
        ],
 )
and to use it in the classpath_spec but still does not work
if I add the path to the main in the 
register_jvm_tool
call, using 
-ldebug
I can see that the classpath used points to 
.pants.d/bootstrap/bootstrap-jvm-tools/tool_cache/shaded_jars/bar.foo-cli.blabla-ShadedToolFingerprintStrategy_e2ba67beff2a.jar
in this jar there is a 
compile_classpath/pants-plugins.scala.foo-cli.txt
that list the compile_classpath but I still have the ClassNotFoundException
this really drives me crazy. Every exemples I look at, it seems to work out of the box. I must be missing an evident but crucial piece
e
OK - re-read a bit more. Look here: 
.../bar.foo-cli.blabla-ShadedToolFingerprintStrategy_e2ba67beff2a.jar
- the "Shaded" is referring to the fact the registering a jvm tool with a main will attempt to automatically shade all elements of the tool's classpath except for the main class. You may need to un-shade portions to get things working. Just a sec for an example...
b
thanks ! I'm looking at the shading options
ok so I tried something very hacky. I changed my 
pants-plugins/scala/foo-cli/BUILD
to
Copy code
java_library(
   name='cli',
   sources = rglobs('*.scala'),
   dependencies = [
         'dependencies'
       ],
   provides = artifact(org='bar',
                       name='foo-cli',
                       repo=snapshots)
   )
target(
  name = 'dependencies',
  dependencies = [
      '3rdparty/jvm/some/dependency'
  ],
)
and then in 
pants-plugin/python/foo/BUILD
:
Copy code
target(
    name='cli',
    dependencies= [
        ':cli-jar',
        'pants-plugins/scala/foo:dependencies'
    ]
    )

jar_library(
    name= 'cli-jar',
    jars = [
        jar(
        org='bar',
        name='foo-cli',
        rev='0.0.1-SNAPSHOT'
        ),
        ],
)
and to uses 
classpath_spec='//pants-plugins/python/foo:cli'
now, I have missing a NoClassDefFoundError on a class of the scala std lib ! it's like it refuses to resolve the transitive dependencies
ok so indeed, whith 
-ldebug
I found the 
fetch-ivy.xml
generated and I found this :
Copy code
<dependency org="bar"
                 name="foo-cli"
                 rev="0.0.1-SNAPSHOT"
                 force="true"
                 transitive="false"
                 
     >
_< no idea why transitive is set to false
e
OK - your python code example above has:
Copy code
JarDependency(org='bar',
                                                name='foo-cli',
                                                rev='0.0.1-SNAPSHOT')
Are you mixing examples?
Note - foo-cli, not foo
b
XD it's a typo
I corrected my example to be consistent
e
So, tool bootstrapping is one of the places ivy is used and not coursier which calls into question your ivysettings config
b
well basically I just have an 
ivysettings.xml
to declare my resolver and credentials.
e
Right - but the local resolver used to get this snapshot is special - not typical m2
b
Copy code
<ivysettings>


    <!-- The ${login} and ${password} values come from a netrc_credentials() object in a BUILD


    -->
    <include file="${user.home}/.ivy2/credentials.xml" />
    <credentials host="<http://sf-nexus.foobar.com|sf-nexus.foobar.com>"
                 realm="Sonatype Nexus Repository Manager"
                 username="${nexus.login}"
                 passwd="${nexus.password}"/>

    <property name="local.ivy.pattern"
              value="[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"
              override="false" />
    <property name="local.artifact.pattern"
              value="[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"
              override="false" />


    <property name="web.ivy.pattern"
              value="[organisation]/[module]/[revision]/[type]s/[artifact].[ext]"
              override="false" />
    <property name="web.artifact.pattern"
              value="[organisation]/[module]/[revision]/[type]s/[artifact].[ext]"
              override="false" />

    <property name="web.maven.pattern"
              value="[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"
              override="false" />

    <settings defaultResolver="chain"/>
    <resolvers>
        <chain name="chain" returnFirst="true">
            <url name="nexus-ivy"
                 descriptor="required">
                <ivy pattern="<https://sf-nexus.foobar.com/content/groups/global/${web.ivy.pattern}>" />
                <artifact pattern="<https://sf-nexus.foobar.com/content/groups/global/${web.artifact.pattern}>" />
            </url>

            <url name="nexus-maven" m2compatible="true"
                 descriptor="required">
                <ivy pattern="<https://sf-nexus.foobar.com/content/groups/global/${web.maven.pattern}>" />
                <artifact pattern="<https://sf-nexus.foobar.com/content/groups/global/${web.maven.pattern}>" />
            </url>

            <ibiblio name="nexus-ivy-ibiblio"
                     descriptor="required"
                     root="<https://sf-nexus.foobar.com/content/groups/global/>" />
            <ibiblio name="nexus-maven-ibiblio"
                     m2compatible="true"
                     descriptor="required"
                     root="<https://sf-nexus.foobar.com/content/groups/global/>" />

            <filesystem name="local" m2compatible="true" local="true" checkmodified="true">
                <ivy pattern="${user.home}/.ivy2/local/${local.ivy.pattern}" />
                <artifact pattern="${user.home}/.ivy2/local/${local.artifact.pattern}" />
            </filesystem>
            <ibiblio name="public" m2compatible="true"
                     descriptor="required"/>
        </chain>

    </resolvers>
</ivysettings>
e
Can you paste your current ivysetting?
coke
b
it's possibly a mess
I just tride to add the two first ibiblio
and I've just read in the ivysettings.xml of pants that 
descriptor="required"
would be a good idea so I put it everywhere XD
e
To confirm, you're publishing to ~/.ivy2/local/ ?
b
no my artifact is already published on 
nexus-maven
e
aha - ok
publishing snapshots is a bit evil! but putting that aside - thinking...
b
yeah ... I agree but It's not my decision
but I think (I hope ?) it's not what causes my problem
e
Which repo is winning? There should be a properties file in the ~/.ivy2/local/.../ dir for the artifact
b
hmm nothing in local but I'm sure it's nexus-maven because I had to configure it to retrieve the dependency
e
Also, the intransitive looks like a red herring. Reviewing the ivy code in pants, it seperates resolves from fetches and all fetches are intransitive since the resolve alreay computes the full list of single jars to fetch.
Can you paste the ClassNotFound stack trace?
b
Copy code
16:02:52 00:01         [routes-gen]
                       ==== stderr ====
                       java.lang.NoClassDefFoundError: play/routes/compiler/RoutesGenerator
                        at com.mediarithmics.playpants.RoutesGen.main(RoutesGen.scala)
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        at java.lang.reflect.Method.invoke(Method.java:498)
                        at com.martiansoftware.nailgun.NGSession.run(NGSession.java:280)
                       Caused by: java.lang.ClassNotFoundException: play.routes.compiler.RoutesGenerator
                        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
                        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
                        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
                        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
                        ... 6 more
                       
                       ==== stdout ====

`
e
And play/routes/compiler/RoutesGenerator is 3rdparty code your cli depends on. So we're back to unshading.
You need to work thorugh which classes in 3rdparty must be seeable by user code and unshade all of those
b
well, I'm not sure it's the problem
when looking in 
-ldebug
/usr/bin/java -cp /home/lorilan/dev/mediarithmics-platform-2/.pants.d/ivy/jars/com.mediarithmics.pants/play-gen-tools/jars/play-gen-tools-0.0.1-SNAPSHOT.jar com.mediarithmics.playpants.RoutesGen
I see this
and no other jars and it does not contain my third party dependencies (shaded or not)
this is what I got using
Copy code
classpath=[
                                 JarDependency(org='bar',
                                               name='foo-cli',
                                               rev='0.0.1-SNAPSHOT'),
                             ],
and not classpath_spec
if I use classpath_spec I got my third party dependencies (shaded) but I lose my jar !
e
main controls shading, spec is not relevant
b
and then there is the "hack" I tried to group my dependency with the third party dependencies in one target (without specifying the main in 
register_jvm_tool
, I specify it when calling 
runjava
so I have no shading problems) but then I still don't get the transitive dependencies (well the n+1 transitive dependency)
e
I think it's easiest to debug without spec, leave that out.
b
yeah so I went back to the setup I've seen the most
e
Is the published pom for your cli correct? Lists the right direct dependencies?
b
I just specify one 
JarDependency
to my published artifbact
yes
e
Correct - good
And you specify a main when registering the tool?
b
ok sorry this is not readable
e
I trust you, no need to paste
b
and "This isn't a standard web page" so firefox does not allow the screenshot XD
anyway
basically this is the source code 
<https://github.com/thesamet/play-pants/blob/master/src/python/play/tasks/routes_gen.py>
we just updated the version and integrated it to our mono - repo for "ease" of maintenance and because we have other build cli-tools to integrate
ha I missed your question
I tried with and without specifying the main
here I'm back without specifying it
I'm really groping around (not sure I'm using the expression well)
may the 
publication_metadata
in
Copy code
provides = artifact(org = 'bar', name='foo', repo= public,
   publication_metadata = ???)
have an impact on dependency resolution ?
because I provide none
I updated pants from 1.9.0 to 1.11.0rc1 but it does not change anything
e
OK - sorry for the silence - just transferred between physical locations.
b
no problem ! I'm already very grateful about the time you granted me
e
I think its best to drill down on the classpath issue. So - to make things easiest to start - I'd get rid of main so there is no shading. Then run 
./pants clean-all && ./pants server --open
, we'll use the 'Latest' run view.
Now run whatever pants command generates the classpath error and we'll look in the server web ui.
b
ok
image.png
e
In the 1st command there is a path "cachepath output to ..."
Can you open that path?
b
i'm in it !
e
What are the contents? Expected? IE: more than 1 jar?
b
there are the following files : classpath classpath.raw fetch-ivy.xml fetch-report-default.xml resolution.json
the classpath contains only one jar
e
resolution.json is probably easiest to view
That's not good!
b
no third dependencies at all
yeah :s
e
found com.mediarithmics.pants#play-gen-tools;0.0.1-SNAPSHOT in nexus-maven
So can you paste the pom from that repo for com.mediarithmics.pants#play-gen-tools;0.0.1-SNAPSHOT ?
b
Copy code
<project xsi:schemaLocation="<http://maven.apache.org/POM/4.0.0>                              <http://maven.apache.org/maven-v4_0_0.xsd%22><modelVersion>4.0.0</modelVersion><groupId>com.mediarithmics.pants</groupId><artifactId>play-gen-tools</artifactId><packaging>jar</packaging><version>0.0.1-SNAPSHOT</version><dependencies><dependency><groupId>com.typesafe.play</groupId><artifactId>routes-compiler_2.12</artifactId><version>2.6.20</version><scope>compile</scope></dependency><dependency><groupId>com.typesafe.play</groupId><artifactId>twirl-compiler_2.12</artifactId><version>1.3.15</version><scope>compile</scope></dependency><dependency><groupId>com.github.scopt</groupId><artifactId>scopt_2.12</artifactId><version>3.7.0</version><scope>compile</scope></dependency></dependencies></project>|http://maven.apache.org/maven-v4_0_0.xsd"><modelVersion>4.0.0</modelVersion><groupId>com.mediarithmics.pants</groupId><artifactId>play-gen-tools</artifactId><packaging>jar</packaging><version>0.0.1-SNAPSHOT</version><dependencies><dependency><groupId>com.typesafe.play</groupId><artifactId>routes-compiler_2.12</artifactId><version>2.6.20</version><scope>compile</scope></dependency><dependency><groupId>com.typesafe.play</groupId><artifactId>twirl-compiler_2.12</artifactId><version>1.3.15</version><scope>compile</scope></dependency><dependency><groupId>com.github.scopt</groupId><artifactId>scopt_2.12</artifactId><version>3.7.0</version><scope>compile</scope></dependency></dependencies></project>>
e
ok
SO then that is not what ivy sees. I predict you had a failed pom download before you edited ivysettings to require metadata
To confirm:
Can you 
tree ~/.ivy2/local/com.mediarithmics.pants/play-gen-tools/
b
Copy code
/home/lorilan/.ivy2/local/com.mediarithmics.pants/play-gen-tools [error opening dir]

0 directories, 0 files
e
Help me out
b
yes
e
I may be spelling the dir structure wrong
b
ha !
ok sorry
e
oh
~/.ivy2/pants/....
b
Copy code
tree ~/.ivy2/local/com/mediarithmics/pants/play-gen-tools 
/home/lorilan/.ivy2/local/com/mediarithmics/pants/play-gen-tools

0 directories, 0 files
e
(we're downloading here not publishing)
b
Copy code
tree ~/.ivy2/pants/com.mediarithmics.pants/play-gen-tools
/home/lorilan/.ivy2/pants/com.mediarithmics.pants/play-gen-tools
├── ivy-0.0.1-SNAPSHOT.xml
├── ivy-0.0.1-SNAPSHOT.xml.original
├── ivydata-0.0.1-SNAPSHOT.properties
└── jars
    └── play-gen-tools-0.0.1-SNAPSHOT.jar

1 directory, 4 files
better 😛
e
cat ivy-0.0.1-SNAPSHOT.xml.original - that's the pom
b
Copy code
cat ~/.ivy2/pants/com.mediarithmics.pants/play-gen-tools/ivy-0.0.1-SNAPSHOT.xml
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
Licensed under the Apache License, Version 2.0 (see LICENSE).
-->
<!-- generated by pants! -->
<ivy-module version="2.0" xmlns:xsi="<http://www.w3.org/2001/XMLSchema-instance>" xsi:noNamespaceSchemaLocation="<http://ant.apache.org/ivy/schemas/ivy.xsd>" xmlns:m="<http://ant.apache.org/ivy/maven>">

  <info organisation="com.mediarithmics.pants" module="play-gen-tools" revision="0.0.1-SNAPSHOT" status="release" publication="20181115160620"/>

  <publications>
    <artifact/>
    <artifact m:classifier="sources"/>
    <artifact type="pom"/>
  </publications>
</ivy-module>
ha
e
Looks bad
b
same
e
So - taking a step back
b
ok so the error is in the publishing ? 😛
e
Was there ever a time when you might have published the jar locally with a bad set of deps?
b
possibly
e
k
now 
rm -rf ~/.ivy2/pants/com.mediarithmics.pants/play-gen-tools
b
but i have 
rm ~/.ivy2/pants -rf
e
./pants clean-all
b
yeah did that
several times 😛
e
Can you 
cat ivydata-0.0.1-SNAPSHOT.properties
b
Copy code
cat ~/.ivy2/pants/com.mediarithmics.pants/play-gen-tools/ivydata-0.0.1-SNAPSHOT.properties 
#ivy cached data file for com.mediarithmics.pants#play-gen-tools;0.0.1-SNAPSHOT
#Fri Nov 16 16:58:02 CET 2018
artifact\:ivy\#ivy.original\#xml\#-157161836.exists=true
artifact\:ivy\#ivy\#xml\#-1591587195.is-local=false
artifact\:ivy\#ivy\#xml\#-1591587195.location=https\://sf-nexus.mediarithmics.com/content/groups/global/com/mediarithmics/pants/play-gen-tools/0.0.1-SNAPSHOT/ivy-0.0.1-SNAPSHOT.xml
resolver=nexus-maven
artifact\:ivy\#ivy\#xml\#-1591587195.exists=true
artifact\:play-gen-tools\#jar\#jar\#-1929109683.exists=true
artifact\:ivy\#ivy.original\#xml\#-157161836.location=https\://sf-nexus.mediarithmics.com/content/groups/global/com/mediarithmics/pants/play-gen-tools/0.0.1-SNAPSHOT/ivy-0.0.1-SNAPSHOT.xml
artifact\:play-gen-tools\#jar\#jar\#-1929109683.location=https\://sf-nexus.mediarithmics.com/content/groups/global/com/mediarithmics/pants/play-gen-tools/0.0.1-SNAPSHOT/play-gen-tools-0.0.1-SNAPSHOT.jar
artifact\:ivy\#ivy\#xml\#-1591587195.original=artifact\:ivy\#ivy.original\#xml\#-157161836
artifact.resolver=nexus-maven
artifact\:ivy\#ivy.original\#xml\#-157161836.is-local=false
artifact\:play-gen-tools\#jar\#jar\#-1929109683.is-local=false
artifact\:ivy\#ivy.original\#xml\#-157161836.original=artifact\:ivy\#ivy.original\#xml\#-157161836
artifact\:play-gen-tools\#jar\#jar\#-1929109683.original=artifact\:play-gen-tools\#jar\#jar\#-1929109683
e
Not good: artifact\:ivy\#ivy.original\#xml\#-157161836.location=https\://sf-nexus.mediarithmics.com/content/groups/global/com/mediarithmics/pants/play-gen-tools/0.0.1-SNAPSHOT/ivy-0.0.1-SNAPSHOT.xml
b
btw, this is the ivysettings for publishing:
Copy code
<ivysettings>


    <credentials host="<http://sf-nexus.mediarithmics.com|sf-nexus.mediarithmics.com>"
                 realm="Sonatype Nexus Repository Manager"
                 username="${login}"
                 passwd="${password}"/>

    <property name="web.ivy.pattern"
              value="[organisation]/[module]/[revision]/[type]s/[artifact].[ext]"
              override="false" />
    <property name="web.artifact.pattern"
              value="[organisation]/[module]/[revision]/[type]s/[artifact].[ext]"
              override="false" />


    <resolvers>
        <!-- For resolving foreign deps only (to create a poms for publishing). -->
        <url name="web">
            <ivy pattern="<https://sf-nexus.mediarithmics.com/content/groups/global/${web.ivy.pattern}>" />
            <artifact pattern="<https://sf-nexus.mediarithmics.com/content/groups/global/${web.artifact.pattern}>" />
        </url>

        <!-- For publishing to maven central only. -->
        <ibiblio name="<http://sf-nexus.mediarithmics.com|sf-nexus.mediarithmics.com>"
                 m2compatible="true"
                 usepoms="true"
                 root="<https://sf-nexus.mediarithmics.com/repository/releases/>"
        />

        <ibiblio name="nexus-snapshot"
                 m2compatible="true"
                 usepoms="true"
                 root="<https://sf-nexus.mediarithmics.com/repository/snapshots>"
        />

    </resolvers>
</ivysettings>
e
So ivy is resolving ivy.xml and not pom
Pants only publishes good poms
I'll re-read your ivysettings but looks like you have 2 styles pointed at same repo - m2 and ivy
b
yeah …
e
That's the problem
b
I'm a bit struggling here
ok
e
Do you need to use ivy.xml instead of pom? In other words, are there any artifacts published to https://sf-nexus.foobar.com/content/groups/global with only ivy.xml metadata and no pom metadata?
b
we'll make the assumption that no
e
If not, delete:
Copy code
<url name="nexus-ivy"
                 descriptor="required">
                <ivy pattern="<https://sf-nexus.foobar.com/content/groups/global/${web.ivy.pattern}>" />
                <artifact pattern="<https://sf-nexus.foobar.com/content/groups/global/${web.artifact.pattern}>" />
            </url>
b
(there is a bunchload of artifacts)
ok, I did it, then 
rm ~/.ivy2/pants -rf
then 
./pants clean-all
then my task …
still in error
e
Help me out - re-investigate the new .properties file
Oh wait
there is alot of repeat in that ivysettings
You have 4 things pointed at the same nexus
b
i rm the first ibiblio
e
Also delete:
Copy code
<ibiblio name="nexus-ivy-ibiblio"
                     descriptor="required"
                     root="<https://sf-nexus.foobar.com/content/groups/global/>" />
Basically you just want 1 resolver per repo
b
yes just did it
e
Just this one:
Copy code
<ibiblio name="nexus-maven-ibiblio"
                     m2compatible="true"
                     descriptor="required"
                     root="<https://sf-nexus.foobar.com/content/groups/global/>" />
Please paste the final ivysettings.xml
b
I rm the url ?
Copy code
<ivysettings>


    <!-- The ${login} and ${password} values come from a netrc_credentials() object in a BUILD


    -->
    <include file="${user.home}/.ivy2/credentials.xml" />
    <credentials host="<http://sf-nexus.mediarithmics.com|sf-nexus.mediarithmics.com>"
                 realm="Sonatype Nexus Repository Manager"
                 username="${nexus.login}"
                 passwd="${nexus.password}"/>

    <property name="local.pattern"
              value="[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"
              override="false" />


    <property name="web.ivy.pattern"
              value="[organisation]/[module]/[revision]/[type]s/[artifact].[ext]"
              override="false" />

    <property name="web.maven.pattern"
              value="[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"
              override="false" />

    <settings defaultResolver="chain"/>
    <resolvers>
        <chain name="chain" returnFirst="true">
             <url name="nexus-maven" m2compatible="true"
                 descriptor="required">
                <ivy pattern="<https://sf-nexus.mediarithmics.com/content/groups/global/${web.maven.pattern}>" />
                <artifact pattern="<https://sf-nexus.mediarithmics.com/content/groups/global/${web.maven.pattern}>" />
            </url>

            <ibiblio name="nexus-maven-ibiblio"
                     m2compatible="true"
                     descriptor="required"
                     root="<https://sf-nexus.mediarithmics.com/content/groups/global/>" />

            <filesystem name="local" m2compatible="true" local="true" checkmodified="true">
                <ivy pattern="${user.home}/.ivy2/local/${local.pattern}" />
                <artifact pattern="${user.home}/.ivy2/local/${local.pattern}" />
            </filesystem>
            <ibiblio name="public" m2compatible="true"
                     descriptor="required"/>
        </chain>

    </resolvers>
</ivysettings>
do I rm also nexus-maven ?
e
Yes
b
done
e
Central ivy lesson - just 1 resolver per repo
Unless you have a really bizarre setup
Now rms and clean-alls and try again
b
still in error 😕
Copy code
cat ~/.ivy2/pants/com.mediarithmics.pants/play-gen-tools/ivydata-0.0.1-SNAPSHOT.properties
#ivy cached data file for com.mediarithmics.pants#play-gen-tools;0.0.1-SNAPSHOT
#Fri Nov 16 17:17:10 CET 2018
artifact\:play-gen-tools\#pom.original\#pom\#-276547021.location=https\://sf-nexus.mediarithmics.com/content/groups/global/com/mediarithmics/pants/play-gen-tools/0.0.1-SNAPSHOT/play-gen-tools-0.0.1-SNAPSHOT.pom
artifact\:ivy\#ivy\#xml\#-1591587195.is-local=false
artifact\:ivy\#ivy\#xml\#-1591587195.location=https\://sf-nexus.mediarithmics.com/content/groups/global/com/mediarithmics/pants/play-gen-tools/0.0.1-SNAPSHOT/play-gen-tools-0.0.1-SNAPSHOT.pom
resolver=nexus-maven-ibiblio
artifact\:play-gen-tools\#pom.original\#pom\#-276547021.is-local=false
artifact\:play-gen-tools\#jar\#jar\#-1929109683.exists=true
artifact\:ivy\#ivy\#xml\#-1591587195.exists=true
artifact\:play-gen-tools\#pom.original\#pom\#-276547021.original=artifact\:play-gen-tools\#pom.original\#pom\#-276547021
artifact\:play-gen-tools\#pom.original\#pom\#-276547021.exists=true
artifact\:play-gen-tools\#jar\#jar\#-1929109683.location=https\://sf-nexus.mediarithmics.com/content/groups/global/com/mediarithmics/pants/play-gen-tools/0.0.1-SNAPSHOT/play-gen-tools-0.0.1-SNAPSHOT.jar
artifact\:ivy\#ivy\#xml\#-1591587195.original=artifact\:play-gen-tools\#pom.original\#pom\#-276547021
artifact.resolver=nexus-maven-ibiblio
artifact\:play-gen-tools\#jar\#jar\#-1929109683.is-local=false
artifact\:play-gen-tools\#jar\#jar\#-1929109683.original=artifact\:play-gen-tools\#jar\#jar\#-1929109683
e
Looks better - fetched a pom
Can you cat the pom?
b
Copy code
cat ~/.ivy2/pants/com.mediarithmics.pants/play-gen-tools/ivy-0.0.1-SNAPSHOT.xml
<?xml version="1.0" encoding="UTF-8"?>
<ivy-module version="2.0" xmlns:m="<http://ant.apache.org/ivy/maven>">
        <info organisation="com.mediarithmics.pants"
                module="play-gen-tools"
                revision="0.0.1-SNAPSHOT"
                status="integration"
                publication="20181115170621"
        >
                <description homepage="" />
        </info>
        <configurations>
                <conf name="default" visibility="public" description="runtime dependencies and master artifact can be used with this conf" extends="runtime,master"/>
                <conf name="master" visibility="public" description="contains only the artifact published by this module itself, with no transitive dependencies"/>
                <conf name="compile" visibility="public" description="this is the default scope, used if none is specified. Compile dependencies are available in all classpaths."/>
                <conf name="provided" visibility="public" description="this is much like compile, but indicates you expect the JDK or a container to provide it. It is only available on the compilation classpath, and is not transitive."/>
                <conf name="runtime" visibility="public" description="this scope indicates that the dependency is not required for compilation, but is for execution. It is in the runtime and test classpaths, but not the compile classpath." extends="compile"/>
                <conf name="test" visibility="private" description="this scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases." extends="runtime"/>
                <conf name="system" visibility="public" description="this scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository."/>
                <conf name="sources" visibility="public" description="this configuration contains the source artifact of this module, if any."/>
                <conf name="javadoc" visibility="public" description="this configuration contains the javadoc artifact of this module, if any."/>
                <conf name="optional" visibility="public" description="contains all optional dependencies"/>
        </configurations>
        <publications>
                <artifact name="play-gen-tools" type="jar" ext="jar" conf="master"/>
                <artifact name="play-gen-tools" type="source" ext="jar" conf="sources" m:classifier="sources"/>
        </publications>
        <dependencies>
                <dependency org="com.typesafe.play" name="routes-compiler_2.12" rev="2.6.20" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
                <dependency org="com.typesafe.play" name="twirl-compiler_2.12" rev="1.3.15" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
                <dependency org="com.github.scopt" name="scopt_2.12" rev="3.7.0" force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
        </dependencies>
</ivy-module>
e
no
b
looks better indeed
what ?
e
.original
Thats an ivy.xml not pom
But yes, looking better
b
Copy code
cat ~/.ivy2/pants/com.mediarithmics.pants/play-gen-tools/ivy-0.0.1-SNAPSHOT.xml.original 
<?xml version="1.0" encoding="UTF-8"?>

<!-- generated by pants! <http://pantsbuild.github.io/> -->
<project xmlns="<http://maven.apache.org/POM/4.0.0>"
         xmlns:xsi="<http://www.w3.org/2001/XMLSchema-instance>"
         xsi:schemaLocation="<http://maven.apache.org/POM/4.0.0>
                             <http://maven.apache.org/maven-v4_0_0.xsd%22>|http://maven.apache.org/maven-v4_0_0.xsd">>

  <modelVersion>4.0.0</modelVersion>

  <groupId>com.mediarithmics.pants</groupId>
  <artifactId>play-gen-tools</artifactId>
  <packaging>jar</packaging>
  <version>0.0.1-SNAPSHOT</version>
  <dependencies>
    <dependency>
      <groupId>com.typesafe.play</groupId>
      <artifactId>routes-compiler_2.12</artifactId>
      <version>2.6.20</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>com.typesafe.play</groupId>
      <artifactId>twirl-compiler_2.12</artifactId>
      <version>1.3.15</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>com.github.scopt</groupId>
      <artifactId>scopt_2.12</artifactId>
      <version>3.7.0</version>
      <scope>compile</scope>
    </dependency>
  </dependencies>
</project>
e
Excellent
So - that was one of your problems
b
yay
e
Now lets pop up the stack to the current ./pants server output
A screenshot was useful before
b
image.png
take a lot of time to upload …
e
What does the cachepath show?
Perhaps 
jq . [the json file]
b
Seems I have a flaky connection on my computer
e
OK. Let me know when you have the cachepath contents.
b
sorry ! I'm back
Copy code
cat /home/lorilan/dev/mediarithmics-platform-2/.pants.d/bootstrap/bootstrap-jvm-tools/7345c2790d72/ivy/b62ecdedc3bb2ee04966616bc517f744a1f4d027-IvyResolveFingerprintStrategy_5b120e9b2989/resolution.json 
{"default": {"target_to_coords": {"//:play-gen-tools": ["com.mediarithmics.pants:play-gen-tools:0.0.1-SNAPSHOT"]}, "coord_to_attrs": {"com.mediarithmics.pants:play-gen-tools:0.0.1-SNAPSHOT": {}}}}
e
looks bad - 1 item
b
indeed
maybe still a red herring but here is the fetch-ivy.xml in the directory :
Copy code
cat fetch-ivy.xml 
<?xml version="1.0"?>

<!--
Copyright 2016 Pants project contributors (see CONTRIBUTORS.md).
Licensed under the Apache License, Version 2.0 (see LICENSE).
-->

<!-- generated by pants! -->
<ivy-module version="2.0"
            xmlns:xsi="<http://www.w3.org/2001/XMLSchema-instance>"
            xsi:noNamespaceSchemaLocation="<http://ant.apache.org/ivy/schemas/ivy.xsd>"
            xmlns:m="<http://ant.apache.org/ivy/maven>">

  <info organisation="internal" module="b62ecdedc3bb2ee04966616bc517f744a1f4d027-IvyResolveFingerprintStrategy_5b120e9b2989-fetch"/>

  <configurations>
    <conf name="default"/>
  </configurations>

  <dependencies>
    <dependency org="com.mediarithmics.pants"
                name="play-gen-tools"
                rev="0.0.1-SNAPSHOT"
                force="true"
                transitive="false"
                
    >
      <conf name="default" mapped="default"/>
      <artifact
        name="play-gen-tools"
        type="jar"
        ext="jar"
        
        
      />
    </dependency>
    <conflict manager="all"></conflict>
  </dependencies>
</ivy-module>
sure the transitive="false" is normal ?
e
Yes fetch-* is not relevant to resolve
b
ok
not sure what additional info I can give you
e
Hrm. A bit stumped. I want to confirm after the ivysettings.xml fixes, you did in fact nuke ~/.ivy2/pants and you ran a ./pants clean-all and can confirm that wiped out .pants.d/?
b
I just ran 
rm -rf ~/.ivy2/pants && ./pants clean-all && ./pants bundle job-management-services/job-front:dist
still failure
and yes
e
And the .original in ~/.ivy2/pants is a pom with deps still like we expect?
b
after ./pants clean-all .pants.d is empty
yes 😕
e
Ok - still thinking
What does this say?:
Copy code
grep play-gen-tools ~/.ivy2/pants/*.{xml,properties}
b
Copy code
grep play-gen-tools ~/.ivy2/pants/*.{xml,properties}
/home/lorilan/.ivy2/pants/internal-b62ecdedc3bb2ee04966616bc517f744a1f4d027-IvyResolveFingerprintStrategy_5b120e9b2989-fetch-default.xml:               <module organisation="com.mediarithmics.pants" name="play-gen-tools">
/home/lorilan/.ivy2/pants/internal-b62ecdedc3bb2ee04966616bc517f744a1f4d027-IvyResolveFingerprintStrategy_5b120e9b2989-fetch-default.xml:                               <metadata-artifact status="successful" details="" size="1144" time="62" location="/home/lorilan/.ivy2/pants/com.mediarithmics.pants/play-gen-tools/ivy-0.0.1-SNAPSHOT.xml" searched="true" original-local-location="/home/lorilan/.ivy2/pants/com.mediarithmics.pants/play-gen-tools/ivy-0.0.1-SNAPSHOT.xml.original" origin-is-local="false" origin-location="<https://sf-nexus.mediarithmics.com/content/groups/global/com/mediarithmics/pants/play-gen-tools/0.0.1-SNAPSHOT/play-gen-tools-0.0.1-SNAPSHOT.pom>"/>
/home/lorilan/.ivy2/pants/internal-b62ecdedc3bb2ee04966616bc517f744a1f4d027-IvyResolveFingerprintStrategy_5b120e9b2989-fetch-default.xml:                                       <artifact name="play-gen-tools" type="jar" ext="jar" status="successful" details="" size="54397" time="80" location="/home/lorilan/.ivy2/pants/com.mediarithmics.pants/play-gen-tools/jars/play-gen-tools-0.0.1-SNAPSHOT.jar">
/home/lorilan/.ivy2/pants/internal-b62ecdedc3bb2ee04966616bc517f744a1f4d027-IvyResolveFingerprintStrategy_5b120e9b2989-fetch-default.xml:                                               <origin-location is-local="false" location="<https://sf-nexus.mediarithmics.com/content/groups/global/com/mediarithmics/pants/play-gen-tools/0.0.1-SNAPSHOT/play-gen-tools-0.0.1-SNAPSHOT.jar>"/>
/home/lorilan/.ivy2/pants/resolved-internal-b62ecdedc3bb2ee04966616bc517f744a1f4d027-IvyResolveFingerprintStrategy_5b120e9b2989-fetch-working@hyrule.xml:    <dependency org="com.mediarithmics.pants" name="play-gen-tools" rev="0.0.1-SNAPSHOT" force="true" transitive="false">
/home/lorilan/.ivy2/pants/resolved-internal-b62ecdedc3bb2ee04966616bc517f744a1f4d027-IvyResolveFingerprintStrategy_5b120e9b2989-fetch-working@hyrule.xml:      <artifact name="play-gen-tools" type="jar" ext="jar"/>
/home/lorilan/.ivy2/pants/resolved-internal-b62ecdedc3bb2ee04966616bc517f744a1f4d027-IvyResolveFingerprintStrategy_5b120e9b2989-fetch-working@hyrule.properties:+organisation\:\#@\#\:+com.mediarithmics.pants\:\#@\#\:+branch\:\#@\#\:+@\#\:NULL\:\#@\:\#@\#\:+module\:\#@\#\:+play-gen-tools\:\#@\#\:+revision\:\#@\#\:+0.0.1-SNAPSHOT\:\#@\#\:=0.0.1-SNAPSHOT integration 0.0.1-SNAPSHOT null
e
OK, how about: 
tree .pants.d/resolve/ivy/current/ivy/
b
Copy code
tree .pants.d/resolve/ivy/current/ivy/                
.pants.d/resolve/ivy/current/ivy/ [error opening dir]

0 directories, 0 files
resolve does not exist
e
Um...
So, let's try this - turn off coursier in pants.ini
s/coursier/ivy/
Then - once more - just a ./pants clean-all since the ~/.ivy2/pants cache looks good now
You use .pants.d right - you're not trying to setup custom pants workdirs or anything like that?
b
no no, regular .pants.d
still failure 😕
e
k
b
still no resolve directory
e
The empty .pants.d/resolve/ivy is not correct
b
Copy code
ls -a .pants.d
.  ..  bootstrap  build_invalidator  gen  jvm-platform-validate  link  logs  native-compile  ng  reports  run-tracker  unpack-jars
e
huh
What happens if you 
./pants resolve some/3rdparty/jar:target
?
Do you then get a .pants.d/resolve/... dir?
b
take some time
Copy code
ls .pants.d/resolve 
ivy
there is one directory
e
and under that?
tree it please
b
log.txt
log.txt
tree .pants.d/resolve > log.txt
e
OK - so resolving via ivy works, jvm tool bootstrapping appears to no-op though.
Ah - reading your ./pants server screen shot more closely, it looks like you use an artifact cache in 
.cache
IIRC all bootstrapped tools are cached in the artifact cache post resolve
b
ok
e
So ... please, 
./pants clean-all && rm -rf .cache
Right now, editing ivysettings.xml does not invalidate the artifact cache unfortunately
So once you had a bad resolve due to a bad ivysettings cached, it wouldn't uncache itself. Lots of cache layers!
b
lol
so I have a different error now
Copy code
18:06:23 00:01       [cache] 
                   No cached artifacts for 1 target.
                   Invalidated 1 target.. .. ..
18:06:23 00:01       [execute]
               Waiting for background workers to finish.
18:06:31 00:09   [complete]
               FAILURE
timestamp: 2018-11-16T18:06:31.734063
Exception caught: (exceptions.TypeError) (backtrace omitted)
Exception message: 'NoneType' object is not iterable
weirdly I feel happy about this new error
e
If you do not, you should probably have:
Copy code
[GLOBAL]
print_exception_stacktrace: True
in pants.ini
While working on a plugin
b
Copy code
Exception caught: (exceptions.TypeError)
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/bin/pants", line 11, in <module>
    sys.exit(main())
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 71, in main
    PantsLoader.run()
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 67, in run
    cls.load_and_execute(entrypoint)
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 60, in load_and_execute
    entrypoint_main()
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/bin/pants_exe.py", line 39, in main
    PantsRunner(exiter, start_time=start_time).run()
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/bin/pants_runner.py", line 62, in run
    return runner.run()
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 158, in run
    self._run()
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 222, in _run
    goal_runner_result = self._maybe_run_v1(run_tracker, reporting)
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 175, in _maybe_run_v1
    return goal_runner_factory.create().run()
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 204, in run
    return self._run_goals()
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 175, in _run_goals
    result = self._execute_engine()
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 163, in _execute_engine
    result = engine.execute(self._context, self._goals)
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/engine/legacy_engine.py", line 26, in execute
    self.attempt(context, goals)
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/engine/round_engine.py", line 233, in attempt
    goal_executor.attempt(explain)
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/engine/round_engine.py", line 49, in attempt
    task.execute()
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/task/simple_codegen_task.py", line 240, in execute
    sources = self._capture_sources((key,))[0]
  File "/home/lorilan/.cache/pants/setup/bootstrap-Linux-x86_64/1.11.0rc1/lib/python2.7/site-packages/pants/task/simple_codegen_task.py", line 292, in _capture_sources
    buildroot_relative_globs = tuple(os.path.join(results_dir_relpath, file) for file in files)
seems like my upgrade from 1.9 to 1.11 was not as smooth as I thought
e
OK - so now you're failing in your plugin code with - presumably a bootstrapped tool
Which is progress past the resolution issues. I think we can call those issues solved, but I'd like to confirm you understand what was wrong and what we fixed.
b
ok I downgraded to 1.9
back to my original setup
no coursier
I suspect it will take some time
e
Well, you can probably turn coursier back on. The only issue in all of that was the bad ivysettings.xml and then not cleaning enough caches after fixing that
b
ok so it compiles a lot more things
the suspense is building
e
Well, nuking .cache will force the zinc scala compiler shims to get rebuilt for one and that is slow
b
yup
still compiling
it works !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
e
Excellent.
b
thank you so very much !!!
e
I want to underscore the central pain here is using coursier a new pants user can get far down the road with things working only to later get super confusing errors because ivysettings is not set up equivalently to coursier.
b
pfiu seriously you are (and the pants slack) so helpful
yeah … but it is worth the pain 😛
e
Well, just pay it back by open sourcing the plugins you work so hard on
b
yes, I'll try to help. First I'll need to update a bit my python game but I will definitively help
4 Views
Open in Slack
PreviousNext
Powered by