https://pantsbuild.org/ logo
Join Slack
Powered by
howdy friends :wave: is there any way to add exclu...
# general
w
howdy friends 👋 is there any way to add excludes to jvm artifact resolution currently? I seem to be hitting an issue around coursier resolution of a specific transitive jar (the coursier json report is listing 
"file": null
for it which the post processing script seems to be choking on)
full output, for the curious
Untitled.txt
and the coursier report json, the bugged part seeming to be
Copy code
{
            "coord": "javax.media:jai_core:1.1.3",
            "file": null,
            "directDependencies":
            [],
            "dependencies":
            []
        },
Untitled.js
h
cc @fast-nail-55400 @witty-crayon-22786 @ancient-vegetable-10556
f
That hasn’t been implemented yet. https://github.com/pantsbuild/pants/issues/13683
h
Hmm, either way we shouldn't be choking on processing the JSON. Or at least not in this way. That seems like an easy fix.
Just automatically ignoring this broken artifact would solve the immediate problem I assume?
@wonderful-yak-59945 would you mind filing a bug report for this at https://github.com/pantsbuild/pants/issues/new/choose ?
👍 1
a
Depends on how that transitive dependency is provided later on. Is there an actual artifact in place?
h
I can take a look today
And also, could you mention this case in https://github.com/pantsbuild/pants/issues/13683 so we know to prioritize it?
Hmm good point @ancient-vegetable-10556, something needs that dep, so excluding may just lead to trouble later on
a
I see that it’s a 
javax
package; I don’t suppose it’s something that comes with a default distribution or something?
h
Ah
Well, it doesn't come from Maven Central! https://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/
The jar isn't there
So the coursier json output makes sense (and we should handle this case with a warning I guess?)
a
This GH issue offers an alternative maven repo to satisfy the dependency: https://github.com/akhikhl/gretty/issues/322
I do not recall whether we have a way to specify alternate maven repos
h
Presumably in the coursier config?
a
Yeah, I am not 100% sure if we have a passthrough for that
let me go looking
w
there is, that specific javax jar is getting pulled in by a geotools jar out of 
<https://repo.osgeo.org/repository/release>
a
Yup!
OK, presently we do not have a way to pass-through the 
COURSIER_REPOSITORIES
envvar into our 
CoursierWrapperProcess
invocations, but it should be easy enough to do.
@happy-kitchen-89482 did you want to take this one on, or did you want me to? I can point you at the right bits of the code, since I touched it last
h
This is done via env var? There is no coursier config file?
There are two things to do here - fix the JSON handling error, and support custom repositories
w
thanks for digging!
a
@happy-kitchen-89482 I do not believe we have a coursier config file
HANG ON
🤦
reads through the actual issue here
😂 1
@wonderful-yak-59945 Can you see what happens if you put the 
osgeo
repo ahead of maven-central in your config file?
w
hrm same result it looks like
that is with
Copy code
[coursier]
repos = [
  "<https://repo.osgeo.org/repository/release>",
  "<https://maven-central.storage-download.googleapis.com/maven2>",
  "<https://repo1.maven.org/maven2>",
]
a
OK, that was worth checking at least 🙂
w
I feel like I'm constantly needing to sanity check everything any time jvm dependency resolves are involved lol
a
maven is… Not Very Good, tbqh
💯 1
w
so far coursier seems significantly more pleasant than ivy at least!
I'll be in and out of meetings today, but will keep an eye on this thread if there's anything else I can do to help
a
Until now, it’s worked reasonably for us!
My working hypothesis here is that Coursier isn’t handling this situation where multiple repos supply 
javax.media:jai-tools
h
That is confusing, I would have thought putting it ahead would work
@witty-crayon-22786 thoughts here?
a
@happy-kitchen-89482 I’m working on reproducing this — it looks like we’re not overriding the default coursier repos
👍 1
so whatever we specify comes after the user’s local 
ivy2
cache and 
<http://repo1.maven.org|repo1.maven.org>
h
Aha! Good find!
a
I’ve added comments to the github thread. There’ll be a fix PR incoming
🙏 1
w
@wonderful-yak-59945: which version of Pants are you all evaluating? things are still changing on an ongoing basis, so 2.10.x is relatively stable, but will not contain multiple-scala-version or multiple-jdk-version support (which are in 2.11.x)
p
I think you meant to ping @wonderful-yak-59945 🙂
🙇 1
w
deerp. sorry about that
w
hi! we're set up with 2.9.0 currently but probably easy enough to bump that to a higher version
we shouldn't need multiple jdk/scala version support for the moment, though that is certainly something that would be relevant down the road
w
got it: thanks. i think that we can probably start by cherry-picking only to 
2.10.x
then, which should stabilize in the next week or so
👍 1
a
@witty-crayon-22786/@happy-kitchen-89482 here’s a CP to 2.10.x. Enjoy: https://github.com/pantsbuild/pants/pull/14584
w
happy to pull down an rc tag to test in the meantime as well
thanks for the help all!
a
@wonderful-yak-59945 You can run 
pants_from_sources
against the 2.10.x branch if you like
👍 1
the CP is there now
Thanks for finding us a weirdness!
w
always happy to find weirdness!
w
you can also consume the SHA via https://www.pantsbuild.org/v2.10/docs/installation#running-pants-from-unreleased-builds once the branch finishes building
with sha 
781bb1bad0916eb33d3a2998675ee9e563aa4f10
… which appears to be available now, actually.
w
ah so! under the fix I still seem to be running into the same issue if I have more than just the 
<https://repo.osgeo.org/repository/release>
repo listed in our coursier config. seems like the "coursier eagerly choosing a bad pom from repo1.maven.org" problem persists if that repo is specified in the pants config
but if I remove those from our config and only specify the osgeo repo, the 
--no-default
patch looks to do its job and pants can download and cache the jar (and subsequent resolves consume that just fine when the other repos are added back)
h
huh, with that osgeo repo ahead of the maven repo in the config?
I think order matters
w
seems like it?
the coursier command does have osgeo first but apparently still didn't choose it? ¯\_(ツ)_/¯
Untitled.txt
I'm at least unblocked having the resolve now cached though
h
That's good, sorry for the trouble though, we'll figure this out.
w
all good!
f
If it were the Pikachu jar, then Coursier probably would have worked. (Because “Pikachu! I (Coursier) choose you!“)
w
lol
I've missed the bad puns
💯 2
h
Hmm, so if I run that coursier command outside Pants:
Copy code
cs  fetch -r=<https://repo.osgeo.org/repository/release> -r=<https://maven-central.storage-download.googleapis.com/maven2> -r=<https://repo1.maven.org/maven2> --no-default --json-output-file=coursier_report.json org.geotools:gt-main:9.2 com.google.geometry:s2-geometry:2.0.0 com.twitter:util-logging_2.11:20.1.0 org.mongodb:bson:3.11.2 joda-time:joda-time:2.10.3 com.fasterxml.jackson.core:jackson-core:2.6.7 com.twitter:util-core_2.11:20.1.0 com.fasterxml.jackson.core:jackson-databind:2.6.7 com.vividsolutions:jts:1.12 com.fasterxml.jackson.module:jackson-module-scala_2.11:2.6.7.1 org.apache.thrift:libthrift:0.9.0 org.codehaus.jackson:jackson-core-asl:1.9.8
It works fine.
So now I'm wondering if there's a race of some kind and order of repos doesn't actually matter
Or, possibly the problem was that the pom was cached via the maven central location, so coursier still expected the jar to come from there too.
@ancient-vegetable-10556 where does Pants place the coursier download cache, is it ~/.cache/pants/coursier?
@wonderful-yak-59945 maybe try temporarily moving that dir (once Chris confirms it as the cache location) and see if re-resolving without cache works?
w
~/.cache/pants/named_caches/coursier/
maybe?
trying that
hm still generates the 
"file":null
json and breaks the post processing script
I am gonna call it for the night but happy to poke more tomorrow
h
Something to try: In a tmpdir install coursier (
brew install coursier/formulas/coursier
and 
cs setup
) and then try the raw 
cs fetch
command I pased above
after nuking 
~/Library/Caches/Coursier
So we can see if the problem happens on your system/network even without Pants in the mix
a
I blew away 
~/.cache/pants
in its entirety (just to be certain ;)), using just the two repos you had configured. It might be worth seeing if it produces the same problematic resolve if you only have osgeo in place
I’m starting to think this is going to be a thing that needs an upstream fix in Coursier, to handle that bogus POM from maven.org…
@wonderful-yak-59945 One other thing I might suggest as a stop-gap is that you provide the 
jai_core
JAR as an explicit 
jvm_artifact
dependency (using the 
jar=
field to a JAR in your repo, or the 
url=
field pointing to the osgeo.org URL for the JAR), and configuring that as an explicit dependency of at least one java/scala source. That will pass the JAR to Coursier explicitly, and it won’t go looking for it from a remote maven repo.
h
I'm still not sure why this works when I run the coursier command directly on my system, but doesn't work when Pants runs the same coursier command on Jacob's system
Is the hypothesis "race condition" ?
a
I haven’t gone through the logs — is there any indication that Coursier is actually attempting a download on Jacob’s logs?
Doesn’t look like it 😕
h
He posted a snippet above where the coursier fetch command leads to that JSON error (so coursier thinks it has resolved correctly from the bad repo)
I ran that snippet with a locally installed 
cs
and empty caches and it succeeded
a
It didn’t download anything
stderr would show logs of trying to download from relevant repos, but there are none
which means it’s re-using a historical resolve
(cache poisoning! woo!)
A log for downloading a non-existent JAR, when there’s an actual download taking place will look like this:
Copy code
/private/var/folders/0m/h2n902qn38b8555xq8z0w_h40000gn/T/process-executionaHl9Ym/.cache/jdk/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar

stderr:
+ coursier_exe=__coursier/./cs-x86_64-apple-darwin
+ shift
+ json_output_file=coursier_report.json
+ shift
++ pwd
+ working_dir=/private/var/folders/0m/h2n902qn38b8555xq8z0w_h40000gn/T/process-executionaHl9Ym
+ __coursier/./cs-x86_64-apple-darwin fetch -r=<https://repo.osgeo.org/repository/geotools-releases> -r=<https://maven-central.storage-download.googleapis.com/maven2> --json-output-file=coursier_report.json org.scala-lang:scala-library:2.13.6 javax.media:jai_core:1.1.3
Downloading <https://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/jai_core-1.1.3.pom>
Downloading <https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.pom>
Downloaded <https://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/jai_core-1.1.3.pom>
Downloaded <https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.pom>
Downloading <https://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/jai_core-1.1.3.jar>
Downloading <https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar>
Downloaded <https://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/jai_core-1.1.3.jar>
Downloading <https://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/jai_core-1.1.3.jar.sha1>
Downloaded <https://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/jai_core-1.1.3.jar.sha1>
Downloaded <https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar>
+ /bin/mkdir -p classpath
+ /Users/chrisjrn/.pyenv/versions/3.8.12/bin/python __coursier/coursier_post_processing_script.py coursier_report.json
Traceback (most recent call last):
  File "__coursier/coursier_post_processing_script.py", line 13, in <module>
    source = PurePath(dep['file'])
  File "/Users/chrisjrn/.pyenv/versions/3.8.12/lib/python3.8/pathlib.py", line 651, in __new__
    return cls._from_parts(args)
  File "/Users/chrisjrn/.pyenv/versions/3.8.12/lib/python3.8/pathlib.py", line 683, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/Users/chrisjrn/.pyenv/versions/3.8.12/lib/python3.8/pathlib.py", line 667, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
That’s is from yesterday — note all the `Downloading`/`Downloaded` lines. Those don’t appear in stderr if Coursier is using its cache
(I suspect this means we aren’t modelling Coursier properly — we may need to have a specific cache based on the configured repos)
h
OK, so the hypothesis is cache poisoning
a
Yes.
The minimal test case would be to attempt to resolve an artifact from 
<http://osgeo.org|osgeo.org>
that has 
jai_core
as a dependency with 
<http://repo1.maven.org|repo1.maven.org>
as the first repo argument, and then re-attempt with 
<http://osgeo.org|osgeo.org>
as the first repo argument, without blowing away the cache
h
Hmm, when I run outside of Pants, I don't see "Downloading/Downloaded", I see download bars
Copy code
<https://maven-central.storage-download.googleapis.com/maven2/org/apache/httpcomponents/httpcomponents-core/4.1.4/httpcomponents-core-4.1.4.pom>
  100.0% [##########] 9.5 KiB (63.7 KiB / s)
etc.
a
If you pipe stderr to a file, I suspect you’ll see flatter output
h
aha makes sense
let me see if I can recreate the cache poisoning outside Pants
Yep!!
That is it
nice deduction Chris
a
OK.
So yeah, @wonderful-yak-59945, you’ll need to blow away your pants cache for the moment
We’ve got a fix to report upstream, and also we’ll need to change how we model Coursier’s cache
@happy-kitchen-89482 Can you write up a ticket with your instructions; I know what changes will need to be made to the cache behaviour, and will be able to write the fix and tests, but you’ve got better info of how to reproduce.
h
So this is basically a coursier bug
a
Right, but it’s one that we can control for, as long as we key the archive cache based on the repos configuration
So the good news is that Pants controls the location of the Coursier archive cache (i.e. where all the maven repos get dumped to)
So I suspect if we start making the 
COURSIER_ARCHIVE_CACHE
value determined by the values of our repositories config, the fix I proposed yesterday should just work
™
h
makes sense!
w
ooh awesome, thanks for the further investigation. sounds like the immediate workaround for us then is: 1. add an explicit jvm_artifact for the problematic jars, and point their 
url
at the correct repo manually 2. blow away 
~/.cache/pants
(and 
~/Library/Caches/Coursier
?) before re-running the resolve is that roughly accurate?
a
@wonderful-yak-59945 Try #2 first, but if that doesn’t help, try #1
👍 1
Looks like we’re going to be able to make caching dependent on our 
--coursier-repos
configuration, without needing to re-download JDKs every time we change the repos configuration. So Pants will, at the very least, be insulated against the cache poisoning
https://github.com/pantsbuild/pants/pull/14603 (@happy-kitchen-89482 @fast-nail-55400)
f
ship it
a
Can I get a review on the cp branch? https://github.com/pantsbuild/pants/pull/14606/files
f
done
a
@wonderful-yak-59945 You can point Pants at 50937c0f0b2192a1fa85aa63d3f08affe1547328 now, which has the cache fixes in place. Enjoy!
w
awesome, thank you!
a
Thanks for the frequent feedback!
Finally, to close this loop, I’ve reported this upstream to Coursier, we’ll see what they have to say about it.
Open in Slack
PreviousNext
Powered by