https://pantsbuild.org/ logo
Powered by
#general
  • h

    happy-kitchen-89482

    02/24/2022, 7:14 AM
    Something to try: In a tmpdir install coursier (
    brew install coursier/formulas/coursier
    and 
    cs setup
    ) and then try the raw 
    cs fetch
    command I pased above
  • h

    happy-kitchen-89482

    02/24/2022, 7:14 AM
    after nuking 
    ~/Library/Caches/Coursier
  • h

    happy-kitchen-89482

    02/24/2022, 7:14 AM
    So we can see if the problem happens on your system/network even without Pants in the mix
  • f

    flat-zoo-31952

    02/24/2022, 3:52 PM
    I'm trying to brainstorm to leverage Pants' dep inference + explicit dep modelling, and I'm wondering if it's possible to run some graph algorithm type stuff on this info? An example: to find the betweenness-centrality to find the most central dependencies. Is there a target type that I could 
    Get
    to use this as an input to this calculation?
    b
    h
    +2
    • 5
    • 35
  • a

    ancient-vegetable-10556

    02/24/2022, 4:11 PM
    I blew away 
    ~/.cache/pants
    in its entirety (just to be certain ;)), using just the two repos you had configured. It might be worth seeing if it produces the same problematic resolve if you only have osgeo in place
  • a

    ancient-vegetable-10556

    02/24/2022, 4:31 PM
    I’m starting to think this is going to be a thing that needs an upstream fix in Coursier, to handle that bogus POM from maven.org…
  • a

    acoustic-librarian-3937

    02/24/2022, 4:31 PM
    Might be a simple question but anyone know what 
    ERROR: Links are not allowed as constraints
    means?
    h
    e
    h
    • 4
    • 29
  • a

    ancient-vegetable-10556

    02/24/2022, 4:32 PM
    @wonderful-yak-59945 One other thing I might suggest as a stop-gap is that you provide the 
    jai_core
    JAR as an explicit 
    jvm_artifact
    dependency (using the 
    jar=
    field to a JAR in your repo, or the 
    url=
    field pointing to the osgeo.org URL for the JAR), and configuring that as an explicit dependency of at least one java/scala source. That will pass the JAR to Coursier explicitly, and it won’t go looking for it from a remote maven repo.
  • h

    happy-kitchen-89482

    02/24/2022, 4:47 PM
    I'm still not sure why this works when I run the coursier command directly on my system, but doesn't work when Pants runs the same coursier command on Jacob's system
  • h

    happy-kitchen-89482

    02/24/2022, 4:47 PM
    Is the hypothesis "race condition" ?
  • a

    ancient-vegetable-10556

    02/24/2022, 4:48 PM
    I haven’t gone through the logs — is there any indication that Coursier is actually attempting a download on Jacob’s logs?
  • p

    polite-garden-50641

    02/24/2022, 4:49 PM
    FYI there is a readm.com outage and that impacts the pants doc site. https://www.readmestatus.com/
    ⤵️ 2
  • a

    ancient-vegetable-10556

    02/24/2022, 4:49 PM
    Doesn’t look like it 😕
  • h

    happy-kitchen-89482

    02/24/2022, 4:49 PM
    He posted a snippet above where the coursier fetch command leads to that JSON error (so coursier thinks it has resolved correctly from the bad repo)
  • h

    happy-kitchen-89482

    02/24/2022, 4:50 PM
    I ran that snippet with a locally installed 
    cs
    and empty caches and it succeeded
  • h

    happy-kitchen-89482

    02/24/2022, 4:50 PM
  • a

    ancient-vegetable-10556

    02/24/2022, 4:50 PM
    It didn’t download anything
  • a

    ancient-vegetable-10556

    02/24/2022, 4:51 PM
    stderr would show logs of trying to download from relevant repos, but there are none
  • a

    ancient-vegetable-10556

    02/24/2022, 4:51 PM
    which means it’s re-using a historical resolve
  • a

    ancient-vegetable-10556

    02/24/2022, 4:51 PM
    (cache poisoning! woo!)
  • a

    ancient-vegetable-10556

    02/24/2022, 4:52 PM
    A log for downloading a non-existent JAR, when there’s an actual download taking place will look like this:
    Copy code
    /private/var/folders/0m/h2n902qn38b8555xq8z0w_h40000gn/T/process-executionaHl9Ym/.cache/jdk/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar

stderr:
+ coursier_exe=__coursier/./cs-x86_64-apple-darwin
+ shift
+ json_output_file=coursier_report.json
+ shift
++ pwd
+ working_dir=/private/var/folders/0m/h2n902qn38b8555xq8z0w_h40000gn/T/process-executionaHl9Ym
+ __coursier/./cs-x86_64-apple-darwin fetch -r=<https://repo.osgeo.org/repository/geotools-releases> -r=<https://maven-central.storage-download.googleapis.com/maven2> --json-output-file=coursier_report.json org.scala-lang:scala-library:2.13.6 javax.media:jai_core:1.1.3
Downloading <https://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/jai_core-1.1.3.pom>
Downloading <https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.pom>
Downloaded <https://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/jai_core-1.1.3.pom>
Downloaded <https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.pom>
Downloading <https://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/jai_core-1.1.3.jar>
Downloading <https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar>
Downloaded <https://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/jai_core-1.1.3.jar>
Downloading <https://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/jai_core-1.1.3.jar.sha1>
Downloaded <https://repo1.maven.org/maven2/javax/media/jai_core/1.1.3/jai_core-1.1.3.jar.sha1>
Downloaded <https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.6/scala-library-2.13.6.jar>
+ /bin/mkdir -p classpath
+ /Users/chrisjrn/.pyenv/versions/3.8.12/bin/python __coursier/coursier_post_processing_script.py coursier_report.json
Traceback (most recent call last):
  File "__coursier/coursier_post_processing_script.py", line 13, in <module>
    source = PurePath(dep['file'])
  File "/Users/chrisjrn/.pyenv/versions/3.8.12/lib/python3.8/pathlib.py", line 651, in __new__
    return cls._from_parts(args)
  File "/Users/chrisjrn/.pyenv/versions/3.8.12/lib/python3.8/pathlib.py", line 683, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/Users/chrisjrn/.pyenv/versions/3.8.12/lib/python3.8/pathlib.py", line 667, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
    That’s is from yesterday — note all the `Downloading`/`Downloaded` lines. Those don’t appear in stderr if Coursier is using its cache
  • a

    ancient-vegetable-10556

    02/24/2022, 4:54 PM
    (I suspect this means we aren’t modelling Coursier properly — we may need to have a specific cache based on the configured repos)
  • h

    happy-kitchen-89482

    02/24/2022, 4:55 PM
    OK, so the hypothesis is cache poisoning
  • a

    ancient-vegetable-10556

    02/24/2022, 4:55 PM
    Yes.
  • a

    ancient-vegetable-10556

    02/24/2022, 4:57 PM
    The minimal test case would be to attempt to resolve an artifact from 
    <http://osgeo.org|osgeo.org>
    that has 
    jai_core
    as a dependency with 
    <http://repo1.maven.org|repo1.maven.org>
    as the first repo argument, and then re-attempt with 
    <http://osgeo.org|osgeo.org>
    as the first repo argument, without blowing away the cache
  • h

    happy-kitchen-89482

    02/24/2022, 4:57 PM
    Hmm, when I run outside of Pants, I don't see "Downloading/Downloaded", I see download bars
  • h

    happy-kitchen-89482

    02/24/2022, 4:57 PM
    Copy code
    <https://maven-central.storage-download.googleapis.com/maven2/org/apache/httpcomponents/httpcomponents-core/4.1.4/httpcomponents-core-4.1.4.pom>
  100.0% [##########] 9.5 KiB (63.7 KiB / s)
  • h

    happy-kitchen-89482

    02/24/2022, 4:57 PM
    etc.
  • a

    ancient-vegetable-10556

    02/24/2022, 4:57 PM
    If you pipe stderr to a file, I suspect you’ll see flatter output
  • h

    happy-kitchen-89482

    02/24/2022, 4:58 PM
    aha makes sense
1...226722682269...2420Latest