Hi here, I’m trying to ship a few PRs (9226, 9227, 9228) but I’m getting no dice on the CI jobs: It seems that some of the python integration tests time out despite the changes affecting scala code. I tried re-running a few times without much luck. Here is an example subset of the output:

12:12:58 [INFO] Tests failed: contrib/awslambda/python/tests/python/pants_test/contrib/awslambda/python
12:13:16 [INFO] Tests failed: tests/python/pants_test/backend/python/tasks:python_repl_integration
12:13:24 [INFO] Tests failed: contrib/go/tests/python/pants_test/contrib/go/subsystems:integration
12:13:30 [INFO] Tests failed: tests/python/pants_test/projects:python_testprojects_integration
12:14:26 [INFO] Tests failed: tests/python/pants_test/backend/python/tasks:pytest_run_integration
12:14:57 [INFO] Tests failed: tests/python/pants_test/backend/python/tasks:python_run_integration
12:15:00 [INFO] Tests failed: tests/python/pants_test/core_tasks:substitute_target_aliases_integration
12:15:40 [INFO] Tests failed: tests/python/pants_test/backend/jvm/tasks:jvm_bundle_integration

(for instance https://travis-ci.com/pantsbuild/pants/jobs/293825763#L2946) Running

./pants test

for any of these targets locally seems to be fine (although I had to

./pants clean-all

first, apparently due to some weird interaction between the PEX upgrade and caching). Are we aware of what is causing this flakyness and do we know of any workaround. Alternatively, is there anything we could revert to make it easier again to contribute code to pants until the situation is improved?

@witty-crayon-22786 / @average-vr-56795: is there any blockers remaining for https://github.com/pantsbuild/pants/pull/8132?

Hi, Pants build works fine in my local machine. When I try to run it through Jenkins, I get the below error.

+ ./pants binary reaper/src/python/:main
[31mNo valid Python interpreter found. Pants requires Python 3.6+ to run.[0m
Build step 'Execute shell' marked build as failure
Finished: FAILURE

How do I fix this? By running it in a docker container with Python 3.6+ installed? Is there a way out without using docker? Thanks!

Hi, yes, you need to ensure that Python 3.6+ is discoverable on the $PATH. Pants needs Python 3.6+ to run, although if you have Python code your own code can still be written in Python 2

I have installed Python 3.7 and update the path variable in Jenkins, I get the below error.

./pants: line 72: /var/jenkins_home/Python3/bin/python3.7: cannot execute binary file: Exec format error
[31mInvalid Python interpreter version for /var/jenkins_home/Python3/bin/python3.7. Pants requires Python 3.6+ to run.[0m
Build step 'Execute shell' marked build as failure
Finished: FAILURE

I am using docker image jenkins/jenkins to run jnkins from the docker hub

this PR is expected to remove messages about dropping connections to the v1 http buildcache, as well as add retries and backoff to avoid having pants users on slow internet wait 30 minutes for cache read errors: https://github.com/pantsbuild/pants/pull/8568. i've asked several times for re-review on this PR, but it's my impression that the http v1 buildcache is a bit of a dark corner, and this addresses a need that is relatively important within twitter due to our heavy dependence on the v1 buildcache (for now). i would love users who have experienced spotty connections to the pants buildcache to weigh in or sign off on this PR, and i would love if any contributor could give it a cursory review on the new

pants.cache

v1 backend introduced, for example. i'd love to get this in by EOW.

not sure if anyone else has seen this -- just noticed it when github offered me the option to use a "template repo" today. @happy-kitchen-89482 i think this example pants v2 python repo is a fantastic idea, thanks for making it! https://github.com/pantsbuild/example-python

Thanks, it's still very very under construction, but working to flesh it out

hey @hundreds-father-404 starting from https://github.com/pantsbuild/pants/commit/6a81a2d74ae38d60fd836bddb144acaf8419ac05, I start to get:

[tw-mbp-yic pants ((6a81a2d...))]$ ./pants goals
Scrubbed PYTHONPATH=/Users/yic/workspace/pants/src/python: from the environment.
timestamp: 2020-03-07T23:55:25.870643
Exception caught: (builtins.TypeError)
  File "/Users/yic/workspace/pants/src/python/pants/bin/pants_loader.py", line 98, in <module>
    main()
  File "/Users/yic/workspace/pants/src/python/pants/bin/pants_loader.py", line 94, in main
    PantsLoader.run()
  File "/Users/yic/workspace/pants/src/python/pants/bin/pants_loader.py", line 90, in run
    cls.load_and_execute(entrypoint)
  File "/Users/yic/workspace/pants/src/python/pants/bin/pants_loader.py", line 83, in load_and_execute
    entrypoint_main()
  File "/Users/yic/workspace/pants/src/python/pants/bin/pants_exe.py", line 32, in main
    PantsRunner(start_time=start_time).run()
  File "/Users/yic/workspace/pants/src/python/pants/bin/pants_runner.py", line 77, in run
    options_bootstrapper = OptionsBootstrapper.create(env=self._env, args=self._args)
  File "/Users/yic/workspace/pants/src/python/pants/option/options_bootstrapper.py", line 170, in create
    initial_bootstrap_options = cls.parse_bootstrap_options(env, bargs, pre_bootstrap_config)
  File "/Users/yic/workspace/pants/src/python/pants/option/options_bootstrapper.py", line 101, in parse_bootstrap_options
    env=env, config=config, known_scope_infos=[GlobalOptions.get_scope_info()], args=args,
  File "/Users/yic/workspace/pants/src/python/pants/option/options.py", line 127, in create
    complete_known_scope_infos = cls.complete_scopes(known_scope_infos)
  File "/Users/yic/workspace/pants/src/python/pants/option/options.py", line 101, in complete_scopes
    for si in copy.copy(ret):
  File "/Users/yic/workspace/pants/build-support/virtualenvs/Darwin/pants_dev_deps.py36.venv/lib/python3.6/copy.py", line 88, in copy
    return copier(x)

Exception message: __copy__() takes 1 positional argument but 2 were given

Any quick fix?

Huh..how did you trigger this?

just via

./pants goals

, let me see if that’s still the case after a git clean -fdx

yeah same.

Is this in upstream Pants or Source? I’ve been able to run things fine for the past few days and CI has been fine, so I’m trying to figure out why this would be failing for you

yes. looks like it’s a difference between 3.6 and 3.7

I put 3.7 on PATH ahead of 3.6, and it works now

Huh, weird. I use 3.6 locally and we use 3.6 in CI

strange indeed. I can see how it performs on docker later. thanks for looking!

Okay, thank you. Iirc we have unit tests to ensure that ordered sets work properly with copy(), so I’m fishing for why this might be happening

ok it worked after upgrading python 3.6 from 3.6.1 to 3.6.8,

Oh awesome, glad that fixed it!

I am working on a pants v2 plugin, is there a way to get pants to run it without pulling it down from pypi ? i.e. I need something similar to

pip -e ~/project/my-library-folder

but for

pants.toml

backend_packages2.add

hey @happy-kitchen-89482 I’m going to remove ivy resolve. One blocker is that coursier is trying to fetch https://github.com/toolchainlabs/binhost/tree/master/java/javac/9%2B181-r4173-1, but there is no pom associated with it. Would you kindly add one there?

alternatively i could add a hardcoded url:

diff --git a/BUILD.tools b/BUILD.tools
index ed7ddc1..88463c4 100644
--- a/BUILD.tools
+++ b/BUILD.tools
@@ -80,7 +80,8 @@ jar_library(
 jar_library(
   name='javac9',
   jars = [
-    jar(org='java', name='javac', rev='9+181-r4173-1'),
+    jar(org='java', name='javac', rev='9+181-r4173-1',
+        url='<https://raw.githubusercontent.com/toolchainlabs/binhost/master/java/javac/9+181-r4173-1/javac-9+181-r4173-1.jar>'),
   ],
 )

Yay to removing Ivy!

Yay indeed to removing Ivy

Let me remind myself of how any of this works 🙂

Are there any way to find deps graph other than

--print-exception-stacktrace

argument?

@fresh-architect-74417 Can you explain exactly what you're looking for? There are a few related things

I was trying to find the dependency cycle for one of the targets. Both of them lived in the same BUILD file. Target A imported target B, and target B imported target A. The command

--print-exception-stacktrace

didn't print the dependency cycle in a clear way. However, I fixed the issue.