pantsbuild #general

Channels

welcome

github-notifications

announce

enough-analyst-54434

01/19/2021, 6:19 PM

Can you confirm this is true and you have a package housed in multiple source roots but with different

__init__.py

contents?

helpful-lunch-92084

01/19/2021, 6:23 PM

hey, so i finally got around to trying out the new pip resolver in pex 2.1.23 via pants 2.3.0.dev0 and it still seems to be failing even though i’m passing in pip-2020-resolver to the

resolver_version

python-setup

. I can confirm that using pex 2.1.23 correctly resolves the following with the new resolver:

boto3==1.15.6

botocore

Pex in action:

Copy code

$ pex --version
2.1.23
$ pex boto3==1.15.6 botocore -o boto3.pex
Failed to resolve compatible distributions:
1: boto3==1.15.6 requires botocore<1.19.0,>=1.18.6 but botocore 1.19.56 was resolved
$ pex --resolver-version=pip-2020-resolver boto3==1.15.6 botocore -o boto3.pex
$

Pants in action:

Copy code

$ cat 3rdparty/python/BUILD.foo
python_requirement_library(
    name="foo",
    requirements=["boto3==1.15.6", "botocore"],
)
$ pants repl 3rdparty/python:foo
...

Engine traceback:
  in select
  in pants.core.goals.repl.run_repl
  in pants.backend.python.goals.repl.create_ipython_repl_request
  in pants.backend.python.util_rules.pex.create_pex (requirements.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/nate/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0.dev0_py38/lib/python3.8/site-packages/pants/engine/process.py", line 255, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building requirements.pex with 2 requirements: boto3==1.15.6, botocore' failed with exit code 1.
stdout:

stderr:
Failed to resolve compatible distributions:
1: boto3==1.15.6 requires botocore<1.19.0,>=1.18.6 but botocore 1.19.56 was resolved

Any thoughts?

enough-analyst-54434

01/19/2021, 6:29 PM

Pants also defaults to the legacy resolver.

enough-analyst-54434

01/19/2021, 6:30 PM

Just a sec for the option....

enough-analyst-54434

01/19/2021, 6:32 PM

Yeah, you want `--python-setup-resolver-version`:

Copy code

$ ./pants help-advanced python-setup
...
--python-setup-resolver-version=<ResolverVersion>
  PANTS_PYTHON_SETUP_RESOLVER_VERSION
  resolver_version
      one of: [pip-legacy-resolver, pip-2020-resolver]
      default: pip-legacy-resolver
      current value: pip-2020-resolver (from pants.toml)
      The resolver implementation to use when resolving Python requirements.
      
      Support for the 'pip-legacy-resolver' will be removed in Pants 2.5; so you're encouraged
      to start using the 'pip-2020-resolver' early. For more information on this change see
      <https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver->
      in-20-2-2020
      Upcoming deprecation in version: 2.3.0.dev1, will be removed in version: 2.5.0.dev1.

helpful-lunch-92084

01/19/2021, 6:34 PM

yah i think i have that set

enough-analyst-54434

01/19/2021, 6:35 PM

Please check using

./pants help-advanced python-setup

. If you read my output above closely it shows the current set value.

helpful-lunch-92084

01/19/2021, 6:35 PM

Copy code

--python-setup-resolver-version=<ResolverVersion>
  PANTS_PYTHON_SETUP_RESOLVER_VERSION
  resolver_version
      one of: [pip-legacy-resolver, pip-2020-resolver]
      default: pip-legacy-resolver
      current value: pip-2020-resolver (from pants.toml)

enough-analyst-54434

01/19/2021, 6:37 PM

Hmm, OK. Sounds like a bug. Give me a few minutes to write one up - I dislike Slack for digging into bugs.

helpful-lunch-92084

01/19/2021, 6:41 PM

helpful-lunch-92084

01/19/2021, 6:41 PM

i’m happy to fill in details if you point me to the issue

enough-analyst-54434

01/19/2021, 6:58 PM

Hrm - I don't repro so I'll have you file if you don't mind once you have a setup you like:

Copy code

$ cat BUILD.nate 
python_requirement_library(
    name="foo",
    requirements=["boto3==1.15.6", "botocore"],
)
$ ./pants repl //:foo
10:55:21.20 [WARN] The constraints file 3rdparty/python/constraints.txt does not contain entries for the following requirements: botocore, boto3
10:55:21.20 [WARN] Ignoring resolve_all_constraints setting in [python_setup] scope because constraints file does not cover all requirements.
10:56:23.37 [INFO] Completed: Building requirements.pex with 2 requirements: boto3==1.15.6, botocore
10:56:23.37 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
...
pants.engine.process.ProcessExecutionFailure: Process 'Building requirements.pex with 2 requirements: boto3==1.15.6, botocore' failed with exit code 1.
stdout:

stderr:
ERROR: Cannot install botocore==1.18.10, botocore==1.18.11, botocore==1.18.12, botocore==1.18.13, botocore==1.18.14, botocore==1.18.15, botocore==1.18.16, botocore==1.18.17, botocore==1.18.18, botocore==1.18.6, botocore==1.18.7, botocore==1.18.8 and botocore==1.18.9 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies>
pid 152051 -> /usr/bin/python3.7 /home/jsirois/.cache/pants/named_caches/pex_root/pip.pex/1b2dfe434f773fd6aaf45cdd941bad0c262b8e13 --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/jsirois/.cache/pants/named_caches/pex_root download --dest /tmp/process-executionYHMAYe/.tmp/tmp10y30qf9/usr.bin.python3.7 --constraint 3rdparty/python/constraints.txt boto3==1.15.6 botocore --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
None

At a baseline we should see the same error and we don't

hundreds-father-404

01/20/2021, 2:40 AM

Hey Zdeněk! It should be. From https://www.pantsbuild.org/docs/macros#how-to-add-a-macro

A macro can create multiple targets:

Copy code

def python23_tests(name, **kwargs):
    kwargs.pop("interpreter_constraints", None)

    python_tests(
        name=f"{name}_py2",
        interpreter_constraints=["==2.7.*"],
        **kwargs,
    )
 
    python_tests(
        name=f"{name}_py3",
        interpreter_constraints=[">=3.5"],
        **kwargs,
    )

narrow-activity-17405

01/20/2021, 7:42 AM

Ok, great! I will give it a try.

❤️ 1

echoing-nightfall-72095

01/20/2021, 7:54 AM

You’re right. I accidentally tried to unzip a .pex file and ended up in the root of my repo. I thought I’d cleaned all up. Thanks 😄

polite-vase-75369

01/20/2021, 9:23 PM

What does interpreter_search_paths actually do for pants? is that for the python that pants uses itself to run? Or the python pants uses to run python code?

polite-vase-75369

01/20/2021, 9:26 PM

In addition, how do pex files find interpreters? When I do PEX_VERBOSE=6, i see it do some magic in finding an interpreter

witty-crayon-22786

01/20/2021, 10:00 PM

https://pantsbuild.slack.com/archives/C18RRR4JK/p1611180036004400

🙌 2

🎉 4

echoing-nightfall-72095

01/21/2021, 12:28 PM

Am I correct that when setting up for CI/CD I need both a

pants.toml

and a

<http://pants.ci|pants.ci>.toml

? I have set the env_var

PANTS_CONFIG_FILES=<http://pants.ci|pants.ci>.toml

. Linting complains that the version should be set in the GLOBAL section in

pants.toml

. But I have specified it in

<http://pants.ci|pants.ci>.toml

Copy code

[GLOBAL]
pants_version = "2.2.0"

I thought maybe it does not use that env_var. I tested setting the env_var and removing pants.ci.toml, then it complains that pant.ci.toml is missing as it should. So the only way to make it work for me is having both and files. I can of course rename the

<http://pants.ci|pants.ci>.toml

pants.toml

and not setting the env_var.

narrow-activity-17405

01/21/2021, 1:09 PM

Awesome. This allowed me to simplify the main BUILD files of the packages a lot.

💯 1

high-egg-2153

01/23/2021, 4:18 PM

A while ago I created the generate_constraints.sh mentioned here: https://www.pantsbuild.org/docs/python-third-party-dependencies I re-ran it this morning and it seems to have caused some issues. Now if I try to run anything I get the following error: Exception message: 1 Exception encountered: Exception: While expanding link "build-support/bin/python": Failed to read link ".../build-support/bin/python3": Absolute symlink: ".../build-support/bin/python3"

enough-analyst-54434

01/23/2021, 5:16 PM

You probably just need to add some ignores to your pants config. For example, Pants itself uses this lockfile recipe and has this ignore (and others) configured here: https://github.com/pantsbuild/pants/blob/e3b3e53b8f31b098ad2334f6537d442f9d453045/pants.toml#L37-L48

high-egg-2153

01/23/2021, 5:33 PM

That was it. I just added an ignore for the entire build-support folder. Thanks!

square-oxygen-75288

01/25/2021, 2:29 PM

Hello! I have an issue with PANTS 1.30 / Scala / Scalatest 3.1+ (again 🙂 ) The problem was partially resolved in this thread: https://pantsbuild.slack.com/archives/C046T6T9U/p1610618752002400 with this PR: https://github.com/slvrtrn/pants-1.30-junit-scalatest-issue/pull/1 submitted by @enough-analyst-54434 However, there is another issue. The tests are not actually executed from the console. It is explicitly said that 0 tests were executed.

Copy code

06:21:43 01:19       [run]
                     Auto-detected 8 processors, using -parallel-threads=8
                     
                     Time: 0.001
                     
                     OK (0 tests)
                     
                     
                   test/scala/com/github/slvrtrn:tests                                             .....   SUCCESS
               Waiting for background workers to finish.
06:21:43 01:19   [complete]
               SUCCESS

The test file itself is correctly recognized and there is no error reported in the console. But the suite is not actually executed.

bored-mechanic-51694

01/25/2021, 4:12 PM

👋 having an issue (i think) passing http_proxy environment to the pip resolver when pants is bootstrapping / resolving plugins in pants 1.26 on a build machine that requires a proxy to access the internet (full logs with -ltrace and PEX_VERBOSE=9 in thread):

Copy code

10:56:38 15:56:38 [INFO] Resolving new plugins...:
10:56:38   pantsbuild.pants.contrib.avro==1.26.0
10:56:38   pantsbuild.pants.contrib.googlejavaformat==1.26.0
10:56:38   pantsbuild.pants.contrib.python.checks==1.26.0
10:56:38   pantsbuild.pants.contrib.scrooge==1.26.0
10:56:38   pantsbuild.pants.contrib.thrifty==1.26.0
10:56:38 pex: Resolving for:
[SNIP]
10:56:39 Starting new HTTPS connection (1): <http://pypi.org:443|pypi.org:443>
10:57:39 Incremented Retry for (url='/simple/pantsbuild-pants-contrib-avro/'): Retry(total=4, connect=None, read=None, redirect=None, status=None)
10:57:39 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fc21492b080>, 'Connection to <http://pypi.org|pypi.org> timed out. (connect timeout=15)')': /simple/pantsbuild-pants-contrib-avro/

helpful-lunch-92084

01/26/2021, 3:58 AM

hey, just ran into this issue: 1. I run pants test :: and it kicks off a bunch of parallel tests via subprocesses. 2. One of the tests fails during requirements.pex building due to a missing dep (like we run our own cheeseshop and don’t fall through to pypi - it’s not great, but we have reasons for this) 3. Pants exits 4. The subprocesses stay active as orphaned procs Eventually they die. Now I don’t know (since i lose their stdout/err) if they die because they also ran into issues, or they completed, or if pants is just delayed in killing them. Since they’re owned by init I feel like it’s the former, in which case pants isn’t cleaning them up properly. Just thought I’d flag. Seems like a kill to the pgid would solve this after handling the initial subprocess exception. Sorry, I don’t have a great test case for this.

witty-crayon-22786

01/26/2021, 5:17 PM

hey folks! at long last, we’ve completed the prioritization effort that we started before the holidays, and we have a roadmap for the first half of 2021! thanks for all of your feedback. https://groups.google.com/g/pants-devel/c/F8Saug3BrFw/m/ZtWDP4YmDwAJ

👖 4

🔥 1

polite-vase-75369

01/26/2021, 6:58 PM

Random q, but if pants v2 doesn’t support scala and doesn’t have it in the roadmap for the first half, how is twitter using it?

busy-vase-39202

01/28/2021, 4:07 AM

Kicking off new #community-outreach channel https://pantsbuild.slack.com/archives/C01L1J4EVTP/p1611806766015000

fierce-france-72742

01/28/2021, 10:03 AM

Since I set a "contraints.txt" file, I realised after calling

package

on a target, calling

run

on it for the first time leads to a long contraints resolution step. I quite don't understand why those constraints are resolved after the ".pex" file was created. Isn't that pex file fully self-contained and hermetic?

👀 1