pantsbuild #general

Channels

welcome

github-notifications

announce

hundreds-father-404

01/05/2022, 6:51 AM

To use the overrides field, yes, you need to upgrade to pants 2.8. The workaround is to remove the problematic requirement from your requirements.txt and use an explicit python_requirement target with

dependencies

set manually. (Target was formerly called python_requirement_library)

rapid-flag-11972

01/05/2022, 6:59 AM

Got it, thanks

cuddly-teacher-45066

01/05/2022, 8:13 AM

I have problems building a pex_binary target with

avro

as a dependency, the host system is a macos machine and it seems this is also not working:

Copy code

pex avro -o /tmp/pex --platform manylinux2014_x86_64-cp-38-cp38

the main goal is to build a docker image using the pex file. I have no problems with numpy surprisingly. An alternative solution I guess could be building the dependency within the container

curved-television-6568

01/05/2022, 10:04 AM

Cache question: does

--no-local-cache

give me the same effect as using the

@_uncacheable_rule

decorator on my rules? (when not using remote execution)

bitter-ability-32190

01/05/2022, 12:52 PM

melodic-thailand-99227

01/05/2022, 12:54 PM

I’m seeing an issue with black & isort, where isort changes it’s mind on what is considered first party vs third party. Can pants help isort with this configuration?

fresh-cat-90827

01/05/2022, 6:37 PM

hey folks. @busy-vase-39202 has suggested to put together a blog post from a Slack message I've posted regarding editing the

BUILD

files programmatically using Bazel build tools. Comments are very welcome 🙂 here's the first draft: https://docs.google.com/document/d/1Y3fCbx_Dap5Qk5L2VRFI6pPat0ctGP92FyYRq5IY6RY/edit?usp=sharing

❤️ 2

🙏 1

shy-advantage-49800

01/05/2022, 6:44 PM

The command:

./pants tailor --check update-build-files --check

is taking 30-40 min to run, any ideas?

Copy code

16:16:04.04 [INFO] Starting: Building black.pex from black_default_lockfile.txt

busy-vase-39202

01/05/2022, 7:13 PM

Looking for an exciting new job in the new year? Check out this thread. https://pantsbuild.slack.com/archives/C046T6T9U/p1641330675115400 Or if you're hiring, feel free to add to it.

orange-beach-75711

01/05/2022, 8:32 PM

@hundreds-father-404 - if we use

overrides

, does it mean that we can leave torch in

requirements.txt

hundreds-father-404

01/05/2022, 8:34 PM

yes, you're changing how the

python_requirement

target gets generated from the

requirements.txt

orange-beach-75711

01/05/2022, 8:36 PM

Got it, thanks for the quick turnaround!

❤️ 1

hundreds-father-404

01/05/2022, 8:38 PM

you're welcome! check out https://www.pantsbuild.org/docs/upgrade-tips for upgrade tips. 2.8 adds

update-build-files

to automate some of it also the release blogs if you haven't yet seen them https://blog.pantsbuild.org

✅ 1

orange-beach-75711

01/05/2022, 8:38 PM

🎉

orange-beach-75711

01/05/2022, 8:38 PM

That looks awesome! Thanks Eric!

❤️ 1

witty-crayon-22786

01/05/2022, 10:57 PM

https://pantsbuild.slack.com/archives/C18RRR4JK/p1641423160055400

full-toothbrush-75676

01/06/2022, 1:01 AM

Hi Pants! Trying to get Pants

v2.9.0.dev1

working on an M1 macbook. We’re on Python 3.8 (we can’t upgrade to Python 3.9 just yet) and have installed Python using the homebrew /

pyenv

using x86_64 arch emulation. We’re using the Pants

docker

plugin and are stumped by the attached error. Does anyone have any insight into this? Thanks in advance!

Copy code

./pants dependencies ::
16:10:34.33 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Parse Dockerfile.' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/private/var/folders/kx/_whrt0l54d9_350g0xwn9zp40000gn/T/process-executionU4DA6T/.cache/pex_root/venvs/5f8edf4d6068dd9c5ff01b0865815af301a9e9d5/05ad48543d1b4172329a34b21ccc64fa30998340/pex", line 157, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/Users/thoyt/.pyenv/versions/3.8.10/lib/python3.8/runpy.py", line 207, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/Users/thoyt/.pyenv/versions/3.8.10/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/thoyt/.pyenv/versions/3.8.10/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/thoyt/.cache/pants/named_caches/pex_root/venvs/s/97bd2daa/venv/lib/python3.8/site-packages/__pants_df_parser.py", line 129, in <module>
    main(cmd, sys.argv[2:])
  File "/Users/thoyt/.cache/pants/named_caches/pex_root/venvs/s/97bd2daa/venv/lib/python3.8/site-packages/__pants_df_parser.py", line 42, in main
    from dockerfile import Command, parse_file, parse_string
ImportError: dlopen(/Users/thoyt/.cache/pants/named_caches/pex_root/venvs/s/97bd2daa/venv/lib/python3.8/site-packages/dockerfile.abi3.so, 0x0002): tried: '/Users/thoyt/.cache/pants/named_caches/pex_root/venvs/s/97bd2daa/venv/lib/python3.8/site-packages/dockerfile.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/dockerfile.abi3.so' (no such file), '/usr/lib/dockerfile.abi3.so' (no such file), '/Users/thoyt/.cache/pants/named_caches/pex_root/venvs/5f8edf4d6068dd9c5ff01b0865815af301a9e9d5/05ad48543d1b4172329a34b21ccc64fa30998340/lib/python3.8/site-packages/dockerfile.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/dockerfile.abi3.so' (no such file), '/usr/lib/dockerfile.abi3.so' (no such file)



Use --no-process-execution-local-cleanup to preserve process chroots for inspection.

better-cricket-91243

01/06/2022, 4:57 AM

Copy code

pex: error: argument --platform: linux_x86_64 is an invalid platform:
Not a valid platform specifier: linux_x86_64

Platform strings must be in one of two forms:
1. Canonical: <platform>-<python impl abbr>-<python version>-<abi>
2. Abbreviated: <platform>-<python impl abbr>-<python version>-<abbr abi>

Given a canonical platform string for CPython 3.7.5 running on 64 bit linux of:
  linux-x86_64-cp-37-cp37m

Where the fields above are:
+ <platform>: linux-x86_64 
+ <python impl abbr>: cp
+ <python version>: 37
+ <abi>: cp37m

The abbreviated platform string is:
  linux-x86_64-cp-37-m

Some other canonical platform string examples:
+ OSX CPython: macosx-10.13-x86_64-cp-36-cp36m
+ Linux PyPy: linux-x86_64-pp-273-pypy_73.

able-match-78954

01/06/2022, 8:09 AM

Hi, can somebody help me with tagging docker images? I tried to follow the documentation and worked out a plugin by trail and error. The goal is to have the image tagged as "latest" and a version determined by our git-versioning-script. When I run "pants package" it works as intended and I get my two tagged immages. But when I run "pants package" the push fails with this error message: "docker push" requires exactly 1 argument." I think that I didnt really understand the plugin system in the right way and am doing it wrong. This is my current plugin-script

Copy code

import logging
from dataclasses import asdict, dataclass

from pants.backend.docker.goals.package_image import DockerFieldSet
from pants.backend.docker.target_types import DockerImageTagsField
from pants.core.goals.package import PackageFieldSet
from pants.engine.rules import collect_rules, rule
from pants.engine.unions import UnionRule

from python_plugins.custom_setup_py import determine_version_from_git

logger = logging.getLogger(__name__)


@dataclass(frozen=True)
class CustomDockerFieldSet(DockerFieldSet):
    ...


@rule
async def custom_docker_tags(request: CustomDockerFieldSet) -> DockerFieldSet:
    version = determine_version_from_git(
        package_name=request.address.target_name,
        build_file_path=request.address.spec_path
    )
    return DockerFieldSet(
        **{
            **asdict(request),
            'tags': DockerImageTagsField(
                raw_value=[
                    # 'latest',
                    str(version)
                ],
                address=request.address
            )
        }
    )


def rules():
    return [
        *collect_rules(),
        UnionRule(PackageFieldSet, CustomDockerFieldSet),
    ]

witty-family-13337

01/06/2022, 11:09 AM

hello, giving it a go to Pants 2.9.0rc3 with the JVM support and I have a couple of questions which I don’t know if they are planned for final 2.9.0 or future releases: 1. Is there a way to pass JVM parameters while running ScalaTest tests? What I’m looking for is an equivalent to SBT’s

Test / javaOptions := List(…)

as we have some tests that require some specific JVM arguments (i.e. allowing reflective access to some Java 11 packages) in order to pass. The

[scalatest].args

section in

pants.toml

seems to be only for passing arguments to ScalaTest, not to the underlying JVM 2. We have some tests that need to load some files when running and they try to access them via the classpath but using a

resource

target as a dependency in the

scalatest_tests

target seems to be unsupported. Pants complains that the

resource

target should have a

resolve

compatible_resolves

field, but that field is also unsupported in the

resource

target. Is there a workaround for this that you may be aware or does it need to be implemented in a future release?

ambitious-student-81104

01/06/2022, 4:26 PM

Hi Pants! We are having a small issue with

./pants fmt

. It says that black and isort made changes, however it doesn't really change any files (the command is just

./pants fmt filepath

). Meanwhile if we run

black filepath

actual changes are made.

full-toothbrush-75676

01/06/2022, 5:55 PM

Hi! A coworker of mine is trying to join this workspace using the invite link on the getting help page but isn’t able to. They are getting a “<their email> doesn’t have an account on this workspace” error. Is the link broken? Thanks!

witty-crayon-22786

01/06/2022, 6:06 PM

bump for https://pantsbuild.slack.com/archives/C18RRR4JK/p1641338890053700: the next few days are a great time to get your ideas and feedback heard!

✔️ 1

cuddly-teacher-45066

01/07/2022, 5:37 AM

when combining the configs from pants.toml and pants.ci.toml, how are list combined? For example if I have different backends in

backend_packages

is a union of the two lists performed?

witty-family-13337

01/07/2022, 8:53 AM

another question regarding the future experimental JVM support in 2.9.0. Having the Google Java Formatter and ScalaFMT backends configured in `pants.toml`(in addition to the ones to regarding other languages). When running

./pants lint ::

I don’t see the JVM ones giving any results but if I run

./pants fmt ::

then they are invoked. Is this intended? Example of

./pants lint ::

Copy code

09:42:27.94 [INFO] Initializing scheduler...
09:42:28.40 [INFO] Scheduler initialized.
09:42:31.48 [INFO] Completed: Lint with shfmt - shfmt succeeded.
09:42:31.48 [INFO] Completed: Lint with gofmt - gofmt succeeded.
09:42:31.80 [INFO] Completed: Lint with Shellcheck - Shellcheck succeeded.
09:42:40.28 [INFO] Completed: Lint with Hadolint - hadolint succeeded.
09:42:40.81 [INFO] Completed: Lint with docformatter - Docformatter succeeded.
09:42:41.35 [INFO] Completed: Lint with isort - isort succeeded.
09:42:43.11 [INFO] Completed: Lint with Black - Black succeeded.
All done! ✨ 🍰 ✨
15 files would be left unchanged.


09:42:48.36 [INFO] Completed: Building flake8.pex from 3rdparty/python/flake8.lock
09:42:49.74 [INFO] Completed: Lint with Flake8 - Flake8 succeeded.

✓ Black succeeded.
✓ Docformatter succeeded.
✓ Flake8 succeeded.
✓ Shellcheck succeeded.
✓ gofmt succeeded.
✓ hadolint succeeded.
✓ isort succeeded.
✓ shfmt succeeded.

Example of

./pants fmt ::

Copy code

09:43:04.21 [INFO] Completed: Format with gofmt - gofmt made no changes.
09:43:04.22 [INFO] Completed: Format with shfmt - shfmt made no changes.
09:43:05.69 [INFO] Completed: Format with Google Java Format - Google Java Format made no changes.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.googlejavaformat.java.JavacTokens$CommentSavingTokenizer (file:/private/var/folders/c9/5v4gbjh17w9_vy73fc90fnch0000gn/T/.tmpIag3cM/4709a472310186fc49e11bd6e6834da83cb4ab66c4d9fb94408faf6702e354dd/com.google.googlejavaformat_google-java-format_1.13.0.jar) to field com.sun.tools.javac.parser.JavaTokenizer.reader
WARNING: Please consider reporting this to the maintainers of com.google.googlejavaformat.java.JavacTokens$CommentSavingTokenizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release


09:43:07.14 [INFO] Completed: Format with Black - Black made no changes.
All done! ✨ 🍰 ✨
15 files left unchanged.


09:43:07.64 [INFO] Completed: Format with docformatter - Docformatter made no changes.
09:43:08.79 [INFO] Completed: Format with isort - isort made no changes.
09:43:21.88 [INFO] Completed: Format with scalafmt - scalafmt made no changes.
        Output from `scalafmt` on 1443 files (.scalafmt.conf):
        Reformatting...
Reformatting... (1.39 %, 20 / 1443)

Reformatting... (6.72 %, 97 / 1443)

Reformatting... (14.35 %, 207 / 1443)

Reformatting... (23.63 %, 341 / 1443)

Reformatting... (33.47 %, 483 / 1443)

Reformatting... (46.08 %, 665 / 1443)

Reformatting... (58.07 %, 838 / 1443)

Reformatting... (67.78 %, 978 / 1443)

Reformatting... (80.39 %, 1160 / 1443)

Reformatting... (93.00 %, 1342 / 1443)

Reformatting... (99.79 %, 1440 / 1443)

Reformatting...



✓ Black made no changes.
✓ Docformatter made no changes.
✓ Google Java Format made no changes.
✓ gofmt made no changes.
✓ isort made no changes.
✓ scalafmt made no changes.
✓ shfmt made no changes.

You can also see in the last one the warning from JVM regarding the reflective access too (not the same I mentioned yesterday, as that was when running some tests, but also something that may need it’s own JVM flags)

refined-addition-53644

01/07/2022, 10:44 AM

Hi, Why are 3rd party dependencies global and not local? Is it more complicated to keep it local like other targets? The issue I am facing right now with a monorepo is that I have to actively maintain 3rdparty dependencies for all individual python packages inside my monorepo.

red-postman-3638

01/07/2022, 12:13 PM

Is is possible to use pip cli options with Pants? Is there any possibility to install

aioeventlet

pip package with Pants? More details in the 🧵.

echoing-farmer-15630

01/07/2022, 3:28 PM

OK, got excited about trying remote caching (our

./pants test ::

step takes ages on CircleCI creating all the wee mini-venvs), and tried a too-simple test: set up a

bazel-remote

instance on an EC2 instance and tried running multiple builds with

remote_cache_X

items set in pants.toml (or not, for the original clean local build). I then simulated a "clean build on a fresh CI node" by

rm -rf

ing

~/.cache/pants

and in the repository both

.pids

and

.pants.d

and doing

pants version

to at least populate the base tool before timing. ...and it seemed to make no difference; ie:

Copy code

Clean build local

________________________________________________________
Executed in  386.20 secs   fish           external
   usr time  461.72 millis  876.00 micros  460.84 millis
   sys time   65.67 millis  240.00 micros   65.43 millis

Clean build remote cache empty but on (for initial population)

________________________________________________________
Executed in  387.25 secs   fish           external
   usr time  450.78 millis  863.00 micros  449.92 millis
   sys time   58.88 millis  239.00 micros   58.64 millis

Clean build remote cache populated and on

________________________________________________________
Executed in  371.61 secs   fish           external
   usr time  449.75 millis    0.00 micros  449.75 millis
   sys time   67.19 millis  1258.00 micros   65.94 millis

... and when I was watching, the bulk of the time (up to 340 seconds) was spent

Resolving constraints.txt

. I'm using 2.10.0.dev0 with

Copy code

[python]
interpreter_constraints = ["CPython==3.9.*"]
requirement_constraints = "constraints.txt"

...the constraints.txt file generated by

pip freeze

and then hand-edited for a few things. Of course, successive runs of

pants test

are nearly instant, but when I can't count on a CI instance having any existing cache (and thus relying on the remote) it doesn't help to the degree I'd hoped. The --stats-log output is

Copy code

local_cache_read_errors: 0
  local_cache_requests: 65
  local_cache_requests_cached: 0
  local_cache_requests_uncached: 65
  local_cache_total_time_saved_ms: 0
  local_cache_write_errors: 0
  local_execution_requests: 61
  local_process_total_time_run_ms: 398394
  remote_cache_read_errors: 0
  remote_cache_requests: 29
  remote_cache_requests_cached: 4
  remote_cache_requests_uncached: 25
  remote_cache_speculation_local_completed_first: 36
  remote_cache_speculation_remote_completed_first: 4
  remote_cache_total_time_saved_ms: 5162
  remote_cache_write_attempts: 59
  remote_cache_write_errors: 0
  remote_cache_write_successes: 57
  remote_execution_errors: 0
  remote_execution_requests: 0
  remote_execution_rpc_errors: 0
  remote_execution_rpc_execute: 0
  remote_execution_rpc_retries: 0
  remote_execution_rpc_wait_execution: 0
  remote_execution_success: 0
  remote_execution_timeouts: 0
  remote_process_total_time_run_ms: 0
  remote_store_blob_bytes_downloaded: 1283423
  remote_store_blob_bytes_uploaded: 115816367
  remote_store_missing_digest: 0

...which implies a lot of misses. So I'm not sure what I'm missing myself (probably a lot; new territory!); why is "Resolving constraints.txt" taking so long, and how could I use remote caching to help, or have I just bolloxed up the setup entirely? (searching slack, it sounds like "Resolving constraints.txt" basically installs every package in constraints.txt; not sure how to speed that or if it's cacheable in any useful sense, so that could be most of the story right there)

witty-crayon-22786

01/07/2022, 5:37 PM

another bump for https://pantsbuild.slack.com/archives/C18RRR4JK/p1641338890053700 ! take a minute to submit an idea or two to let us know where to improve this year! submissions close on wednesday.

🙌 2

📝 2

few-arm-93065

01/07/2022, 7:29 PM

Hi @curved-television-6568, is there currently a way to set up a docker registry in my pants.toml with default=true, but specify that the registry should not be published to?