cool, then use this:

[GLOBAL]
pants_ignore.add = ["my-dir/"]

I have successfully migrated a java project within my monorepo to pants (the rest of the monorepo is python). I have a

deploy_jar

target in my java project. The jar is required by one of python classes. Is there any way to represent this type of dependency a BUILD file?

Great to hear! How is the JAR loaded? As a resource file?

So, we are using Apache Flink's pyflink. We build some custom components using the flink java api that we need to make available to python. That jar is loaded locally by the client using py4j and is uploaded to the flink cluster when deploying jobs. Until now, we were building the jar and copying it into a lib dir within our python source tree. We created a resources target which includes the jar. I wasn't sure if there was a way to update the resources target to point to the jar that the package goal creates from a deploy_jar target.

Something I am used to in gradle but not sure how to do it in pants, or if it is supported.

I'm thinking the best way to approach this for now is via a new "codegen" plugin. You can have

deploy_jar

generate

ResourceSourceField

so that when you depend on it, it's like depending on

resource

but where it gets generated by ~running

./pants package

first https://www.pantsbuild.org/docs/plugins-codegen To do this, you'll want to add a private

sources

field to

deploy_jar

like this: https://github.com/pantsbuild/pants/blob/fe112bf20e9e7cfe4a2c746316ebc2655c428697/src/python/pants/core/target_types.py#L142-L145. You're just doing this to hook into codegen. Use a "plugin field" to extend the existing

deploy_jar

target: https://www.pantsbuild.org/docs/target-api-extending-targets Then, in your codegen rule, you'll call

await Get(BuiltPackage, DeployJarFieldSet, field_set)

I think. This essentially calls

./pants package

. You can then return the resulting

BuiltPackage.digest

in the

GeneratedSources

for your codegen rule We're happy to help more if you have questions along the way 🙂 Feel free to ask questions here or in #plugins We may want to generalize that cross-language relationship in core Pants, but in the meantime

Cool - thanks for pointing me in the right direction. I'll dig in on my end 👍

pants==2.8.0

I need multiple tags for every docker image. pants build them corectllt. When I want to push them

./pants publish ::

I get an error from docker.

11:56:49.04 [INFO] Initializing scheduler...
11:56:49.12 [INFO] Scheduler initialized.
11:56:51.83 [INFO] Completed: Building docker image dataprocessor_simple:local-sha +1 additional tag.
11:56:51.83 [INFO] Built docker images: 
  * dataprocessor_simple:local-sha
  * dataprocessor_simple:local-branch
"docker push" requires exactly 1 argument.
See 'docker push --help'.

Usage:  docker push [OPTIONS] NAME[:TAG]

Push an image or a repository to a registry

𐄂 dataprocessor_simple:local-sha failed.
𐄂 dataprocessor_simple:local-branch failed.

My 2c: Today if a

python_sources

, say, depends on a

pex_binary

, then IIRC we package that PEX and make it available as a runtime resource for the python sources. This seems no different in principle, so we almost certainly want to support this in core Pants.

I thought that we only do that for

python_distribution

and then we install that into the venv via your PR from a few months ago?

Right but apart from that I thought we treated pexes as resources? Did I remember wrong?

Is there a plan to do a

1.31

release? Would be great to see this bug in the

lint.jvm-dep-check.unnecessary_deps

get fixed. Also after hacking around it, it seems to fire a lot of false positives. I added

scope=forced

for all the generated code ones, but it fires in many other places in my scala codebase (mix of many 1stparty, 3rdparty, and internal deps). I have

strict_deps=true

everywhere. Anybody else running into this?

./pants lint src/scala/::
                   (If you're seeing this message in error, you might need to change the `scope` of the dependencies.)
<a lot of deps>

Hi Pants community! 2021 was another massive year for the project, even as the pandemic continued to disrupt our lives in so many challenging ways. We grew our Python user base significantly, engaged in critical developer relations efforts, gained new community members, including several new maintainers and contributors, and added support for new languages and frameworks, including Shell, Go, Java, Scala and Docker. 2022 will be another busy year, in which we plan to continue to grow the user community, add further language support, and improve performance. Thanks to all who contributed this past year, whether by patches, documentation, community building, bug reports or feature suggestions. And to the many new teams who adopted Pants this year - thanks for your faith in the project, and my apologies for any bumps in the road. Your feedback allows us to continue to develop and improve the system. Our goal continues to be to create the best build system, with the most supportive community, out there. Your help is crucial in realizing that goal! I hope you all remain safe through this latest pandemic surge. Happy new year!

Hi, I am trying to use pants and have weird issue - mac os monterey 12.0.1, apple silicon:

./pants --version                                                       
Bootstrapping Pants using /opt/homebrew/opt/python@3.8/bin/python3.8
Creating the virtualenv PEX.
Downloading the Pex PEX.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   640  100   640    0     0   2413      0 --:--:-- --:--:-- --:--:--  2452
100 3529k  100 3529k    0     0  3800k      0 --:--:-- --:--:-- --:--:-- 5971k
SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.42/pex> verified.
Installing pantsbuild.pants==1.30.4 into a virtual environment at /Users/pawel/.cache/pants/setup/bootstrap-Darwin-arm64/1.30.4_py38
created virtual environment CPython3.8.12.final.0-64 in 138ms
  creator CPython3Posix(dest=/Users/pawel/.cache/pants/setup/bootstrap-Darwin-arm64/pants.UyuuvG/install, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/pawel/Library/Application Support/virtualenv)
    added seed packages: pip==21.1.2, setuptools==57.0.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Requirement already satisfied: pip in /Users/pawel/.cache/pants/setup/bootstrap-Darwin-arm64/pants.UyuuvG/install/lib/python3.8/site-packages (21.1.2)
Collecting pip
  Using cached pip-21.3.1-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.2
    Uninstalling pip-21.1.2:
      Successfully uninstalled pip-21.1.2
Successfully installed pip-21.3.1
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==1.30.4 (from versions: 0.0.17, 0.0.18, 0.0.20, 0.0.21, 0.0.22, 0.0.23, 0.0.24, 0.0.25, 0.0.26, 0.0.27, 0.0.28, 0.0.29, 0.0.30, 0.0.31, 0.0.32, 0.0.33, 0.0.34, 0.0.35, 0.0.36, 0.0.37, 0.0.38, 0.0.39, 0.0.40, 0.0.41, 0.0.42, 0.0.43, 0.0.44, 0.0.45, 0.0.46, 0.0.47, 0.0.48, 0.0.49, 0.0.50, 0.0.51, 0.0.52, 0.0.53, 0.0.54, 0.0.55, 0.0.56, 0.0.57, 0.0.58, 0.0.59, 0.0.60, 0.0.61, 0.0.62, 0.0.63, 0.0.64, 0.0.65, 0.0.66, 0.0.67, 0.0.68, 0.0.69, 0.0.70, 0.0.71, 0.0.72, 0.0.73, 0.0.74, 0.0.75, 0.0.76, 0.0.77, 0.0.79, 0.0.80, 0.0.81, 0.0.82, 1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0)
ERROR: No matching distribution found for pantsbuild.pants==1.30.4

is anyone able to help? tried to run: curl -L -O https://static.pantsbuild.org/setup/pants && chmod +x ./pants rm -rf ~/.cache/pants/

Hmm... playing with the docker functionality again (massive progress for us!) one of the arguments we were passing was

--build-arg CONTAINER_GID={os.getgid()}

. (this was for a development container so that it ran as the existing user rather than as root or some fixed user ID). It's not clear how to do this with

extra_build_args

in pants (or generally, how to pass in a variable like that). I obviously can't

import os

to call `getgid`; I could pass it in as an env variable, but I can't see how to use env variables within a

BUILD

right now. Possible or not?

Yeah, env vars are not available in BUILD files.

You can consume the env var (if it would be ok passing it in as env var, was unclear to me if you knew how, but didn’t want to, or ok with, but need to know how) 😉

docker_image(extra_build_args=["CONTAINER_GID"])

CONTAINER_GID=$(group_id) ./pants package ..

Ah! Thanks--I even saw that in the docs and forgot it

That should do the trick; thanks

Cool!

@echoing-farmer-15630 oh, and while I have your attention (sort of), did you see my ping here https://github.com/pantsbuild/pants/pull/13959#issuecomment-1004275888 (in case you have any input regarding the expected stability of the calculated hash value for the image tag)

Thanks 🙂

I didn't, thanks--but I've been testing our repo with the current hash and the two changes (target stage and ssh) and it's worked wonderfully so far, just taking a while to migrate things and will do some benchmarks for builds shortly. I'll take a look at the hash issue!

OK, just some jolly benchmarks moving from "serially building all docker containers with a build script" to "parallel builds using pants": we went from 28.8 minutes (1730 sec) to 611.88 seconds, about a 2.8x improvement (and there's one spectacularly long-build container that's responsible for the long tail; most were built much faster so if it weren't for that laggard this would look much better). And that's not even counting the impressive dependency checking, etc that goes with a proper build system. Very pleased with this; going to be working on moving it to production. Thanks for the fantastic support!

Amazing! Thanks for sharing and all the bug reports / feedback along the way! 😄

Would you mind telling a bit more about how does your codebase looks like? Why are you building so many docker images? Is it some Microservice architecture?

Hi team! Quick question - have people used new relic java agents with pants v1? We have been struggling to get some reliable instrumentations of our jdbc connections and are wondering if it could be related to the packaging step with pants.

It is, although some of them are more "macroservices" because python dependencies can get a little heavyweight with data science and math packages (numpy and scipy are bad enough, but if a developer tries to include torch you're looking at gigabyte-plus packages). So we have an API gateway that drops tasks into a distributed task queue (celery) and a "dispatcher" that handles some serialization and dispatching tasks to hand them off to the right workers, and then some heavyweight workers to actually handle the tasks. Then there is a scheduler which just injects periodic cleanup tasks into the system, a "janitor" which handles some of the cleanup tasks when they're requested, a web service front-end, an admin front-end, a little websocket dingus that handles small bits of plumbing... and by far most of the changes are in the 2-3 heavyweight workers, so our deployment system has been working too hard...

How to fix errors like

Unable to import

generated by

pylint

when running

./pants lint ::

This is from an internal pants plugin

12:40:29.39 [ERROR] Completed: Lint using Pylint - Pylint failed (exit code 30).
************* Module pants-plugins.internal_plugins.register
pants-plugins/internal_plugins/register.py:1:0: C0114: Missing module docstring (missing-module-docstring)
pants-plugins/internal_plugins/register.py:1:0: E0401: Unable to import 'pants.backend.python.goals.setup_py' (import-error)
pants-plugins/internal_plugins/register.py:2:0: E0401: Unable to import 'pants.engine.fs' (import-error)
pants-plugins/internal_plugins/register.py:3:0: E0401: Unable to import 'pants.engine.rules' (import-error)
pants-plugins/internal_plugins/register.py:4:0: E0401: Unable to import 'pants.engine.target' (import-error)
pants-plugins/internal_plugins/register.py:5:0: E0401: Unable to import 'pants.engine.unions' (import-error)
pants-plugins/internal_plugins/register.py:8:0: C0115: Missing class docstring (missing-class-docstring)
pants-plugins/internal_plugins/register.py:10:4: C0116: Missing function or method docstring (missing-function-docstring)
pants-plugins/internal_plugins/register.py:8:0: R0903: Too few public methods (1/2) (too-few-public-methods)
pants-plugins/internal_plugins/register.py:58:0: C0116: Missing function or method docstring (missing-function-docstring)