pantsbuild #general

Channels

welcome

github-notifications

announce

brave-planet-28195

10/05/2022, 8:44 PM

Hey guys, I have the following question: In

pants

v1 there was a dedicated approach for managing jvm dependencies -> https://v1.pantsbuild.org/3rdparty_jvm.html In

pants

v2 a new concept of dependencies inference is introduced. (https://blog.pantsbuild.org/automatically-unlocking-concurrent-builds-and-fine-grained-caching-on-the-jvm-with-dependency-inference/) What I'm trying to understand is how pant v2 suggests to handle transitive dependencies & conflicts? Should transitive dependencies. be listed explicitly?

✅ 1

happy-kitchen-89482

10/05/2022, 9:35 PM

Funny you should ask, I’ve been meaning to look into this.

👍 1

happy-kitchen-89482

10/05/2022, 9:36 PM

Right now there isn’t, the lambda implementation uses

lambdex

under the covers, as a shortcut, but it ~~absolutely~~ probably doesn’t need to

happy-kitchen-89482

10/06/2022, 1:03 AM

Soliciting option re output paths for exporting venvs under `dist/`:

👀 1

bumpy-noon-80834

10/06/2022, 9:20 AM

I may be too much of a perfectionist, and it breaks my heart to see a 500k overhead. 😅 But the truth is that it works well and I can definitely live with it!

bumpy-noon-80834

10/06/2022, 10:21 AM

Did someone find a way to run Pulumi with Pants? I tried to manually set PYTHONPATH to the exported virtualenv to satisfy the "pulumi" command but it still fails. With the same trick, I tried to

import pulumi

, and it fails with:

ImportError: cannot import name 'cygrpc' from 'grpc._cython'

. However, I can

import pulumi

without any issue when I provision a shell with

./pants repl ::

. In this case, cygrpc is imported from

~/.cache/pants/named_caches/pex_root/installed_wheels/b821403907e865e8377af3eee62f0cb233ea2369ba0fcdce9505ca5bfaf4eeb3/grpcio-1.47.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/grpc/_cython/cygrpc.cpython-39-x86_64-linux-gnu.so

I'll try to write a standalone Pulumi program and run it as a PEX. But that's sub-optimal as I'd loose a lot of Pulumi's tooling. Any suggestion?

bumpy-noon-80834

10/06/2022, 1:18 PM

So I guess there is no way to avoid bundling boto3? I'd like to do that because its already available remotely (ref: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html)

happy-kitchen-89482

10/06/2022, 3:05 PM

Not yet, but this is work we’d like to do

👍 1

happy-kitchen-89482

10/06/2022, 3:06 PM

You can at least exclude

boto3

using the

!!

syntax in the `python_awslambda`’s

dependencies=

happy-kitchen-89482

10/06/2022, 3:07 PM

E.g.,

happy-kitchen-89482

10/06/2022, 3:07 PM

dependencies=['!!3rdparty/python#boto3', ...]

bumpy-noon-80834

10/06/2022, 3:08 PM

Where can I find docs about this syntax? Would that keep boto3 in the lockfile?

ancient-vegetable-10556

10/06/2022, 3:15 PM

@broad-processor-92400 Hey, I noticed you’re doing a bunch of work on the bootstrapping side of Pants. I’ve got a partially-working stand-alone binary implementation using PyOxidizer; is that something that you might be interested in helping getting to work/would it being more complete be of use to you?

happy-kitchen-89482

10/06/2022, 3:42 PM

Yeah, it means your tests and everything else still use boto3 from the lockfile, but the packaged lambda would not

happy-kitchen-89482

10/06/2022, 3:42 PM

See here: https://www.pantsbuild.org/docs/targets

bumpy-noon-80834

10/06/2022, 3:45 PM

Awesome, thanks!

hundreds-father-404

10/06/2022, 4:45 PM

A new "good first issue": eagerly error if a plugin author adds multiple

sources

fields to a target type. Pants won't work as expected, but we don't eagerly error in this case, which is a gotcha. Should be a couple lines to implement 🙂 https://github.com/pantsbuild/pants/issues/17132

🙌 1

✅ 1

brave-planet-28195

10/06/2022, 5:56 PM

Hey guys, me again. I'm currently struggling to migrate my private POC project to use pants v2. That POC projects uses most popular libs from Scala world. The problem is that managing dependencies in JVM world is kinda a headache (and it has always been a headache). For example if you run

cs resolve -t org.typelevel:log4cats-slf4j_2.13:2.5.0

for a small library

log4cats

it will already show you conflicts among transitive dependencies (and its a single lib, not a group of them). Conflicts resolution is very very important topic for every build tool that is used for Java/Scala project (if project is not

Hello world

of cause).... I'm pretty sure JVM community will be keen on adopting pants v2 if there was any support for conflict resolution strategy. For example doing something like this would be very helpful.

Copy code

versionReconciliation ++= Seq(
  "org.typelevel" %% "cats-core" % "relaxed",
  "*" % "*" % "strict"
)

https://www.scala-lang.org/2019/10/17/dependency-management.html Anyways thank u for great tool! Looking forward for new features (heard that

scala_library

is on its way....)

busy-vase-39202

10/06/2022, 6:06 PM

📣 Recruiting? If you want team members who prioritize code quality, developer productivity, and excellent tooling, then this is a great place to find them! Job opening posts are welcome in #general or #random, whichever you prefer.

👀 1

brash-student-40401

10/06/2022, 8:06 PM

What's the proper Pants way to replace an existing

data_files

option from a

setup.py

? For background, I am moving internal packages to a Monorepo. We have

packageA

, which had the

data_files

. I moved that code fine, and can run it with toy examples no problem. But then I move over

packageB

, which calls code that relies on those files. But Pants has no knowledge of them at all, so of course they don't end up where they're supposed to. The documentation states that

you cannot use data_files, install_requires, namespace_packages, package_dir, package_data, or packages because Pants will generate these for you, based on the data derived from your code and dependencies.

, but when it's not generated for me how am I supposed to get the files where they need to be?

careful-address-89803

10/06/2022, 11:45 PM

Hey folks, I'm probably doing amending silly, but I keep getting "no module named ..." for modules that need remapping. For example, azure-identity provides azure.identity, so I have in module_mapping

"azure-identity": ["azure.identity"]

. The dependency on that package shows up in

./pants dependencies ...

, And the package shows up in the message about which packages it's putting in the pex, and the whl for that package shows up in the pex sandbox. This doesn't seem to be a problem for tests, though, as they run just fine. I'm pretty sure it has something to do with resolves, as some of them work with

./pants run

and others don't. Short of binary-searching my dependencies, is there anything I can do to debug what's going on?

✅ 1

brave-planet-28195

10/07/2022, 9:55 AM

Not sure if it's a bug or not.... In jvm you need to use

jvm_artifact.excludes

heavily to fix transitive dependencies conflicts. That brings the idea of grouping several jvm_artifacts together to form

working groups

(groups of artifacts that 100% have no conflicts and can work nicely together) Based on pants documentation here - "*target* ...This can be used as a generic "bag of dependencies", i.e. you can group several different targets into one single target so that your other targets only need to depend on one thing...." But if I try to include empty target in the dependencies of

scala_library

I get this:

Copy code

ClasspathSourceMissing: No JVM classpath providers (from: CompileScalaSourceRequest, CoursierFetchRequest, DeployJarClasspathEntryRequest, JvmResourcesRequest) were compatible with the combination of inputs:
  * src/main/scala/com/testingzone/test:test    (target)

Should this be logged as issue? Or there is a better way of grouping scala/java dependencies and then using them? P.S: As a temporary workaround I created folder with empty package object

depends/package.scala

and defined dependencies like this:

Copy code

scala_sources(
    name = "fs2",
    sources = ["package.scala"],
    dependencies = [
        "3rdparty/jvm/co/fs2:fs2-core",
        "3rdparty/jvm/co/fs2:fs2-io",
    ]
)

👍 1

refined-addition-53644

10/07/2022, 2:42 PM

Hi, what’s the way to do module mapping for modules which are added under

extra_requirements

. My exact case is with mypy. I added some stub for boto3 called

boto3-stubs

but the package itself is

mypy_boto3_batch

when I need to use the import. e.g.

Copy code

from mypy_boto3_batch.client import BatchClient

Hence pants is generating warnings about not being able to infer the owner of such import. This is the relevant section of

pants.toml

Copy code

[mypy]
version = "mypy>=0.971"
extra_requirements.add = [
  "pydantic>=1.9.1",
  "pandas-stubs>=1.4.4.220919",
  "types-requests>=2.28.11",
  "boto3-stubs[s3,batch]>=1.24"
]
extra_type_stubs = ["pandas-stubs>=1.4.4.220919", "types-requests>=2.28.11", "boto3-stubs[s3,batch]>=1.24"]
lockfile = "build-support/mypy.lock"

happy-kitchen-89482

10/07/2022, 4:30 PM

Happy Friday folks! For those not yet aware - if you’re want to speed up your CI workflows, we at Toolchain offer a remote cache service for Pants. The pantsbuild/pants repo itself uses this for dramatic speedups! You can get started with a free trial here: https://toolchain.com/pricing, and/or DM me for more details.

🚀 2

🤸 3

🚤 2

curved-television-6568

10/07/2022, 6:59 PM

Pants TOD (Tip Of the Day): If you ever felt like it would be nice if Pants had this little extra feature.. but don’t want to invest in writing a custom plugin to implement another goal? Todays tip will present an alternative approach that gives the same feel but with a very lightweight integration. Consider this UX:

Copy code

$ ./pants my-feat --help
Usage: python -m my-feat [OPTIONS]

Options:
  --feat-1
  --another
  --help
...

This is possible by using

[cli.alias]

pants.toml

and a

pex_binary

target some where in your first party code base (could be a utility tree living outside your regular application code, it just needs to be in a source root).

Copy code

[cli.alias]
my-feat = "run src/utils/my_feat.py --"

Then in

src/utils/BUILD

Copy code

pex_binary(.., entry_point="my_feat.py")

Then implement

my_feat.py

as any regular cli tool, using

click

or what makes most sense to you.

👍 1

🙌 5

✅ 1

bumpy-noon-80834

10/07/2022, 8:49 PM

In some tests, I use pytest as a library (e.g.

pytest.mark.parametrize

./pants lint ::

fails because pytest is not available in the pylint environment. I fixed it by adding a

python_requirement()

for pytest. But this feels weird, as pytest is the standard test runner used by Pants and because now I could end up using a different version of pytest as a library vs. as a test runner. And I have to repeat that standard requirement almost everywhere. Any better way to handle this?

ripe-processor-39843

10/08/2022, 12:00 AM

i’m seeing an issue running

./pants lint --changed-since=origin/master

that just started happening out of the blue, i think it might have started happening after an osx system update. error output is:

Copy code

16:58:10.09 [INFO] Initializing scheduler...
16:58:10.26 [INFO] Scheduler initialized.
16:58:11.23 [ERROR] The `--changed-*` options are only available if Git is used for the repository.
Traceback (most recent call last):
  File "/Users/lee/.cache/pants/setup/bootstrap-Darwin-arm64/pants.jKTvcH/install/lib/python3.9/site-packages/pants/bin/daemon_pants_runner.py", line 125, in single_daemonized_run
    runner = LocalPantsRunner.create(
  File "/Users/lee/.cache/pants/setup/bootstrap-Darwin-arm64/pants.jKTvcH/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 152, in create
    specs = calculate_specs(
  File "/Users/lee/.cache/pants/setup/bootstrap-Darwin-arm64/pants.jKTvcH/install/lib/python3.9/site-packages/pants/init/specs_calculator.py", line 57, in calculate_specs
    raise InvalidSpecConstraint(
pants.init.specs_calculator.InvalidSpecConstraint: The `--changed-*` options are only available if Git is used for the repository.

Pretty stumped on this, the directory i’m in is definitely a git repo. Like I said this was working fine for me for a long period of time, and only started breaking recently i think related to an osx update, but yea, having trouble getting to the bottom of it. anyone see something like this before or have any suggestions?

bumpy-noon-80834

10/08/2022, 10:36 PM

I've written a Lambda function using pandas, s3fs and fastparquet. I defined it as:

Copy code

python_awslambda(
    name = "lambda",
    dependencies = [
        "!!:types-requests",
        ":s3fs",
        ":fastparquet",
    ],
    handler = "lambda.py:handler",
    runtime = "python3.9",
)

However, when running it on AWS, I get:

Copy code

{
  "errorMessage": "Install s3fs to access S3",
  "errorType": "ImportError",
[...]

I tried to run it locally and I get:

Copy code

$ lambdex test ./dist/packages.data-fred.src.data_fred/lambda.zip /tmp/event.json 
Traceback (most recent call last):
  File "/home/pmuller/.local/bin/lambdex", line 8, in <module>
    sys.exit(main())
  File "/home/pmuller/.local/lib/python3.9/site-packages/lambdex/bin/lambdex.py", line 291, in main
    args.func(args)
  File "/home/pmuller/.local/lib/python3.9/site-packages/lambdex/bin/lambdex.py", line 207, in test_lambdex
    runner = EntryPoint.parse("run = %s" % lambdex_entry_point).resolve()
  File "/home/pmuller/.local/lib/python3.9/site-packages/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 2481, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/pmuller/.pex/unzipped_pexes/2c5fccf932b3d8caf958b1b9e39ecf41464b6f78/data_fred/lambda.py", line 6, in <module>
    from pandas import DataFrame
  File "/home/pmuller/.pex/installed_wheels/0e2227a9f4be0a9ef2995a85f08c068f142d53f2fd8289da259c29ee633290de/pandas-1.5.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/pandas/__init__.py", line 16, in <module>
    raise ImportError(
ImportError: Unable to import required dependencies:
numpy: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    <https://numpy.org/devdocs/user/troubleshooting-importerror.html>

Please note and check the following:

  * The Python version is: Python3.9 from "/usr/bin/python3.9"
  * The NumPy version is: "1.21.5"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

However, I tried to package it with pex and it works fine:

Copy code

pex_binary(
    name = "pex",
    dependencies = [
        ":s3fs",
        ":fastparquet",
    ],
    entry_point = "lambda.py",
)

Any idea what I did wrong? (simple lambdas, without heavy dependencies like pandas & co work fine)

bumpy-noon-80834

10/09/2022, 9:56 PM

I consider adding a few TypeScript-based projects in a Pants/Python monorepo. I understand Pants doesn't support TypeScript so far. On one side, I enjoy Pants for Python, on the other, I enjoy yarn for TypeScript. Does anyone has any experience/idea how to integrate those together? I am not even sure what to look for. Just looking for inspiration. I guess that by default, I'll manage a /python prefix with Pants and a /typescript prefix with yarn.

adamant-magazine-16751

10/10/2022, 11:37 AM

Hi, is it possible to override an undeclared first-party dependency? I have a directory containing 2 python source files, say

a.py

and

b.py

a.py

depends on

b.py

without explicitly importing it. I also have a BUILD file containing

python_sources

target. Is there a way to tell pants that

depends on

so that it's properly packaged in pexes?