pantsbuild #general

Channels

welcome

github-notifications

announce

curved-manchester-66006

01/06/2023, 12:39 AM

I've been trying out the Helm backend. It is really spiffy! Since it is experimental/preview, are tickets still the right way to do bugs/feature-requests? Or is that premature?

curved-television-6568

01/06/2023, 2:24 AM

yes, issues on GH is perfectly OK, thanks! 🙂

dazzling-elephant-33766

01/06/2023, 12:32 PM

Hi everyone, I’m evaluating migrating a monorepo to pants. So far I’ve managed to get everything building and it’s looking great. A lot of services were written under the assumption a single DB connection would be kept/shared amongst all the tests for that service (I’m sure this is not uncommon in python webdev) i.e.

Copy code

service/a/test/test_a.py
service/a/test/test_b.py 
service/a/test/test_c.py

Will re-use the same DB. Since pants runs each pytest in a separate process I had to create a unique DB name for each and run any initial migrations xN times. This overhead unfortunately makes the tests slower than they were before, even with parallel execution. Is there’s any mechanism to force pytest to run tests in a single process? Passing

--debug

still appears to run the DB setup/teardown once per test file. Whereas ideally I’d like it to run once for the whole suite.

busy-vase-39202

01/06/2023, 2:10 PM

More than OK, welcomed and much appreciated. Thank you for contributing to improving the Helm support!

faint-dress-64989

01/06/2023, 3:29 PM

Using pants publish for shipping container to a container registry works beautifully! I was wondering if publish could also be used for moving pex files to s3 for different distribution use cases?

loud-laptop-89838

01/06/2023, 4:00 PM

I'm curious what is the typical approach to managing requirements in a pants monorepo. In particular, a new requirement. So let's say I open a python file and I want to use a new library. Pants hasn't seen it before, so it won't be in my `./pants export`ed venv. It's also not in my requirements file. So do I add a line to my requirements file, then

export

, then my interpreter should be updated? Just wondering what everyone is using as a workflow. And what about separating requirements for different areas of the repo? E.g. do you have a top level requirements file, then one in each module? Are you using

pyproject.toml

? Again, in each module? Or just one for the whole repo.

plain-night-51324

01/06/2023, 9:29 PM

anyway to specify a default for test like

Copy code

./pants --changed-since=HEAD test

but default to changed-since

green-match-66337

01/07/2023, 1:53 PM

Hey, I have a request on a repo I manage to include tests within the sdist… can’t figure out how to get pants to include the tests and the tests dependencies (some json files) https://github.com/ghandic/jsf/pull/53

green-match-66337

01/07/2023, 1:57 PM

Build seems to work fine - but when I extract the sdist it’s missing all the tests

green-match-66337

01/07/2023, 2:01 PM

Only way around I found was to explicitly make a target to define the tests and then use that in the dependency for the distribution target

prehistoric-motorcycle-70707

01/08/2023, 4:50 AM

Hey all, Im trying to use Pants to build an AWS Lambda function. Ive been able to pretty much get it all working, except for one catch. Our code relies transiently on the

cryptography

library, and as I understand it, AWS Lambdas require a specific binary for that (and probably other) libraries. I was able to resolve that pre-Pants by following the instructions on this Github issue, but Im struggling to figure out how to get that working with the

python_awslambda

target in Pants. Pre-Pants I can run this pip command to fetch the correct binary

pip install --platform manylinux2014_x86_64 --implementation cp --python 3.9 --only-binary=:all: --upgrade --target ./ cryptography==38.0.1

. But I dont understand how to translate this information to Pants and Pex. Im doing my best to understand, but pretty much everything Im doing is a stab in the dark 😂 I assume I need to do something with the

complete_platforms

argument in the

python_awslambda

target. So I created a

file

target to a

lambda_platform.json

file, and provided that to the

complete_platform

argument. The json looks like this:

Copy code

{
  "marker_environment": {
    "platform_release": "manylinux2014_x86_64",
    "implementation_name": "cp",
    "python_version": "3.9"
  },
  "compatible_tags": [
    "cp39-abi3-manylinux2014_x86_64"
  ]
}

The package goal prints the correct platform file, however if I look at the cryptography dist info in the generated lambda, the

WHEEL

file differs from what is generated when I run the pip command, and it doesnt seem to include the

manylinux2014_x86_64

platform. So it doesnt appear to have worked. Does anyone have any advice or thoughts here? Am I on the right track? (Side note, changing the lambda_platform.json contents doesnt trigger a new package. I would've thought Pants would re-build if that dependency changed)

bored-energy-25252

01/08/2023, 5:00 PM

Created a Github Organization to host: Community Driven Projects for democratizating Data+ML (AI in 202x) https://github.com/liga-ai All projects will adopt Pants as the build tool for Python and Scala!

🎉 4

👖 2

modern-monkey-78364

01/09/2023, 7:21 AM

Hi, I facing an unusual issue. Looking for some feedback. I have working branch with all the test cases running. I had added

--no-binary=protobuf

in my requirements.txt file to resolve this issue. But unfortunately, after adding this, I am not able to decode some of my protobuf objects at runtime, and hence I am removing this part. But after removing one of my test case doesn’t run at all. It keeps showing something like:

Copy code

[INFO] Long running tasks:
             Run Pytest for src/backend/unit_test.py:tests

So, looking at some help in debugging this, as the new commit has no other change except removal and --no-binary=protobuf and subsequent change in pytest_lockfile.txt after running

./pants generate-lockfiles --resolve=pytest

✅ 1

breezy-agent-54573

01/09/2023, 12:08 PM

Hey everyone, I am trying to use pants to generate openapi python code from a swagger file. I have looked at this page https://www.pantsbuild.org/docs/plugins-codegen , but it seems a little bit complex just for a couple of shell commands. What I tried to do is to generate the code using experimental_shell_command and then load it using python_source. the first step works(i have confirmed it in the created sandbox) but the second step ignores the files that were generated by that command(I have added the first command as dependency for the second one). My question is, Is there a way to include those generate files as sources or maybe some way to generate python code other than that way (https://www.pantsbuild.org/docs/plugins-codegen) ? the BUILD file i used:

Copy code

experimental_shell_command(
    name="generate_external_client_sources",
    command='echo $PWD; docker run --rm -v ${PWD}/..:/local openapitools/openapi-generator-cli:v6.0.1 generate -i /local/external_brain_swagger.yaml -o /local/ -g python --package-name brain_client.external --additional-properties="generateSourceCodeOnly=True"',
    tools=["bash", "docker"],
    outputs=["external/"],
    dependencies=["Brain:generate_external_swagger"],
)

experimental_shell_command(
    name="generate_internal_client_sources",
    command='echo $PWD; docker run --rm -v ${PWD}/..:/local openapitools/openapi-generator-cli:v6.0.1 generate -i /local/internal_brain_swagger.yaml -o /local/ -g python --package-name brain_client.internal --additional-properties="generateSourceCodeOnly=True"',
    tools=["bash", "docker"],
    outputs=["internal/"],
    dependencies=["Brain:generate_internal_swagger"],
)

python_sources(
    name="brain_client",
    dependencies=[
        ":generate_external_client_sources",
        ":generate_internal_client_sources",
    ],
)

✅ 1

icy-umbrella-50644

01/09/2023, 5:12 PM

Hi, I recently came across a problem in Bazel where you have a rule that outputs a directory of randomly named files instead of a few known files. This means you can't add it to the build graph because that only deals in files, not directories. The currently recommended workaround is to zip/unzip the directory before/after running the rule. Not idea. I was wondering if Pants handles this any better?

chilly-tailor-75063

01/09/2023, 5:21 PM

Hi all, maybe it’s me, but I continue on my pants path of 3 steps forward and 2 back. 😉 I briefly had pants working for me, and I’m not sure what changed, but now when I run

./pants lint ::

it reports a bunch of

E0401: Unable to import

errors for my own custom modules.

chilly-tailor-75063

01/09/2023, 5:21 PM

And as usual, I’m not sure what I should look at to resolve this

prehistoric-wire-35783

01/09/2023, 7:09 PM

Hello everyone! Is there a way to make

./pants check

behave equivalently to

mypy --install-types --non-interactive

? Reading through the past chats, I think the answer is no, but I just wanted to double check. I presume the current solution was put in place to create the pex environment. While is not bad, it would be nice if it can support install type stubs on the fly for minimal maintenance...

freezing-lamp-12123

01/09/2023, 9:07 PM

is pants suited for running services / long running processes, in conjunction with e2e testing of a system?

aloof-appointment-30987

01/10/2023, 1:03 AM

Cross compiling goLang for Docker... Is there a way to specify the foreign platform when generating the
go_binary
? I've confirmed that if I build using

Copy code

env GOOS=linux GOARCH=arm go build .

then I create a

file

target that references the binary result of the above command and copy that file into a

Dockerfile

image and run using

./pants run path/to/project:docker_image

The docker image executes the binary as expected. If I instead copy the output of

go_binary

into the

Dockerfile

image, I get the failure

exec /app: exec format error

This indicates the go binary is built for the wrong platform. I've tried

Copy code

env GOOS=linux GOARCH=arm ./pants run path/to/project:docker

without success.

fresh-cat-90827

01/10/2023, 7:01 AM

Working with the options semantics for the JSON schema of our

pants.toml

file. Is there any special case where an option that represents a collection won't be supporting the following types 🧵

prehistoric-wire-35783

01/10/2023, 11:38 AM

Hello I am encountering an exception coming from Pants's Rust engine in CI/CD. Would anyone know a workaround?

Copy code

Exception: Snapshot failed: Failed to digest inputs: Throw { val: Failed to watch filesystem for `/some/path`: Error { kind: MaxFilesWatch, paths: ["/some/path"] }, python_traceback: "Traceback (no traceback):\n  <pants native internals>\nException: Failed to watch filesystem for `/some/path`: Error { kind: MaxFilesWatch, paths: [\"/some/path\"] }", engine_traceback: ["digest_file"] }

abundant-analyst-12845

01/10/2023, 11:55 AM

hi just a quick question. If I wanted to introduce pants for a poetry project that has has an existing

poetry.lock

file. should the

python.resolves

block point to the existing lock file like

common = poetry.lock

or more like

common = "3rdparty/python/common.lock"

am hoping to use this global poetry file to manage many "sub-projects" which will host lambda

echoing-farmer-15630

01/10/2023, 2:17 PM

I think while I've been away from

pants

, the dependency analyzer got too good 🙂 Silly problem: I'm trying to extract a subset of a library which tried to hide its dependencies by putting imports under

if

statements, which I think used to work (but I may be wrong) ie something like:

Copy code

def hide_imports(library: str):
    if library == "numpy":
        import numpy as result
    elif library == "pandas":
        import pandas as result
    return result

but

pants

cannot be fooled:

Copy code

❯ ./pants dependencies --no-dependencies-transitive src/.../test.py
//:root_requirements#numpy
//:root_requirements#pandas
...

(the situation is more complex with internal dependencies, but you get the idea; what if we wanted to build a library which only included

numpy

as a dependency and would be ok if someone called

hide_imports("pandas")

?) Is there any way to build a

python_distribution

pex_binary

with "only obvious" dependencies enclosed or expressly excluding some dependencies? Trying to build a subset of a library for a client without exposing more IP than necessary.

refined-addition-53644

01/10/2023, 3:12 PM

Hi I have been seeing some tests failing locally on my macos (not M1). They pass in our CI but locally I get errors like. Not sure where these exit codes are coming from? Same test passes when run directly using pytest even locally. I am on pants 2.14.0

Copy code

16:09:38.20 [ERROR] Completed: Run Pytest - tests/test_main.py failed (exit code -11).

abundant-analyst-12845

01/10/2023, 3:12 PM

hi is it possible to ensure

pants tailor ::

only generate BUILD files for directories mentioned within root_pattern block instead of everything

brash-student-40401

01/10/2023, 3:24 PM

This is hopefully a quick/silly error on my part, but I can't seem to get past it - my

./pants publish src/services/app/lambdas/svc1/Dockerfile

is failing with

Copy code

stderr:
COPY failed: file not found in build context or excluded by .dockerignore: stat src/services/app/lambdas/svc1/requirements.txt: file does not exist

Anything referenced in that directory, which does in fact exist, is failing. What's the proper format for referencing stuff in the Dockerfile?

glamorous-nail-59544

01/10/2023, 4:03 PM

Hello friends, I hit a new wall and would appreciate your help 🙂 So I am following the example-django repo to migrate many django microservices to pants, the problem I am having now is when running tests that access the DB, or when I try to run

makemigrations

I am getting

django.db.migrations.exceptions.NodeNotFoundError: Migration app.0010_auto_20211020_1418 dependencies reference nonexistent parent node ('matching', '0009_auto-20211015')

I have double checked dependencies and everything is configured correctly AFAIK with the right dependencies showing. The error is showing for a different migration file everytime I run it, so it is not a problem with one specific migration file.

happy-kitchen-89482

01/10/2023, 4:23 PM

If you run

./pants dependencies path/to/manage.py

do you see `matching`'s migrations in there?

busy-vase-39202

01/10/2023, 8:05 PM

📣 Interested in blogging about Pants? You're welcome to pitch topics for the Pants Blog. (DM me!) We also encourage writing for your company blog or personal blog. Some topics that are particularly good:

🫅 4