pantsbuild #general

Channels

welcome

github-notifications

announce

bored-energy-25252

08/17/2022, 9:58 AM

Is there a Pants plugin for Databricks Integration? Like: • uploading the whl/jar to databricks libraries • manage the databricks clusters Or could pants community start a project like that?

bored-energy-25252

08/17/2022, 10:00 AM

For Data/ML engineering, Scala/Python/SQL is the first class language. I think adding SQL to the supported language is a good idea.

bored-energy-25252

08/17/2022, 10:07 AM

Pants monorepo for a company is not as attractive as Bazel monorepo. Python is the first class language of Pants. Modern Data/ML Stack relies heavily on Python. Let Data/ML dance with Pants!

👯 6

late-ram-12573

08/17/2022, 1:35 PM

Hi Everyone, I'm just getting started with pants, trying to move my team to a monorepo. I've been experimenting with setting up a test project to get a feel for pants, but my updates to files are not getting picked up. For example, my source root pattern is "src/*". if I run pants roots, I see 3 directories I made, A, B, and C. If I add directory D and rerun pants roots, I still only see A, B, and C. I noticed that if I make an adjustment to my pants.toml file, it reinitializes the scheduler and the changes are then picked up. After making an adjustment and rerunning pants roots, I now see A, B, C and D. This is just an example with roots, but the same issue is happening with any changes I make to BUILD files, which is incredibly frustrating. Has anyone else experienced this? I'm currently on Windows 10 and using WSL

👋 2

cool-yacht-37128

08/17/2022, 2:25 PM

Is there a way to pass in an environment variable to

experimental_shell_command

hundreds-father-404

08/17/2022, 3:27 PM

Everyone, please welcome @sparse-lifeguard-95737 as the newest Pants Contributor! Contributions include: • pytest-xdist support •

export

includes Python tools • updating Pants to use Py37+ by default • adding env var support to

experimental_shell_command

Dan is also quite active in this Slack, often helping others and sharing valuable feedback. Thanks Dan for making the community a better place to be! 🎉 Posted in #announce

🎉 5

🚀 2

👖 2

flat-zoo-31952

08/17/2022, 5:40 PM

🚲 🏚️ "Experimental" name bikeshed time 🥼 🧫 In follow up from the last contributor meeting, it's time to talk about the "experimental" prefix we're attaching to some goals, options, and targets to suggest that they will change. While it's good that we have a way to get feedback while explicitly saying that we're not pushing a final version, I don't think just prefixing things with "experimental" is the way forward. Especially if it's included as part of "normal" output: it makes Pants as a whole look a bit more experimental than it needs to be. But there would need to be alternatives to this if we changed it. More in 🧵

bored-glass-58755

08/17/2022, 7:12 PM

Hi, I am trying to publish a python library using pants and believe I should be using the publish goal but can’t find any relevant documentation for [publish] scope in the config file. Could anyone point me to the right direction?

bitter-ability-32190

08/17/2022, 7:32 PM

New PR ready for having plugins which format (and subsequently lint) your BUILD files: https://github.com/pantsbuild/pants/pull/16560 🙂

💯 2

clever-ghost-87030

08/17/2022, 7:45 PM

I'm using Pants 2.12 in a self-hosted github runner and running in to an issue. I set up python versions for 3.9 and 3.10 on the runner but I get the following error when it runs

./pants generate-lockfiles fmt ::

ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython<4.0,>=3.9' failed with exit code 102.

It seems to think the interpreter is broken:

Copy code

Skipped the following broken interpreters:
1.) /mnt/github_actions_runner/_work/_tool/Python/3.10.6/x64/bin/python3.10:
/mnt/github_actions_runner/_work/_tool/Python/3.10.6/x64/bin/python3.10: error while loading shared libraries: libpython3.10.so.1.0: cannot open shared object file: No such file or directory
2.) /mnt/github_actions_runner/_work/_tool/Python/3.9.13/x64/bin/python3.9:
/mnt/github_actions_runner/_work/_tool/Python/3.9.13/x64/bin/python3.9: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory

Not sure how to fix this, any help would be appreciated

incalculable-grass-76623

08/18/2022, 12:12 AM

I am running to conversions of a few repos to Pants and am wondering if there are techniques for performance improvements in how or what I am doing ...

Copy code

./pants test libs/somelib/tests/decorators/test_foobar.py

After this begins,

Pants

starts by building a

pex

from the requirements.txt .. (and tests_requirements.txt)

Copy code

10:06:26.17 [INFO] Canceled: Building requirements.pex with 31 requirements: ... etc ... (381 characters truncated)
10:06:32.25 [INFO] Completed: Building pytest.pex from pytest_default.lock
10:07:08.96 [INFO] Completed: Building requirements.pex with 31 requirements: ... etc ... (381 characters truncated)

the requirements are in the

BUILD

file as

Copy code

python_requirements(
    name="requirements"
)

python_requirements(
    name="test_requirements",
    source="test_requirements.txt",
)

two questions. 1. why does Pants "Cancel" the first build but then proceed immediately to build it. 2. how can I get Pants to cache this file, it hasn;t changed and everytime i run the test, it rebuilds the requirements.pex which is 60s.

cold-sugar-54376

08/18/2022, 2:48 AM

is there a way to upgrade the pip version that pants uses or pass flags into pip? getting this error when trying to package an app that uses

numpy

(for example):

Copy code

ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly
  WARNING: You are using pip version 20.3.4; however, version 22.2.2 is available.

This is on an M1 mac (12.5) with pants 2.12.0

✅ 1

happy-kitchen-89482

08/18/2022, 3:38 AM

Funny you should ask: @enough-analyst-54434 is currently working on supporting newer versions of pip.

happy-kitchen-89482

08/18/2022, 3:39 AM

But how sure are you that this is the problem?

happy-kitchen-89482

08/18/2022, 3:40 AM

What versions of Python and numpy are in play here?

cold-sugar-54376

08/18/2022, 3:50 AM

python 3.9.10, numpy 1.23.1

cold-sugar-54376

08/18/2022, 3:51 AM

i’m not sure why its trying to build the numpy wheels since they are available on pypi

cold-sugar-54376

08/18/2022, 3:51 AM

wonder if it has something to do with the fact that i’m on 12.5

happy-kitchen-89482

08/18/2022, 4:23 AM

Hmm that numpy has a wheel that should be compatible: numpy-1.23.1-cp39-cp39-macosx_11_0_arm64.whl

happy-kitchen-89482

08/18/2022, 4:25 AM

I’m macOS 12.4 on an M2, and that wheel is installed if I

pip install numpy==1.23.1

in a virtualenv

happy-kitchen-89482

08/18/2022, 4:25 AM

What happens if you bypass pants/pex entirely on your system and use pip directly in a python3.9 virtualenv?

cold-sugar-54376

08/18/2022, 4:43 AM

mine builds from source

cold-sugar-54376

08/18/2022, 4:47 AM

Copy code

Collecting numpy==1.23.1
  Downloading numpy-1.23.1.tar.gz (10.7 MB)
     |████████████████████████████████| 10.7 MB 14.7 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: numpy
  Building wheel for numpy (PEP 517) ... done
  Created wheel for numpy: filename=numpy-1.23.1-cp39-cp39-macosx_12_0_arm64.whl size=5080188 sha256=40bc478bef790e163c1e73a5a4dd3ada5bee6a29a66ff35c12a5a800ef9dc665
  Stored in directory: /Users/ashu/Library/Caches/pip/wheels/e9/08/af/c32ddeb26bf7cf91805eb7b3db6db127625aed4c121f7ad811
Successfully built numpy
Installing collected packages: numpy
Successfully installed numpy-1.23.1

cold-sugar-54376

08/18/2022, 5:03 AM

fwiw this is the error I get when building w/ pex:

Copy code

assert tag in supported_tags, "would build wheel with unsupported tag {}".format(tag)
    AssertionError: would build wheel with unsupported tag ('cp39', 'cp39', 'macosx_12_5_arm64')

cold-sugar-54376

08/18/2022, 5:03 AM

looks like its using the wrong osx deployment target

average-flag-94768

08/18/2022, 8:43 AM

Hi, I'm having some trouble with generate-lockfiles hanging forever and leaving a process using 100% of a core in the background after I cancel the goal. Is there something I could be doing wrong to cause this or should I raise a bug?

bored-glass-58755

08/18/2022, 1:27 PM

Could anyone please help me understand the difference between backend_packages.add and backend_packages2.add? Is the latter to be used for pants v2? Are those interchangeable? I am using pants 2.12

✅ 1

😮 1

brief-dinner-12483

08/18/2022, 1:58 PM

Hi all. I'm trying to build a Docker container where I am installing tools that I need and modifying the

PATH

in the

Dockerfile

. Looks something like this:

Copy code

# Dockerfile

...
RUN git clone <https://github.com/asdf-vm/asdf.git> /root/.asdf --branch v0.10.0
ENV PATH=/root/.asdf/shims:/root/.asdf/bin:$PATH

COPY .tool-versions ./.tool-versions
RUN asdf plugin-add python && \
	asdf plugin-add nodejs && \
	asdf plugin-add yarn && \
	asdf install python && \
	asdf install nodejs && \
	asdf install yarn
...
RUN yarn run gulp compile
...

So I'm setting the

/root/.asdf/bin

path to be part of the

PATH

for the Dockerfile, but when the line

RUN yarn run gulp compile

executes in the

docker_image

target, I get the following error:

Copy code

Traceback (most recent call last):
  File "/pants/setup/bootstrap-Linux-x86_64/2.13.0rc0+git30af7f9c_py38/lib/python3.8/site-packages/pants/engine/process.py", line 272, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Searching for `python3` on PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/runner/.local/bin' failed with exit code 127.

python3

can't be found and

PATH

is missing

/root/.asdf/bin

. I can't tell if that's the

PATH

from my local host or if that's inside the container but I know that on my local I have:

Copy code

[python-bootstrap]
# All pants users should configure asdf for 3.8.
search_path = [
    "<ASDF_LOCAL>",
]

in my

pants.toml

, so pants should know to check for asdf locally, and I've set the

PATH

in the context of the container build. I'm not sure what my next steps are and I'd appreciate any help!

quaint-forest-8735

08/18/2022, 2:33 PM

Is there a way to specify the location of the

pex

executable for

pants_jupyter_plugin

instead of fetching it from GitHub? (ie. https://github.com/pantsbuild/pants-jupyter-plugin/blob/ae67015a7d011315973ccc4416fdabd115a79197/pants_jupyter_plugin/pex.py#L45-L50) We were attempting to use pex + pants_jupyter_plugin to isolate user notebook dependencies from the python environment used to launch jupyter notebook server in a deployment that was behind a heavily restricted firewall (that in particular was denying access to

<http://github.com|github.com>

*.<http://githubusercontent.com|githubusercontent.com>

), and the kernel failed to connect if the pex executable wasn't present on startup. Unfortunately we needed to stop using this in the near-term to unblock, but would love to figure out a path forward since using pex to isolate nb deps is pretty ideal for us

fresh-cat-90827

08/18/2022, 3:06 PM

I am trying to build a Python distribution (wheel) using the

python_distribution

target using a handwritten

setup.py

file. The docs say that

If you use a handwritten setup.py ... Pants will bundle whatever the script tells it to.

Running

python3 setup.py bdist_wheel

produces a wheel with expected contents, whereas

./pants package //:say-hello

produces a wheel with

dist-info

metadata only. Have I misunderstood how Pants runs the existing

setup.py

✅ 1