pantsbuild #general

Channels

welcome

github-notifications

announce

mysterious-waiter-14207

09/26/2022, 11:59 AM

For JVM languages, I wonder why pants chose to produce jar files and not class files? I think some performance will be gained by not zipping the class files, stripping the jar file and then for the scalac/javac to uncompress the jars. Plus I think if Pants will eventually want to batch the compilation phase, splitting the class files in a digest might be easier than splitting a jar file. I tried that locally and the switch was pretty simple, numbers look too good at the moment (50%) so I try to figure out the real improvement (I changed the cache scope of scalac to PerSession to test the improvement).

bitter-car-24102

09/26/2022, 12:14 PM

Apologies if this is the wrong place to post this, but I am having trouble getting started on MacOS 12.5.1. I am using python3.9 but not working.

Copy code

Installing pantsbuild.pants==2.13.0 into a virtual environment at /Users/ian/.cache/pants/setup/bootstrap-Darwin-arm64/2.13.0_py39
++ scrub_env_vars
++ pex_env_vars=(${!PEX_@})
++ local -r pex_env_vars
++ [[ ! 0 -eq 0 ]]
++ '[' -n '' ']'
++ /Users/ian/.pyenv/shims/python3.9 /Users/ian/.cache/pants/setup/bootstrap-Darwin-arm64/virtualenv-20.4.7/virtualenv.pex --quiet --no-download /Users/ian/.cache/pants/setup/bootstrap-Darwin-arm64/pants.QDZmgq/install
++ /Users/ian/.cache/pants/setup/bootstrap-Darwin-arm64/pants.QDZmgq/install/bin/pip install --quiet -U pip 'setuptools<58'
++ /Users/ian/.cache/pants/setup/bootstrap-Darwin-arm64/pants.QDZmgq/install/bin/pip install --quiet --progress-bar off pantsbuild.pants==2.13.0
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==2.13.0 (from versions: 0.0.17, 0.0.18, 0.0.20, 0.0.21, 0.0.22, 0.0.23, 0.0.24, 0.0.25, 0.0.26, 0.0.27, 0.0.28, 0.0.29, 0.0.30, 0.0.31, 0.0.32, 0.0.33, 0.0.34, 0.0.35, 0.0.36, 0.0.37, 0.0.38, 0.0.39, 0.0.40, 0.0.41, 0.0.42, 0.0.43, 0.0.44, 0.0.45, 0.0.46, 0.0.47, 0.0.48, 0.0.49, 0.0.50, 0.0.51, 0.0.52, 0.0.53, 0.0.54, 0.0.55, 0.0.56, 0.0.57, 0.0.58, 0.0.59, 0.0.60, 0.0.61, 0.0.62, 0.0.63, 0.0.64, 0.0.65, 0.0.66, 0.0.67, 0.0.68, 0.0.69, 0.0.70, 0.0.71, 0.0.72, 0.0.73, 0.0.74, 0.0.75, 0.0.76, 0.0.77, 0.0.79, 0.0.80, 0.0.81, 0.0.82, 1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0)
ERROR: No matching distribution found for pantsbuild.pants==2.13.0

bitter-car-24102

09/26/2022, 12:15 PM

Looking through the release files on pypi, I tried to install the macos 11 wheel but without luck:

bitter-car-24102

09/26/2022, 12:15 PM

Copy code

% /Users/ian/.cache/pants/setup/bootstrap-Darwin-arm64/pants.QDZmgq/install/bin/pip install <https://files.pythonhosted.org/packages/f9/b1/874493df9662137994a3d1311c2aa8861eee2bc41d4a6ec2a39ea0576954/pantsbuild.pants-2.13.0-cp39-cp39-macosx_11_0_arm64.whl>
ERROR: pantsbuild.pants-2.13.0-cp39-cp39-macosx_11_0_arm64.whl is not a supported wheel on this platform.

bitter-car-24102

09/26/2022, 12:16 PM

Pants looks great, just trying to figure out how to get started...

bitter-car-24102

09/26/2022, 1:29 PM

Building locally using sources works for me for now: https://www.pantsbuild.org/docs/running-pants-from-sources

handsome-sunset-98068

09/26/2022, 1:41 PM

I can't figure out why the github action is continuously failing. • Pant v2.12 • This action is run by a colleague in the team. • the workflow file:

Copy code

name: ci

on:
  pull_request:
    branches:
    - develop

concurrency:
  group: ci-${{ github.head_ref }}
  cancel-in-progress: true

jobs:
  build:
    name: Perform CI Checks
    env:
      PANTS_CONFIG_FILES: pants.ci.toml
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.9]
    steps:
    - name: Get the latest code 
      uses: actions/checkout@v3 # checkout latest commit 
    - name: Cache the artefacts
      uses: actions/cache@v3 # cache the pants, test results, pex binaries
      id: cache
      with:
        path: |
          ~/.cache/pants/setup
          ~/.cache/pants/lmdb_store
          ~/.cache/pants/named_caches
        key: ${{ runner.os }}-
    - name: Setup Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}
    - name: Bootstrap Pants
      run: |
        ./pants --version
    
    - name: Check BUILD files
      run: ./pants tailor --check update-build-files --check
    - name: Lint and typecheck all targets
      run: |
        ./pants fmt lint ::
    - name: Test all targets
      run: |
        ./pants test ::
    - name: Upload pants log
      uses: actions/upload-artifact@v3
      with:
        name: pants-log
        path: .pants.d/pants.log
      if: always()  # We want the log even on failures.

polite-garden-50641

09/26/2022, 3:20 PM

A couple of things you can try: Make sure you have the latest pants bootstrap script checked in your repo: https://github.com/pantsbuild/setup/blob/gh-pages/pants Enable debug mode for that script by changing this line https://github.com/pantsbuild/setup/blob/gh-pages/pants#L13 to be

set -eoxu pipefail

polite-garden-50641

09/26/2022, 3:22 PM

also as a side note, it is recommended to use the pants version is a cache key so

~/.cache/pants/setup

will get wiped when you upgrade pants. see: https://github.com/toolchainlabs/remote-api-tools/blob/main/.github/workflows/build.yaml#L24 as an example.

worried-painter-31382

09/26/2022, 3:34 PM

Hi! We still have trouble with running pants in a python subprocess. Some new findings: 1. It looks like

pantsd

isn't being re-used. We get

Copy code

17:26:09.72 [INFO] Initializing scheduler...
17:26:09.85 [INFO] Scheduler initialized.

repeatedly for subprocesses. 2. It looks like python2.7 is used for some pex unzip thingy? Is this expected? 3. We're having cache misses everytime the subprocess is rerun. Could it be due to pyenv, XDG_CACHE_HOME_DIR or something else? It's hard to figure out what settings causes pants to disregard its cache.

busy-vase-39202

09/26/2022, 4:25 PM

@careful-address-89803 please feel free to also open an issue, to request the additional info in

-ldebug

handsome-sunset-98068

09/26/2022, 4:52 PM

Thanks, it worked after upgrading the script and the pants version as well. And point taken for updating the cache key as well. 🙂

👍 1

brainy-solstice-27042

09/26/2022, 7:18 PM

I'm converting our system to use pants 2.13 instead of pants 1.30 and cannot seem to get the BUILD file configuration right to allow me to simply run a pex file without it turning into a python terminal OR reporting

ModuleNotFoundError: No module named 'main'

. So far I've tried setting the

entry_point='main.py'

entry_point='main.py:main'

, and even added an

__init__.py

at the base of the folder my service is in. Does the entry point path have to equal the path that pants generates the pex as? Or can it just be main.py?

broad-processor-92400

09/26/2022, 8:06 PM

I think this will be the same problem as https://github.com/pantsbuild/actions/issues/5, where the setup cache contains an invalid symlink due to the Python minor version being changed behind the scenes.

ambitious-actor-36781

09/26/2022, 8:46 PM

It’d be neat if generated code was run through formatters (selectively?) for you. We’re currently running our codegen output through BlackRequest->FmtResult and while it works it’s pretty janky due to how the rule(s) are cached and output is controlled. But short of copying the rule, it’s doesn’t seem that easy to do it in a cache-friendly way.

plain-carpet-73994

09/26/2022, 11:01 PM

I'm doing some work with pytorch. Sometimes I need to depend on the version of PyTorch that's CPU-only and sometimes I need GPU support (depending on if I'm building for a computer with or without a GPU). The dependencies are teh same in both cases; the only thing that's different is the

--extra-index-url

. For example, to run on a CPU you'd do

pip3 install torch --extra-index-url <https://download.pytorch.org/whl/cpu>

but for a GPU you'd do

pip3 install torch --extra-index-url <https://download.pytorch.org/whl/cu116>

. I know how to add an extra index URL to

[python-repos]

pants.toml

but I have no idea how to do that conditionally. Any ideas?

handsome-sunset-98068

09/27/2022, 8:15 AM

How to correctly include a non-py (.png) file into the binary, though it correctly works running locally? I am getting this error after deploying on lambda

Copy code

Response
{
  "errorMessage": "[Errno 2] No such file or directory: 'newcamera.png'",
  "errorType": "FileNotFoundError",
  "requestId": "2276ab1d-46e1-4c5f-ad34-18a5f75e3968",
  "stackTrace": [
    "  File \"/var/task/lambdex_handler.py\", line 58, in handler\n    return __RUNNER(*args, **kwargs)\n",
    "  File \"/var/task/test.py\", line 9, in lambda_handler\n    file_length()\n",
    "  File \"/var/task/test.py\", line 2, in file_length\n    with open('newcamera.png', 'rb') as f:\n"
  ]
}

repo link for reference.

✅ 1

cold-vr-15232

09/27/2022, 10:17 AM

Hi! I'm not sure if it's something I do wrong or if there is an issue: I started using a Python monorepo, and some of the test modules in different applications have the same name. Running mypy via "./pants check ::" gives an error because of that. Is there a way to get around that? I tried running separately on each source root, but that's one level too high, would need to list all directories under each root also.

cold-vr-15232

09/27/2022, 1:29 PM

Is there a way to create a pex binary whose name or path includes a version tag? I mean, through already existing means. I saw that pex has the bdist_pex command for setuptools, that seems to do what I want. How can I use it from Pants? And does it work for other build backends? I also saw that this can be done for Docker tags. My guess is that it is possible with a plugin, but that has not been written yet 🙂

cold-vr-15232

09/27/2022, 2:24 PM

I have one more question for today. I had my project working with all goals and all looked nice. I then added a dependency to a library (pydantic, but I don't think it matters) and the apps still run nicely, but the linting throws an error (No name 'BaseModel' in module 'pydantic'). I don't understand why would that happen, nor what to change to make it work again. Unfortunately, thins kind of issues are what make people anxious about trying a new tool, so I'd like to be prepared when I try to sell the idea to my team.

✅ 1

ancient-france-42909

09/27/2022, 3:08 PM

I’m trying to run pants on arm, and I built the wheel myself, but the docs aren’t very helpful in saying how I get the pants script to pick that up. Also, as a side note, the newer docs have the link to “this is how you build pants”, but they don’t have the actual docs anymore 🙂

✅ 1

wooden-thailand-8386

09/27/2022, 3:08 PM

Hey, qq - a coworker and I are trying to update the runtime for the Google Cloud Function to

python310

and we noticed that there’s a list of available targets that are being defined on

pants

. By checking out the code we found this ( https://github.com/pantsbuild/pants/blob/2fbd8bb618cbc06bb242e461be9e9942e7699e65/[…]thon/pants/backend/google_cloud_function/python/target_types.py ) would the change be as easy as adding the new runtime there? Also, this is more out of curiosity, but how does

pants

handles runtimes that are a different version than the python defined on

pants.toml

? Tks!

little-zoo-57121

09/27/2022, 3:36 PM

Hello all 👋 we're using the rc version of pants 2.14 (for the docker tags plugin capabilities), and it seems

pants filter

may be broken? When using filter values that worked in 2.12 or trying anything different we just get the following cryptic error. The only value that does not result in this is

--help

✅ 1

busy-vase-39202

09/27/2022, 4:36 PM

Thanks to @fresh-cat-90827 it recently came to our attention that the website's current description of the Pants Maintainer role sounds like more work than we actually intend. So @ancient-vegetable-10556 has proposed revised language. If you've ever been curious about the Maintainer role, please take a look at the proposed language and share your impressions. Does it feel like something you'd want to do in the future? That you'd feel qualified to do? Does it sound easier or harder to attain that role? Can you better see fitting it in with your job, personal life, etc? Does it leave you with unanswered questions? Please help us to communicate well about this influential role! ❤️

refined-addition-53644

09/27/2022, 5:37 PM

Hi, not sure what’s wrong but pants output lost colour in the terminal (VS code) suddenly. I am on pants 2.14.0rc1

boundless-zebra-79556

09/27/2022, 8:35 PM

I'm trying to run

mypy

check with pants. On my console it works fine. But on GitHub Action, I get this weird error message:

Copy code

Error: 9.77 [ERROR] Completed: Typecheck using MyPy - mypy failed (exit code 2).
.cache/pex_root/venvs/2dc0bc0943d3614000af98704b489da83561978d/f8dfd366c5bae0ea109e2675cda4aa418ef670f0/lib/python3.10/site-packages/numpy/__init__.pyi:636: error: Positional-only parameters are only supported in Python 3.8 and greater

It's weird because I'm using Python 3.10, and pants should be able to infer the version, right? Also weird that its checking a third-party package. I'm using pants 2.13.0 and mypy.ini settings are:

Copy code

[mypy]
namespace_packages = true
explicit_package_bases = true
ignore_missing_imports = true

high-yak-85899

09/27/2022, 8:46 PM

I did a tiny bit of browsing and didn't see what I needed. Is there a way to include architecture constraints for

generate-lockfiles

? For an example,

generate-lockfiles

found psycard==2.0.4 for me, but

export

was unhappy because that release doesn't have a supported linux option. While I can fix this manually in

requirements.txt

, I'm wondering if there's a more robust way to feed architecture constraints in ahead of time.

plain-carpet-73994

09/27/2022, 11:47 PM

I have a large package that I depend on (pytorch CUDA) and my CI keeps failing due to:

Copy code

There was 1 error downloading required artifacts:
1. torch 1.12.1+cu116 from <https://download.pytorch.org/whl/cu116/torch-1.12.1%2Bcu116-cp310-cp310-linux_x86_64.whl>
    Executing /home/companion/.cache/pants/named_caches/pex_root/venvs/ed3764586d1d1a9f27f2d6aa23d6ce7741735e1a/35dbbcf0d165ce1faf5f10df88cd6c8a72fc9216/bin/python -sE /home/companion/.cache/pants/named_caches/pex_root/venvs/ed3764586d1d1a9f27f2d6aa23d6ce7741735e1a/35dbbcf0d165ce1faf5f10df88cd6c8a72fc9216/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /home/companion/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-TxaSjj/.tmp/pex-pip-log.tsa4zqe_/pip.log download --dest /home/companion/.cache/pants/named_caches/pex_root/downloads/b6bc31244aa2818929fbb30c483c221df471e9d856e805c5a1ff72b131ae9e7b.9bdef5f1ebe741c6adc310711d52f843 --no-deps <https://download.pytorch.org/whl/cu116/torch-1.12.1%2Bcu116-cp310-cp310-linux_x86_64.whl> --index-url <https://pypi.org/simple> --extra-index-url <https://download.pytorch.org/whl/cu116> --retries 5 --timeout 15 failed with -9

I think it's just timing out as the download is big. Is there a way to change the

--timeout

? Looked through

./pants help-advanced python

and didn't see any options that seemed relevant.

high-yak-85899

09/28/2022, 3:09 AM

Question about pex lockfiles. I have

rtl_tools@ <git+ssh://git@git.corp.astranis.com/source/rtl_tools.git@1508cf5ef4f0e645e65bfe1af5343fe1b35f0422>

in my requirements file. I'm seeing

"url": "git+ssh://****@git.corp.astranis.com/source/rtl_tools.git@1508cf5ef4f0e645e65bfe1af5343fe1b35f0422"

end up in the constraints file. What's going on with the

****

ancient-breakfast-45303

09/28/2022, 3:26 AM

Just ran into a rough bug with direct wheel references over http with a local version specifier: https://github.com/pantsbuild/pex/issues/1919