pantsbuild #general

Channels

welcome

github-notifications

announce

gorgeous-winter-99296

03/10/2023, 8:18 AM

How do I specify the python version that Pex uses while building? I've switched our repo to minimum version 3.9 but some Pex targets still use 3.8 on CI.

Copy code

ProcessExecutionFailure: Process 'Building 9 requirements for ...foo.pex from the locks/pants.lock resolve: ...' failed with exit code 1.
[2023-03-09T21:49:32Z] stdout:
[2023-03-09T21:49:32Z]
[2023-03-09T21:49:32Z] stderr:
[2023-03-09T21:49:32Z] There was 1 error downloading required artifacts:
[2023-03-09T21:49:32Z] 1. emote-rl 23 from git+<https://github.com/EmbarkStudios/emote@ts/memory-logging-proxy-wrapper#egg=emote-rl>
[2023-03-09T21:49:32Z]     pid 2670012 -> /root/.cache/pants/named_caches/pex_root/venvs/eac612bf592de69ec2b624a7a30f7fec79e2c3a3/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/python -sE /root/.cache/pants/named_caches/pex_root/venvs/eac612bf592de69ec2b624a7a30f7fec79e2c3a3/ddab8011daaee380698ac2fb9701af18c90c03f6/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /root/.cache/pants/named_caches/pex_root/pip_cache download --dest /root/.cache/pants/named_caches/pex_root/downloads/resolver_download.tbpzqg62/usr.bin.python3.8 --no-binary :all: --no-deps git+<https://github.com/EmbarkStudios/emote@ts/memory-logging-proxy-wrapper#egg=emote-rl> --index-url <https://pypi.org/simple/> --extra-index-url <https://download.pytorch.org/whl/cpu/> --retries 5 --timeout 15 exited with 1 and STDERR:
[2023-03-09T21:49:32Z] ERROR: Package 'emote-rl' requires a different Python: 3.8.5 not in '>=3.9'

Note that the resolver uses python3.8. I've added the following environment variables:

Copy code

PANTS_PYTHON_INTERPRETER_CONSTRAINTS="['==3.9.*']"
PY="python3.9"
PYTHON_BIN_NAME="python3.9"

and

Copy code

[python]
interpreter_constraints = ["==3.9.*"]

Not sure if this is a caching issue (due to changing version) or something I need to configure. This is on 2.15.0.

clever-father-91273

03/10/2023, 12:53 PM

Hello there, I'm having trouble executing

pants publish

in Google Cloud Build. I'm getting the following error:

unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials

However, this does not seem to be the case, since if I only use Pants to build the image (

pants package

), I can then push them with

docker push

, so the credentials seem to be in place. The

~/.docker/config.json

also looks correct. What am I missing here?

ripe-gigabyte-88964

03/10/2023, 8:53 PM

Is there a way to list the current formatters applied when running

pants fmt

? I'm trying to figure out why the terraform formatter does not seem to be running.

silly-queen-7197

03/11/2023, 12:26 AM

Is there a way to opt out of generating pex files during development* when trying to run scripts / modules? Unfortunately I have some large dependencies like scipy, etc, and whenever I make a slight change to a first party library (like add a print) it takes a really long time for

./pants run foo:cli

to run. Having

Building foo/cli.pex with 20 requirements:...

take 20 seconds really kills being able to establish a fast feedback loop

silly-queen-7197

03/11/2023, 12:51 AM

I'm trying to run

./pants run --debug-adapter foo/cli.py

. In my

pants.toml

I have

interpreter_constraints = ["CPython==3.10.*"]

yet I see

Copy code

pex.environment.ResolveError: A distribution for sqlalchemy could not be resolved for /usr/bin/python3.7.
Found 1 distribution for sqlalchemy that do not apply:
1.) The wheel tags for SQLAlchemy 1.4.27 are cp310-cp310-manylinux_2_17_x86_64, cp310-cp310-manylinux_2_5_x86_64, cp310-cp310-manylinux1_x86_64, cp310-cp310-manylinux2014_x86_64 which do not match the supported tags of /usr/bin/python3.7:
cp37-cp37m-manylinux_2_28_x86_64
... 484 more ...

purple-plastic-57801

03/11/2023, 6:50 PM

Curious. What is the current state of pants and JavaScript?

proud-dentist-22844

03/12/2023, 1:58 AM

How does

./pants export --resolve=...

(or more likely the underlying

pex

?) determine which version of

pip

and

setuptools

gets installed in the exported venv? The version of

pip

is not locked, so I'm not surprised to see a different version when exported with any of

python3.6

python3.7

, and

python3.8

. But

setuptools

is in the lockfile. When I export with

python3.6

python3.7

, I get the version of setuptools in my lockfile. With

python3.8

, I get a newer version of

setuptools

. NB: This is with the default
--py-resolve-format=mutable_virtualenv
. With
--py-resolve-format=symlinked_immutable_virtualenv
then all of them have the correct locked version of
setuptools
and do not have
pip
. I'm on pants 2.16.0a0

silly-queen-7197

03/12/2023, 4:42 AM

Here's a potentially dumb question. On https://www.pantsbuild.org/docs/installation it suggests removing

./pants

. Ok, I'll run

Copy code

curl --proto '=https' --tlsv1.2 -fsSL <https://static.pantsbuild.org/setup/get-pants.sh>

as suggested but this doesn't install anything in

~/bin

. All it seems to do is print a bunch of stuff out? I'm running on Ubuntu. The output looks like

Copy code

...
ARCH="$(calculate_arch)"
URL="<https://github.com/pantsbuild/scie-pants/releases/${version}/scie-pants-${OS}-${ARCH}>"
dest="${bin_dir}/${base_name}"

log "Downloading and installing the pants launcher ..."
install_from_url "${URL}" "${dest}"
green "Installed the pants launcher from ${URL} to ${dest}"
if ! command -v "${base_name}" > /dev/null; then
  warn "${dest} is not on the PATH."
  log "You'll either need to invoke ${dest} explicitly or else add ${bin_dir} to your shell's PATH."
fi

green "\nRunning \`pants\` in a Pants-enabled repo will use the version of Pants configured for that repo."
green "In a repo not yet Pants-enabled, it will prompt you to set up Pants for that repo."

wide-midnight-78598

03/13/2023, 1:33 AM

My memory is failing me on this, was going to try it tonight, but if someone can remind me - can I build a pex for a Linux x86 box using my x86 Mac (using platform/complete_platforms)? Or do I need docker involved?

high-yak-85899

03/13/2023, 5:13 PM

There's a nice script for cache size maintenance in the docs for using pants in CI. I've noticed that this script can take many multiple minutes to run even though we call it on two directories. Is that what others have experienced or are we likely doing something wrong?

blue-football-92560

03/13/2023, 5:49 PM

Generating lockfiles takes 22 minutes for our repo. Is there a way to speed it up? Or at least get a rough idea of what is taking the most time?

Copy code

generate-lockfiles  0.27s user 0.05s system 0% cpu 22:26.65 total

Building a normal virtualenv with the same requirements only takes 6 minutes.

Copy code

pip install --no-cache-dir -r requirements.txt  268.47s user 34.02s system 89% cpu 5:38.85 total

wide-midnight-78598

03/13/2023, 6:53 PM

@ancient-vegetable-10556 Regarding this awesome

adhoc_tool

workflow (https://github.com/pantsbuild/example-adhoc/blob/main/javascript/BUILD), how should a tool like

pnpm

be handled, which places all installed deps into a global content store, and then hardlinks them into the local repo? https://pnpm.io/faq#why-does-my-node_modules-folder-use-disk-space-if-packages-are-stored-in-a-global-store

clever-hamburger-59716

03/13/2023, 6:59 PM

Hi Folks, What is the state of the art when it comes to packaging (fully contained) python applications to run on Windows? We would like to ship our app (which we package into a pex today) to end users on Windows. For Linux and Mac it looks like we can build a scie (with a pex bundled), what's the story on the windows side? Is there a mechanism like PyOxidizer where I can point to a distribution?

purple-plastic-57801

03/14/2023, 12:07 AM

Dumb question: Can pants manage the version of python for you? For instance in recent versions of rules_python for bazel it will hermetically manage the python toolchain for you (if you configure it that way)

purple-plastic-57801

03/14/2023, 12:38 AM

Maybe it’s bazelTrauma™️ … but pants feels too easy. What are some gotchas? Foot guns? Etc?

refined-hydrogen-47967

03/14/2023, 3:03 AM

Hi, I’m curious about some strange behavior when compiling Scala I’ve noticed while updating from 1.30.4 to 2.15. According to https://docs.scala-lang.org/style/files.html it seems it’s common to put sealed traits and their case classes in the same file, which compiles fine with 1.30.4, but in 2.15 it complains that there’s a dependency cycle. I can workaround this by explicitly excluding the target as a dependency from itself, but that seems a bit awkward. Is this intended behavior?

brave-hair-402

03/14/2023, 11:48 AM

Am I correct that if I have a base pants.toml file, and additional pants.ci.toml (set via environmental variable PANTS_CONFIG_FILES), the will both be used, but configs in pants.ci.toml will take precedence?

✅ 1

ripe-gigabyte-88964

03/14/2023, 12:48 PM

Is there a way to disable a specific goal for a specific backend e.g. I only want to use the

fmt

goal from this backend and not the

check

adamant-magazine-16751

03/14/2023, 12:58 PM

Hi. I'm testing out pants 2.15 and the Target Environments feature specifically. I must say it looks really good and will definitely replace our previous approach of using docker-in-docker and a builder image (because of the necessity of running pants in the same environment as the target). I have a question though. Is there a way to specify the target platform for a

docker_image

target? An example would be building a linux/amd64 image on an m1 mac. I managed to build a pex using

platform=linux_x86_64

but I suppose that a docker_image built on an arm will target

arm64

. Do I understand correctly that this example workflow from the docs considers the case with an x86 mac - the same architecture?https://www.pantsbuild.org/docs/environments#use-a-docker_environment-to-build-the-inputs-to-a-docker_image I'm not even sure if this is something I should expect to be available. This is a very specific case of quick-prototyping - building a dev version of an image locally, publishing and running it in the test environment that is based on x86

ripe-gigabyte-88964

03/14/2023, 2:55 PM

How does the

gha-cache-key

work here? What's the use case for busting the cache? Would we want to bust the cache every time we increment versions or can GHA caches be updated?

wide-midnight-78598

03/14/2023, 3:02 PM

Does Pants (in some way/shape/form) consume dotenvs (https://www.npmjs.com/package/dotenv, https://pypi.org/project/python-dotenv/)? Use case is that I have several .env files for local/dev/rc/prod cases, and I'd like to build

pex

packages for each env for easier distribution. Currently, I need to add a file/resource and capture the .envs to send them along, or I can manually write my .env parameters in pants.toml or the target - but I'm not committing any of those values

👀 1

adamant-magazine-16751

03/14/2023, 3:56 PM

Is there a built-in way to change the naming scheme of

tailor

? Or would I need to write a plugin for that?

high-yak-85899

03/14/2023, 6:15 PM

Any existing ideas for how we could use bazel outputs in the pants-side of our repo? Could

experimental_shell_command

work its way in here to call out to build bazel things for us?

cold-vr-15232

03/14/2023, 8:17 PM

Hi! Should I worry about warnings like "Executor shutdown took unexpectedly long: tasks were likely leaked"? This happens in CI, but things are not taking long, the whole build might take 4 seconds or so (when all is cached)

square-laptop-33187

03/14/2023, 8:27 PM

Is there a way to reuse previous builds? More specifically, I want

pants package Dockerfile

to finish immediately if run twice without changing any dependencies, similar to

docker build

. I looked into

--changed-since

and it looks useful but for a slightly different scenario. Any pointers will be appreciated.

ripe-gigabyte-88964

03/14/2023, 8:35 PM

Can you use variables in

BUILD

files? Or some other wildcard substitution mechanism?

abundant-autumn-67998

03/14/2023, 10:16 PM

When running pex, is there a way to redirect the pip log output to stdout when running pex? sometimes it prints backtracking warnings and would be nice see them on the console.

high-yak-85899

03/14/2023, 11:54 PM

So I think I noticed a safety issue with the new symlink feature. It seems like you can call

export

with the symlink flag and it rightfully says it puts an immutable venv in place. However, if you do this without clearing the export that didn't use the symlink, you still have

pip

and

python

in the

bin

directory. Is that expected?

breezy-apple-27122

03/15/2023, 8:41 AM

maybe someone asked this already, but let me try: how can python_sources() be recursive instead of adding one BUILD file in each subfolder?

✅ 1

adamant-magazine-16751

03/15/2023, 10:32 AM

I have a requirements.txt file with around 70 requirements. Some of them have minimum versions

>=

, some use exact versions and for some the version is unspecified. I've ran into OOM errors a couple of times already when trying to do

pants generate-lockfiles

on a machine with 8GB of ram. IIRC last time I resolved it by pinning everything to an exact version in requirements. Is there a better way to work around it? I'm not even sure why I run into these memory errors in the first place