Ah 🤔 so no way to cache pip dependencies so we don’t have to reinstall them on each run?

Also I’ve been unsuccessful in checking locally how do I know when things are is coming from cache vs being generated

Hello! I’m having a difficult time with Golang 1.18 and pants 2.14. When running tests or packaging, I get a handful of errors stating

pack: open __pkg__.a: permission denied

. It appears for a couple of the dependencies I’m using (namely golang.org/x/sys/unix and github.com/cespare/xxhash/v2). My theory is there is a name collision or something between these dependencies and others, anyone have an idea for how to deal with this/get around it and/or if my intuition is wrong?

Hi! I’m very new to pants, trying it out in an evaluation. I’m trying to get my pytest to work, and I’ve included a distributable as a dependency, but I can’t get the test to find the module, which is strange. E.g.

python_distribution(
    name="dist",
    dependencies=[
      "iai_fl_client:reqs",
      "iai_auth:dist",
      "iai_fl_client/src/iai_fl_client",
      "iai_fl_client/src/iai_fl_client/utils",
      "iai_fl_client/src/iai_fl_client/client",
    ],
    provides=python_artifact(
        name="iai-fl-client",
        version="2.0.24",
    ),
    wheel_config_settings={"--global-option": ["--python-tag", "py37.py38.py39"]},
)

and in the tests BUILD

python_tests(
    name="tests",
    timeout=120,
    runtime_package_dependencies=[
      "iai_fl_client:dist",
      "iai_auth:dist",
    ]
)

And when running tests it can’t import

iai_fl_client.utils

- what am I doing wrong?

_________________ ERROR collecting tests/test_client_utils.py __________________
ImportError while importing test module '/private/var/folders/xn/yvv0m2ln4dg_vyr8rp8_3xzh0000gn/T/pants-sandbox-dNO8Kw/iai_fl_client/tests/test_client_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/nasron/.pyenv/versions/3.8.13/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
iai_fl_client/tests/test_client_utils.py:3: in <module>
    from iai_fl_client.utils.util import poll_server_info
E   ModuleNotFoundError: No module named 'iai_fl_client.utils'

anyone know off the top of their head which directory pants places the generated pkg.a files for go? I’m thinking I may have a permissions issue

I've got bare bones Ubuntu running inside VirtualBox and I'm having trouble running pants. The error is related to evaluating go dependecies (I believe) The command is

./pants --print-stacktrace -ldebug list ::

Using version 2.16.0.dev3

Traceback (most recent call last):
  File "/home/vagrant/.cache/pants/setup/bootstrap-Linux-x86_64/2.16.0.dev3_py38/lib/python3.8/site-packages/pants/backend/go/util_rules/build_pkg.py", line 774, in required_built_go_package
    raise Exception(
Exception: Failed to compile main:
./main.go:18:2: could not import encoding/json (open : no such file or directory)
./main.go:19:2: could not import fmt (open : no such file or directory)
...
./main.go:27:2: too many errors

it looks to me like pants is not picking up the default $GOPATH (I'm not a GoLang expert however...) I show default GOPATH

vagrant@linux:~$ go env GOPATH
/home/vagrant/go

Same command using pants version 2.16.0.dev5

File "/home/vagrant/.cache/pants/setup/bootstrap-Linux-x86_64/2.16.0.dev5_py38/lib/python3.8/site-packages/pants/util/frozendict.py", line 51, in __getitem__
    return self._data[k]
KeyError: 'unicode/utf8'

Full debug log with stacktrace for both pants versions attached

log using pants 2.16.0.dev6

Can I expect the syntax

extra_build_args=['FURY_AUTH=%(env.FURY_AUTH)s']

to work in a

BUILD

file, I'm trying to create a Docker image target. I can do

[docker]
build_args = ['FURY_AUTH=%(env.FURY_AUTH)s']

in

pants.toml

but I haven't had any luck figuring out the corresponding syntax in a

BUILD

file.

I am encountering errors when running pants commands in my local pyenv environment. The same commands work fine in CI/CD, and other users can run them locally. The errors occur when trying to build

requirements.pex

from lock files, and certain packages cannot be installed. Here's and example of the error logs:

21:45:36.76 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building 22 requirements for requirements.pex from the build-support/databricks_lock.txt resolve: boto3==1.16.7, cleanco<2.2, enigma-data-catalog-sdk, enigma-enrichment, enigma-namedframes~=1.0.2, enigma-pyspark-commons, fuzzywuzzy, jellyfish, matplotlib==3.4.2, mlflow==1.20.2, numpy<1.24,>=1.20, pandas==1.2.4, plotly==5.1.0, probablepeople, protobuf==3.17.2, pyarrow==4.0.0, pyspark-test, pyspark==3.1.2, pytest, scikit-learn==0.24.1, scipy~=1.6.0, tldextract' failed with exit code 1.
stdout:

stderr:
There were 4 errors downloading required artifacts:
1. pandas 1.2.4 from <https://files.pythonhosted.org/packages/e8/81/f7be049fe887865200a0450b137f2c574647b9154503865502cfd720ab5d/pandas-1.2.4.tar.gz>
    ERROR: Command errored out with exit status 1: /Users/tom/.cache/pants/named_caches/pex_root/venvs/cac1718c056bb509f51fcdcc0c376b33deaaa8ec/80d537a6fbdf98a843c78d59de5b2b2db73ee10d/bin/python /Users/tom/.cache/pants/named_caches/pex_root/venvs/cac1718c056bb509f51fcdcc0c376b33deaaa8ec/80d537a6fbdf98a843c78d59de5b2b2db73ee10d/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /Users/tom/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-build-env-nq8fgxoc/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i <https://pypi.org/simple/> --extra-index-url https://****@repo.artifactory.enigma.com/artifactory/api/pypi/pypi-local/simple -- setuptools wheel 'Cython>=0.29.21,<3' 'numpy==1.16.5; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version=='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.5; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version=='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy; python_version>='"'"'3.9'"'"'' Check the logs for full command output.

the packages that fail are

pandas==1.2.4

,

pyarrow==4.0.0

and

scikit-learn=0.24.1

Hi, I gradually migrate a huge repo we have from a Docker-based infra we have to Pants. Lately, we've started to have sporadic cases in which CI fails due to networking issues. We've found that it could happen from over-using Docker. We currently replace in the testing phase a Pytest Docker with

pants test

. Just to make sure: does Pants/Pex's sandboxing has a virtual network interface or anything related that could have a race-condition with Docker when creating a container?

Wondering about one thing: Is there a reference for the

pants.toml

file somewhere? Would be nice to be able to find all of the available options for

[docker]

in one place for example.

Hi all - Does anyone know how to get the pants

--changed-since

flag to work with a Github Action triggered by a Pull Request? I keep encountering issues such as "fatal origin/stage...HEAD: no merge base" , but don't know how to correct the issue. Can anyone help me? 🙏🤞

Hi all, does anyone know if there’s already a Pants plugin for Python’s futurize? I’m working on moving some Python2 into my brand new monorepo ✨ and I’m missing that piece. Otherwise I will try to create one

I have a question about Pex binaries, which may or may not make sense… we’re using Flyte (https://flyte.org/) to deploy ML workflows. Flyte expects to be able to import our code from <some location> in a docker image (I’m not totally sure where this is, but I can figure it out). My understanding is that Pex files are meant to be executables… but is there a way to import from a pex file? If not, what would be the idomatic way to use pants to put my code into a docker image?

I'm hitting an error with

generate-lockfiles

on a specific package publicly available via pypi (

lexid

) - I'm trying to figure out how to debug this and not having much luck. I searched slack history and other times this has come up has been around private packages, so this seems different. I have a repo up with an isolated minimal test case https://github.com/rhysyngsun/pants-lexid-test, but the short of it is

python_requirement(requirements=["lexid"])

is enough to trigger it. I'll throw the stacktrace I'm getting in a threaded response to this message.

With experimental python backend: I’m wondering about the

repositories

param of

python_distributions

. Per https://www.pantsbuild.org/docs/python-publish-goal (linked from the bottom of https://www.pantsbuild.org/docs/python-distributions) it could include aliases like

@pypi

or private indexes. What about

@testpypi

vs

@pypi

? It might be interesting to publish first to

@testpypi

and then

@pypi

- If I put

python_distributions(repositories=["@pypi", "@testpypi"], …)

does that mean that it will publish to both repos? Or I can use TWINE_* vars to select which one to publish to? background: this is for StackStorm, an open source project. So, I don’t need to worry about publishing to private indexes.

One of my engineers uses the package

tensorflow-ranking

which has a dependency on

tensforflow

. On M1 laptops, engineers install

tensforflow-macos

not

tensorflow

. We get around this by

pip

installing it with

--no-deps

. I tried checking to see if I could specify that flag in

requirements.txt

and solve this outside of

pants

but haven't found anything. Is there a way I can add

--no-deps

to

tensorflow-ranking

in

pants

?

My local

lmdb_store

directory is currently at 46GiB, which is a fairly hefty proportion of my full disk. AIUI, pantsd is meant to be doing some GC on it, but, if so, there's a lot of non-garbage. Can I introspect the GC process? Is there a way to force a (harder) GC?

running into issues when upgrading flake8 to 6.0.0:

ERROR: Could not find a version that satisfies the requirement flake8==6.0.0
ERROR: No matching distribution found for flake8==6.0.0

./pants --changed-since=HEAD~1 --changed-dependees=transitive publish \
          --docker-build-args="[ \
            'DD_ENV=${STAGE}', \
            'DD_VERSION=${COMMIT_REF}', \
            'STAGE=${STAGE}', \
            'ECR_REGISTRY=${ECR_REGISTRY}', \
            'IMAGE_TAG=${COMMIT_REF}', \
            'A_REPOSITORY=${A_REPOSITORY}', \
            'B_REPOSITORY=${B_REPOSITORY}', \
            'C_REPOSITORY=${C_REPOSITORY}', \
          ]"

I am using this command to deploy docker to ecs(actually ecr) 1. I want to deploy when something changed in the container. Is that the right way I am doing now? Is there better way? 2. I want to add more repository here but I am worried that cd speed is a bit slower. What is the best way to deploy multiple docker image? 3. Is there any way to check updated image, so that I can use the output for the next step? (to ecs run)

Is there an "easy" way of temporarily overriding a resolved dependency to use a local version of a package? For example, we might have found a bug in foo-pkg from PyPi, and would want to fix that bug and also test it with our full stack.

Hey all, I am looking to run Pants within our CI system. We want to run Pants in a docker image. Is there any resources around how to build a docker file for this? I would like to avoid having to download pants each time the CI runs, and cache it in the docker file. I see that releases has the pexes (but I am looking for 2.15.0rc5 in particular).

Hi folks, I'm trying to use

--debug-adapter

with a python target in VS Code at Pants version

2.16.0.dev5

. It starts up and connects ok, but in one of my projects it fails on importing a (generated) protobuf module. In another simple project breakpoints simply aren't hit. Both of these things seem to point to the debugger not being aware of the sandboxing. I tried adding

run_goal_use_sandbox=False,

to my target to no avail (which maybe wouldn't work with generated protobuf modules anyway?). If I revert to

2.14.0

things seem to work ok. Any ideas to try? @bitter-ability-32190

Hi all, I’m trying to get my head around resolves, interpreter constraints and requirements. I have a library that I want to make Py2/Py3 compatible, and I have defined two python_sources, each with the corresponding interpreter_constraints and resolve, but when it comes to the library dependencies, I don’t know how to say that, the py2 sources depend on these versions, and py3 sources on these other ones. I don’t even know if it makes sense at all. Has anyone faced this problem?

Hey Pants Team, curious if there's a way to have the BUILD files changed as input to a

goal_rule

. Using

targets: Targets

doesn't seem to include BUILD files.

Getting an error trying to use

pytest_psql

when running tests within the pex sandbox:

/Users/nasron/.cache/pants/named_caches/pex_root/venvs/s/e101a3aa/venv/lib/python3.8/site-packages/pytest_pgsql/plugin.py:88: in database_snapshot
    return database.create_database_snapshot(pg_engine)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

connectable = Engine(<postgresql://postgres@127.0.0.1:63347/test>)

    def create_database_snapshot(connectable):
        """Create a snapshot of the current state of the database so that we can
        restore it to this state when the test exits.
    
        Arguments:
            connectable (`sqlalchemy.engine.Connectable`):
                The engine, connection, or other Connectable to use to take this
                snapshot.
    
        Returns (dict):
            A dictionary with three keys:
    
            * ``schemas``: A tuple of the names of all the schemas present.
            * ``tables``: A list of all of the tables present. Each item in the list
              is a dictionary with the schema name, table name, and table OID.
            * ``extensions``: A tuple of the names of all the extensions currently
              installed.
        """
>       execute = connectable.execute
E       AttributeError: 'Engine' object has no attribute 'execute'

I think

pyscopg2-binary

isn’t getting installed despite being in the dependency list - but I’m note sure.

How should I go about getting

./pants test

to include doctests? I added

python_tests(
    name="tests",
    sources=["*.py"],
)

as a target in my directory and ran

./pants test foobar/:: -- --doctest-modules

but it fails on a bunch of files without tests

Are any

./pants export

improvements slated for 2.15 release? I've been getting a growing number of reports internally on our use of export hanging for a long time. I know from the past there's been some back and forth about taking the risk of using some pants internals for the exported venv instead of generating it fresh.

Hi, if I want to build a docker image that uses the lockfile for dependencies, is using the pex binary the only way? If I use built distributions (wheels, sdist), the specific locked dependencies wont be followed in the docker build.

I am looking to integrate Pants into our CI/CD. I have built a docker image that can run Pants with all the dependencies for our environment. I am able to run the tailor, check, lint, and test commands. Now I want to be able to build docker images in that container. I am using docker-out-of-docker approach and linking the /var/log/docker.sock to the image. However, I am having some trouble getting the docker to run. Is there any advice on how to integrate into CI this way?