pantsbuild #general

Channels

welcome

github-notifications

announce

damp-quill-59187

02/07/2020, 8:54 PM

@happy-kitchen-89482 if I'm understanding correctly it does not. This is an excerpt from the build file:

Copy code

python_tests(
     name="test",
     sources=globs("**/test_*.py", "**/*_test.py", "conftest.py"),
     dependencies = [
         ":lib",
         "3rdparty/python:spark-testing-base",
         "3rdparty/python:pyspark",
     ]
 )

python_library(
    name="lib",
    dependencies=[
        '3rdparty/python:google-auth',
        '3rdparty/python:google-api-python-client',
        'src/common:lib',
    ],
    sources=rglobs("*.py","*.yml"),
    provides=setup_py(
        name="our.google.project",
        version=get_version(),
        zip_safe=True,
    )
)

happy-kitchen-89482

02/07/2020, 11:09 PM

Looks like it doesn't. So now I'm not sure where that setuptools constraint is coming from. Pants itself depends on setuptools, but since you're not depending on Pants as a library (you're just using it as a tool) that shouldn't signify. Hmm.

happy-kitchen-89482

02/07/2020, 11:11 PM

Ok, so it would be coming from pex-builder-wrapper, but your

Copy code

[pex-builder-wrapper]
setuptools_version: 45.1.0

should have taken care of that

aloof-angle-91616

02/07/2020, 11:12 PM

pex_build_util

will only add the

setuptools

dependency if it's not already been added to the pex builder

happy-kitchen-89482

02/07/2020, 11:12 PM

also true

aloof-angle-91616

02/07/2020, 11:13 PM

not sure how to make use of this fact yet however

crooked-tent-69497

02/07/2020, 11:51 PM

another coverage question, it looks like coverage will not run under pants unless i also include --coverage=, even if I specify command line arguments to be passed to the underlying pytest call as in -

./pants test :: — --cov=

. That in and of itself isn’t a huge problem, but we’re running tests across many targets, and I’m having little luck using any kind of wildcard when it comes to specifying modules. For example, if we have targets a-z, each with submodules, I’m only able to get some coverage results doing something like

./pants test --coverage=modulesa.a,a.b,a.c,a.d,…,z.x,z.y,z.z

. Am I just doing things wrong? I’ve tried paths as well, but generally I still get “module a was not included, ignoring”

witty-crayon-22786

02/08/2020, 12:07 AM

--coverage=auto

should do the equivalent of "all the things"

👍 2

witty-crayon-22786

02/08/2020, 12:08 AM

Copy code

$ ./pants help test.pytest

test.pytest:

  --test-pytest-coverage=<str> (default: None)
      Emit coverage information for specified packages or directories (absolute or
      relative to the build root).  The special value "auto" indicates that Pants
      should attempt to deduce which packages to emit coverage for.
  --[no-]test-pytest-coverage-include-test-sources (default: False)
      Whether to include test source files in coverage measurement.
  --test-pytest-coverage-output-dir=<DIR> (default: None)
      Directory to emit coverage reports to. If not specified, a default within
      dist is used.
  --test-pytest-coverage-reports=<str> (--test-pytest-coverage-reports=<str>) ..., --test-pytest-coverage-reports="[<str>, <str>, ...]", --test-pytest-coverage-reports="+[<str>, <str>, ...]" (one of: [xml, html] default: ['xml','html'])
      Which coverage reports to emit.

crooked-tent-69497

02/08/2020, 12:10 AM

the docs seemed to imply the default would be to include directories in which the tests live, which isn’t true for some of our targets, i’ll try it out

crooked-tent-69497

02/08/2020, 12:11 AM

yeah, it didn’t work for those targets

crooked-tent-69497

02/08/2020, 12:11 AM

does pants support appending coverage ? there doesn’t seem to be a pants flag for that

crooked-tent-69497

02/08/2020, 12:29 AM

i would have expected auto would use the build files to figure out what to measure

happy-kitchen-89482

02/09/2020, 10:01 PM

Can some committer give me a shipit on this revert in the pex repo https://github.com/pantsbuild/pex/pull/879 ? Thanks

rich-helicopter-79006

02/11/2020, 1:29 PM

I’m getting a weird problem when I try to import sources/docs with an IntelliJ pants project. Specifically, an `Exception caught: (pants.backend.jvm.ivy_utils.IvyResolveMappingError)`:

Copy code

Exception caught: (pants.backend.jvm.ivy_utils.IvyResolveMappingError)

 File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 85, in main
    PantsLoader.run()
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 81, in run
    cls.load_and_execute(entrypoint)
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 74, in load_and_execute
    entrypoint_main()
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/bin/pants_exe.py", line 39, in main
    PantsRunner(exiter, start_time=start_time).run()
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/bin/pants_runner.py", line 88, in run
    return runner.run()
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 231, in run
    self._run()
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 292, in _run
    goal_runner_result = self._maybe_run_v1()
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 259, in _maybe_run_v1
    return goal_runner_factory.create().run()
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 209, in run
    return self._run_goals()
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 180, in _run_goals
    result = self._execute_engine()
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 168, in _execute_engine
    result = engine.execute(self._context, self._goals)
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/engine/legacy_engine.py", line 26, in execute
    self.attempt(context, goals)
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/engine/round_engine.py", line 234, in attempt
    goal_executor.attempt(explain)
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/engine/round_engine.py", line 50, in attempt
    task.execute()
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/task/console_task.py", line 56, in execute
    for value in self.console_output(targets) or tuple():
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/backend/project_info/tasks/export.py", line 414, in console_output
    graph_info = self.generate_targets_map(targets, classpath_products=classpath_products)
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/backend/project_info/tasks/export.py", line 180, in generate_targets_map
    classpath_products = self.resolve_jars(targets)
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/backend/project_info/tasks/export.py", line 156, in resolve_jars
    confs=confs)
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/backend/jvm/tasks/ivy_task_mixin.py", line 145, in resolve
    pinned_artifacts=artifact_set))
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/backend/jvm/tasks/ivy_task_mixin.py", line 181, in _resolve_subset
    for target, resolved_jars in self.add_directory_digests_for_jars(resolved_jars_per_target):
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/backend/jvm/tasks/resolve_shared.py", line 38, in add_directory_digests_for_jars
    targets_and_jars=list(targets_and_jars)
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/backend/jvm/ivy_utils.py", line 433, in resolved_jars_for_each_target
    ivy_info = self._ivy_info_for(conf)
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/backend/jvm/ivy_utils.py", line 453, in _ivy_info_for
    return IvyUtils.parse_xml_report(conf, report_path)
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/backend/jvm/ivy_utils.py", line 926, in parse_xml_report
    ret.add_module(ivy_module)
  File "/Users/allen/.cache/pants/setup/bootstrap-Darwin-x86_64/1.16.0_py27/lib/python2.7/site-packages/pants/backend/jvm/ivy_utils.py", line 599, in add_module
    .format(ref_unversioned, module.ref.rev))

Exception message: Already defined module IvyModuleRef(net.sourceforge.f2j:arpack_combined_all:latest.integration:javadoc:jar), as rev 0.1!

The command line being executed is:

Copy code

pants --target-spec-file=/private/var/folders/zn/04ckg50s6c70p76y0s564r7c0000gn/T/pants_target_specs7.in --no-quiet export --output-file=/private/var/folders/zn/04ckg50s6c70p76y0s564r7c0000gn/T/pants_depmap_run1245.out --formatted --export-libraries-sources --export-libraries-javadocs

Running this command on the command-line triggers the same error, so it doesn’t seem like it’s IntelliJ/IntelliJ-pants related, but it’s definitely because I’m exporting sources/javadocs. Is this a problem somewhere in my deps?

brief-coat-32007

02/11/2020, 6:22 PM

hi, question I am currently using pants build for my go project I may have encounter 2 issues 1. how can i access a resource file during test run? e.g. schema file for local database setup to run test against database layer 2. it seems that pants build does not allow a go dir with only test files? ref for 1: https://github.com/pantsbuild/pants/issues/5132 https://github.com/pantsbuild/pants/issues/2590 ref for 2: https://github.com/pantsbuild/pants/issues/5131

brief-coat-32007

02/11/2020, 6:23 PM

for 1, just putting the schema file in the test dir does not really work, since there will be an integration test referencing it in another dir

brief-coat-32007

02/11/2020, 8:46 PM

nvm, made it work by adding a dummy go file and ref it in the test files

early-needle-54791

02/11/2020, 9:58 PM

I’m writing a jvm task test for a task that uses an invalidated block. Unfortunately the invalidated block tries to fingerprint the task, which involves fingerprinting all the subsystems that it needs, which don’t exist in the test context. Is there a well known way to work around this, or should I just mock out the fingerprinting?

elegant-nightfall-29735

02/13/2020, 10:54 AM

Has anyone gotten Pants to run tests on CI? I’m currently trying to runs tests that also include docker-compose. Its turning out to be way more difficult that I imagined.

elegant-nightfall-29735

02/13/2020, 10:54 AM

They’re python tests if that helps

happy-kitchen-89482

02/13/2020, 3:31 PM

I think pretty much everyone runs Pants in CI, including the Pants project itself... Can you provide more information on what you're trying to do and what problems you're encountering?

happy-kitchen-89482

02/13/2020, 3:32 PM

For example, which CI service are you using, and how does docker-compose fit in to that

happy-kitchen-89482

02/13/2020, 4:03 PM

What pants commands you're running in CI, what errors you're seeing, etc.

elegant-nightfall-29735

02/13/2020, 5:40 PM

Actually, the issue now was with using the pantsbuild/centos6 docker image from your docker hub. I can’t seem to get docker and docker-compose to work on the image. My latest error is this bad boy:

Copy code

docker-compose: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /tmp/_MEIPxHDvM/libz.so.1)

elegant-nightfall-29735

02/13/2020, 5:40 PM

But thats a recent error change - which generally indicates progress 😉

hundreds-father-404

02/13/2020, 5:42 PM

Hm, we solely use

pantsbuild/centos6

for the sake of building the

pantsbuild.pants

wheel to be

manylinux1

compatible. We don’t expect users to actually use

pantsbuild/centos6

. Were you using that docker image for a particular reason?

elegant-nightfall-29735

02/13/2020, 5:45 PM

Frankly, I dont even know what im doing. I was using the

docker/compose:1.25.1

(a debian) image before - but I was getting all sorts of errors when I’d try to download pants version 1.24.0. It said the only available versions went up to 1.8.0. I can look for a stack trace.

elegant-nightfall-29735

02/13/2020, 5:47 PM

Sorry for the big ole text dump:

Copy code

$ ./pants --version
   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                  Dload  Upload   Total   Spent    Left  Speed
 100   122  100   122    0     0    903      0 --:--:-- --:--:-- --:--:--   897
 100   281  100   281    0     0    897      0 --:--:-- --:--:-- --:--:--   897
   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 100 3626k  100 3626k    0     0  5270k      0 --:--:-- --:--:-- --:--:-- 5270k
 Using base prefix '/usr'
 New python executable in /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.GPeIco/install/bin/python3.7
 Also creating executable in /root/.cache/pants/setup/bootstrap-Linux-x86_64/pants.GPeIco/install/bin/python
 Installing setuptools, pip, wheel...
 done.
 Collecting pip
   Using cached <https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl>
 Installing collected packages: pip
   Found existing installation: pip 19.0.3
     Uninstalling pip-19.0.3:
       Successfully uninstalled pip-19.0.3
 Successfully installed pip-20.0.2
 ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==1.24.0 (from versions: 0.0.17, 0.0.18, 0.0.20, 0.0.21, 0.0.22, 0.0.23, 0.0.24, 0.0.25, 0.0.26, 0.0.27, 0.0.28, 0.0.29, 0.0.30, 0.0.31, 0.0.32, 0.0.33, 0.0.34, 0.0.35, 0.0.36, 0.0.37, 0.0.38, 0.0.39, 0.0.40, 0.0.41, 0.0.42, 0.0.43, 0.0.44, 0.0.45, 0.0.46, 0.0.47, 0.0.48, 0.0.49, 0.0.50, 0.0.51, 0.0.52, 0.0.53, 0.0.54, 0.0.55, 0.0.56, 0.0.57, 0.0.58, 0.0.59, 0.0.60, 0.0.61, 0.0.62, 0.0.63, 0.0.64, 0.0.65, 0.0.66, 0.0.67, 0.0.68, 0.0.69, 0.0.70, 0.0.71, 0.0.72, 0.0.73, 0.0.74, 0.0.75, 0.0.76, 0.0.77, 0.0.79, 0.0.80, 0.0.81, 0.0.82, 1.0.0rc1, 1.0.0rc2, 1.0.0rc3, 1.0.0, 1.0.1rc0, 1.0.1, 1.1.0rc0, 1.1.0rc1, 1.1.0rc2, 1.1.0rc3, 1.1.0rc4, 1.1.0rc5, 1.1.0rc6, 1.1.0rc7, 1.1.0rc8, 1.1.0rc9, 1.1.0, 1.2.0.dev0, 1.2.0.dev1, 1.2.0.dev2, 1.2.0.dev3, 1.2.0.dev4, 1.2.0.dev5, 1.2.0.dev6, 1.2.0.dev8, 1.2.0.dev9, 1.2.0.dev10, 1.2.0.dev11, 1.2.0.dev12, 1.2.0rc0, 1.2.0rc1, 1.2.0rc2, 1.2.0rc3, 1.2.0rc4, 1.2.0, 1.2.1rc0, 1.2.1, 1.3.0.dev0, 1.3.0.dev1, 1.3.0.dev2, 1.3.0.dev3, 1.3.0.dev4, 1.3.0.dev5, 1.3.0.dev6, 1.3.0.dev7, 1.3.0.dev8, 1.3.0.dev9, 1.3.0.dev10, 1.3.0.dev11, 1.3.0.dev12, 1.3.0.dev13, 1.3.0.dev14, 1.3.0.dev15, 1.3.0.dev16, 1.3.0.dev17, 1.3.0.dev18, 1.3.0.dev19, 1.3.0rc0, 1.3.0rc1, 1.3.0rc2, 1.3.0rc3, 1.3.0rc4, 1.3.0, 1.3.1rc0, 1.3.1rc1, 1.3.1rc2, 1.4.0.dev0, 1.4.0.dev1, 1.4.0.dev3, 1.4.0.dev4, 1.4.0.dev5, 1.4.0.dev6, 1.4.0.dev7, 1.4.0.dev8, 1.4.0.dev9, 1.4.0.dev10, 1.4.0.dev11, 1.4.0.dev13, 1.4.0.dev14, 1.4.0.dev15, 1.4.0.dev16, 1.4.0.dev17, 1.4.0.dev18, 1.4.0.dev19, 1.4.0.dev20, 1.4.0.dev22, 1.4.0.dev23, 1.4.0.dev24, 1.4.0.dev25, 1.4.0.dev26, 1.4.0.dev27, 1.4.0rc0, 1.4.0rc1, 1.4.0rc2, 1.4.0rc3, 1.4.0rc4, 1.4.0rc5, 1.4.0, 1.5.0.dev0, 1.5.0.dev1, 1.5.0.dev2, 1.5.0.dev3, 1.5.0.dev4, 1.5.0.dev5, 1.5.0rc0, 1.5.0rc1, 1.5.0, 1.5.1rc0, 1.5.1rc1, 1.5.1rc2, 1.6.0.dev0, 1.6.0.dev1, 1.6.0.dev2, 1.6.0rc0, 1.6.0rc1, 1.6.0rc2, 1.6.0rc3, 1.6.0, 1.7.0.dev0, 1.7.0.dev1, 1.7.0.dev2, 1.7.0rc0, 1.7.0rc1, 1.7.0, 1.8.0.dev0, 1.8.0.dev1, 1.8.0.dev2)
 ERROR: No matching distribution found for pantsbuild.pants==1.24.0
 New virtual environment successfully created at /root/.cache/pants/setup/bootstrap-Linux-x86_64/1.24.0_py37.
 /root/.cache/pants/setup/bootstrap-Linux-x86_64/1.24.0_py37/bin/python: can't open file '/root/.cache/pants/setup/bootstrap-Linux-x86_64/1.24.0_py37/bin/pants': [Errno 2] No such file or directory

elegant-nightfall-29735

02/13/2020, 5:48 PM

I’m running everything with Gitlab CI. The error is happening as part of this:

Copy code

.test_base:
    image:
        name: docker/compose:1.25.1
        entrypoint: ["/bin/sh", "-c"]
    extends: .dind
    variables:
        DOCKER_HOST: <tcp://localhost:2375>
    before_script:
        - docker info
        - docker-compose --version
        - apk add --update libffi-dev linux-headers musl-dev python3 py3-pip gcc bash curl
#        - apk add --update libffi-dev linux-headers musl-dev python3-dev py3-pip gcc bash curl
        - pip3 install --upgrade pip
        - which pip
        - pip --version
        - echo $PATH
        - export PATH=$PATH:/usr/lib/:/root/
        - echo $PATH
        # It appears that theres usr/lib/python3.7 and usr/bin/python3.7
        - which python3.7
        - alias python=python3.7
        - export VERSION=$(cat VERSION)-build${CI_PIPELINE_ID}
        - docker login $DOCKER_REGISTRY -u $DOCKER_USER -p $DOCKER_PASS
        - curl -L -O <https://pantsbuild.github.io/setup/pants> && chmod +x pants
        - ./pants --version
        - 'printf "[GLOBAL]\npants_version: $(./pants --version)\n" > pants.ini'
        - pip3.7 install --disable-pip-version-check --cache-dir=.pip -r requirements.txt
    tags:
    - linux
    timeout: 15m
    artifacts:
      name: "$CI_COMMIT_REF_NAME"
      when: always
      paths:
        - logs
        - pants.ini
      expire_in: 1 week

It happens on

./pants --version