pantsbuild #general

Channels

welcome

github-notifications

announce

wide-zoo-86070

03/30/2022, 6:10 PM

Assume that I am using conda to manage my dependencies in the past where I specify the following, and my code uses Python 3.9 features

Copy code

dependencies:
  - python=3.9.5

My question is, to use pants, do I have to put interpreter_constraints = [“==3.9.*“]. Or I can put interpreter_constraints = [“>=3.6.*“] (for example). I guess, my question becomes, when pants run my codes, will it get (somehow : ) ) python39 instead of python36? If yes, how to make it do it? or it wont, it will just use the interpreter it found.

echoing-farmer-15630

03/30/2022, 7:33 PM

Okay, hmm... I'm not sure what interaction is doing this or what I'm missing, but 2.11.0rc0, python interpreter_constraints = ["CPython==3.9.*"], system python is 3.10, pyenv enabled "local" python is 3.9.9, I'm in a virtualenv is 3.9.9, and ... ...lockfile generation consistently fails because it's using 3.10 (system python)...?

Copy code

stderr:
ERROR: Package 'aiofiles' requires a different Python: 3.10.2 not in '<4.0,>=3.6'

...neither of which makes sense to me (it shouldn't be resolving with 3.10, and 3.10.2 is in fact between 3.6 and 4.0...)

hundreds-father-404

03/30/2022, 7:36 PM

which lockfile is being generated?

echoing-farmer-15630

03/30/2022, 7:41 PM

python-default

echoing-farmer-15630

03/30/2022, 7:42 PM

hmm, it's clearly using the pyenv version (in the generated __run.sh I see "/home/vputz/.pyenv/versions/3.10.2/bin/python") but I've set pyenv local and global to 3.9.9 to try and force the issue. That's... odd

echoing-farmer-15630

03/30/2022, 7:46 PM

huh; yeah,

Copy code

Process 'Generate lockfile for python-default' failed with exit code 1.

echoing-farmer-15630

03/30/2022, 7:47 PM

and

Copy code

/tmp/process-executionKTQOsz for "Generate lockfile for python-default"

echoing-farmer-15630

03/30/2022, 7:49 PM

Copy code

/tmp/process-executionKTQOsz
❯ grep python __run.sh
/home/vputz/.pyenv/versions/3.10.2/bin/python ./pex --tmpdir .tmp lock ...

happy-kitchen-89482

03/30/2022, 7:54 PM

Ah I see. In that case, that is exactly what interpreter_constraints is for

happy-kitchen-89482

03/30/2022, 7:54 PM

That is what forces Pants to limit the interpreters it will use

✅ 1

hundreds-father-404

03/30/2022, 8:01 PM

the relevant option https://www.pantsbuild.org/docs/reference-python#section-resolves-to-interpreter-constraints default should work, but could try it

echoing-farmer-15630

03/30/2022, 8:41 PM

Sigh... yeah, I don't get this at all.

interpreter_constraints

["CPython==3.9.*"]

resolves_to_interpreter_constraints="{'python-default': ['==3.9.*']}"

, and yet trying

Copy code

./pants --no-process-cleanup generate-lockfiles --generate-lockfiles-resolve="['python-default']"

...gets me that

ERROR: Package 'aiofiles' requires a different Python: 3.10.2 not in '<4.0,>=3.6'

error. The resulting execution still referenced something in my ~/.cache/pants directory so I nuked that,

./.pants.d

and

.pids

and started from scratch.... same thing. Weird. I'll try it with rc1 tomorrow (late over here). Very strange.

❗ 1

busy-vase-39202

03/30/2022, 11:11 PM

Hey, y'all, quick get-to-know-our-community-better Q&A: how many teams and individuals are using Pants at your company?

freezing-area-97131

03/31/2022, 3:13 AM

Hi! I encountered this error when upgrading pants:

Copy code

[WARN] /home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.23.0_py36/lib/python3.6/site-packages/pex/pep425tags.py:274: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp

**** Failed to install importlib_metadata-4.11.3 (caused by: NonZeroExit("received exit code 1 during execution of `['/usr/bin/python2.7', '-s', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpwfro9psz']` while trying to execute `['/usr/bin/python2.7', '-s', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpwfro9psz']`",)
):
stdout:

stderr:
Traceback (most recent call last):
  File "<stdin>", line 13, in <module>
IOError: [Errno 2] No such file or directory: 'setup.py'

Thanks @happy-kitchen-89482 for pointing out I should fix the

importlib-metadata

version to

2.1.3

. But I don’t know which package depends on it. How do I pin the version? I added

importlib-metadata==2.1.3

in requirement.txt, but it did not work.

rapid-exabyte-76685

03/31/2022, 7:28 AM

Is there a way to get

list

to show the tags that have been applied to targets/files?

ambitious-actor-36781

03/31/2022, 8:46 PM

does anyone have experience running pants behind a corpo firewall? Getting errors downloading the pex release "invalid certificate contents: invalid peer cert: unknown issuer" when running

package

✅ 2

rapid-exabyte-76685

04/01/2022, 1:49 AM

To address some problems with certain tests not currently working under Pants vs the current environment (probably related to parallelism?) I tried defining two

python_tests

in the directory that has the tests in question... 🧵

Copy code

python_tests(
  name="test",
  source=[
    "test_*.py",
    "!test_that_is_deadlocking1.py",
    "!test_that_is_deadlocking2.py"
  ]
)

python_tests(
  name="deadlocking_tests",
  source=[
    "test_that_is_deadlocking1.py",
    "test_that_is_deadlocking2.py"
  ]
)

freezing-area-97131

04/01/2022, 5:48 AM

Hi! I was upgrading pants. Now I am in

1.23.0

./pants --version

works fine, and also

./pants test ...

works fine. But I encountered a weird bug. I found out that some of my unit tests got run twice, and that caused error. The folder structure has a

__init__.py

and the

BUILD

file is like this:

Copy code

python_tests( 
    name='xxxxx', 
    sources=globs('*.py'), 
    dependencies=[ ... ], 
)

I guess that’s why the same file get run two times. And due to some cache or memory issue, the variables in the first run were not cleaned up, so when the second run start, the value was not as expected, so the assertion failed. This didn’t happen in previous version. Is it due to some upgrade issue? I fixed it by deleting the

__init__.py

file, but I then thought modify

BUILD

file might work as well (?

echoing-farmer-15630

04/01/2022, 9:49 AM

Hmm. Yesterday with 2.11.0.rc0 I couldn't generate lockfiles because pants was picking up my python interpreter incorrectly. That's fixed in 2.11.0rc1, but now I seem to be having some problems with my 2.11.0rc1 pex lockfile generation

Copy code

1976.7s Generate lockfile for python-default

I eventually gave up after those 30 minutes and took out

lockfile_generator="pex"

and poetry spit out a 212k lockfile in 25 sec. Admittedly I have a pretty unclean setup (I'm trying to generate requirements from what was my old constraints.txt file, which represents a

pip freeze

of the project, so it's sort of generating a lockfile from a lockfile). But that difference was pretty striking and if the new pex-based JSON-style lockfiles are faster to use after generation, I'd love to make the switch. Any suggestions as to what may be going on?

freezing-area-97131

04/01/2022, 10:53 AM

Hi~ Now I have upgraded to

1.25.0

, and got this error when running

./pants test …

. How do I find out which packages cause this conflict?

Copy code

Exception message: Could not satisfy all requirements for pytest<4.7,>=4.6.6:
    pytest<4.7,>=4.6.6, pytest<5.4,>=5.3.5

👀 1

little-easter-51691

04/01/2022, 3:21 PM

Hello, I'd like to add the output (a dir with some files) of a python script (which has its own python dependencies) to a folder of an archive of another project. One possible way is to build a pex binary and call it with the experimental_shell step and use its output, but here the pex needs to be on the path to be found by the experimental_shell command. Is there an idiomatic way to achieve this?

wide-zoo-86070

04/01/2022, 3:33 PM

Hello. I would like to know the different between cached locally and memoized? For example, when using pants in our CI, sometimes I see

Copy code

project_a/test_helloworld/unit/translator_test.py succeeded in 0.61s (cached locally).

sometime it is

Copy code

project_a/test_helloworld/unit/translator_test.py succeeded in 0.61s (memoized).

high-yak-85899

04/01/2022, 5:46 PM

I think this is somewhat of an antipattern but I'm curious to get ideas: Is there any way to bundle up a bunch of python sources neatly into one pex binary-like thing? I think what I'm describing is a distribution now that I type it out. Basically, I have a bunch of modules that individually are executable but semantically should be grouped together. They'll all share roughly the same dependencies so packaging them all up as individual pex binaries would be pretty inefficient. I'd like to distribute them (and any transitive third party dependencies) as a unit. Should I look at Pants's wheel building capabilities for this or is there another direction folks would recommend?

rapid-exabyte-76685

04/02/2022, 6:47 AM

Does anyone have any best practices for applying CircleCI’s caching capabilities to the Pants

HOME/.cache/pants/

cache directories? 🧵

billions-bear-56484

04/02/2022, 2:40 PM

Hi guys, I was wondering if it is possible, to define script arguments in the pex_binary target? Currently my target looks like this and have to pass the arguments manually:

Copy code

pex_binary(
    name="alembic",
    script="alembic",
    dependencies=[
        ":alembic_requirements",
    ],
)

My goal is to have something like this:

Copy code

pex_binary(
    name="makemigrations",
    script="alembic -n accounts revision --autogenerate",
    // or entry_point="alembic -n accounts revision --autogenerate",
    dependencies=[
        ":alembic_requirements",
    ],
)

Thanks and have a great weekend 🎉

freezing-area-97131

04/03/2022, 7:11 PM

Hi~ I’m upgrading from pants 1.25.0 to 1.26.0.

./pants --version

works fine, but when I build my binary, I encounter this error:

Copy code

ERROR: Command errored out with exit status 1:
   command: /usr/bin/python2.7 /tmp/tmp35l8p0cf/.deps/pip/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpPhYaZM
       cwd: /tmp/pip-download-Ku1TIN/gevent
  Complete output (10 lines):
  Traceback (most recent call last):
    File "/tmp/tmp35l8p0cf/.deps/pip/pip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/tmp/tmp35l8p0cf/.deps/pip/pip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/tmp/tmp35l8p0cf/.deps/pip/pip/_vendor/pep517/_in_process.py", line 85, in get_requires_for_build_wheel
      backend = _build_backend()
    File "/tmp/tmp35l8p0cf/.deps/pip/pip/_vendor/pep517/_in_process.py", line 76, in _build_backend
      obj = getattr(obj, path_part)
  AttributeError: 'module' object has no attribute '__legacy__'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python2.7 /tmp/tmp35l8p0cf/.deps/pip/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpPhYaZM Check the logs for full command output.
19:05:57 [DEBUG] pants.process.lock:pid=7609: releasing lock: <pants.process.lock.OwnerPrintingInterProcessFileLock object at 0x7d76780ecd68>

               Waiting for background workers to finish.
12:05:57 00:32   [complete]
               FAILURE
timestamp: 2022-04-03T12:05:57.308688
Exception caught: (pex.resolver.Unsatisfiable)
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/pants_loader.py", line 94, in main
    PantsLoader.run()
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/pants_loader.py", line 90, in run
    cls.load_and_execute(entrypoint)
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/pants_loader.py", line 83, in load_and_execute
    entrypoint_main()
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/pants_exe.py", line 32, in main
    PantsRunner(start_time=start_time).run()
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/pants_runner.py", line 115, in run
    return runner.run()
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 293, in run
    self._run()
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 389, in _run
    goal_runner_result = self._maybe_run_v1()
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 330, in _maybe_run_v1
    self._exiter,
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/goal_runner.py", line 230, in run
    return self._run_goals()
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/goal_runner.py", line 201, in _run_goals
    result = self._execute_engine()
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/bin/goal_runner.py", line 189, in _execute_engine
    result = engine.execute(self._context, self._goals)
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/engine/legacy_engine.py", line 21, in execute
    self.attempt(context, goals)
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/engine/round_engine.py", line 252, in attempt
    goal_executor.attempt(explain)
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/engine/round_engine.py", line 50, in attempt
    task.execute()
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/backend/python/tasks/resolve_requirements.py", line 29, in execute
    pex = self.resolve_requirements(interpreter, self.context.targets(has_python_requirements))
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/backend/python/tasks/resolve_requirements_task_base.py", line 96, in resolve_requirements
    pex_builder.add_requirement_libs_from(req_libs, platforms=platforms)
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/python/pex_build_util.py", line 202, in add_requirement_libs_from
    self.add_resolved_requirements(reqs, platforms=platforms)
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/python/pex_build_util.py", line 275, in add_resolved_requirements
    distributions = self.resolve_distributions(reqs, platforms=platforms)
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/python/pex_build_util.py", line 251, in resolve_distributions
    self._builder.interpreter, list(deduped_reqs), platforms, list(find_links),
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pants/python/pex_build_util.py", line 339, in _resolve_multi
    manylinux=python_setup.manylinux,
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pex/resolver.py", line 722, in resolve
    return list(resolve_request.resolve_distributions(ignore_errors=ignore_errors))
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pex/resolver.py", line 521, in resolve_distributions
    raise_type=Unsatisfiable):
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pex/resolver.py", line 392, in _run_parallel
    max_jobs=self._max_parallel_jobs
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/1.26.0_py37/lib/python3.7/site-packages/pex/jobs.py", line 219, in execute_parallel
    raise error

Looks like a python2 syntax error again, though I have no idea how to fix this. Thanks!

rapid-exabyte-76685

04/04/2022, 5:30 AM

Is there any way to get Pants to somehow 'share' the binaries built from an sdist between the PEXs that get created for different sets of tests? Or is the correct way to do this to build a wheel/bdist for the dependency that is otherwise expressed as an sdist (accessed via a git reference)

refined-addition-53644

04/04/2022, 9:19 AM

Hi, would it be possible to maintain multiple poetry based (i.e. multiple

pyproject.toml

) python projects using multiple lockfiles "resolves"? I guess the issue could be if these projects have some common third party dependencies. We have a monorepo but it contains multiple python packages generated using poetry which have their own dependencies.

bitter-ability-32190

04/04/2022, 3:39 PM

Just wanted to share a win. Previously when we needed scripts from packages to be invoked as regular dev workflow we had to add it to our monumental list of requirements and run it through the user's virtual environment 🤮 Now, we have: • An isolated resolve for 3rdparty tools to be used directly as scripts (like `pip-tools`'s

pip-compile

) • An easy way to make sure the packaging is consistent and portable (

pex_binary

) ◦ With macros to make this easy for devs to declare • A consistent and portable way to invoke the tool (

./pants run ...

)

🙌 4

busy-vase-39202

04/04/2022, 4:16 PM

The lesson here is add Pants to your LinkedIn! https://pantsbuild.slack.com/archives/C046T6TA4/p1649081375622519

👖 2