https://pantsbuild.org/ logo
Join Slack
Powered by
I’m trying to create a special `package` rule that...
# plugins
a
I’m trying to create a special 
package
rule that works with 
python_library
as target. What I did was create a target
Copy code
class DagsDependencies(Dependencies):
    """Simple inheritance for dependencies."""


class Dags(Target):
    """DAG tasks generated by a library."""

    alias = "dags"
    core_fields = (*COMMON_TARGET_FIELDS, DagsDependencies)
and then
Copy code
@dataclass(frozen=True)
class DagsFieldSet(PackageFieldSet):
    required_fields = (DagsDependencies,)

    output_path: OutputPathField
    dependencies: DagsDependencies


@rule(level=<http://LogLevel.INFO|LogLevel.INFO>)
async def package_dags(field_set: DagsFieldSet) -> BuiltPackage:
    output_filename = field_set.output_path.value_or_default(
        field_set.address, file_ending="yaml"
    )
    ???
however, what I want no is to access the sources in the 
dependencies
, ie, loop over the dependencies and get the source files of those that are 
python_library
. I have been trying and reading source code, and I couldn’t manage. Any hints?
c
a
Yes, that could work!
I’m wondering if this is the best approach, because this gives me the source files of the library, but no 3rd party deps right? I will probably need to build a pex or something like that to have everything installed
c
If you need to package a dependency, an example of that is on the lines following. Apologies for not being more helpful. I'm on the phone…
a
Oh I see, this works in master though, doesn’t seem to be released? Looking specifically at 
FileSourceField
is it recommended to develop on master?
Looking at the code, what I think I need to do is for each of the dependencies create a 
PexEnvironment
, run a few functions and get the output. I’ve gone through the docs and everything and I can’t really figure out how to do this. Are there any other docs or tutorials I can look at? The examples of 
package
for example don’t even use 
Dependencies
and they cover quite a simple case…
c
Ah, apologies. Yes, the code I pointed at should work the same on released code, with the exception for the field type.
At a computer now, so will look a bit closer at your question, see if I can find a better reply 🙂
a
Thanks! I think I’m looking at it all wrong… Looking at it again, what I’m trying to do is put together several python libraries, and this requires importing code. So what I should do: 1. Create a target for each of the libraries (
task
) that creates a pex environment, imports them and creates a yaml file with some data. How to do that in a streamlined way? 2. Create a target that has requires the libraries (or the 
task
targets?) as dependencies, imports those yaml files and does operations. No clue if it’s better task or library as dependency, but the rest should be reasonably straightforward
c
So, is the second task dependent on the yaml data from the first task? Did I understand that correctly.. ?
(Created a gist for your initial question, although it may not be relevant any longer.. https://gist.github.com/kaos/378d80196c40480425880c1d23504203 )
a
So, is the second task dependent on the yaml data from the first task? Did I understand that correctly.. ?
YEs! and ideally I should be able to run onlt on “changed” libraries if possible, but I don’t understand caching mechanisms
👍 1
c
I think the new 
experimental_shell_command
and possibly 
experimental_run_shell_command
are targets for you. I’ll run some experiments.
OK, first, these are experimental commands, and requires pants 
2.8.0.dev5
and have a few rough edges still. But. Here’s an example to show what will be possible. Feedback most welcome. Files:
Copy code
.
├── pants
├── pants.toml
└── src
    └── example
        ├── BUILD
        ├── __init__.py
        ├── task1.py
        └── task2.py
Copy code
# src/example/BUILD
python_library(name="example")
python_requirement(name="yaml", requirements=["pyyaml"])

pex_binary(name="task1-bin", entry_point="example.task1:main")
pex_binary(name="task2-bin", entry_point="task2.py:main")

experimental_shell_command(
    name="task1",
    command="../../src.example/task1-bin.pex > task.yaml",
    tools=["bash", "python3.7", "cut", "sed"],
    log_output=True,
    dependencies=[":task1-bin"],
    outputs=["task.yaml"],
    timeout=2,
)

experimental_run_shell_command(
    name="task2",
    command="tree {chroot} && cat {chroot}/src/example/task.yaml | {chroot}/src.example/task2-bin.pex",
    dependencies=[":task1", ":task2-bin"],
)
Copy code
# src/example/task1.py
import yaml

def main():
    data = dict(task="data")
    print(yaml.safe_dump(data))
Copy code
# src/example/task2.py
import sys
import yaml


def main():
    data = yaml.safe_load(sys.stdin)
    print("Read data:", data)
Example output:
Copy code
$ ./pants run src/example:task2
/Users/aadt/src/github/kaos/dags/.pants.d/tmpk6nwqjyr
├── src
│   └── example
│       ├── task.yaml
│       ├── task1.py
│       └── task2.py
└── src.example
    ├── task1-bin.pex
    └── task2-bin.pex

3 directories, 5 files
Read data: {'task': 'data'}
Copy code
# pants.toml
[GLOBAL]
pants_version = "2.8.0.dev5"
backend_packages.add = [
  "pants.backend.python",
  "pants.backend.shell",
]
a
oh, I see! So simply creating the 
pex
with the command I want and running it
then I could in principle write simple scripts
and they would be “pants-independent”
c
Right. So, 
./pants package pex:target
will put your pex file in 
./dist/...
by default.
a
Thanks! Now I’m reallymoving forward! I was looking into the example
Copy code
experimental_shell_command(
    name="task1",
    command="../../src.example/task1-bin.pex > task.yaml",
    tools=["bash", "python3.7", "cut", "sed"],
    log_output=True,
    dependencies=[":task1-bin"],
    outputs=["task.yaml"],
    timeout=2,
)
Why does the command have 
../../?
What’s the root of the running?
I ask because I can’t get my pex to be executed “No such file or directory”
Tree
Copy code
BUILD
app/data_grabber
├── BUILD
This is in the data_grabber 
BUILD
Copy code
resources(name="config", sources=["data_grabber/config_data/sources/*"])

python_sources(
    name="grabbers",
    sources=["data_grabber/grabbers/*.py"],
)

python_sources(
    name="data_grabber",
    sources=["data_grabber/**/*.py", "!data_grabber/grabbers/*.py"],
    dependencies=[":config", ":grabbers"],
)

pex_binary(
    name="app",
    entry_point="data_grabber/cli.py:cli",
    dependencies=[":data_grabber"],
)

python_tests(name="unit", sources=["tests/*.py"], tags=["unit"])

python_requirements(module_mapping={"python-dateutil": ["dateutil"]})

experimental_shell_command(
    name="tasks",
    command="./dist/app.data_grabber/app.pex --no-log-to-file list-tasks tasks.yaml",
    tools=[
        "bash",
        "python",
    ],
    log_output=True,
    dependencies=["app/data_grabber:app"],
    outputs=["tasks.yaml"],
    timeout=10,
)
And the root 
BUILD
Copy code
experimental_run_shell_command(
    name="test",
    command="tree {chroot} && cat {chroot}/src/example/tasks.yaml > dag.yaml",
    dependencies=["app/data_grabber:tasks"],
)
c
Ah, yes, that is one of the rough edges. It is because the command is executed in the same relative directory as the BUILD file, in which it is defined. Where as the packaged pex is in 
src.example/task.pex
from the build root. so the 
../../
is to get back up to build root.
a
weird, this should work and it doesn’t then
with the build files above ^
c
What is really helpful is to run with 
--no-process-execution-local-cleanup
and inspect the chroot for your shell command.
command=“./dist/app.data_grabbe
This should most likely be 
../../app.data_grabber/app.pex …
a
Copy code
$ ./pants run :test --no-process-execution-local-cleanup
14:45:36.69 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-execution2Fdf4K for "Test binary /Users/albert/.pyenv/shims/python."
14:45:36.69 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionjffSjl for "Test binary /Users/albert/.pyenv/shims/python3."
14:45:36.86 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionTY695M for "Running experimental_shell_command app/data_grabber:tasks"
14:45:36.98 [INFO] Completed: Running experimental_shell_command app/data_grabber:tasks
14:45:36.98 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 236, in _run_inner
    return self._perform_run(goals)
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 175, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 192, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/init/engine_initializer.py", line 133, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 548, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 516, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Running experimental_shell_command app/data_grabber:tasks' failed with exit code 127.
stdout:

stderr:
/bin/bash: app.data_grabber/app.pex: No such file or directory




Use --print-stacktrace for more error details and/or -ldebug for more logs.
See <https://www.pantsbuild.org/v2.8/docs/troubleshooting> for common issues.
Consider reaching out for help: <https://www.pantsbuild.org/v2.8/docs/getting-help>


╭─14:45:37 albert@dc10 ~/Arxiu/Work/Projects/Ongoing/python-apps-pants 1 3.8.6
│ python-apps-pants±feature/pants!?                                                                                                                                                                                                     1.64s
$ tree  /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionTY695M
/private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionTY695M
├── __run.sh
├── app
│   └── data_grabber
│       ├── data_grabber
│...
├── app.data_grabber
│   └── app.pex
I see, it works with the 
../..
and no dist
so it goes in the chroot and then execute a the BUILD level
c
Yes, the 
experimental_shell_command
executes in the BUILD folder, where as the 
experimental_run_shell_command
executes at the project root.
It doesn’t really make sense here as to why, but if you have source dependencies (from the same directory) it does… still working out how to make this intuitive for any kind of use case…
a
I see, yes
now, next issue is timeout
the pex runs quickly if run with 
run
c
Are you using pyenv?
a
but it’s timing out
yes, my setup is with 
pyenv
c
Then add 
cut
and 
sed
to your list of 
tools
..
a
oooh wow
c
This is also a corner case, which presents itself if you execute the __run.sh script from the chroot of the shell command, but not otherwise.. unfortunately..
a
wow , now I’m interested 🙂
c
The process invocation fails, but for some reason the stdout/stderr are not propagated in that case, so the error message is buried.
a
still timeout
Copy code
experimental_shell_command(
    name="tasks",
    command="../../app.data_grabber/app.pex --no-log-to-file list-tasks tasks.yaml",
    tools=["bash", "python3.8", "sed", "cut"],
    log_output=True,
    dependencies=["app/data_grabber:app"],
    outputs=["tasks.yaml"],
    timeout=30,
)
Copy code
$ ./pants run :test
14:52:24.93 [INFO] Completed: Running experimental_shell_command app/data_grabber:tasks
14:52:24.94 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 236, in _run_inner
    return self._perform_run(goals)
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 175, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 192, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/init/engine_initializer.py", line 133, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 548, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 516, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Running experimental_shell_command app/data_grabber:tasks' failed with exit code -15.
stdout:
Exceeded timeout of 30.0 seconds when executing local process: Running experimental_shell_command app/data_grabber:tasks
stderr:
c
Ouch.. hmm… ok. Yeah, try running it with the 
__run.sh
from the chroot (which you get when you use the no process cleanup flag I posted earlier)
a
Had to kill it, seems like 
ln
is misbehaving
Copy code
$ ./__run.sh
./__run.sh: line 3: export: `python3.8=/Users/albert/.pyenv/shims/python3.8': not a valid identifier
ln: .bin/mkdir: File exists
ln: .bin/sed: File exists
ln: ./.bin: File exists
ln: .bin/ln: File exists
ln: .bin/cut: File exists
ln: .bin/bash: File exists
^C
./__run.sh  13.86s user 12.55s system 94% cpu 27.846 total
c
Oh, yeah, you can rm -rf .bin before re-running should help with those ln issues.
from within the chroot
Oh, I see… the tool has a period in there… sorry, haven’t anticipated that
a
let me try without
I wanted to just use 
python
bur didn’t work
yeah, the dot is killng it
c
mm… I’ve had some issues getting a correct python version too… does 
python3
work?
a
nope
the pex wants 
python3.8
Copy code
env: python3.8: No such file or directory
c
😕 I’ll put up a fix for the dot issue.. I’ll see if I can think of a work around in the mean time..
When you have a period in the tool name, what does your 
__run.sh
contain?
mine looks like:
Copy code
$ cat __run.sh 
#!/bin/bash
# This command line should execute the same process as pants did internally.
export TOOLS=$'sed mkdir bash python3.7 ln cut' bash=/usr/local/bin/bash cut=/usr/bin/cut ln=/bin/ln mkdir=/bin/mkdir python3.7=/Users/x/.pyenv/shims/python3.7 sed=/usr/bin/sed
cd /private/var/folders/8j/c8jf_msj009947wyw82xvdkw0000gn/T/process-executionOZ0BSk/src/example
/bin/bash -c $'$mkdir -p .bin;for tool in $TOOLS; do $ln -s ${!tool} .bin; done;export PATH="$PWD/.bin";../../src.example/task1-bin.pex > task.yaml'
Nvm, I get the same message as you now…
Although, when running it with pants, it does work. It’s just the __run.sh script that has issues, it would seem.
Right, another pyenv question.. what version do you get with 
pyenv version
?
I had the same issue (but with python3.7 instead..) and had to work around it with setting 
pyenv local 3.7.12
(as the 
experimental_run_shell_command
uses the python version you have in your env) so if 
python3.8 --version
doesn’t work with your path, the run shell command won’t either..
a
mmm that should work, let me try
pyenv local works
so I avoided the 3.8 problem with a simple workaround
Copy code
experimental_shell_command(
    name="tasks",
    command="python ../../app.data_grabber/app.pex --no-log-to-file list-tasks tasks.yaml",
    tools=["bash", "python", "sed", "cut"],
    log_output=True,
    dependencies=["app/data_grabber:app"],
    outputs=["tasks.yaml"],
    timeout=30,
)
🙏 1
👍 1
running 
__run.sh
doesn’t work though
it stays in a silent loop
unless I run 
pyenv local
Copy code
╭─15:38:11 albert@dc10 /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executioneyXrwn                                                                                                                                 0.03s
$ time ./__run.sh
^C
./__run.sh  93.99s user 58.12s system 95% cpu 2:39.32 total

╭─15:40:57 albert@dc10 /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executioneyXrwn SIGINT(2)                                                                                                                       4.13s
$ pyenv local 3.8.6

╭─15:41:00 albert@dc10 /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executioneyXrwn 3.8.6                                                                                                                           0.09s
$ time ./__run.sh
ln: .bin/mkdir: File exists
ln: .bin/sed: File exists
ln: .bin/python: File exists
ln: .bin/ln: File exists
ln: .bin/cut: File exists
ln: .bin/bash: File exists
./__run.sh  2.57s user 0.61s system 101% cpu 3.118 total
(and running with pants doesn’t work)
because my default python is 3.9, and this runs on 3.8 I guess
c
Yeah, there’s a few issues in the __run.sh script. Will address them. But running using pants ought to work..
a
times out
c
Sorry for the troubles with this experimental feature, though. It’s great feedback what works and not.
a
happy to help, and thanks for your help!!
❤️ 1
the problem is that the correct python is not being passed
Copy code
experimental_shell_command(
    name="tasks",
    command="python3 ../../app.data_grabber/app.pex --no-log-to-file list-tasks tasks.yaml",
    tools=["bash", "python3", "sed", "cut"],
    log_output=True,
    dependencies=["app/data_grabber:app"],
    outputs=["tasks.yaml"],
    timeout=30,
)
The code should run in 3.8.6 but I get
Copy code
$ ./pants run :test
15:45:29.80 [INFO] Completed: Running experimental_shell_command app/data_grabber:tasks
15:45:29.81 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 236, in _run_inner
    return self._perform_run(goals)
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 175, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 192, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/init/engine_initializer.py", line 133, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 548, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 516, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Running experimental_shell_command app/data_grabber:tasks' failed with exit code 1.
stdout:

stderr:
Failed to find compatible interpreter on path /Users/albert/.pyenv/versions/3.9.7/bin:/usr/local/Cellar/pyenv/2.0.6/libexec:/usr/local/Cellar/pyenv/2.0.6/plugins/python-build/bin:/private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executioniPTfl1/app/data_grabber/.bin.

Examined the following interpreters:
1.) /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/bin/python3.9 CPython==3.9.7

No interpreter compatible with the requested constraints was found:
  Version matches CPython==3.8.6
Copy code
╭─15:45:29 albert@dc10 ~/Arxiu/Work/Projects/Ongoing/python-apps-pants 1 3.8.6
│ python-apps-pants±feature/pants!?                                                                                                                                                                                                     3.03s
$ cat .python-version
3.8.6

╭─15:46:38 albert@dc10 ~/Arxiu/Work/Projects/Ongoing/python-apps-pants 3.8.6
│ python-apps-pants±feature/pants!?                                                                                                                                                                                                     0.04s
$ which python3
/Users/albert/.pyenv/shims/python3

╭─15:46:46 albert@dc10 ~/Arxiu/Work/Projects/Ongoing/python-apps-pants 3.8.6
│ python-apps-pants±feature/pants!?                                                                                                                                                                                                     0.03s
$ python3
Python 3.8.6 (default, Feb 10 2021, 19:43:41)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
c
what search pants paths and interpreter constraints do you have in your 
pants.toml
?
a
Copy code
[python-setup]
interpreter_constraints = ["CPython==3.8.6"]
interpreter_search_paths = ["<PYENV_LOCAL>", "<PYENV>", "<PATH>"]
c
Hah.. that looks good to me… hmm..
Oh, right. Perhaps this is the process looking for a 
python3
on your path, and finds the python3.9 one.. so this is not to bootstrap and run pants, and in that case, perhaps specifying a shell binary path could help: https://www.pantsbuild.org/v2.8/docs/reference-shell-setup
Not sure if 
<PYENV>
et al are supported here…
There ought to be a way to specify/prefer the interpreter currently used for running the pants process…
Ok, so for step 1, to make the 
__run.sh
script runnable: https://github.com/pantsbuild/pants/pull/13293 This will also support 
.
in the tool names, which potentially would also fix the other issue w python interpreter you have.. will continue to work on improving this situation, however..
a
I’ll test this ASAP
💯 1
Oh wait, what’s the correct way of testing this? Run a development version of pants?
c
Ah, yes easiest will be once it has been merged. Then you can run it with the sha from the latest commit on main by setting PANTS_SHA (see the pants boot script)
👍 1
OK, it has been merged, so using 
PANTS_SHA=4702f46187502fb70d00758852ef35556d165ec7
should allow you to try this out.
🙏 1
a
Finally got back to this! I updated pants
Copy code
$ PANTS_SHA=4702f46187502fb70d00758852ef35556d165ec7 ./pants --version
Bootstrapping Pants using /Users/albert/.pyenv/shims/python3.9
Installing pantsbuild.pants==2.8.0.dev5+git4702f461 into a virtual environment at /Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/2.8.0.dev5+git4702f461_py39
created virtual environment CPython3.9.7.final.0-64 in 453ms
...
12:00:10.81 [INFO] Initializing scheduler...
12:00:10.95 [INFO] Scheduler initialized.
2.8.0.dev5+git4702f461
But I ran
Copy code
$ ./pants run :test --no-process-execution-local-cleanup
12:02:35.08 [INFO] Initialization options changed: reinitializing scheduler...
12:02:35.28 [INFO] Scheduler initialized.
12:02:35.32 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionMSP1O9 for "Searching for `bash` on PATH=/usr/bin:/bin:/usr/local/bin"
12:02:35.50 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-execution4qRUkZ for "Test binary /bin/bash."
12:02:35.94 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionzK8dUS for "Searching for `python2` on PATH=/Users/albert/.pyenv/versions/3.8.6/bin:/Users/albert/.pyenv/versions/3.9.7/bin:/Users/albert/.pyenv/shims:/usr/local/bin:/usr/local/sbin:/usr/local/opt/go/libexec/bin:/usr/local/opt/go/bin/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Users/albert/.scripts:/usr/local/Caskroom/miniconda/base/bin"
12:02:35.94 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionpTSkFI for "Searching for `python` on PATH=/Users/albert/.pyenv/versions/3.8.6/bin:/Users/albert/.pyenv/versions/3.9.7/bin:/Users/albert/.pyenv/shims:/usr/local/bin:/usr/local/sbin:/usr/local/opt/go/libexec/bin:/usr/local/opt/go/bin/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Users/albert/.scripts:/usr/local/Caskroom/miniconda/base/bin"
12:02:35.95 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionJl41YD for "Searching for `python3` on PATH=/Users/albert/.pyenv/versions/3.8.6/bin:/Users/albert/.pyenv/versions/3.9.7/bin:/Users/albert/.pyenv/shims:/usr/local/bin:/usr/local/sbin:/usr/local/opt/go/libexec/bin:/usr/local/opt/go/bin/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Users/albert/.scripts:/usr/local/Caskroom/miniconda/base/bin"
12:02:36.04 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionYFq30t for "Test binary /usr/bin/python."
12:02:36.05 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionfLEOWB for "Test binary /Users/albert/.pyenv/shims/python."
12:02:36.05 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionxKZRIX for "Test binary /Users/albert/.pyenv/versions/3.8.6/bin/python."
12:02:36.05 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionijqEaU for "Test binary /usr/local/Caskroom/miniconda/base/bin/python."
12:02:36.10 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionsmS28U for "Test binary /usr/bin/python2."
12:02:36.14 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-execution02U4by for "Test binary /Users/albert/.pyenv/versions/3.8.6/bin/python3."
12:02:36.14 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionWBGSGO for "Test binary /Users/albert/.pyenv/versions/3.9.7/bin/python3."
12:02:36.15 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionW3hyLL for "Test binary /Users/albert/.pyenv/shims/python3."
12:02:36.15 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionJ5KVMv for "Test binary /usr/local/Caskroom/miniconda/base/bin/python3."
12:02:36.15 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executioniy0EtA for "Test binary /usr/local/bin/python3."
12:02:36.16 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-execution4UOmWt for "Test binary /usr/bin/python3."
12:02:36.27 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionc6LYSH for "Find interpreter for constraints: CPython==3.8.6"
12:02:38.12 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionOE9Ghr for "Searching for `sed` on PATH=/Users/albert/.pyenv/shims:/usr/local/bin:/usr/local/sbin:/usr/local/opt/go/libexec/bin:/usr/local/opt/go/bin/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Users/albert/.scripts:/usr/local/Caskroom/miniconda/base/bin"
12:02:38.12 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionN0EDIA for "Searching for `python3.8` on PATH=/Users/albert/.pyenv/shims:/usr/local/bin:/usr/local/sbin:/usr/local/opt/go/libexec/bin:/usr/local/opt/go/bin/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Users/albert/.scripts:/usr/local/Caskroom/miniconda/base/bin"
12:02:38.12 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executiontttZWv for "Searching for `mkdir` on PATH=/Users/albert/.pyenv/shims:/usr/local/bin:/usr/local/sbin:/usr/local/opt/go/libexec/bin:/usr/local/opt/go/bin/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Users/albert/.scripts:/usr/local/Caskroom/miniconda/base/bin"
12:02:38.12 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionUoI5BJ for "Searching for `bash` on PATH=/Users/albert/.pyenv/shims:/usr/local/bin:/usr/local/sbin:/usr/local/opt/go/libexec/bin:/usr/local/opt/go/bin/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Users/albert/.scripts:/usr/local/Caskroom/miniconda/base/bin"
12:02:38.13 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionllP8Ih for "Searching for `ln` on PATH=/Users/albert/.pyenv/shims:/usr/local/bin:/usr/local/sbin:/usr/local/opt/go/libexec/bin:/usr/local/opt/go/bin/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Users/albert/.scripts:/usr/local/Caskroom/miniconda/base/bin"
12:02:38.13 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionA2Oqxn for "Searching for `cut` on PATH=/Users/albert/.pyenv/shims:/usr/local/bin:/usr/local/sbin:/usr/local/opt/go/libexec/bin:/usr/local/opt/go/bin/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin:/Users/albert/.scripts:/usr/local/Caskroom/miniconda/base/bin"
12:02:38.56 [INFO] Preserving local process execution dir /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-execution9c0q1P for "Running experimental_shell_command app/data_grabber:tasks"
12:03:08.67 [INFO] Completed: Running experimental_shell_command app/data_grabber:tasks
12:03:08.67 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 236, in _run_inner
    return self._perform_run(goals)
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 175, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/bin/local_pants_runner.py", line 192, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/init/engine_initializer.py", line 133, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 548, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/albert/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.Q0kpzs/install/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 516, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Running experimental_shell_command app/data_grabber:tasks' failed with exit code -15.
stdout:
Exceeded timeout of 30.0 seconds when executing local process: Running experimental_shell_command app/data_grabber:tasks
stderr:




Use --print-stacktrace for more error details and/or -ldebug for more logs.
See <https://www.pantsbuild.org/v2.8/docs/troubleshooting> for common issues.
Consider reaching out for help: <https://www.pantsbuild.org/v2.8/docs/getting-help>
and it seems 
__run.sh
is still the same?
Copy code
╭─12:03:41 albert@dc10 ~/Arxiu/Work/Projects/Ongoing/python-apps-pants 3.8.6
│ python-apps-pants±feature/pants+!?                                                                                                                                                                                                   0.02s
$ /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-execution9c0q1P

╭─12:03:46 albert@dc10 /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-execution9c0q1P                                                                                                                                0.03s
$ ls
__run.sh         app              app.data_grabber lib              requirements.txt

╭─12:03:48 albert@dc10 /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-execution9c0q1P                                                                                                                                0.03s
$ cat __run.sh
#!/bin/bash
# This command line should execute the same process as pants did internally.
export TOOLS=$'python3.8 mkdir bash ln cut sed' bash=/bin/bash cut=/usr/bin/cut ln=/bin/ln mkdir=/bin/mkdir python3.8=/Users/albert/.pyenv/shims/python3.8 sed=/usr/bin/sed
cd /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-execution9c0q1P/app/data_grabber
/bin/bash -c $'$mkdir -p .bin;for tool in $TOOLS; do $ln -s ${!tool} .bin; done;export PATH="$PWD/.bin";python3.8 ../../app.data_grabber/app.pex --no-log-to-file list-tasks tasks.yaml'
👀 1
c
That is very strange. It not only works for me, but I also get a properly cleaned up 
__run.sh
script..
Copy code
13:03:05.16 [INFO] Preserving local process execution dir /private/var/folders/8j/c8jf_msj009947wyw82xvdkw0000gn/T/process-execution3QVCaA for "Building src.example/task1-bin.pex with 1 requirement: pyyaml"
13:03:09.27 [INFO] Completed: Building src.example/task1-bin.pex with 1 requirement: pyyaml
13:03:09.28 [INFO] Preserving local process execution dir /private/var/folders/8j/c8jf_msj009947wyw82xvdkw0000gn/T/process-executionSip8xd for "Running experimental_shell_command src/example:task1"
13:03:10.08 [INFO] Completed: Running experimental_shell_command src/example:task1
13:03:10.09 [INFO] Preserving local process execution dir /private/var/folders/8j/c8jf_msj009947wyw82xvdkw0000gn/T/process-executionM0iGWm for "Building src.example/task2-bin.pex with 1 requirement: pyyaml"
13:03:13.67 [INFO] Completed: Building src.example/task2-bin.pex with 1 requirement: pyyaml
/Users/.../.pants.d/tmp82j6qjwo
├── src
│   └── example
│       ├── task.yaml
│       ├── task1.py
│       └── task2.py
└── src.example
    ├── task1-bin.pex
    └── task2-bin.pex

3 directories, 5 files
Read data: {'task': 'data'}

$ cat /private/var/folders/8j/c8jf_msj009947wyw82xvdkw0000gn/T/process-executionSip8xd/__run.sh 
#!/bin/bash
# This command line should execute the same process as pants did internally.
export TOOLS=$'ln sed bash python3_7 mkdir cut' bash=/usr/local/bin/bash cut=/usr/bin/cut ln=/bin/ln mkdir=/bin/mkdir python3_7=/Users/aadt/.pyenv/shims/python3.7 sed=/usr/bin/sed
cd /private/var/folders/8j/c8jf_msj009947wyw82xvdkw0000gn/T/process-executionSip8xd/src/example
/bin/bash -c $'$mkdir -p .bin;for tool in $TOOLS; do $ln -sf ${!tool} .bin; done;export PATH="$PWD/.bin";../../src.example/task1-bin.pex > task.yaml'

$ PANTS_SHA=4702f46187502fb70d00758852ef35556d165ec7 ./pants --version
13:04:29.37 [INFO] Initialization options changed: reinitializing scheduler...
13:04:29.96 [INFO] Scheduler initialized.
2.8.0.dev5+git4702f461
There’s a 2.8.0rc0 in the process of being released right now… so when that’s out, see if that makes any difference for you.
a
oh, ok, I’ll wait 🙂
I saw it was out, now 
__run.sh
is correct I think
Copy code
#!/bin/bash
# This command line should execute the same process as pants did internally.
export TOOLS=$'mkdir cut bash sed python3_8 ln' bash=/bin/bash cut=/usr/bin/cut ln=/bin/ln mkdir=/bin/mkdir python3_8=/Users/albert/.pyenv/shims/python3.8 sed=/usr/bin/sed
cd /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionfeKloM/app/data_grabber
/bin/bash -c $'$mkdir -p .bin;for tool in $TOOLS; do $ln -sf ${!tool} .bin; done;export PATH="$PWD/.bin";python3.8 ../../app.data_grabber/app.pex --no-log-to-file list-tasks tasks.yaml'
But doesn’t run, it just spins 
bash
processes that stay in the background, but doesn’t return
even running it directly
running with 
pants run
works
Copy code
$ ./pants run app/data_grabber:app -- --no-log-to-file list-tasks test.yaml
20:08:23.88 [INFO] Initialization options changed: reinitializing scheduler...
20:08:24.32 [INFO] Scheduler initialized.
20:08:53.47 [INFO] Completed: Building app.pex with 17 requirements: PyYAML==5.4, atlassian-python-api==3.13.0, bs4>=0.0.1, click>=7.1, clickhouse_driver==0.2.1, confluent-kafka==1.5.0, google-api-python-client==2.4.0, google-auth... (185 characters truncated)
$
what else can I do to debug?
c
Does the app.pex start at all? Does it run any subprocesses? Try running the python interpreter with the same path as setup in the __run script.
a
something starts because I get a rogue bash process in the background
ok
interesting
Copy code
╭─20:30:28 albert@dc10 /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionHhFG3q                                                                                                                                0.22s
$ export TOOLS=$'mkdir cut bash sed python3_8 ln' bash=/bin/bash cut=/usr/bin/cut ln=/bin/ln mkdir=/bin/mkdir python3_8=/Users/albert/.pyenv/shims/python3.8 sed=/usr/bin/sed
cd /private/var/folders/6k/8y0zj1hn2jg7qmk631zj5nr80000gn/T/process-executionHhFG3q/app/data_grabber
/bin/bash

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit <https://support.apple.com/kb/HT208050>.
bash-3.2$ which python3.8
/Users/albert/.pyenv/shims/python3.8
bash-3.2$ $mkdir -p .bin;echo "made bin"; for tool in $TOOLS; do $ln -sf ${!tool} .bin; echo "Linked ${!tool}"; done
made bin
Linked /bin/mkdir
Linked /usr/bin/cut
Linked /bin/bash
Linked /usr/bin/sed
Linked /Users/albert/.pyenv/shims/python3.8
Linked /bin/ln
bash-3.2$ python3.8
Python 3.8.5 (default, Sep  4 2020, 02:22:02)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
It picks up a completely random python
something from anaconda which doesn’t make any sense
Copy code
bash-3.2$ python3.8
Python 3.8.5 (default, Sep  4 2020, 02:22:02)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
because my python is 3.8.6
so it is still struggling with pyenv
in my normal situation I get
Copy code
╭─20:34:22 albert@dc10 ~/Arxiu/Work/Projects/Ongoing/python-apps-pants 3.8.6
│ python-apps-pants±feature/pants+!?                                                                                                                                                                                                   0.03s
$ python3.8
Python 3.8.6 (default, Feb 10 2021, 19:43:41)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
3 Views
Open in Slack
PreviousNext
Powered by