https://pantsbuild.org/ logo
Join Slack
Powered by
Hello again. I am trying to convert my “python_app...
# general
s
Hello again. I am trying to convert my “python_app” targets to “archive” targets and im running into this error.
Copy code
15:08:58.38 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/briandesimone/.cache/pants/setup/bootstrap-Darwin-x86_64/2.1.1_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 255, in run
    engine_result = self._run_v2(goals)
  File "/Users/briandesimone/.cache/pants/setup/bootstrap-Darwin-x86_64/2.1.1_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 182, in _run_v2
    return self._maybe_run_v2_body(goals, poll=False)
  File "/Users/briandesimone/.cache/pants/setup/bootstrap-Darwin-x86_64/2.1.1_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 204, in _maybe_run_v2_body
    poll_delay=(0.1 if poll else None),
  File "/Users/briandesimone/.cache/pants/setup/bootstrap-Darwin-x86_64/2.1.1_py36/lib/python3.6/site-packages/pants/init/engine_initializer.py", line 126, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/Users/briandesimone/.cache/pants/setup/bootstrap-Darwin-x86_64/2.1.1_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 569, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/briandesimone/.cache/pants/setup/bootstrap-Darwin-x86_64/2.1.1_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 539, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Create db.data_cache_db.lexio-migrations-data-cache.tar' failed with exit code 1.
stdout:

stderr:
Usage:
  List:    tar -tf <archive-filename>
  Extract: tar -xf <archive-filename>
  Create:  tar -cf <archive-filename> [filenames...]
  Help:    tar --help
This doesn’t give me much to go on here and even with debug mode on this is the extent of the error message. Here is my BUILD file:
Copy code
python_library(
    name="lib",
    dependencies=[
        "3rdparty/python:alembic",
        "3rdparty/python:SQLAlchemy",
        "lib/python_core/src",
    ],
    sources=["**/*.py"],
    tags={"code", "db", "migration", "python"},
)
pex_binary(
    name="alembic", entry_point="alembic.config", dependencies=[":lib"], zip_safe=False
)
archive(
    name="lexio-migrations-data-cache",
    packages=[":alembic"],
    files=[":files"],
    format="tar",
    tags={"code", "db", "migration", "python"},
    output_path="db.data_cache_db.lexio-migrations-data-cache.tar",
)
files(
  name="files",
  sources=[
    "alembic.ini",
    "env.py",
    "versions/*.py",
  ],
)
Any help here would be appreciated
h
Hm I suspect that we are invoking 
tar
in an illegal way on your machine. What is your environment, and what does 
tar --version
say?
s
Copy code
bsdtar 2.8.3 - libarchive 2.8.3
im in a pyenv running python 3.6.8
👍 1
h
K, the command we run is 
/path/to/tar caf result.tar f1.ext f2.ext ...
. I suspect tar 2.8 doesn’t like that for some reason. Could you please try running something like the above directly on the CLI?
s
yeah one second
w
in 
-ldebug
there should be a bit more information about the process that is launched: it should have the complete argument list
👍 1
will look like:
Copy code
12:06:19.39 [DEBUG] spawned local process as 17884 for Process { argv: ["/Users/stuhood/.pyenv/versions/3.7.7/bin/python3.7", "./pytest.pex", "--no-header", "-k", "test_cache_scope_successful", "src/python/pants/engine/process_test.py"], env: {"LANG": "en_US.UTF-8", "LC_CTYPE": "en_US.UTF-8", "PATH": "/Users/stuhood/src/pants/build-support/virtualenvs/Darwin/pants_dev_deps.py37.venv/bin:/Users/stuhood/src/google-cloud-sdk/bin:/Users/stuhood/.cargo/bin:/Users/stuhood/.pyenv/shims:/Users/stuhood/.pyenv/bin:/Users/stuhood/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin", "PEX_EXTRA_SYS_PATH": "src/python", "PEX_IGNORE_RCFILES": "true", "PEX_INHERIT_PATH": "false", "PEX_ROOT": "/Users/stuhood/.cache/pants/named_caches/pex_root", "PYTEST_ADDOPTS": "--color=yes"}, working_directory: None, input_files: Digest(Fingerprint<7f3ffe3931443d29371d1517757f49d31bcca776d58b14e3a472e84a0b3abc59>, 431), output_files: {}, output_directories: {}, timeout: Some(90s), execution_slot_variable: Non
“spawned local process as …”
s
yeah i do have that from the debug logs. I think its a permissions issue on my dist dir.. let me check something
… and thats a no. thought it was because when i ran that above command i had some permission denied errors when trying to use a tar
h
That 
tar
process will run in a tmpdir and write the file there. Then, Pants will capture the result, and Pants will write to 
dist/
. This error is happening when invoking 
tar
, so permissions on the 
dist/
folder is almost certainly a red herring as we haven’t hit that step yet
s
ah got it. fair point.
Copy code
cannot execute binary file
here is the stderr from trying to execute above command
i just made a tar and then ran what you had above
here is my debug logs if this helps
Copy code
spawned local process as 82972 for Process { argv: ["/usr/bin/tar", "caf", "db.data_cache_db.lexio-migrations-data-cache.tar", "db/data_cache_db/alembic.ini", "db/data_cache_db/env.py", "db/data_cache_db/versions/006652de6637_add_day_of_fiscal_quarter_function.py", "db/data_cache_db/versions/0937f76df8bd_add_topic_feeds_table.py", "db/data_cache_db/versions/25a71be92e10_add_fiscal_year_function.py", "db/data_cache_db/versions/2f6e248acf13_update_card_title_and_card_category_to_.py", "db/data_cache_db/versions/355b0fd2f88b_feed_items_jsonb_not_null.py", "db/data_cache_db/versions/6843323245c1_add_fiscal_quarter_year_function.py", "db/data_cache_db/versions/6f8531f3d5b6_refactor_fiscal_quarter_year_function.py", "db/data_cache_db/versions/7103915533d9_adding_chart_content_to_topic_feed_.py", "db/data_cache_db/versions/73ff0eea1886_add_day_of_fiscal_year_function.py", "db/data_cache_db/versions/78a04f5f3c39_drop_newsfeed_v1_tables.py", "db/data_cache_db/versions/85902519c1fe_add_metric_column.py", "db/data_cache_db/versions/a105975e3483_add_card_category_and_card_label_.py", "db/data_cache_db/versions/c892ac9c500e_add_day_of_quarter_function.py", "db/data_cache_db/versions/d0c55c9df3f5_create_feed_tables.py", "db/data_cache_db/versions/d81d044d33ac_add_story_config_to_topic_feed_items.py", "db/data_cache_db/versions/fe29d0c81a77_add_plpgsql_extension.py", "db.data_cache_db/alembic.pex"], env: {"PATH": "/usr/bin:/bin:/usr/local/bin"}, working_directory: None, input_files: Digest(Fingerprint<c1958cd0d6a590c150c407b4bbc9e07daf312f5f9705986f0ad885de70372182>, 166), output_files: {RelativePath("db.data_cache_db.lexio-migrations-data-cache.tar")}, output_directories: {}, timeout: None, execution_slot_variable: None, description: "Create db.data_cache_db.lexio-migrations-data-cache.tar", level: Debug, append_only_caches: {}, jdk_home: None, target_platform: None, is_nailgunnable: false, cache_scope: Successful }
h
Thanks, what was the full output of running 
/usr/bin/tar caf db.data_cache_db.lexio-migrations-data-cache.tar db/data_cache_db/alembic.ini db/data_cache_db/env.py
directly? (Prob fine to leave off those other files for now)
s
Copy code
Brian-DeSimone-MacBook-Pro---NS0311:talos briandesimone$ /usr/bin/tar caf db.data_cache_db.lexio-migrations-data-cache.tar
Usage:
  List:    tar -tf <archive-filename>
  Extract: tar -xf <archive-filename>
  Create:  tar -cf <archive-filename> [filenames...]
  Help:    tar --help
👀 1
h
Okay, and then the same thing but running with these permutations please instead of `caf`: - 
-caf
- 
cf
- 
-cf
(Trying to confirm it’s a bad argv)
s
yep i see where you are going one sec
Copy code
Brian-DeSimone-MacBook-Pro---NS0311:talos briandesimone$ /usr/bin/tar -caf db.data_cache_db.lexio-migrations-data-cache.tar
Usage:
  List:    tar -tf <archive-filename>
  Extract: tar -xf <archive-filename>
  Create:  tar -cf <archive-filename> [filenames...]
  Help:    tar --help
Brian-DeSimone-MacBook-Pro---NS0311:talos briandesimone$ /usr/bin/tar -cf db.data_cache_db.lexio-migrations-data-cache.tar
tar: no files or directories specified
Brian-DeSimone-MacBook-Pro---NS0311:talos briandesimone$ /usr/bin/tar cf db.data_cache_db.lexio-migrations-data-cache.tar
tar: no files or directories specified
h
excellent, so the 
a
flag must not exist in tar 2.8. That’s an easy fix and we can cherry-pick it
🔥 1
Ahh, we do need to recreate the functionality somehow:
Copy code
-a, --auto-compress
             (c mode only) Use the archive suffix to decide a set of the format and the
             compressions.  As a simple example,
                   tar -a -cf archive.tgz source.c source.h
             creates a new archive with restricted pax format and gzip compression,
                   tar -a -cf archive.tar.bz2.uu source.c source.h
             creates a new archive with restricted pax format and bzip2 compression and
             uuencode compression,
                   tar -a -cf archive.zip source.c source.h
             creates a new archive with zip format,
                   tar -a -jcf archive.tgz source.c source.h
             ignores the ``-j'' option, and creates a new archive with restricted pax for-
             mat and gzip compression,
                   tar -a -jcf <http://archive.xxx|archive.xxx> source.c source.h
             if it is unknown suffix or no suffix, creates a new archive with restricted
             pax format and bzip2 compression.
That’s doable though, the user tells us what they wanted to use
In the meantime to us getting this fix out, you might want to use 
zip
as an archive format
s
i wonder if this is only an issue for me cause im testing on my local. we use centos dockers to actually run this in our production pipelines.
h
depends if your prod has tar 3+ or not, I suspect. But this is enough of a gotcha that it’s worth us fixing - we don’t want to assume you have a particular tar version
s
yeah exactly. i agree with you here
h
3 Views
Open in Slack
PreviousNext
Powered by