Hello, I was having some issues last week regardin...
# general
p
Hello, I was having some issues last week regarding a specific test case that fails despite it being ran/implemented the same way as my other tests. After further inspection it looks like it doesn’t pass on my machine (Mac M1) where as in our build pipeline it passes (x86). I’m using the correct scie_pants binaries for their respective platforms as well. In addition, our build pipeline experiences a seperate issue where we receive
No source for code:
for a file thats never been an issue before. This is all with the intended goal of going from 2.15 -> 2.19.
👀 1
b
Hi, sorry for the trouble. I don't think we'll be able to help you much without some more details. 1. Can you construct a reduced reproducer that you can share publicly? 2. If not, can you share your
pants.toml
, the relevant
BUILD
files, and even the test contents? 3. Following https://www.pantsbuild.org/2.19/docs/using-pants/troubleshooting-common-issues#debug-tip-inspect-the-sandbox-with---keep-sandboxes to use
--keep-sandboxes=on_failure
to explore what Pants is "actually" running is often a good start for debugging these problems. 4.
No source for code
doesn't seem to appear in the Pants repository, so we're definitely going to need some more info to know what's going on there.
p
Thanks for the response! I’ll start with the
pants.toml
that exists for both local and cicd environments
Copy code
[GLOBAL]
pants_version = "2.19.0"
backend_packages = [
  'pants.backend.python',
  'pants.backend.python.lint.black',
  'pants.backend.python.lint.flake8',
  'pants.backend.python.typecheck.mypy',
]
pantsd_invalidation_globs.add = ["**/requirements.txt"]
# use_deprecated_python_macros = false

[source]
marker_filenames = ["SOURCE_ROOT"]

[python]
interpreter_constraints = ["CPython==3.11.8"]
enable_resolves = true
#default_resolve = "myresolve"

# [python.resolves]
# myresolve = "3rdparty/myresolve_lockfile.txt"

[python-bootstrap]
# search_path = ["<PYENV>", "/opt/python3"]
search_path = ["<PATH>", "<PYENV>"]

[test]
# Note - Pants strips external env vars, so any additional values must be added here, globally, or per test. <https://www.pantsbuild.org/docs/python-test-goal#setting-environment-variables>
# When boto3 is used in tests, it reads the local config file first. This doesn't work in CI. 
# To make it consistent for local and CI, disable it here by pointing to non-existent file and set the region to a default.
extra_env_vars = ["AWS_DEFAULT_REGION=us-east-1", "AWS_CONFIG_FILE=non-existent.txt"]

[pytest]
args = ["--no-header"]


[anonymous-telemetry]
enabled = false

[mypy]
args = "--namespace-packages --explicit-package-bases"
# extra_type_stubs = ["types-requests==2.30.0.0", "types-chardet==5.0.4.6", "types-urllib3==1.26.25.13", "boto3-stubs==1.26.135"]

[python-infer]
# 2.17 is transitioning to a new, faster parser for dependency inference:
use_rust_parser = true
The commented out portions are items we had from <=2.15.
For my local issues, here is the build file:
Copy code
python_sources()

python_tests(
    name="tests",
    dependencies=[
        ":publisher_pkg",
        "//:reqs#ddtrace",
        "//:reqs#synapse",
        "//:reqs#mammoth",
        "//:reqs#boto3",
        "//:reqs#python-json-logger",
    ],
    sources=[
        "test_*.py"
    ]
)

files(
    name="publisher_pkg",
    sources=[
        "./publisher.yml",
        "./storm/commands/*",
        "./storm/modules/*,",
        "./test_data/*",
    ],
)
And the folder structure
Screenshot 2024-03-21 at 9.24.56 AM.png
Along with the path with respect to project root directory:
services/storm-services/publisher/service-code/publisher/test_service.py
So after saving the sandbox, I noticed that the test_data that gets copied down didn’t have the same system permissions that exist in the original code. I changed the permissions and was able to get it to pass, is there anything that effects that when pants generates these files?
Here’s some additional output showing before changing permissions manually in the sandbox and after
sers/david.patrick/.cache/pants/named_caches/pex_root/venvs/s/3a87d9ce/venv/lib/python3.11/site-packages/synapse/common.py405 PermissionError ------------------------------------------------------------------------------------------ generated xml file: /private/var/folders/b1/dns6z6cd7m1dsbxmbhjbqhhh0000gn/T/pants-sandbox-ywTaBX/services.storm-services.publisher.service-code.publisher.test_service.py.tests.xml ------------------------------------------------------------------------------------------ ======================================================================================================================================================================== short test summary info ========================================================================================================================================================================= ERROR services/storm-services/publisher/service-code/publisher/test_service.py:TestPublisher:test_help_exists - PermissionError: [Errno 13] Permission denied: ‘/private/var/folders/b1/dns6z6cd7m1dsbxmbhjbqhhh0000gn/T/pants-sandbox-ywTaBX/services/storm-services/publisher/service-code/publisher/test_data/testdoc.docx’ ERROR services/storm-services/publisher/service-code/publisher/test_service.py:TestPublisher:test_convert_to_markdown - PermissionError: [Errno 13] Permission denied: ‘/private/var/folders/b1/dns6z6cd7m1dsbxmbhjbqhhh0000gn/T/pants-sandbox-ywTaBX/services/storm-services/publisher/service-code/publisher/test_data/testdoc.docx’ ERROR services/storm-services/publisher/service-code/publisher/test_service.py:TestPublisher:test_extracts_metadata - PermissionError: [Errno 13] Permission denied: ‘/private/var/folders/b1/dns6z6cd7m1dsbxmbhjbqhhh0000gn/T/pants-sandbox-ywTaBX/services/storm-services/publisher/service-code/publisher/test_data/testdoc.docx’ ERROR services/storm-services/publisher/service-code/publisher/test_service.py:TestPublisher:test_poc_upload - PermissionError: [Errno 13] Permission denied: ‘/private/var/folders/b1/dns6z6cd7m1dsbxmbhjbqhhh0000gn/T/pants-sandbox-ywTaBX/services/storm-services/publisher/service-code/publisher/test_data/testdoc.docx’ =========================================================================================================================================================================== 4 errors in 10.38s =========================================================================================================================================================================== david.patrick@AMAC0V4KH2272 pants-sandbox-ywTaBX % chmod 665 services/storm-services/publisher/service-code/publisher/test_data/testdoc.docx david.patrick@AMAC0V4KH2272 pants-sandbox-ywTaBX % ./__run.sh ========================================================================================================================================================================== test session starts =========================================================================================================================================================================== collected 4 items services/storm-services/publisher/service-code/publisher/test_service.py .... [100%] ------------------------------------------------------------------------------------------ generated xml file: /private/var/folders/b1/dns6z6cd7m1dsbxmbhjbqhhh0000gn/T/pants-sandbox-ywTaBX/services.storm-services.publisher.service-code.publisher.test_service.py.tests.xml ------------------------------------------------------------------------------------------ =========================================================================================================================================================================== 4 passed in 17.91s =========================================================================================================================================================================== david.patrick@AMAC0V4KH2272 pants-sandbox-ywTaBX %
b
Does the permission in the sandbox differ to the permissions of the files in you repo?
p
Yep, full permissions in the repo
And all permissions in the same among the other microservices that pants test services
Only 1/14 produce the error above
b
Hm, weird. If you can narrow this down to a self-contained reproducer, that'd help a lot
q
Any updates on this? I have run into a similar issue where I get the
Permission denied
error for reading/writing files from/to disk in the sandbox directory.