Having some trouble, I guess I don't understand ho...
# general
a
Having some trouble, I guess I don't understand how inferred dependencies work, or I'm doing something wrong. Will include tree and context in a reply, but my pants list . is not inferring all dependencies, but is inferring some. Its not inferring my first party dep, and some third party deps.
Tree: . ├── app │ ├── emu │ ├── healthdash │ ├── BUILD │ ├── Dockerfile │ ├── healthdash │ │ ├── alert_tickets.csv │ │ ├── init.py │ │ ├── NOC_Pod_Overview.py │ │ ├── pages │ │ ├── pycache │ │ ├── shared │ │ └── values.py │ ├── poetry.lock │ ├── pyproject.toml │ ├── README.md │ ├── TESTING.md │ └── tests │ └── init.py ├── dist │ └── app.healthdash ├── find.sh ├── lib │ └── skykick │ ├── BUILD │ ├── doc │ └── openapi-spec │ ├── poetry.lock │ ├── pyproject.toml │ ├── README.md │ ├── skykick │ │ ├── auth.py │ │ ├── backup.py │ │ ├── common.py │ │ ├── exceptions.py │ │ ├── init.py │ │ ├── models.py │ │ └── pycache │ └── tests │ ├── init.py │ ├── pycache │ ├── test_auth.py │ └── test_models.py ├── mypy.ini ├── pants.toml ├── pants.toml.bak ├── poetry.toml ├── pyrightconfig.json ├── README.md ├── requirements.yml ├── ruff.toml └── third-party └── README.md
Sorry I had to edit it a bit so formatting is a bit weird
healthdash BUILD: python_sources( name="healthdash_sources", sources=["healthdash/**/*.py"], ) pex_binary( name="healthdash_app", script="streamlit", args=["run", "healthdash/NOC_Pod_Overview.py"], execution_mode="venv", dependencies=[ ":healthdash_sources", ], ) poetry_requirements(name="poetry") healthdash pyproject.toml: [tool.poetry] name = "healthdash" version = "0.1.0" description = "" authors = [] readme = "README.md" [tool.poetry.dependencies] python = ">=3.10,<4.0" streamlit = ">=1.28.1" httpx = { extras = ["http2"], version = ">=0.25.1" } logicmonitor-sdk = ">=3.0.172" confz = ">=2.0.1" pymssql = ">=2.2.11" altair = "^5.5.0" skykick = { path = "../../lib/skykick/", develop = true } [build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api" pants.toml [GLOBAL] pants_version = "2.23.0" backend_packages = [ "pants.backend.python", "pants.backend.docker", "pants.backend.experimental.python.lint.ruff.check", ] [source] root_patterns = ['app/*', 'lib/*'] _# marker_filenames = ["pyproject.toml"]_ [python] interpreter_constraints = ['>=3.10,<3.12'] _*NOC_POD_OVERVIEW.py (truncated)*_ import logging import re from collections import defaultdict from datetime import datetime, timedelta from http import HTTPStatus from typing import Any, cast from zoneinfo import ZoneInfo import altair as alt import httpx import logicmonitor_sdk as lm import pandas as pd import streamlit as st from healthdash import CACHE_TIME, Conf from healthdash.shared.connectwise.main import query_cw_db from healthdash.shared.logicmonitor.main import ( get_lm_clusterlatency_data, get_lm_clusterstorage_data, get_logicmonitor_all_devices, get_logicmonitor_sdts, get_logicmonitor_suppressed_devices, parse_properties, ) from lib.skykick.skykick import common pants list . output: app/healthdash:healthdash_app app/healthdash:poetry#altair app/healthdash:poetry#confz app/healthdash:poetry#httpx app/healthdash:poetry#logicmonitor-sdk app/healthdash:poetry#pymssql app/healthdash:poetry#streamlit app/healthdash/pyproject.toml:poetry
Initially, I was getting an error about skykick not existing, but at some point when I changed the source roots I think, I now get "file does not exist healthdash/NOC_Pod_Overview.py"
b
Sorry for the trouble. I think you’re on the right track with your mention of source roots, although it’s hard to be 100% sure without any error output. In particular it looks like your imports are
import healthdash…
which would imply that
src
itself should be the root, not
src/healthdash
(via
src/*
). The source roots contain directories that are the libraries/importable objects, they aren’t the importable objects themselves. Can you try changing the source roots to remove the globs?
a
No change to
pants list
with that change unfortunately.
Is it possible I'm having so many issues because everything is structured like
app/app1/app1/__init__.py
vs
app/app1/__init__.py
? Does pants get confused about the dependency tree/is something incorrect in my BUILD/source roots?
I notice also on a blank repository with a similar structure, running
tailor ::
, I get two
BUILD
files, one inside of `app1`(e.g. next to
pyproject.toml
, and one inside of
app1/app1
(e.g. next to my
__init__.py
), though it seems to build fine if I remove the internal BUILD file.
b
I wouldn’t expect
pants list
output to change based on dependency inference. That is listing the individual targets rather than any metadata about them. for getting this working, maybe something like
pants dependencies path/to/broken/file.py
would be better. (I acknowledge your other question/note, but I’m on mobile at the moment so can’t digest/respond yet)
Is it possible I'm having so many issues because everything is structured like
app/app1/app1/__init__.py
vs
app/app1/__init__.py
? Does pants get confused about the dependency tree/is something incorrect in my BUILD/source roots?
Pants should be able to work with this, but you might need your source roots to be
["src", "app/*"]
, i.e.
src/a/b/c.py
is imported as
import a.b.c
and
app/a/b/c.py
is imported as
import b.c
I notice also on a blank repository with a similar structure, running
tailor ::
, I get two
BUILD
files, one inside of `app1`(e.g. next to
pyproject.toml
, and one inside of
app1/app1
(e.g. next to my
__init__.py
), though it seems to build fine if I remove the internal BUILD file.
Yeah, by default Pants will have one BUILD file per directory, with targets just globbing the files within that directory, rather than a "top-level" BUILD file with
**/*.py
-style recursive globs. Both should work okay.
s
@alert-beard-13327 i work in a project where everything is structured like apps/app1/src and apps/app2/src I set apps/* to be the source roots and then there is a flag python-infer.ambiguity_resolution that you can set to “by_source_root” This will make pants prefer any module within the current source root with the same name as one outside of it, solved all my problems with clashes
a
Can you clarify where you're setting that flag? @straight-alligator-6122
Oh nevermind, got it.
Okay, now I'm stuck on this:
Copy code
poetry_requirements(
    name="poetry",
)


pex_binary(
    name="healthdash_app", 
    script="streamlit",
    args=["run", "app/healthdash/healthdash/NOC_Pod_Overview.py"],
    execution_mode="venv", 
    dependencies=[
        ":poetry", "//app/healthdash/healthdash/NOC_Pod_Overview.py",
    ], 
)
My assumption would be that it attempts to run the NOC_Pod_Overview that exists inside the PEX file, but instead I have to run it in a way that the relative path I provide as args exists, meaning, its using the repo version of that file and not the... bundled/packaged version? Which doesn't make sense to me.
b
Happy to help, but I'm not sure what the problem is there. Are you seeing an error?
a
Yes, sorry, when I run the pex, if I don't run it from a directory where the args path is relative to the pex file, it says the NOC_Pod_Overview.py file doesn't exist. e.g. if I run
./dist/app-healthdash/healthdash.pex
from anywhere except the repo root, it cant find the file.
Copy code
.
└── monorepo/
    ├── .github/
    │   └── workflows/
    │       ├── build_app1.yaml
    │       └── build_app2.yaml
    ├── app/
    │   └── healthdash/
    │       ├── healthdash/
    │       │   ├── NOC_Pod_Overview.py
    │       │   └── __init__.py
    │       ├── BUILD
    │       ├── Dockerfile
    │       ├── pyproject.toml
    │       └── poetry.lock
    ├── dist/
    │   └── healthdash-app/
    │       └── healthdash-app.pex
    └── pants.toml
Meaning, lets say I cd into
dist
and run the pex from that directory, it can't find the
.py
file, but if I go up a directory, e.g.
monorepo
, it runs, but its not using the bundled version of that file, its using the actual file that exists in
app/healthdash/healthdash/
b
ah yeah, okay. I see. I'd guess it's because the
streamlit
args are being consumed relative to the process's current working directory, while the files in the PEX will be placed in some "random" venv. That is, the pex isn't run from where its own sources are. I think others have encountered this, but I'm not sure how they solved it (I personally haven't used streamlit). Maybe try digging through a slack search for streamlit?
ah, one more question: how are you planning to deploy/run this?
s
Ive got streamlit working in pex, you just need to point to the app entrypoint in lib/python3.x/site-packages
Youll need to manually add the pages dir python sources if you use it
a
@broad-processor-92400 eventually, in a docker container, but was having this issue initially when building it in so stepped back to get streamlit working on its own
@straight-alligator-6122 can you clarify what you mean by point the entrypoint to site-packages?
b
Ah, okay, if you're deploying in a fixed docker container, a possibility might be sure it is arranged so that it work in the docker container (and when running from the root directory locally), but not necessarily in other circumstances... not great, but maybe unblocks things and can be improved iteratively.