polite-garden-5064110/21/2021, 1:22 AM
config in pyproject.toml? this is what I see:
worried-salesclerk-3783410/21/2021, 2:10 AM
fresh-cat-9082710/21/2021, 6:40 AM
inside some artifact (say a Docker image or an archive)? If there are 5 scripts, each depending on
, each script == one entry point == one
, I get now 5
files, each containing the same
wheel binaries. I’ve looked carefully at the docs, and PEX can only have a single entry point. And say I don’t want to have 5 Docker images / 5 archives, each with a single PEX packaged.
modern-wolf-3622810/21/2021, 9:54 AM
kind-alligator-3848610/21/2021, 4:04 PM
kind-alligator-3848610/21/2021, 4:05 PM
kind-alligator-3848610/21/2021, 4:06 PM
pantsbuild-sample ➤ ./pants --version git:master* 12:05:36.26 [INFO] Initializing scheduler... 12:05:36.34 [INFO] Scheduler initialized. 12:05:36.36 [WARN] Please either set `enabled = true` in the [anonymous-telemetry] section of pants.toml to enable sending anonymous stats to the Pants project to aid development, or set `enabled = false` to disable it. No telemetry sent for this run. An explicit setting will get rid of this message. See <https://www.pantsbuild.org/v2.7/docs/anonymous-telemetry> for details. 2.7.0
kind-alligator-3848610/21/2021, 4:07 PM
flat-zoo-3195210/21/2021, 5:37 PM
This whole thing fits very naturally as a graph of dependencies. We could model it this way and execute nodes in the graph as their dependencies are satisfied, and if nodes fail we can retry N times before marking them completely failed.
I built a little test executor for this using networkx and it looked like it could work pretty well. It basically worked like:
- build graph, eg base image depends on container, provider image depends on base image, provider upload depends on provider image etc.
Then, to execute:
- gather all stages that are runnable (dependencies are satisfied)
- submit to threadpool to run
- walk the graph again and find any stages that are now runnable and submit to threadpool to run
- continue until we can't progress any more or everything is doneIf only he knew there was tooling that already basically did this, but in a little more refined way. 😀 Looks like I've got another person I can make a good case for Pants to.
busy-vase-3920210/21/2021, 6:05 PM
hundreds-father-40410/21/2021, 7:15 PM
loud-laptop-1794910/21/2021, 11:49 PM
curved-television-656810/22/2021, 5:40 AM
? I think they’re similar, but haven’t used
. Am curious about if there’s any major pros/cons between them?
stale-nightfall-2980110/22/2021, 8:16 AM
stale-nightfall-2980110/22/2021, 9:17 AM
is working fine (it's tests run and it packages from what I can tell) and it's BUILD file looks like this:
python_library( name = "core", sources = ["core/*.py"], ) python_tests( name="test", sources = ["test/test_*.py"], dependencies=["core"], ) python_distribution( name="dist", dependencies=["core"], setup_py_commands=["bdist_wheel", "sdist"], provides=setup_py( name='core', version="1.0.0", description='blah, blah, blah', ), )
which depends on core can not find core as per this error:
________________ ERROR collecting foo/test/test_cherrypy.py _________________ ImportError while importing test module '/tmp/process-executionLlK5rO/foo/test/test_cherrypy.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /home/james/.pyenv/versions/3.8.12/lib/python3.8/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) foo/test/test_cherrypy.py:11: in <module> from core.test_utils import token E ModuleNotFoundError: No module named 'core.test_utils'
's BUILD looks like this:
My expectation is that the
python_library( name = "foo", sources = ["foo/jwtutils.py", "foo/jwthug.py", "foo/jwtcherry.py"], dependencies = ["core:core"] ) python_tests( name="test", sources = ["test/test_*.py"], dependencies=[ "core:core", "thirdparty/python:cherrypy" ], ) python_distribution( name="dist", dependencies=["foo"], setup_py_commands=["bdist_wheel", "sdist"], provides=setup_py( name='foo', version="1.0.0", description='foo library', ), )
library available to foo, is this assumption incorrect?
curved-television-656810/22/2021, 9:26 AM
curved-television-656810/22/2021, 9:27 AM
tall-truck-6785910/22/2021, 11:17 AM
I build Classify like this :
12:41:45.93 [ERROR] Completed: Run Pytest - src/python/classify/tests/test_get_model.py failed (exit code 4). ImportError while loading conftest '/tmp/process-executiondZwPlt/src/python/classify/tests/conftest.py'. src/python/classify/tests/conftest.py:17: in <module> from tests.utils.email import get_test_email E ModuleNotFoundError: No module named 'tests'
I use this command to run tests :
cat <<EOT >src/python/classify/BUILD python_sources( dependencies=[ "src/python/classify/main.py", "//:python-multipart", "//:scikit-learn" ] ) python_tests( name="tests", sources=["tests/*.py"], dependencies=[ ":classify", "//:pg8000" ] ) python_distribution( name="wheel", dependencies=[":classify"], provides=setup_py( name="classify", version="0.0.0", ), wheel=True, ) pex_binary( name="binary", entry_point="__main__.py", dependencies=["//:pg8000"], )
So far I tried everything I thought that could work and I need some help. Thanks !
- image: recital/classify custom: - command: ./pants test src/python/classify/tests:tests
stale-nightfall-2980110/22/2021, 12:12 PM
are folders side-by-side and
tests depend on a
curved-television-656810/22/2021, 12:39 PM
tall-truck-6785910/22/2021, 12:45 PM
enough-analyst-5443410/22/2021, 1:52 PM
Pants needs to know
from core.test_utils import token E ModuleNotFoundError: No module named 'core.test_utils'
in the filesystem path
. Pants call
a "source root". This is equivalent to a
entry for Python. The list of "source roots" Pants knows about can be found by running
is not in the list printed out, you need to teach Pants about
as described here: https://www.pantsbuild.org/docs/source-roots#configuring-source-roots
stale-nightfall-2980110/22/2021, 2:53 PM
if I'm in
and I do
is that correctly telling me that it's
and it should successfully be able to find
from each other ... OR should
be showing me
. core foo
enough-analyst-5443410/22/2021, 3:04 PM
, that says the file
will be importable as
stale-nightfall-2980110/22/2021, 3:10 PM
would be at
which is not what I want ... the "outer"
holds the REAME, dockerfile, etc ...
is the "deployable" code
enough-analyst-5443410/22/2021, 3:27 PM
happy-kitchen-8948210/22/2021, 3:49 PM
parallel source tree is officially recommended. In the Pants repo, for example, we mostly have tests right alongside the code they test (so
is tested by
. Pants allows you to do this because it has the information to know not to bundle tests in the distribution, and so on. We like it because it's really easy to find the tests for some module, and vice versa.
happy-kitchen-8948210/22/2021, 3:49 PM
stale-nightfall-2980110/22/2021, 4:03 PM
file at the root of each library/project folder.