gorgeous-winter-99296
02/13/2023, 2:00 PMconsole_scripts
to work in a pex_binary
that I'm not sure how to solve. Essentially I've got a package that has two entry_points;
* server
* worker
server is the main entrypoint, and spawns workers when requested via GRPC. I can set the server as the main entrypoint in a pex-binary, and that works fine. However; the console_script for the worker is never available. I've also tried this: https://github.com/pantsbuild/pants/discussions/17308, but while it shows the entrypoints they aren't on the path. I would expect this to work if I could install the packages into the pex; but it doesn't quite work either due to how the install happens.
Our source layout is
src/
py/
a/
pyproject.toml
README.md
a/
..
b/
pyproject.toml
README.md
b/
...
With py/a
, py/b
being individual source roots (marker_filenames = ["pyproject.toml"]
). However; when depending on a:package
and b:package
the pex cannot be built due to the README.md colliding.
Exception: Can only merge Directories with no duplicates, but found 2 duplicate entries in :
`README.md`: 1.) file digest=d31d7752e98a52694aaa502b3ce9e49ef02aeb2459a4dcad81eba6456567d077 size=855:
...
`README.md`: 2.) file digest=964e4cf6af0fa0dfe069e7cd383792d591eb94ad56d41bd16aad4f72f36d6816 size=3501:
...
Which makes sense. But I don't think there's anything wrong with our setup either. Any thoughts?enough-analyst-54434
02/13/2023, 2:08 PMpython_distribution
just to get console script entry point metadata, you can write down entry point metadata: https://github.com/pantsbuild/scie-pants/blob/main/tools/BUILD
If you also publish a wheel for some reason this will probably be unappealing since it is non DRY.gorgeous-winter-99296
02/13/2023, 2:11 PMenough-analyst-54434
02/13/2023, 2:12 PMpex_binary
target and describe the target type of each of it's dependencies?python_distribution
in mine.python_distribution
target.gorgeous-winter-99296
02/13/2023, 2:16 PMpex_binary(
name="testing",
entry_point="server.py",
dependencies=[
"//src/py:mock-dist-info",
"//src/py/server-plugins:package",
]
)
enough-analyst-54434
02/13/2023, 2:17 PMpython_distribution
targets. Are they?gorgeous-winter-99296
02/13/2023, 2:17 PMenough-analyst-54434
02/13/2023, 2:17 PMgorgeous-winter-99296
02/13/2023, 2:18 PMb
above, unrelated to server/worker (both in a
)enough-analyst-54434
02/13/2023, 2:18 PMexecution_mode="venv"
pex_binary
1st and foremost.gorgeous-winter-99296
02/13/2023, 2:19 PMenough-analyst-54434
02/13/2023, 2:20 PMpython_resources
target to own it. 0 python_distribution
targets in sight.gorgeous-winter-99296
02/13/2023, 2:24 PMenough-analyst-54434
02/13/2023, 2:24 PMsys.executable -mthis
gorgeous-winter-99296
02/13/2023, 2:26 PMpex_binary(
name="dummy",
entry_point="erupt.entry_points.server:main",
dependencies=[
"//src/py:mock-dist-info",
"//src/py/erupt-gym:package",
],
execution_mode="venv",
)
and
✦5 ❯ cat src/py/anyname.dist-info/entry_points.txt
[console_scripts]
erupt-runner = erupt.entry_points.runner:main
erupt-server = erupt.entry_points.server:main
But subprocess fails with
FileNotFoundError: [Errno 2] No such file or directory: 'erupt-runner'
enough-analyst-54434
02/13/2023, 2:30 PMgorgeous-winter-99296
02/13/2023, 2:34 PManyname.dist-info
folder nor the erupt
are in that site-packages directory, nor can I find the binaries in the bin directory. Is there a difference between 2.14 and 2.15 here?enough-analyst-54434
02/13/2023, 2:35 PMgorgeous-winter-99296
02/13/2023, 2:35 PMpants run cmd:dummy -- some_args
enough-analyst-54434
02/13/2023, 2:37 PMgorgeous-winter-99296
02/13/2023, 2:38 PM$ cat src/py/BUILD
resources(name="mock-dist-info", sources=[
"erupt.dist-info/**/*",
"erupt.dist-info/*"
])
I changed the name from anyname.dist-info
to erupt.dist-info
just to ensure it wasn't some weird thing happening there with it not belonging to a package.enough-analyst-54434
02/13/2023, 2:40 PMgorgeous-winter-99296
02/13/2023, 2:41 PMresources(name="package_data", sources=["pyproject.toml"])
python_distribution(
name="package",
dependencies=[
":package_data",
"./erupt_gym:erupt_gym",
],
provides=python_artifact(
name="erupt-gym",
version="0.1.0",
long_description_content_type="markdown",
),
entry_points={
"erupt.engines": {"Gym": "erupt_gym.engine:GymEngine"},
},
long_description_path="src/py/erupt/README.md",
)
enough-analyst-54434
02/13/2023, 2:42 PMgorgeous-winter-99296
02/13/2023, 2:43 PMenough-analyst-54434
02/13/2023, 2:44 PMgorgeous-winter-99296
02/13/2023, 2:45 PMerupt
), runner
and server
.enough-analyst-54434
02/13/2023, 2:46 PMgorgeous-winter-99296
02/13/2023, 2:50 PMenough-analyst-54434
02/13/2023, 2:51 PMgorgeous-winter-99296
02/13/2023, 2:53 PMenough-analyst-54434
02/13/2023, 2:53 PMpex_binary
target, it should now only have dependencies on just the dist info resource and the server / runner code. Is that right?gorgeous-winter-99296
02/13/2023, 3:00 PMpex_binary(
name="dummy",
entry_point="erupt.entry_points.server:main",
dependencies=[
"//src/py:mock-dist-info",
"//src/py/erupt/erupt/entry_points/runner.py"
],
execution_mode="venv",
)
enough-analyst-54434
02/13/2023, 3:02 PMOk, that's fine. It's more complicated to debug ...I will say that, if we get this working, it is definitely alot of ceremony, files and indirection to avoid
sys.executable -m ...
which any new hire with Python experience could read, grok, and find what is being invoked.gorgeous-winter-99296
02/13/2023, 3:06 PMenough-analyst-54434
02/13/2023, 3:07 PM./pants run ...
and instead just ./pants package ... && ./dist/my.pex
- I'm pulling in the complication removal card.gorgeous-winter-99296
02/13/2023, 3:10 PMenough-analyst-54434
02/13/2023, 3:11 PMgorgeous-winter-99296
02/13/2023, 3:13 PMenough-analyst-54434
02/13/2023, 3:15 PMgorgeous-winter-99296
02/13/2023, 3:15 PMenough-analyst-54434
02/13/2023, 3:18 PMgorgeous-winter-99296
02/13/2023, 3:20 PMenough-analyst-54434
02/13/2023, 3:20 PMgorgeous-winter-99296
02/13/2023, 3:22 PMsys.executable
does work I can just use that, right? Should side-step the whole thing. And console_scripts would still work for installing packaged wheels.enough-analyst-54434
02/13/2023, 3:23 PMgorgeous-winter-99296
02/13/2023, 3:27 PMenough-analyst-54434
02/13/2023, 3:28 PMusage: pex-tools PATH venv [-h] [--scope {all,deps,srcs}] [-b {false,prepend,append}] [-f] [--collisions-ok] [-p] [--copies] [--compile] [--prompt PROMPT] [--rm {pex,all}]
[--non-hermetic-scripts] [--emit-warnings] [--pex-root PEX_ROOT] [--disable-cache] [--cache-dir CACHE_DIR] [--tmpdir TMPDIR] [--rcfile RC_FILE]
PATH
Creates a venv from the PEX file.
positional arguments:
PATH The directory to create the virtual environment in.
options:
...
--collisions-ok Don't error if population of the venv encounters distributions in the PEX file with colliding files, just emit a warning. (default: False)
./pants package ... && PEX_TOOLS=1 ./dist/my.pex venv --collisions-ok right/here && ./right/here/pex
gorgeous-winter-99296
02/13/2023, 3:33 PMenough-analyst-54434
02/13/2023, 3:33 PMgorgeous-winter-99296
02/13/2023, 3:39 PMpants package
show the README being included.enough-analyst-54434
02/13/2023, 3:40 PM./pants run ...
?gorgeous-winter-99296
02/13/2023, 3:41 PMenough-analyst-54434
02/13/2023, 3:42 PM./pants run ...
and `./pants package ... && ./my.pex`should behave the same. Any deviation is alomst certainly a bug.run
machinery in the last few releases and I've lost track a bit of what works like what when.gorgeous-winter-99296
02/13/2023, 3:44 PMenough-analyst-54434
02/13/2023, 3:44 PMgorgeous-winter-99296
02/13/2023, 3:49 PM❯ pants package src::
10:52:56.64 [INFO] Wrote dist/a-project-0.1.0.tar.gz
10:52:56.64 [INFO] Wrote dist/a_project-0.1.0-py3-none-any.whl
10:52:56.64 [INFO] Wrote dist/b-project-0.1.0.tar.gz
10:52:56.64 [INFO] Wrote dist/b_project-0.1.0-py3-none-any.whl
❯ zipinfo dist/a_project-0.1.0-py3-none-any.whl
Archive: dist/a_project-0.1.0-py3-none-any.whl
Zip file size: 1206 bytes, number of entries: 5
-rw-r--r-- 2.0 unx 9 b- defN 16-Jan-01 00:00 a/__init__.py
-rw-r--r-- 2.0 unx 33 b- defN 16-Jan-01 00:00 a/aardvark.py
?rw------- 2.0 unx 87 b- defN 16-Jan-01 00:00 a_project-0.1.0.dist-info/WHEEL
?rw------- 2.0 unx 224 b- defN 16-Jan-01 00:00 a_project-0.1.0.dist-info/METADATA
?rw------- 2.0 unx 346 b- defN 16-Jan-01 00:00 a_project-0.1.0.dist-info/RECORD
5 files, 699 bytes uncompressed, 558 bytes compressed: 20.2%
❯ zipinfo dist/b_project-0.1.0-py3-none-any.whl
Archive: dist/b_project-0.1.0-py3-none-any.whl
Zip file size: 1203 bytes, number of entries: 5
-rw-r--r-- 2.0 unx 9 b- defN 16-Jan-01 00:00 b/__init__.py
-rw-r--r-- 2.0 unx 33 b- defN 16-Jan-01 00:00 b/bowling.py
?rw------- 2.0 unx 87 b- defN 16-Jan-01 00:00 b_project-0.1.0.dist-info/WHEEL
?rw------- 2.0 unx 224 b- defN 16-Jan-01 00:00 b_project-0.1.0.dist-info/METADATA
?rw------- 2.0 unx 345 b- defN 16-Jan-01 00:00 b_project-0.1.0.dist-info/RECORD
5 files, 698 bytes uncompressed, 557 bytes compressed: 20.2%
❯ pants package cmd::
10:53:15.75 [ERROR] 1 Exception encountered:
Engine traceback:
in `package` goal
Exception: Can only merge Directories with no duplicates, but found 2 duplicate entries in :
`README.md`: 1.) file digest=96d7f2c61ad10f92e449136b94433c482ad88ef52f8d9513dd1426c4fb3ea17e size=44:
# B's README
Dummy placeholder lorem ipsum
`README.md`: 2.) file digest=028373478b36dc388b32c231aacb2962b6530618c38ae32af9aad1e7ace29323 size=44:
# A's README
Dummy placeholder lorem ipsum
enough-analyst-54434
02/14/2023, 10:05 AMgorgeous-winter-99296
02/14/2023, 10:07 AMpackaging the pex and running the pex behave the same (README duplication)So we never get that far.
enough-analyst-54434
02/14/2023, 10:07 AM./pants run
vs ./pants package && ./dist/my.pex
gorgeous-winter-99296
02/14/2023, 10:11 AMenough-analyst-54434
02/14/2023, 10:16 AMgorgeous-winter-99296
02/14/2023, 10:25 AMenough-analyst-54434
02/14/2023, 10:27 AMgorgeous-winter-99296
02/14/2023, 10:33 AMenough-analyst-54434
02/14/2023, 5:07 PMgorgeous-winter-99296
02/14/2023, 5:28 PMenough-analyst-54434
02/14/2023, 5:45 PMpython_distribution
target when depending on one of these in some other target?python_distribution
targets - which do not make it in the wheel built from it - do make it into a LocalDists PEX usages in leaf rules and thus verbs acting on the upstream targets depending on the python_distribution
. This tanks the engine with 2 different digests for same file - README.gorgeous-winter-99296
02/15/2023, 8:02 PMenough-analyst-54434
02/15/2023, 8:32 PMgorgeous-winter-99296
02/15/2023, 8:54 PMenough-analyst-54434
02/15/2023, 8:59 PMhappy-kitchen-89482
02/16/2023, 1:36 AMpython_library
A depends on python_distribution
B, we take that to mean "build the wheel for B and have A consume that". But - when we compute the transitive dependency closure of A, we naively follow the edge to B the same as we would through any other target. So now we have a single big pile of files in the closure, and we subtract out the ones that are in B's wheel, because we know B provides them. We can't subtract out any other files we reached via B because A might need them (via some other dep path).
So I think what we really should be doing is not following python_distribution
B's deps during the dep graph traverse. We should treat everything behind it as provided by it. So if some file is still in the closure we know we reached it some other way, but not by traversing through B.enough-analyst-54434
02/16/2023, 1:39 AMhappy-kitchen-89482
02/16/2023, 1:48 AMenough-analyst-54434
02/16/2023, 2:07 AMhappy-kitchen-89482
02/16/2023, 6:42 PM