rough-minister-58256
01/05/2021, 9:53 PMwitty-crayon-22786
01/05/2021, 9:59 PM--venv
execution mode for pex… the initial usecase is for lower latency execution rather than for export, but export is something for the medium term to better support IDEs/notebooksrough-minister-58256
01/05/2021, 10:04 PM!pip install some.pypi.dep==1.0 some.monorepo.target==2.0
and then mutate at will with seamless transitive dep compatibility (i.e. export).witty-crayon-22786
01/05/2021, 10:05 PMrough-minister-58256
01/05/2021, 10:06 PMwitty-crayon-22786
01/05/2021, 10:07 PMrough-minister-58256
01/05/2021, 10:07 PM!pip install monorepo.src.python.coolthing
-> with materialization unrolling inner source and 3rdparty deps.witty-crayon-22786
01/05/2021, 10:08 PMrough-minister-58256
01/05/2021, 10:09 PMwitty-crayon-22786
01/05/2021, 10:11 PMrough-minister-58256
01/05/2021, 10:12 PMwitty-crayon-22786
01/05/2021, 10:13 PMrough-minister-58256
01/05/2021, 10:13 PMwitty-crayon-22786
01/05/2021, 10:15 PMrough-minister-58256
01/05/2021, 10:15 PMwitty-crayon-22786
01/05/2021, 10:16 PMrough-minister-58256
01/05/2021, 10:17 PMprint(module.__file__)
-> indicates src sha.witty-crayon-22786
01/05/2021, 10:22 PMpip
is part of the requirements, that’s one thing… if it’s not, i’d still maybe say that leaning in on using the remote execution CAS api as a glorified, persistent rsync would work fairly well (or just rsync)rough-minister-58256
01/05/2021, 10:22 PM0.0.0.devN#<sha>
versioning scheme where N monotonically increments + inexact version matching on the pip side (iirc, local version identifiers can be glob matched as e.g. .*
).pip
here is fundamental.pip
needs to manage the remainder of the virtual env (composed of both pypi packages + monorepo sourced packages) in a consistent way.pip
invocations may change the env (mutable) and/or want to seek “newer” versions of the monorepo src packages to resolve conflicts (e.g. if a 3p version is bumped in the monorepo and wants to be realized in the venv).witty-crayon-22786
01/05/2021, 10:31 PMrough-minister-58256
01/05/2021, 10:36 PMwitty-crayon-22786
01/05/2021, 10:37 PMrough-minister-58256
01/05/2021, 10:38 PMwitty-crayon-22786
01/05/2021, 10:39 PMa->b->c
, if c
changes, need to publish new copies of b
and c
rough-minister-58256
01/05/2021, 10:40 PMwitty-crayon-22786
01/05/2021, 10:41 PMb
and c
need to be refetched into the workspace, even if they only changed transitively)rough-minister-58256
01/05/2021, 10:41 PMwitty-crayon-22786
01/05/2021, 10:45 PMrough-minister-58256
01/05/2021, 10:46 PMwitty-crayon-22786
01/05/2021, 10:46 PMrough-minister-58256
01/05/2021, 10:47 PMtags = {'pip.transitive'}
witty-crayon-22786
01/05/2021, 10:49 PMrough-minister-58256
01/05/2021, 11:26 PMenough-analyst-54434
01/05/2021, 11:27 PMrough-minister-58256
01/05/2021, 11:28 PMwitty-crayon-22786
01/05/2021, 11:29 PMenough-analyst-54434
01/05/2021, 11:29 PMrough-minister-58256
01/05/2021, 11:30 PMpip install -e ./dist/xxx
?enough-analyst-54434
01/05/2021, 11:37 PM"“vanilla jupyter notebook on GCP” having trivial reach to the monorepo"You'll have to excuse me, but this is where you lose me. Maybe you could write down some command lines a rando like me could run that give the current crappy solution and then a description of what you'd like to happen? I have no clue what a GC or any of the other stuff is.
rough-minister-58256
01/05/2021, 11:40 PM!pip install pex
<cmd-enter>
import pex; print(pex.__version__)
<cmd-enter>
then the goal state is: !pip install monorepo.src.python.a.b
with similar results for a mono-repo sourced python_library
vs pypi package.enough-analyst-54434
01/05/2021, 11:41 PMrough-minister-58256
01/05/2021, 11:46 PM!pip install
additional 3p/monorepo libraries iteratively and have them stay consistent)enough-analyst-54434
01/05/2021, 11:47 PMpip install -e monorepo/src/python/... seem like the two obvious ways.
rough-minister-58256
01/05/2021, 11:48 PMenough-analyst-54434
01/05/2021, 11:48 PMwitty-crayon-22786
01/05/2021, 11:49 PMenough-analyst-54434
01/05/2021, 11:49 PMwitty-crayon-22786
01/05/2021, 11:51 PMenough-analyst-54434
01/05/2021, 11:51 PMwitty-crayon-22786
01/05/2021, 11:51 PMrough-minister-58256
01/05/2021, 11:51 PMenough-analyst-54434
01/05/2021, 11:53 PMrough-minister-58256
01/05/2021, 11:54 PMpython_library(…, deps=['src/python/x/y']…)
export as requires=['monorepo.src.python.x.y==X']
enough-analyst-54434
01/05/2021, 11:55 PMrough-minister-58256
01/05/2021, 11:55 PMenough-analyst-54434
01/05/2021, 11:56 PMwitty-crayon-22786
01/05/2021, 11:57 PMrough-minister-58256
01/05/2021, 11:57 PMwitty-crayon-22786
01/05/2021, 11:57 PMenough-analyst-54434
01/05/2021, 11:57 PMwitty-crayon-22786
01/05/2021, 11:58 PMrough-minister-58256
01/05/2021, 11:58 PMenough-analyst-54434
01/05/2021, 11:59 PMrough-minister-58256
01/06/2021, 12:00 AMenough-analyst-54434
01/06/2021, 12:01 AMrough-minister-58256
01/06/2021, 12:02 AMwitty-crayon-22786
01/06/2021, 12:05 AMenough-analyst-54434
01/06/2021, 12:06 AMrough-minister-58256
01/06/2021, 12:07 AMenough-analyst-54434
01/06/2021, 12:08 AMrough-minister-58256
01/06/2021, 12:10 AM