Having a couple issues getting started with builds. Have a test project called casper w/

./src/casper/<pkg, mod, etc.>

structure and

"src/"

pattern for root. When using

./pants run <target>

located in

./src/casper/pkg/subpkg

, I get an module not found error on

import casper.pkg.subpkg

I would have assumed Pants would place

./src

in the

PYTHONPATH

(or perhaps install the package in the hermetic env to make it self-importable)? I don't know if it makes a difference, but the import was in

__init__.py

of

pkg

. The second issue is that when I make any changes to the offending file, it seems Pants does not pick those changes up -- build and run goals keep using the same depended on file.

1st issue: run

./pants roots

to see what Pants guesses for

PYTHONPATH

entries.

but the import was in init.py of pkg

It might be this gotcha about

__init__.py

files: https://www.pantsbuild.org/v2.8/docs/python-backend Have you seen that yet?

@enough-analyst-54434 I get back

src

and

tests

and so

"./src"

should be on the path, thus

import casper.pkg.subpkg

possible? Curiously, when packaging, it gives me

./dist/src.casper.pkg.subpkg/target_module.pex

. Is the

src.

prefix an indication that it only put

./

on the path?

./dist/src.casper.pkg.subpkg/target_module.pex .  Is the src. prefix an indication that it only put ./  on the path?

That part is a red herring. See the

output_path

field https://www.pantsbuild.org/v2.8/docs/reference-pex_binary#codeoutput_pathcode

@hundreds-father-404 Gotcha. Per your suggestion, I enabled the first option, but the second option python.tailor_ignore_solitary_init_files = false appears to only be available on 2.8 (using 2.7.1). Even with the first option enabled and recreating BUILD files and trying to change the init file into a blank, the build/package goals still use the original init file with the bad import. Is pants not invalidating the cache of that file when I make changes?

Yeah - we do not do magic - you have to declare 3rdparty dependencies via requirements or elsewise - we will not guess a version for you.

Yeah no plans to backport, we'll be doing a stable release of 2.8 likely on Monday assuming no bugs are reported before then Taylor to check, are things working now? Anything we can help with?

I found the macro to use requirements.txt, but again stuck in the situation where Pants won't invalidate its caches when I make file changes. I wonder if this has anything to do with my files being checked out in my NFS home directory, but the .cache/ being on the local host in /var/tmp?

I don't think it has to do with separate locations. I'm almost positive it is exactly due to NFS.

Is there any way to “wake it up” (like an event loop)?

discussion on stack overflow points at it being because NFS does not have protocol support for notification events: https://stackoverflow.com/questions/4231243/inotify-with-nfs

Right, but can Pants be instructed to scan/recompute hashes for a given file, dir, project manually? Or maybe it needs a polling feature. For example, PyCharm complains about file syncing because it can't place file watchers on NFS mounts, but we still manage because it periodically polls the project files and you can force resync.

probably needs an opt-in polling feature

In the meantime, kill whatever server processes listening for file changes Pants gins up so it is forced to reexamine the files?

Yeah, which is easier accomplished with

--no-pantsd

. Performance will not be good though.

Does that option disable caching? Is caching and killing more optimal if willing to trade longer init times w/ faster goal completion?

It does not disable caching on-disk, only memoization in-memory. But that memoization is pretty important to avoid doing things like "Creating module mapping" for dependency inference Rather than killing the server, it'd be simpler to use

--no-pantsd

so you don't start the server in the first place It is implemented in Rust. I'm not familiar enough w/ that part of the code to know how tricky it would be, but definitely possible!

@hundreds-father-404 Thanks! I'll add this request to my list of things I'm keeping as we delve into this over the weekend. I think local performance is really important (for developers to run goals before committing), but our CI jobs would always be starting from disk (at best) and so the initialization lag is already going to be a fact of life there. They download the Git workspace and at some point run your custom build logic. The intention is to have Pants already bootstrapped on an NFS drive the build uid can access, so when it starts running goals using the pants script, it will not need to re-bootstrap itself and have access to the cache, which will hopefully be in a state based on the last successful release. Still need to sort out the details though. If you have any examples of people deploying Pants under similar constraints that would be helpful!

I'm slightly confused by your scenario description since it sounds ~opposite of what we've been working through here, where the code repo is on NFS. Here you describe ~/.cache/pants on NFS IIUC. Pants has native support for a remote / shared cache: https://www.pantsbuild.org/docs/remote-caching-execution. The Pants OSS CI jobs dogfood this with a remote cache hosted / donated by Toolchain in fact.