Found the cause of my issues. python_tests seems to behave differently based on where the BUILD file is. https://github.com/pantsbuild/pants/issues/6398