You want to securely automate nightly jobs that read non-public batch files from Cloud Storage, run a Spark Scala job on Dataproc, and write results to BigQuery. How should the workload run securely?

The correct approach is to run the job under a service account with the principle of least privilege. This means granting only the necessary permissions: read access to Cloud Storage for input files and write access to BigQuery for results. This minimizes the potential blast radius if the service account were ever compromised. Locking down the Cloud Storage bucket to only your user account is impractical for automated jobs and doesn't allow the Dataproc cluster to access the data. Granting the Project Owner role to a service account is a security anti-pattern as it provides excessive permissions, violating the principle of least privilege. Using a user account with Project Viewer on the Dataproc cluster is insufficient, as Project Viewer does not grant read/write access to Cloud Storage or BigQuery, and user accounts are not ideal for automated workloads.