You run a GKE cluster serving multiple teams. The ML team needs Nvidia Tesla P100 GPUs for model training with minimal effort and cost. What should you do?

Adding a GPU-enabled node pool to the existing GKE cluster is the most efficient and cost-effective solution. This allows the ML team to utilize GPUs without impacting other teams or incurring the overhead of managing a separate cluster. The nodeSelector ensures their pods are scheduled only on nodes with the required P100 GPUs. Asking the ML team to add an annotation is insufficient as it doesn't provision the necessary hardware. Recreating every node is disruptive, expensive, and unnecessary since only the ML team needs GPUs. Creating a separate cluster for the ML team adds management overhead and increases costs compared to integrating into the existing cluster.