Code currently inserts many small rows into BigQuery. Which change will improve throughput?

Including multiple rows per insert request, also known as batching, significantly improves BigQuery insert throughput. BigQuery is optimized for large-scale data operations, and each API call incurs overhead. By sending multiple rows in a single request, you reduce the number of API calls and the associated overhead, making the process more efficient. Performing inserts in parallel using multiple threads can help, but if each thread is still inserting single rows, the fundamental overhead per row remains high. Batching within each thread would be more effective. Writing each row to Cloud Storage individually, then loading, introduces significant latency and overhead due to numerous small file operations and the two-step process. Similarly, writing each row to Cloud Storage in parallel still suffers from the overhead of many small files and the extra loading step, which is less efficient than direct batched inserts into BigQuery for streaming data.