I am a newbie, I just started a query where I have ~1 million rows on bigquery and it has 25 columns. Rows have the type is RowIterator
I wrote a script in Python to loop them and process data. I used:
client = bigquery.Client()query_job = client.query(query)rows = query_job.result() (~1 million records)df = rows.to_dataframe() (*)dict_rows = df.to_dict(orient="records")for row in dict_rows: # process data
(*) which takes around 5-6 minutes. This is not good for me.
Any suggestions on how can I process it faster?