Quantcast
Channel: Active questions tagged row - Stack Overflow
Viewing all articles
Browse latest Browse all 479

How to loop almost 1 million rows from the bigquery with python

$
0
0

I am a newbie, I just started a query where I have ~1 million rows on bigquery and it has 25 columns. Rows have the type is RowIterator

I wrote a script in Python to loop them and process data. I used:

client = bigquery.Client()query_job = client.query(query)rows = query_job.result()    (~1 million records)df = rows.to_dataframe()    (*)dict_rows = df.to_dict(orient="records")for row in dict_rows:    # process data

(*) which takes around 5-6 minutes. This is not good for me.

Any suggestions on how can I process it faster?


Viewing all articles
Browse latest Browse all 479

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>