Inserting into partitioned tables in BigQuery

[bogdanstate]

So, generally speaking I am used to using partitions for data management in noSQL tables. For instance, Google BigQuery supports partitions, as does HDFS, etc. I am a bit confused as to what the easiest way to insert into a partition (e.g. via a BigQueryTask) might actually be. Anyone have a minimal example?

[bogdanstate]

Answered my own question. The steps seem to be:

1. CreateBigQueryTable task, specifying a time_partitioning. I note that range partitioning doesn't seem to be supported currently. It would be cool if that were supported though.

2. A downstream BigQueryTask that fulfills the following requirements:

   • destination_table has $[DATEID] appended afterwards, where DATEID is in YYYYMMDD format (e.g. 20200101, not 2020-01-01).
   • query contains a DATE '[DATEID]' AS ds column. Here 'DATEID' is in YYYY-MM-DD format (e.g. 2020-01-01).