Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

how to use BigQueryIO.read using SerializableFunction() instead of BigQueryIO.readTableRows()

How to use BigQueryIO.read using SerializableFunction() instead of BigQueryIO.readTableRows() ? As I've heard that readTableRows is time consuming.

Our objective is to build dataflow to execute bigquery select query and write results to bigquery and GCS.To achieve that, I have used BigQueryIO.readTableRows() to read data, but I've heard it is slow so what to use BigQueryIO.read using SerializableFunction() to retrieve complete row. Could you please help me?

// to read data from bigquery  
p.apply(
  BigQueryIO.readTableRows()
    .fromQuery("myquery")
    .usingStandardSql()
    .withoutValidation()
    .withQueryLocation("EU")
// to write result set to bigquery 
  .apply(
    BigQueryIO.writeTableRows()
      .to(tableSpec)
      .withJsonSchema(tableSchemaJson)
      .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
      .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
How to achieve same thing using BigQueryIO.read using SerializableFunction()?
                Have you found readTableRows to be slow? Where have you heard that it's slow? I think it must be acceptable - but do you have evidence to the contrary?
– Pablo
                Sep 24, 2019 at 21:05
                apache beam java tutorial documents says below statement:      readTableRows returns a PCollection of BigQuery TableRow objects. Each element in the PCollection represents a single row in the table. Integer values in the TableRow objects are encoded as strings to match BigQuery’s exported JSON format. This method is convenient, but can be 2-3 times slower in performance compared to read(SerializableFunction).
– sivateja
                Oct 11, 2019 at 7:54
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.