Afaik you keep the connection open during the complete processing of the current query, that would make perfect sense. But the Spark App may have multiple Impala queries and processing in between not related to IMPALA.

In Spark (3) you have this JDBC parameter:

queryTimeout The number of seconds the driver will wait for a Statement object to execute to the given number of seconds. Zero means there is no limit. In the write path, this option depends on how JDBC drivers implement the API setQueryTimeout, e.g., the h2 JDBC driver checks the timeout of each query instead of an entire JDBC batch. It defaults to 0.

But this is less relevant.

On the server side - impalad, you have have various settings. See https://docs.cloudera.com/documentation/enterprise/latest/topics/impala_timeouts.html#impalad_timeout/

These parameters are either set these to 0 or not for resource pools.

When you provide the jdbcURL for IMPALA via Spark, ensure you request a pool (request pool) with the appropriate settings, so you do not get a timeout. I am not sure if you can do this via Spark via sessionInitStatement (for query level), most likely you can, but you may have different queries with different characteristics.

So, it is more a config issue.

Thanks for contributing an answer to Stack Overflow!

• Please be sure to answer the question . Provide details and share your research!

But avoid …

• Asking for help, clarification, or responding to other answers.
• Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers .