To return expected results, you can:
Reduce the number of search terms.
Each term you use focuses the search further.
Check your spelling.
A single misspelled or incorrectly typed term can change your result.
Try substituting synonyms for your original terms.
For example, instead of searching for "java classes", try "java training"
Did you search for an IBM acquired or sold product ?
If so, follow the appropriate link below to find the content you need.
org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 1 tasks (1477.3 MiB) is bigger than spark.driver.maxResultSize (1024.0 MiB)
Could you tell me how to avoid this issue?
Serialization is the processing of converting data objects into a series of Bytes during transferring across the network. Serialized data is using for
data transfer across executors or also between driver and executor.
The
spark.driver.maxResultSize
parameter in spark session configuration defines the maximum limit of the total size of the serialized result for Spark actions across all partitions. The spark actions include actions such as collect() to the driver node, toPandas(), or saving a large file to the driver local file system.
-
The default value of
spark.driver.maxResultSize
is 1Gb. The minimum value is 1Mb, and the maximum is 0 (unlimited).
-
If the size of the serialized result sent to the driver is greater than its max result size, then the spark job fails with memory exceptions.
-
It is always better to have a proper limit instead of unlimited setting to protect the driver from out-of-memory errors.
[{"Type":"MASTER","Line of Business":{"code":"LOB10","label":"Data and AI"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSHUT6","label":"IBM Watson Studio Premium Cartridge for IBM Cloud Pak for Data"},"ARM Category":[{"code":"a8m3p000000hBziAAE","label":"Analytics Engine"}],"ARM Case Number":"TS011884551","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]
