spark-submit --conf spark.driver.memory=18g
Very important note, this property will not be taken into consideration if you set it from code, according to Spark Documentation - Dynamically Loading Spark Properties:
Spark properties mainly can be divided into two kinds: one is related to deploy, like “spark.driver.memory”, “spark.executor.instances”, this kind of properties may not be affected when setting programmatically through SparkConf in runtime, or the behavior is depending on which cluster manager and deploy mode you choose, so it would be suggested to set through configuration file or spark-submit command line options; another is mainly related to Spark runtime control, like “spark.task.maxFailures”, this kind of properties can be set in either way.
Broadly speaking, spark Executor JVM memory can be divided into two parts. Spark memory and User memory. This is controlled by property spark.memory.fraction - the value is between 0 and 1.
When working with images or doing memory intensive processing in spark applications, consider decreasing the spark.memory.fraction. This will make more memory available to your application work. Spark can spill, so it will still work with less memory share.
The second part of the problem is division of work. If possible, partition your data into smaller chunks. Smaller data possibly needs less memory. But if that is not possible, you are sacrifice compute for memory. Typically a single executor will be running multiple cores. Total memory of executors must be enough to handle memory requirements of all concurrent tasks. If increasing executor memory is not a option, you can decrease the cores per executor so that each task gets more memory to work with.
Test with 1 core executors which have largest possible memory you can give and then keep increasing cores until you find the best core count.
The location to set the memory heap size (at least in spark-1.0.0) is in conf/spark-env.
The relevant variables are SPARK_EXECUTOR_MEMORY & SPARK_DRIVER_MEMORY.
More docs are in the deployment guide
Also, don't forget to copy the configuration file to all the slave nodes.
–
–
Did you dump your master gc log? So I met similar issue and I found SPARK_DRIVER_MEMORY only set the Xmx heap. The initial heap size remains 1G and the heap size never scale up to the Xmx heap.
Passing "--conf "spark.driver.extraJavaOptions=-Xms20g" resolves my issue.
ps aux | grep java and the you'll see the follow log:=
24501 30.7 1.7 41782944 2318184 pts/0 Sl+ 18:49 0:33 /usr/java/latest/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Xmx30g -Xms20g
I have few suggession for the above mentioned error.
● Check executor memory assigned as an executor might have to deal with partitions requiring more memory than what is assigned.
● Try to see if more shuffles are live as shuffles are expensive operations since they involve disk I/O, data serialization, and network I/O
● Use Broadcast Joins
● Avoid using groupByKeys and try to replace with ReduceByKey
● Avoid using huge Java Objects wherever shuffling happens
From my understanding of the code provided above, it loads the file and does map operation and saves it back. There is no operation that requires shuffle. Also, there is no operation that requires data to be brought to the driver hence tuning anything related to shuffle or driver may have no impact. The driver does have issues when there are too many tasks but this was only till spark 2.0.2 version. There can be two things which are going wrong.
There are only one or a few executors. Increase the number of executors so that they can be allocated to different slaves. If you are using yarn need to change num-executors config or if you are using spark standalone then need to tune num cores per executor and spark max cores conf. In standalone num executors = max cores / cores per executor .
The number of partitions are very few or maybe only one. So if this is low even if we have multi-cores,multi executors it will not be of much help as parallelization is dependent on the number of partitions. So increase the partitions by doing imageBundleRDD.repartition(11)
Simple if you are using a script or juyter notebook then set only config path when you start build a spark session...
spark = SparkSession.builder.master('local[*]').config("spark.driver.memory", "15g").appName('testing').getOrCreate()
heap space errors generally occur due to either bringing too much data back to the driver or the executor.
In your code it does not seem like you are bringing anything back to the driver, but instead you maybe overloading the executors that are mapping an input record/row to another using the threeDReconstruction() method. I am not sure what is in the method definition but that is definitely causing this overloading of the executor.
Now you have 2 options,
edit your code to do the 3-D reconstruction in a more efficient manner.
do no edit code, but give more memory to your executors, as well as give more memory-overhead. [spark.executor.memory or spark.driver.memoryOverhead]
I would advise being careful with the increase and use only as much as you need. Each job is unique in terms of its memory requirements, so I would advise empirically trying different values increasing every time by a power of 2 (256M,512M,1G .. and so on)
You will arrive at a value for the executor memory that will work. Try re-running the job with this value 3 or 5 times before settling for this configuration.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
