Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Is just the max value .The goal is to calculate OVERHEAD as a percentage of real executor memory, as used by RDDs and DataFrames

--executor-memory/spark.executor.memory
controls the executor heap size, but JVMs can also use some memory off heap, for example for interned Strings and direct byte buffers. 
The value of the spark.yarn.executor.memoryOverhead
property is added to the executor memory to determine the full memory request to YARN for each executor. It defaults to max(executorMemory * 0.10, with minimum of 384).
The executors will use a memory allocation based on the property of spark.executor.memoryplus an overhead defined by spark.yarn.executor.memoryOverhead
                Thanks for your reply.I still wondering ,I have applied memory from yarn for my job  then I set the value of 'spark.yarn.executor.memoryOverhead', does it means  it will add extra memory for my job?
– liyong
                Dec 9, 2016 at 10:50
                By default, spark.yarn.am.memoryOverhead is AM memory * 0.10, with a minimum of 384. This means that if we set spark.yarn.am.memory to 777M, the actual AM container size would be 2G. This is because 777+Max(384, 777 * 0.10) = 777+384 = 1161, and the default yarn.scheduler.minimum-allocation-mb=1024, so 2GB container will be allocated to AM.
– Indrajit Swain
                Dec 9, 2016 at 12:21
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.