Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

What is spark.local.ip ,spark.driver.host,spark.driver.bindAddress and spark.driver.hostname?

the ApplicationMaster connect to spark Driver by spark.driver.host

spark Driver bind to bindAddress on the client machine

by examples

1 example of port binding

.config('spark.driver.port','50243')
then netstat -ano on windows
TCP    172.18.1.194:50243     0.0.0.0:0              LISTENING       15332
TCP    172.18.1.194:50243     172.18.7.122:54451     ESTABLISHED     15332
TCP    172.18.1.194:50243     172.18.7.124:37412     ESTABLISHED     15332
TCP    172.18.1.194:50243     172.18.7.142:41887     ESTABLISHED     15332
TCP    [::]:4040              [::]:0                 LISTENING       15332
The nodes in the cluster 172.18.7.1xx are in the same network as my development machine 172.181.1.194 as my netmask is 255.255.248.0
2 example of specify ip from ApplicationMaster to Driver
.config('spark.driver.host','192.168.132.1')
then netstat -ano
TCP    192.168.132.1:58555    0.0.0.0:0              LISTENING       9480
TCP    192.168.132.1:58641    0.0.0.0:0              LISTENING       9480
TCP    [::]:4040              [::]:0                 LISTENING       9480
however the ApplicationMaster cannot connect and reported error
Caused by: java.net.NoRouteToHostException: No route to host
because this ip is a VM bridge on my development machine
3 example of ip bind
.config('spark.driver.host','172.18.1.194')
.config('spark.driver.bindAddress','192.168.132.1')
then netstat -ano
TCP    172.18.1.194:63937     172.18.7.101:8032      ESTABLISHED     17412
TCP    172.18.1.194:63940     172.18.7.102:9000      ESTABLISHED     17412
TCP    172.18.1.194:63952     172.18.7.121:50010     ESTABLISHED     17412
TCP    192.168.132.1:63923    0.0.0.0:0              LISTENING       17412
TCP    [::]:4040              [::]:0                 LISTENING       17412
Detailed Version
Before explain in detail, there are only these three related conf variables:
spark.driver.host
spark.driver.port
spark.driver.bindAddress
There are NO variables like spark.driver.hostname or spark.local.ip. But there IS a environment variable called SPARK_LOCAL_IP
and before explain the variables, first we have to understand the application submition process
Main Roles of computers:
development machine
master node (YARN / Spark Master)
worker node
There is an ApplicationMaster for each application, which takes care of resource request from cluster and status monitor of jobs(stages)
The ApplicationMaster is in the cluster, always.
Place of spark Driver
development machine: client mode
within the cluster: cluster mode, same place as the ApplicationMaster
Let's say we are talking about client mode
The spark application can be submitted from a development machine, which act as a client machine of the application, as well as a client machine of the cluster.
The spark application can alternatively submitted from a node within the cluster (master node or worker node or just a specific machine with no resource manager role)
The client machine might not be placed within the same subnet as the cluster and this is one case that these variables try to deal with. Think about your internet connection, it is often not possible that your laptop can be accessed from anywhere around the globe just as google.com.
At the beginning of the application submission process, the spark-submit on the client side would upload necessary files to the spark master or yarn, and negotiate about resource requests. In this step the client connect to the cluster, and the cluster address is the destination address that the client tries to connect.
Then the ApplicationMaster starts on the allocated resource.
The resource allocated for ApplicationMaster is by default random, and cannot control by these variables. It is controlled by the scheduler of the cluster, if you're curious about this.
Then the ApplicationMaster tries to connect BACK to the spark Driver. This is the place that these conf variables take effects.
                Why doesn't the cluster (master) tell AppliationMaster where the driver is? The cluster knows the driver's ip when the driver connects to it.
– colinfang
                Sep 30, 2020 at 12:48
                @colinfang your local system cannot be accessed from outside. You need to submit the tasks from a system that is accesible back from the master
– Ja8zyjits
                Dec 17, 2020 at 20:13
                @colinfang by default it(master) knows the client ip. However this answer solves the question where the client ip is mapped by routing OR port is not accessible, and needs port/ip forwarding. Ja8zyjits explains this scenario, where an edge node is used to connect from master to client.
– cdarlint
                Dec 30, 2020 at 2:09
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.