With hostNetworking: true for the controller pods restarting them is difficult with a small number of nodes #563

/kind bug

What happened?
I was trying to make some changes to the IAM Roles that the controller uses and in order for these changes to be picked up the pods had to be restarted. The cluster I was working with had 2 nodes available. When I issued the kubectl rollout restart deployment efs-csi-controller a new pod was spun up that got stuck in a Pending State because it could not bind to a port on either of the 2 nodes available as they were both in use. In order to get things to work I had to scale the deployment down to 0 replicas, at which point both pods were killed so the ports were released and then scale it back up again manually.

What you expected to happen?
A restart would be possible.

How to reproduce it (as minimally and precisely as possible)?
Install the driver into a cluster with 2 or fewer nodes and try and run the command contained above.

Anything else we need to know? :
I have seen this manifest in Helm Chart 2.1.0 (App Version: 1.3.0) so this may be resolved in later versions. This also overlaps with another issue whereby if you have fewer than 2 nodes the driver will fail to install properly unless you specifically vary the replicaCount at the start.

Obviously there are some easy workarounds:

But it feels as though this kind of behaviour is a bug when it comes down to it, the driver should respond to traditional Kubernetes idioms in a way that makes sense. In terms of a solution I've had a look at the various Kubernetes Deployment Update Strategies and there doesn't seem to be anything that makes sense to use. What we really want here is the behaviour to be more like a stateful set where each pod is killed, so it gives up its claim on the ports and then is recreated but modelling the controller as a StatefulSet seems like the wrong move so I'm a bit stumped as to the answer.

Environment

Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.2-eks-0389ca3", GitCommit:"8a4e27b9d88142bbdd21b997b532eb6d493df6d2", GitTreeState:"clean", BuildDate:"2021-07-31T01:34:46Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Driver version: v.1.3.0 (Helm Chart 2.1.0)