By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

We have a kubernetes service that points to an external service ip. External service IP is the ec2 instance internal IP address where the mysql service is running at 3306

NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                       AGE
mysql                ExternalName   <none>           172.31.21.116   3306/TCP                      20m
  • When using CORE DNS, the external IP service do not resolve
  • / # nslookup mysql.letmetry
    Server:    100.64.0.10
    Address 1: 100.64.0.10 kube-dns.kube-system.svc.cluster.local
    nslookup: can't resolve 'mysql.letmetry'
    
  • When using KUBE DNS, the external IP service resolves
  • / # nslookup mysql.letmetry.svc.cluster.local.
    Server:    100.64.0.10
    Address 1: 100.64.0.10 kube-dns.kube-system.svc.cluster.local
    Name:      mysql.letmetry.svc.cluster.local.
    Address 1: 172.31.21.116 ip-172-31-21-116.ap-south-1.compute.internal
    

    Please suggest why this is happening.
    The logs of CORE DNS when this request is made -

    100.96.109.30:41248 - [19/Nov/2018:14:56:25 +0000] 4 "AAAA IN mysql.letmetry.default.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 110.391µs
    100.96.109.30:42034 - [19/Nov/2018:14:56:25 +0000] 5 "AAAA IN mysql.letmetry.svc.cluster.local. udp 50 false 512" NXDOMAIN qr,rd,ra 143 91.647µs
    100.96.109.30:41174 - [19/Nov/2018:14:56:25 +0000] 6 "AAAA IN mysql.letmetry.cluster.local. udp 46 false 512" NXDOMAIN qr,rd,ra 139 93.808µs
    100.96.109.30:34985 - [19/Nov/2018:14:56:25 +0000] 7 "AAAA IN mysql.letmetry.ap-south-1.compute.internal. udp 60 false 512" NXDOMAIN qr,rd,ra 60 1.458123ms
    100.96.109.30:45287 - [19/Nov/2018:14:56:25 +0000] 8 "A IN mysql.letmetry. udp 32 false 512" NXDOMAIN qr,rd,ra 108 102.671µs
    100.96.109.30:36956 - [19/Nov/2018:14:56:25 +0000] 9 "A IN mysql.letmetry.default.svc.cluster.local. udp 58 false 512" NXDOMAIN qr,rd,ra 151 91.6µs
    100.96.109.30:51749 - [19/Nov/2018:14:56:25 +0000] 10 "A IN mysql.letmetry.svc.cluster.local. udp 50 false 512" NXDOMAIN qr,rd,ra 143 92.572µs
    100.96.109.30:54959 - [19/Nov/2018:14:56:25 +0000] 11 "A IN mysql.letmetry.cluster.local. udp 46 false 512" NXDOMAIN qr,rd,ra 139 87.971µs
    100.96.109.30:42924 - [19/Nov/2018:14:56:25 +0000] 12 "A IN mysql.letmetry.ap-south-1.compute.internal. udp 60 false 512" NXDOMAIN qr,rd,ra 60 1.420079ms
    

    System info

  • It is a default installation of Core DNS with kops. No custom configuration was made. We just tried to replace kube dns with core dns using kops and are stuck here.
  • image: gcr.io/google_containers/coredns:1.1.3
  • kubernetes version: 1.9.8
  • kops version: 1.10.0
  • CoreFile
  • Corefile: |-
        .:53 {
            errors
            health
            kubernetes cluster.local. in-addr.arpa ip6.arpa {
              pods insecure
              upstream
              fallthrough in-addr.arpa ip6.arpa
            prometheus :9153
            proxy . /etc/resolv.conf
            cache 30
            reload
    
    / # nslookup mysql.letmetry
    Server:    100.64.0.10
    Address 1: 100.64.0.10 kube-dns.kube-system.svc.cluster.local
    Name:      mysql.letmetry
    Address 1: 172.31.21.116 ip-172-31-21-116.ap-south-1.compute.internal
    
  • Output of kubectl -n letmetry get service mysql -o yaml
  • $ kubectl -n letmetry get service mysql -o yaml
    apiVersion: v1
    kind: Service
    metadata:
      creationTimestamp: 2018-11-19T16:42:39Z
      name: mysql
      namespace: letmetry
      resourceVersion: "105997025"
      selfLink: /api/v1/namespaces/letmetry/services/mysql
      uid: 205f30f2-ec1a-11e8-8bba-02ebb8238b60
    spec:
      externalName: 172.31.21.116
      ports:
      - port: 3306
        protocol: TCP
        targetPort: 3306
      sessionAffinity: None
      type: ExternalName
    status:
      loadBalancer: {}
    
  • Core dns does not work with FQDN also
  • OK, the issue is, I believe, that the external name field is supposed to hold a DNS name, not an IP. For example, per the Cluster DNS spec, this field is expected to be a DNS name because it must be used to construct a CNAME record (which can only contain a DNS name target).

    If you want to point a cluster service to an external IP, you can do so with a headless service and static endpoint.

    Also, if it works for kube-dns then it should work for core-dns as well to maintain the backward compatibility.

    Perhaps. Although strictly speaking, kube-dns is violating the Cluster DNS Spec in behaving that way.

    IMO, it's not intended that the ExternalName field should be able to contain an IP address. All documentation I can find assumes it will be a name, not an IP. The API should probably not allow IPs in that field.

    Diving into the k8s API validation code, it looks like the Service validation is letting IPv4s slip through because it thinks they are domain names (after all, an IPV4 address is a series of valid dns labels delimited by dots). For example, looking at the code, an IPv6 address would fail validation because they contain colons. In other words, the API appears to be accepting an IPv4 address as a domain name, not as an IP. This is the regex the API uses ... [a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)* (used in IsDNS1123Subdomain() which is used when validating the externalName field for ExternalName services).

    Got into this "issue" from the ingress-nginx controller perspective - version 0.18.0 or later outputs the following line:

    2018/11/19 15:53:42 [error] 110379#110379: *24398057 [lua] dns.lua:61: resolve(): server returned error code: 3: name error, context: ngx.timer

    This happens when any externalName is an IP, and it tries to query the nameserver with the IP as an A record.

    Both CoreDNS and ingress-nginx are working according to spec, and this detail should be in the official Kubernetes docs for clarity.

    Got into this "issue" from the ingress-nginx controller perspective - version 0.18.0 or later outputs the following line:

    2018/11/19 15:53:42 [error] 110379#110379: *24398057 [lua] dns.lua:61: resolve(): server returned error code: 3: name error, context: ngx.timer

    This happens when any externalName is an IP, and it tries to query the nameserver with the IP as an A record.

    Both CoreDNS and ingress-nginx are working according to spec, and this detail should be in the official Kubernetes docs for clarity.

    @nikopen - We are facing the same issue. We created the k8s cluster with Kubeadm and deployed nginx-ingress through helm chart. Can you please let me know how you fixed this issue?

    @nvenky this happens when you have some Services with externalName as an IP instead of a hostname, as mentioned above.

    Turn the IPs into hostnames to fix this.

    I too am using externalname services with internal IP addresses.

    In Kubernetes, IP addresses are not supposed to be allowed in the in ExternalName services. Kubernetes only permits FQDNs, as such an IP would be interpreted as a FQDN not an IP, resulting in unexpected behavior. IMO Kubernetes field validation could be improved here, by screening out numeric TLDs.

    @sylr This is the Kubernetes spec and CoreDNS follows it as intended. Docs have been equally updated to make this detail visible.

    If you want to push for this change, a good place for that would be SIG-Network in Kubernetes. If accepted and implemented there, only then I assume CoreDNS would propagate the change.