az acr check-health --name <myregistry> --ignore-errors --yes
If a problem is detected, it provides an error code and description. For more information about the errors and possible solutions, see Health check error reference.
If you get Helm or Notary related errors, it doesn't mean that you have an issue with Container Registry or AKS. It only indicates that Helm or Notary isn't installed, Azure CLI isn't compatible with the current installed version of Helm or Notary, and so on.
To validate whether the container registry is accessible from the AKS cluster, run the following az aks check-acr command:
az aks check-acr --resource-group <MyResourceGroup> --name <MyManagedCluster> --acr <myacr>.azurecr.io
The following sections help you troubleshoot the most common errors that are displayed in Events in the output of the kubectl describe pod command.
Cause 1: 401 Unauthorized error
An AKS cluster requires an identity. This identity can be either a managed identity or a service principal. No matter what the identity is, the proper authorization that's used to pull an image from a container registry is necessary. Otherwise, you may get the following "401 Unauthorized" error:
Failed to pull image "<acrname>.azurecr.io/<repository:tag>": [rpc error: code = Unknown desc = failed to pull and unpack image "<acrname>.azurecr.io/<repository:tag>": failed to resolve reference "<acrname>.azurecr.io/<repository:tag>": failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized
A number of solutions can help you resolve this error, subject to the following constraints:
Solutions 2, 3, and 4 are applicable only to AKS clusters that use a service principal.
Solutions 1, 2, and 3 are applicable for the Azure method of creating the role assignment at Container Registry level for AKS's identity.
Solutions 4 and 5 are applicable for the Kubernetes method of pulling a Kubernetes secret.
Solution 1: Ensure AcrPull role assignment is created for identity
The integration between AKS and Container Registry creates an AcrPull role assignment at container registry level for the AKS cluster's identity. Ensure that the role assignment is created.
To check if the AcrPull role assignment is created, use one of the following ways:
Run the following command:
az role assignment list --scope /subscriptions/<subscriptionID>/resourceGroups/<resourcegroupname>/providers/Microsoft.ContainerRegistry/registries/<acrname> -o table
Check in the Azure portal by selecting Azure Container Registry > Access control (IAM) > Role assignments. For more information, see List Azure role assignments using the Azure portal.
Besides the AcrPull role, some built-in roles and custom roles can also contain the "Microsoft.ContainerRegistry/registries/pull/read" action. Check those roles if you've got any of them.
If the AcrPull role assignment isn't created, create it by configuring Container Registry integration for the AKS cluster with the following command:
az aks update -n <myAKSCluster> -g <myResourceGroup> --attach-acr <acr-resource-id>
Solution 2: Ensure service principal isn't expired
Ensure that the secret of the service principal that's associated with the AKS cluster isn't expired. To check the expiration date of your service principal, run the following commands:
SP_ID=$(az aks show --resource-group <myResourceGroup> --name <myAKSCluster> \
--query servicePrincipalProfile.clientId -o tsv)
az ad sp credential list --id "$SP_ID" --query "[].endDate" -o tsv
For more information, see Check the expiration date of your service principal.
If the secret is expired, update the credentials for the AKS cluster.
Solution 3: Ensure AcrPull role is assigned to correct service principal
In some cases, for example, when the service principal of the AKS cluster is replaced with a new one, the container registry role assignment still refers to the old service principal. To ensure that the container registry role assignment refers to the correct service principal, follow the steps:
To check the service principal that's used by the AKS cluster, run the following command:
az aks show --resource-group <myResourceGroup> \
--name <myAKSCluster> \
--query servicePrincipalProfile.clientId \
--output tsv
To check the service principal that's referenced by the container registry role assignment, run the following command:
az role assignment list --scope /subscriptions/<subscriptionID>/resourceGroups/<resourcegroupname>/providers/Microsoft.ContainerRegistry/registries/<acrname> -o table
Compare the two service principals. If they don't match, integrate the AKS cluster with the container registry again.
Solution 4: Ensure service principal is correct and secret is valid
If you pull an image by using an image pull secret, and that Kubernetes secret was created with the values of a service principal, make sure that the associated service principal is correct and the secret is still valid. Follow these steps:
Run the following kubectl get and base64 command to see the values of the Kubernetes secret:
kubectl get secret <secret-name> --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode
Check the expiration date by running the following az ad sp credential list command. The username is the service principal value.
az ad sp credential list --id "<username>" --query "[].endDate" --output tsv
If necessary, reset the secret of that service principal by running the following az ad sp credential reset command:
az ad sp credential reset --name "$SP_ID" --query password --output tsv
Update or re-create the Kubernetes secret accordingly.
Solution 5: Ensure the Kubernetes secret has the correct values of the container registry admin account
If you pull an image by using an image pull secret, and that Kubernetes secret was created with values of container registry admin account, make sure that the values in the Kubernetes secret are the same as the values of the container registry admin account. Follow these steps:
Run the following kubectl get and base64 command to see the values of the Kubernetes secret:
kubectl get secret <secret-name> --output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode
In the Azure portal, search for and select Container registries.
In the list of container registries, select your container registry.
In the navigation pane for the container registry, select Access keys.
In the Access keys page for the container registry, compare the container registry values with the values in the Kubernetes secret.
If the values don't match, then update or re-create the Kubernetes secret accordingly.
If a Regenerate password operation happened, an operation that's named "Regenerate Container Registry Login Credentials" will be displayed in the Activity log page of the container registry. The Activity log has a 90-day retention period.
Cause 2: Image not found error
Failed to pull image "<acrname>.azurecr.io/<repository:tag>": [rpc error: code = NotFound desc = failed to pull and unpack image "<acrname>.azurecr.io/<repository:tag>": failed to resolve reference "<acrname>.azurecr.io/<repository:tag>": <acrname>.azurecr.io/<repository:tag>: not found
Solution: Ensure image name is correct
If you get this error, ensure that the image name is fully correct. You should check the registry name, registry login server, the repository name, and the tag. A common mistake is that the login server is specified as "azureacr.io" instead of "azurecr.io".
When the image name isn't fully correct, the 401 Unauthorized error may also occur because AKS always tries anonymous pull no matter whether the container registry has enabled anonymous pull access.
Cause 3: 403 Forbidden error
Failed to pull image "<acrname>.azurecr.io/<repository:tag>": rpc error: code = Unknown desc = failed to pull and unpack image "<acrname>.azurecr.io/<repository:tag>": failed to resolve reference "<acrname>.azurecr.io/<repository:tag>": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
Solution 1: Ensure AKS virtual network link is set in the container registry's Private DNS zone
If the network interface of the container registry's private endpoint and the AKS cluster are in different virtual networks, make sure that the virtual network link for the AKS cluster's virtual network is set in the Private DNS zone of the container registry. (That link is named "privatelink.azurecr.io" by default.) If the virtual network link isn't in the Private DNS zone of the container registry, add it by using one of the following ways:
In the Azure portal, select the private DNS zone "privatelink.azurecr.io", select Virtual network links > Add under the Settings panel, and then select a name and the virtual network of the AKS cluster. Select OK.
It's optional to check the "Enable auto registration" feature.
Create a virtual network link to the specified Private DNS zone by using Azure CLI.
Solution 2: Add AKS Load Balancer's public IP address to allowed IP address range of the container registry
If the AKS cluster connects publicly to the container registry (NOT through a private link or an endpoint) and the public network access of the container registry is limited to selected networks, add AKS Load Balancer's public IP address to the allowed IP address range of the container registry:
Confirm that the public network access is limited to selected networks.
In the Azure portal, navigate to the container registry. Under Settings, select Networking. On the Public access tab, Public network access is set to Selected networks or Disabled.
Obtain the AKS Load Balancer's public IP address by using one of the following ways:
In the Azure portal, navigate to the AKS cluster. Under Settings, select Properties, select one of the virtual machine scale sets in the infrastructure resource group, and check the public IP address of the AKS Load Balancer.
Run the following command:
az network public-ip show --resource-group <infrastructure-resource-group> --name <public-IP-name> --query ipAddress -o tsv
Allow access from the AKS Load Balancer's public IP address by using one of the following ways:
Run az acr network-rule add command as follows:
az acr network-rule add --name acrname --ip-address <AKS-load-balancer-public-IP-address>
For more information, see Add network rule to registry.
In the Azure portal, navigate to the container registry. Under Settings, select Networking. On the Public access tab, under Firewall, add the AKS Load Balancer's public IP address to Address range and then select Save. For more information, see Access from selected public network - portal.
If Public network access is set to Disabled, switch it to Selected networks first.
Cause 4: 443 timeout error
Failed to pull image "<acrname>.azurecr.io/<repository:tag>": rpc error: code = Unknown desc = failed to pull and unpack image "<acrname>.azurecr.io/<repository:tag>": failed to resolve reference "<acrname>.azurecr.io/<repository:tag>": failed to do request: Head "https://<acrname>.azurecr.io/v2/<repository>/manifests/v1": dial tcp <acrprivateipaddress>:443: i/o timeout
The "443 timeout" error only occurs when you connect privately to a container registry by using Azure Private Link.
Solution 1: Ensure virtual network peering is used
If the network interface of the container registry's private endpoint and the AKS cluster are in different virtual networks, ensure that virtual network peering is used for both virtual networks. You can check virtual network peering by running the Azure CLI command az network vnet peering list --resource-group <MyResourceGroup> --vnet-name <MyVirtualNetwork> --output table or in the Azure portal by selecting the VNETs > Peerings under the Settings panel. For more information about listing all peerings of a specified virtual network, see az network vnet peering list.
If the virtual network peering is used for both virtual networks, ensure that the status is "Connected". If the status is Disconnected, delete the peering from both virtual networks, and then re-create it. If the status is "Connected", see the troubleshooting guide: The peering status is "Connected".
For further troubleshooting, connect to one of the AKS nodes or pods, and then test the connectivity with the container registry at TCP level by using the Telnet or Netcat utility. Check the IP address with the nslookup <acrname>.azurecr.io command, and then run the telnet <ip-address-of-the-container-registry> 443 command.
For more information about connecting to AKS nodes, see Connect with SSH to Azure Kubernetes Service (AKS) cluster nodes for maintenance or troubleshooting.
Solution 2: Use Azure Firewall Service
If the network interface of the container registry's private endpoint and the AKS cluster are in different virtual networks, in addition to virtual network peering, you may use Azure Firewall Service to set up a Hub-spoke network topology in Azure. When you set up the firewall rule, you need to use network rules to explicitly allow the outbound connection to the container registry private endpoint IP addresses.
The host operating system (node OS) is incompatible with the image that's used for the pod or container. For example, if you schedule a pod to run a Linux container on a Windows node, or a Windows container on a Linux node, the following error occurs:
Failed to pull image "<acrname>.azurecr.io/<repository:tag>":
rpc error:
code = NotFound
desc = failed to pull and unpack image "<acrname>.azurecr.io/<repository:tag>": no match for platform in manifest: not found,
This error can occur for an image that's pulled from any source, as long as the image is incompatible with the host OS. The error isn't limited to images that are pulled from the container registry.
Specify the correct nodeSelector field in the configuration settings of your pod or deployment. The correct value for this field's kubernetes.io/os setting ensures that the pod will be scheduled on the correct type of node. The following table shows how to set the kubernetes.io/os setting in YAML:
Container type
YAML setting
For example, the following YAML code describes a pod that needs to be scheduled on a Linux node:
apiVersion: v1
kind: Pod
metadata:
name: aspnetapp
labels:
app: aspnetapp
spec:
containers:
- image: "mcr.microsoft.com/dotnet/core/samples:aspnetapp"
name: aspnetapp-image
ports:
- containerPort: 80
protocol: TCP
nodeSelector:
"kubernetes.io/os": linux
If the troubleshooting guidance in this article doesn't help you resolve the issue, here are some other things to consider:
Check the network security groups and route tables associated with subnets, if you've got any of those items.
If a virtual appliance like a firewall controls the traffic between subnets, check the firewall and Firewall access rules.
Third-party information disclaimer
The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.
If you have questions or need help, create a support request, or ask Azure community support. You can also submit product feedback to Azure community support.
