@DanijelMalik , Thanks for your feedback. We are looking into this query and will update you as soon as possible.

@DanijelMalik , Could please provide us with the powershell output for this error. Which look similar to this,
Note: Please make sure to mask confidential areas like subscription ID or secrets.
{
"status": "Failed",
"error": {
"code": "ApplicationGatewayKeyVaultSecretException",
"message": "Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions/XXXXXXXXXXXXXXXX/resourceGroups/XXXXXXXXXXXX/providers/Microsoft.Network/applicationGateways/applicationGatewayV2'. See details below:",
"details": [
{
"code": "ApplicationGatewayKeyVaultSecretAccessDenied",
"message": "Access denied for KeyVault Secret 'XXXXXXXXXXXXXXXXX' for Application Gateway '/subscriptions/XXXXXXXXXXX/resourceGroups/XXXXXXXXXXXXX/providers/Microsoft.Network/applicationGateways/applicationGatewayV2'. Make sure that Identity assigned to Application Gateway has access to the KeyVault associated with secret."
}
]
}
}

@DanijelMalik , Just checking whether you got some time to view the previous response.

@DanijelMalik, Since we have not heard back from you we will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly reopen the issue and continue the discussion.

Same issue here.

For me this was resolved by enabling soft deletion on the key vault.

There is an existing issue about this requirement not being documented: #34382

Hi, I'm experiencing the same issue. What seems to be the problem is that although the keyvault allows access from the appGw subnet (via the Firewalls and virtual networks settings blade in the keyvault configuration); and the appGw subnet is being input to New-AzApplicationGateway via the -GatewayIpConfigurations argument the appGw do not seem to be able to access the keyvault (the identity has access rights in the keyvault access policies as well).

When I allow access to the keyvault from "all networks" it seems to work fine.

Perhaps in the appGw creation process the keyvault is being accessed before the appGw is configured with the subnet and the keyvault then sees this access from somewhere else than the subnet?

I'm running the powershell script locally but my IP has access to the key vault as well so that should not be the problem.

I see the same issue

@SubhashVasarapu-MSFT

I have the same issue. I'm using Azure CLI 2.0.76. My application gateway and key vault are in different resource groups in the same subscription. The key vault has soft delete enabled, can be accesses from all networks and has an access policy for the application gateway's assigned user assigned identity with the get secrets permission.

az network application-gateway ssl-cert create -g XXX --gateway-name XXX --name XXX --key-vault-secret-id https://XXX.vault.azure.net/secrets/XXX --debug

results in (abbreviated)

msrest.http_logger : {
  "status": "Failed",
  "error": {
    "code": "ApplicationGatewayKeyVaultSecretException",
    "message": "Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX'. See details below:",
    "details": [
      {
        "code": "ApplicationGatewayKeyVaultSecretAccessDenied",
        "message": "Access denied for KeyVault Secret 'https://XXX.vault.azure.net/secrets/XXX' for Application Gateway '/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX'. Make sure that Identity assigned to Application Gateway has access to the KeyVault associated with secret."
      }
    ]
  }
}
msrest.exceptions : Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX'. See details below:
cli.azure.cli.core.util : Deployment failed. Correlation ID: 623f5539-b652-49fc-9a29-326bcadaa055. Access denied for KeyVault Secret 'https://XXX.vault.azure.net/secrets/XXX' for Application Gateway '/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX'. Make sure that Identity assigned to Application Gateway has access to the KeyVault associated with secret.
Deployment failed. Correlation ID: 623f5539-b652-49fc-9a29-326bcadaa055. Access denied for KeyVault Secret 'https://XXX.vault.azure.net/secrets/XXX' for Application Gateway '/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX'. Make sure that Identity assigned to Application Gateway has access to the KeyVault associated with secret.

If I edit the application gateway HTTP listener in Azure Portal and try to get the SSL certificate from the key vault there, I get this error:

Failed to save configuration changes to application gateway 'XXX'. Error: Invalid value for the identities 'XXX/providers/Microsoft.ManagedIdentity/userAssignedIdentities/XXX'. The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property.

Hi, I'm experiencing the same issue. What seems to be the problem is that although the keyvault allows access from the appGw subnet (via the Firewalls and virtual networks settings blade in the keyvault configuration); and the appGw subnet is being input to New-AzApplicationGateway via the -GatewayIpConfigurations argument the appGw do not seem to be able to access the keyvault (the identity has access rights in the keyvault access policies as well).

When I allow access to the keyvault from "all networks" it seems to work fine.

Perhaps in the appGw creation process the keyvault is being accessed before the appGw is configured with the subnet and the keyvault then sees this access from somewhere else than the subnet?

I'm running the powershell script locally but my IP has access to the key vault as well so that should not be the problem.

This worked for me too. Only change I made from failure to success was to allow all networks

If I edit the application gateway HTTP listener in Azure Portal and try to get the SSL certificate from the key vault there, I get this error:

Failed to save configuration changes to application gateway 'XXX'. Error: Invalid value for the identities 'XXX/providers/Microsoft.ManagedIdentity/userAssignedIdentities/XXX'. The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property.

@wolfganggallo I'm getting this exact same error. My azure portal seems to be stuck in an error state now as well and does not let me make any changes. Did you manage to fix this?

Att: @SubhashVasarapu-MSFT -- please reopen.

I"m getting the same issue: "The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property", and my -- keyvault in a different resource group and vnet. I think in my case the vnets are not peered, so there is no route between the APIM instance and the keyvault at runtime, but the Azure Portal UI will still list the keyvault as available to use, and allow it to be linked to the listener. I am going to create a temporary vnet peering to see if it resolves the issue so that I may remove the listener. My Appgw is currently broken as a result:

UPDATE Yes, that's the problem. The Azure Portal will show you key vaults that are not actually available at runtime, and saving the listener's settings will break your Appgw. The quick fix is to go to the keyvault and temporarily open it to "all networks / internet" then re-save your listener. What you do then is up to you - copy the cert to a local keyvault, or add a vnet peering, or add the missing subnet. Ultimately the portal should validate the network accessibility between appgw and the kyvault. Bug.

2nd UPDATE Just to be clear: This completely BREAKS the Portal UI. AppGw CANNOT be fixed over the portal. Subsequent saves fail.

I just hit this error as well in Portal UI. Soft delete is enabled. Allowing all networks work-around worked.

Please reopen, being able to use the whitelist is good security.

@SubhashVasarapu-MSFT Please reopen. We are coping with a similar same issue. Portal UI is completely broken.

Every operations fails with the following message:
Set-AzApplicationGateway : Either Data or KeyVaultSecretId must be specified for Certificate '/subscriptions/********-****-****-****-************/resourceGroups/**-***-ApplicationGateway-RG/providers/Microsoft.Network/ applicationGateways/**-***-Shared-WAF/sslCertificates/wildcard2022' in Application Gateway.

@PgInsight @oising Did you only need to set the "Allow All Networks" to get the AppGw working again? We have a problem with the same symptoms, but we have the KeyVault in the same subnet, possibly a different root cause.

I cannot do anything using UI, PowerShell, CLI or Resource Explorer.

@PgInsight @oising Did you only need to set the "Allow All Networks" to get the AppGw working again? We have a problem with the same symptoms, but we have the KeyVault in the same subnet, possibly a different root cause.

I cannot do anything using UI, PowerShell, CLI or Resource Explorer.

Setting all networks Allow on the KeyVault and we also needed to set the Soft Delete flag on the Key Vault using PowerShell cmd.

($resource = Get-AzureRmResource -ResourceId (Get-AzureRmKeyVault -VaultName "YOUR-VAULT-NAME").ResourceId).Properties | Add-Member -MemberType "NoteProperty" -Name "enableSoftDelete" -Value "true"

Set-AzureRmResource -resourceid $resource.ResourceId -Properties $resource.Properties

I had the same issue. However I was able to solve issue by removing the Key Vault certificate using PowerShell and not resource explorer using explorer showed the below error:
{
"error": {
"code": "MissingIdentityIds",
"message": "The identity ids must not be null or empty for 'UserAssigned' identity type."
}
}

used the below sample script to remove certificate and listener and the app gateway went back into working state
#delete listener and cert from app gw
$AppGw = Get-AzApplicationGateway -Name "app-gw-ssl-key" -ResourceGroupName "lab"
Remove-AzApplicationGatewayHttpListener -ApplicationGateway $AppGw -Name "https"
Remove-AzApplicationGatewaySslCertificate -ApplicationGateway $AppGW -Name "victor-cer"
#save changes
set-azapplicationgateway -ApplicationGateway $AppGW

when the app gateway was out of failed state, I checked the access policy of the key vault and saw that the identity of the app gw was in another category that's not "APPLICATION". I guess this was the cause of the issue.
Added the identity again to the access policy from the key vault setting and it was able to show as "APPLICATION". This solved the issue for me and I was able to add listener with cert from key vault without issues.

@SubhashVasarapu-MSFT This needs to be reopened. The App Gateway should never end up in a disabled/broken state like this -- I've hit the same issue again. How can this be escalated? This is not a documentation issue. This is a product quality issue.

To summarize:

When configuring a listener to use an SSL certificate in a remote keyvault, the portal UI will let you configure the listener with a keyvault that is unavailable at runtime to the appgw due to network restrictions (not open to internet, selected subnets only). After you do this, the App Gateway portal interface is BROKEN for all listeners.

I will be taking this forward to the respective team.

@dfendit @Patrik-Berglund can you two provide some details on your setups/goal and the steps you take to hit the problem?

@mscatyao The note makes it seem like you could do the following:

• Deny access to all networks in KeyVault
• Allow Application Gateway to bypass this rule via allowing trusted services to access the KeyVault

In other words: https://docs.microsoft.com/en-us/azure/application-gateway/media/key-vault-certs/key-vault-firewall.png I can just check d) and not bother with adding the application gateway via b) and c).

However, according to my tests and Microsoft support, this is not correct:

• Application gateway is not a trusted service - see here for a list which does not contain it: https://docs.microsoft.com/en-us/azure/key-vault/general/overview-vnet-service-endpoints#trusted-services
• You will need to follow all steps shown here https://docs.microsoft.com/en-us/azure/application-gateway/media/key-vault-certs/key-vault-firewall.png, in particular the gateway subnet must be allowed in the firewall (why I still need to allow trusted services is beyond me, but fine).

So there seems to be some improvement over the old status quo where we had to allow all networks, but the note https://user-images.githubusercontent.com/14315890/112624676-3abc7080-8e2e-11eb-8645-342bd48b8408.png is at best misleading, if not wrong.

Furthermore, when trying to deploy an Application Gateway without networking access to the KeyVault, it ends up in a failed state with a generic error message that something went wrong. This could also be improved.

I've been struggling with this issue for two days straight.

The one difference I've found (not I have not seen mentioned here) is that when I use keyVaultSecretId from the arm template provisioning the app gateway, and the key vault and app gateway are in the same resource group, it works.

Here is a snippet from my app gateway ARM code.

{
    "type": "Microsoft.Network/applicationGateways",
    ...
    "identity":{
        "type": "UserAssigned",
          "userAssignedIdentities": {
            "[variables('managedIdentity')]": {}
          }
    },
    ...
    "sslCertificates": [
          {
            "name": "[variables('appGatewayCertName')]",
            "properties": {            
              "keyVaultSecretId": "[variables('keyVaultSecretId')]"
            }
          }
    ],
}

I'm assigning a managed identity (already set up/provisioned) to the app gateway. The managed identity has Get and List permissions for the key vault. I'm then using the sslCertificates array to reach out to the key vault using keyVaultSecretId (https://[KeyVaultUri]/secrets/[CertName]) to obtain the certificate.

However, when I try to provision the app gateway in a different resource group than the key vault, it fails with this error message:

Status Message: At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details. (Code: DeploymentFailed)
 - {
  "status": "Failed",
  "error": {
    "code": "ResourceDeploymentFailure",
    "message": "The resource operation completed with terminal provisioning state 'Failed'.",
    "details": [
      {
        "code": "ApplicationGatewayKeyVaultSecretException",
        "message": "Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions/<subscription>/resourceGroups/<resourceGroup>/providers/Microsoft.Network/applicationGateways/<appGateway>'. See details below:",
        "details": [
          {
            "code": "ApplicationGatewaySslCertificateInvalidData",
            "message": "Data or Password for certificate /subscriptions/<subscription>/resourceGroups/<resourceGroup>/providers/Microsoft.Network/applicationGateways/<appGateway>/sslCertificates/<certName> is invalid."
          }
        ]
      }
    ]
  }
} (Code:Conflict)

The Key Vault's Networking is set to "All Network's", it has soft delete enabled, and the Days to retain deleted vaults is set to 90. the App Gateway and Key Vault are in the same region (Central US).

I am running the deployment as a user who has the "Owner" role, so I should have the appropriate permissions to deploy anything across the entire subscription.

I've tried following the rather convoluted new guidance on how this should be done here and it does not work. More specifically, this section

Where it claims: "The values of appGatewaySSLCertificateData and appGatewaySSLCertificatePassword are looked up from the key vault as described in the section Reference secrets with dynamic ID."

I tried to call out to the key vault using template parameters:

"appGatewaySSLCertificateData":{
            "reference": {
                "keyVault": {
                    "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroup>/providers/Microsoft.KeyVault/vaults/<keyVaultName>"
                },
                "secretName": "<secretName>"
            }
}

And change my sslCertficates array in my ARM code with this code:

    "sslCertificates": [
        {
        "name": "[variables('appGatewayCertName')]",
        "properties": {            
        "data": "[parameters('appGatewaySSLCertificateData')]",
        "password": "<certPassword>"
        }
        }
],

And it did not work.

@mscatyao The note makes it seem like you could do the following:

• Deny access to all networks in KeyVault
• Allow Application Gateway to bypass this rule via allowing trusted services to access the KeyVault

In other words: https://docs.microsoft.com/en-us/azure/application-gateway/media/key-vault-certs/key-vault-firewall.png I can just check d) and not bother with adding the application gateway via b) and c).

However, according to my tests and Microsoft support, this is not correct:

• Application gateway is not a trusted service - see here for a list which does not contain it: https://docs.microsoft.com/en-us/azure/key-vault/general/overview-vnet-service-endpoints#trusted-services
• You will need to follow all steps shown here https://docs.microsoft.com/en-us/azure/application-gateway/media/key-vault-certs/key-vault-firewall.png, in particular the gateway subnet must be allowed in the firewall (why I still need to allow trusted services is beyond me, but fine).

So there seems to be some improvement over the old status quo where we had to allow all networks, but the note https://user-images.githubusercontent.com/14315890/112624676-3abc7080-8e2e-11eb-8645-342bd48b8408.png is at best misleading, if not wrong.

Furthermore, when trying to deploy an Application Gateway without networking access to the KeyVault, it ends up in a failed state with a generic error message that something went wrong. This could also be improved.

"when trying to deploy an Application Gateway without networking access to the KeyVault"

@Martin-Idel-SI, what does this mean specifically?

Do you mean key vault Networking is set to "Private endpoint and selected networks" instead of "All Networks" and the vnet/subnet that the app gateway lives in is not whitelisted for the key valut?

@mgmccarthy
Yes, exactly. I set networking to "Private endpoint and selected networks" and the subnet of the app gateway is not whitelisted and no private endpoint exists that the application gateway has access to.

If the app-gateway subnet is whitelisted - everything is working as expected on my end.
If the app-gateway has access to a private endpoint for the Keyvault - everything is working as expected on my end.

@mgmccarthy
Yes, exactly. I set networking to "Private endpoint and selected networks" and the subnet of the app gateway is not whitelisted and no private endpoint exists that the application gateway has access to.

If the app-gateway subnet is whitelisted - everything is working as expected on my end.
If the app-gateway has access to a private endpoint for the Keyvault - everything is working as expected on my end.

what kind of access does app-gateway have to private endpoint for keyvault in your last line?
Is it the same as allowing virtual networks in c) here https://docs.microsoft.com/en-us/azure/application-gateway/media/key-vault-certs/key-vault-firewall.png ?

I have also been testing it today and it simply does not work without adding the virtual network/subnet where application gateway lives in.

@Martin-Idel-SI and @nikidandwani, turns out the root cause of my issue was the person who was responsible for creating and then uploading the certificate to key vault was doing it in PEM format, instead of PFX format. Apparently, app gateway's don't support PEM formatted certs, but key vault can have either one uploaded to it.

After those certs were changed to PFX format and uploaded to key vault, everything worked as expected. Just to clarify what that means, this code in app gateway ARM:

"sslCertificates": [
      {
        "name": "[variables('appGatewayCertName')]",
        "properties": {            
          "keyVaultSecretId": "[variables('keyVaultSecretId')]"
        }
      }
]

Where kevVaultSecretId is this URL (I put it together as a variable at the top of the app gateway arm template):

https://[Key Vault Name].vault.azure.net/secrets/[cert name]

works perfectly.

Note, even though I'm pointing to "secrets" in the url path, it is picking up the certificates under the Certificates blade in key vault.

It appears, anecdotally, that the transient error which renders an Application Gateway using Key Vault (both in a private network) inaccessible has been addressed?

Does anyone know if Microsoft actively fixed this, or is it simply a case that the people on this thread do not tear-down & stand-up their environments often enough to have been affected by it?

@mgmccarthy
Yes, exactly. I set networking to "Private endpoint and selected networks" and the subnet of the app gateway is not whitelisted and no private endpoint exists that the application gateway has access to.
If the app-gateway subnet is whitelisted - everything is working as expected on my end.
If the app-gateway has access to a private endpoint for the Keyvault - everything is working as expected on my end.

what kind of access does app-gateway have to private endpoint for keyvault in your last line?
Is it the same as allowing virtual networks in c) here https://docs.microsoft.com/en-us/azure/application-gateway/media/key-vault-certs/key-vault-firewall.png ?

I have also been testing it today and it simply does not work without adding the virtual network/subnet where application gateway lives in.

@nikidandwani

For me, it works like this:

• I have a VNET with two subnets: Gateway Subnet + Private link subnet
• The Application Gateway uses the Gateway Subnet, the Key Vault has a private link in the private link subnet
• Network Security Rules allow mutual access
• The Application Gateway has an identity assigned which has access to the Key Vault

Note that the private link consists of multiple parts:

• Private endpoint in the VNET
• Private link
• private DNS zone for Key Vault + A record for Key Vault
• private DNS zone is linked to VNET

Setup is done via terraform.

Hey here!

We have some other issues when we try to attach a self-signed certificate (PEM, X509) stored in KeyVault to Application Gateway in Listener.

• Application Gateway and user has access to Key Vault.
• Application Gateway has an identity assigned which has access to the Key Vault
• Key Vault networking set to: "All networks"

Error:

"statusMessage": "{\"status\":\"Failed\",\"error\":{\"code\":\"ResourceOperationFailure\",\"message\":\"The resource operation completed with terminal provisioning state 'Failed'.\",\"details\":[{\"code\":\"ApplicationGatewayKeyVaultSecretException\",\"message\":\"Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions/XYZ/resourceGroups/xyz-resource-group/providers/Microsoft.Network/applicationGateways/xyz-gateway-001'. See details below:\",\"details\":[{\"code\":\"ApplicationGatewaySslCertificateInvalidData\",\"message\":\"Data or Password for certificate /subscriptions/XYZ/resourceGroups/xyz-resource-group/providers/Microsoft.Network/applicationGateways/xyz-gateway-001/sslCertificates/xyz-listener-httpsvaultCert is invalid.\"}]}]}}",

https://docs.microsoft.com/en-us/azure/application-gateway/key-vault-certs

This is getting to be ridiculous. How am I meant to reconcile the above information with an email we just received (yes, our keyvault is configured to allow trusted services):

Here is our setup:

Certificate generated and signed in Azure Key Vault by an external CA, public access enabled on Keyvault (no network restrictions). EDIT Certificat was generated in PEM format.
Application Gateway v2, Standard_v2 with public and private frontend.
Managed identity is set on App Gateway and has correct rights on Keyvault.

When trying to link keyvault with App Gateway, I have the same error as OP.

To fix my issue:
I downloaded our certificate from the Keyvault, added a password to the certificate ( "openssl rsa -aes256 -in certificate_from_kv.pem -out certificate_from_kv_with_pwd.pem" when prompted, enter password ). Then import the new certificate (with password) into a new certificate in Keyvault.

Application Gateway is too dumb to accept a non-password protected certificate (there is no password on a certificate created in Azure Keyvault, duh!), so we have to set a password to be able to use the certificate created in the Keyvault.

I hope this will get fixed by Microsoft, what a lost of time to find this out...

EDIT We tried a certificate generated with Azure Keyvault in PKCS12 format (first try was with a PEM generated certificate), it did work without having to export/add password/import. WOW

Our team has bee banging heads against the wall for almost two days with this issue on V2_WAF. Doesn't help that deleting and provisioning an appgw takes 15 minutes each and the error messages usually just say "Failed" or "InternalServerError" without any more explanation.

We finally solved it by allowing the Virtual Network for the AppGW in the KeyVault Networking Firewall.
In conclusion the Allow trusted Microsoft services to bypass this firewall option is still NOT working. Despite being listed as a trusted service and the claim above that this was added March 15th.

Some other conclusion: Our keyvault has a shorter retention policy from the 90 day, so that workaround used further up in this thread is likely no longer needed.

Our team has bee banging heads against the wall for almost two days with this issue on V2_WAF. Doesn't help that deleting and provisioning an appgw takes 15 minutes each and the error messages usually just say "Failed" or "InternalServerError" without any more explanation.

We finally solved it by allowing the Virtual Network for the AppGW in the KeyVault Networking Firewall. In conclusion the Allow trusted Microsoft services to bypass this firewall option is still NOT working. Despite being listed as a trusted service and the claim above that this was added March 15th.

Some other conclusion: Our keyvault has a shorter retention policy from the 90 day, so that workaround used further up in this thread is likely no longer needed.

This doc says that service endpoint is must:

I assume integration will not work if you didn't add a service endpoint.

Thank you all for the feedback and sharing the details of the issue that you encountered. Based on your comments, we have updated the documentation to clarify the below points.

• In addition to the Trusted Services setting, you will need to allow your Application Gateway's subnet with Service endpoint configuration to grant access for all the scenarios.
• Supported file format for the certificate.
• Leveraging Azure Advisor for investigating and identifying Key Vault error for your Application Gateway resource.

Apologies for the inconvenience.

@SubhashVasarapu-MSFT can you close the issue?

Is this functional in Azure Gov Virginia?

Even with the Trusted Services setting, we're still having to temporarily open up the firewall on the key-vault in order for this to work.

we are still seeing inconsistencies with leveraging the KV for Certs and the App GW. Especially when certs are not from the same KV or even manually uploaded into the AppGW.

I also have this issue - with allow all networks, trusted services, soft delete enabled 90 days etc

Most of my time of having the appgw deployed in code was troubleshooting and finding workarounds

Am experiencing the same issue, i will try to read all the comments here and see if something will help. On a side note am using Terraform to deploy application gateway v2, key vault, keyvault ceriticate, keyvault secret and managed identity that is used to access keyvault.

@Martin-Idel-SI and @nikidandwani, turns out the root cause of my issue was the person who was responsible for creating and then uploading the certificate to key vault was doing it in PEM format, instead of PFX format. Apparently, app gateway's don't support PEM formatted certs, but key vault can have either one uploaded to it.

After those certs were changed to PFX format and uploaded to key vault, everything worked as expected. Just to clarify what that means, this code in app gateway ARM:

"sslCertificates": [
      {
        "name": "[variables('appGatewayCertName')]",
        "properties": {            
          "keyVaultSecretId": "[variables('keyVaultSecretId')]"
        }
      }
]

Where kevVaultSecretId is this URL (I put it together as a variable at the top of the app gateway arm template):

https://[Key Vault Name].vault.azure.net/secrets/[cert name]

works perfectly.

Note, even though I'm pointing to "secrets" in the url path, it is picking up the certificates under the Certificates blade in key vault.

Yeah, we had "network all" and the correct permissions to certificates and secrets. However, I had initially uploaded a PEM with the key, cert and CA intermediates in it. Doing the same but with a PFX and AppGW worked fine without any other changes.

My suggestion is the "pop-up" in the AppGW for adding from a keyvault gets a nice big "this only works with imported PFX files" message.

Hello Team,

Trying to consolidate down this thread, as it's a bit lengthy. We've updated the Key Vault documentation quite a bit to reflect network permissions and key vault permissions, so I hope those changes address the items covered in this thread.

To clarify a few statements made on this thread:

1. Deployment of AppGW + Key Vault is supported via Portal, PowerShell, CLI, ARM, Bicep, and Terraform.
2. Certificates should be in PKCS #​12 format (pfx) and the private key must be exportable from key vault.
3. You can successfully configure the integration of Key Vault and AppGW with the Networking -> Firewalls and Virtual Networks -> "Allow access from" option set to Private endpoints and selected networks. "All networks" access is not required for this to work, however it is required to set "Allow trusted Microsoft services to bypass this firewall" to Yes.
4. Key Vault + Application Gateway is supported in all Azure regions, including Gov Cloud.
5. Ensure proper DNS configuration is applied to the subnet. If you are using key vault with private endpoints, you will need to ensure that DNS resolves to the private IP vs public IP. This has been a common oversight for customers that enable private endpoints but forget to link the private zone to the vnet or update DNS within their defined DNS servers.

If the documentation and items outlined in this thread do not work, please open a support case so we can determine root cause. This will allow us to ensure we get you up and running as well as enable us to document additional gotchas for configuring this setup.

Thank you!
Jack

#please-close

Here's my setup

• AppGW v2 using a User-Assigned Managed Identity, Certificates that reference KeyVault
• KeyVault using RBAC permissions, Private Endpoint, Firewall fully locked down

Deploying AppGW via ARM Template, deployment hangs and errors out after 30 minutes.

If I enabled Service Endpoints and add the AppGW's subnet to the KeyVault Firewall, it works!

Based on the documentation the AppGW should be able to talk to the KeyVault over its Private Endpoint, but that is not what I'm seeing in action.

@nnellanspdl, it sounds like traffic is still trying to egress via public IP. I would verify DNS configuration is correct, with the Azure DNS zone associated to the VNET that contains Application Gateway. If you still see the issue, please open a support case so we can further investigate. Thanks!

I had the same problem and now it's resolved.
@nnellanspdl, the gotcha, for which you hardly find hints in the docus, is that when you use Private Endpoint, the keyVaultSecretId in sslCertificates should not be the Key Vault Secret Id itself, as is the case without Private Endpoint; because the Gateway would still try to reach to your Key Vault via public network. Instead, the keyVaultSecretId here should be the URL to the certificate such as https://[Key Vault Name].vault.azure.net/secrets/[cert name]. Only this way your Gateway understands that it should go through that dedicated subnet.
A very helpful reference is @mgmccarthy 's post here above.
If you deploy with Terraform, it's the same.

I ran into this issue, and found something else that'd be useful to be added to the documentation

I had everything correct, except that the user account I was logged in with, didn't have the 'Get' Secret permission in the Key vault Access policies 😮‍💨

So it's not only the managed identity that requires the 'Get' permision, but also the user account that you are using ...

Can anyone post a concrete example for Terraform or ARM as we are also struggling with Private Endpoint + RBAC.
From looking through comments this seems to work on a random basis...

I had the same issue. However I was able to solve issue by removing the Key Vault certificate using PowerShell and not resource explorer using explorer showed the below error:

{
"error": {
"code": "MissingIdentityIds",
"message": "The identity ids must not be null or empty for 'UserAssigned' identity type."
}
}

used the below sample script to remove certificate and listener and the app gateway went back into working state

#delete listener and cert from app gw
$AppGw = Get-AzApplicationGateway -Name "app-gw-ssl-key" -ResourceGroupName "lab"
Remove-AzApplicationGatewayHttpListener -ApplicationGateway $AppGw -Name "https"
Remove-AzApplicationGatewaySslCertificate -ApplicationGateway $AppGW -Name "victor-cer"
#save changes
set-azapplicationgateway -ApplicationGateway $AppGW
when the app gateway was out of failed state, I checked the access policy of the key vault and saw that the identity of the app gw was in another category that's not "APPLICATION". I guess this was the cause of the issue. Added the identity again to the access policy from the key vault setting and it was able to show as "APPLICATION". This solved the issue for me and I was able to add listener with cert from key vault without issues.

Just following up for posterity - we encountered this issue in GCC High today, and the fix was this solution. We had to forcibly remove the failed SSL cert via CLI, then remove the AGW permissions on the keyvault. Re-add permissions, then readd the cert. After getting the AGW out of the failed state, it worked as expected.