azure - Can't deploy key vault key or secret with Terraform using private endpoints the first time - Stack Overflow

I want to deploy a key or secret with Terraform into a key vault that is created in the same deployment, but I get an error everytime on the first try. When the key vault already exists or if I run the terraform code the 2nd time it works. The error I get is:

Error: checking for presence of existing Key "example-key1" (Key Vault "/"): keyvault.BaseClient#GetKey: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Public network access is disabled and request is not from a trusted service nor via an approved private link.\r\nCaller: appid=xxx;oid=xxx;iss=xxx: xx-xxx-xxx;location=westeurope" InnerError={"code":"ForbiddenByConnection"}

with azurerm_key_vault_key.example, on main.tf line 97, in resource "azurerm_key_vault_key" "example": 97: resource "azurerm_key_vault_key" "example" {

Public access must be disabled at all times. I also use the existing vaultcore private dns zone with the private endpoint. We use a hub spoke model so the private dns zone has a link with the vnet hub and the vnet spoke that i'm using is connected to the vnet hub.

I want it to work the first time

I tried putting a wait in the code of 1,2,5 and even 20 minutes, but it also didn't work. I also tried to link the private dns zone also to the vnet spoke i'm using, but that didn't work either.

It doesn't matter if I run it in Visual studio code or through a self-hosted agent that is in the same vnet, I get the error every first time.

The code I use is:

data "azurerm_client_config" "current" {}

data "azurerm_resource_group" "resource_group" {
  name = "test-rg"
}

data "azurerm_subnet" "example" {
  name                 = "test-subnet"
  virtual_network_name = "test-vnet"
  resource_group_name  = "test-vnet-rg"
}

data "azurerm_private_dns_zone" "vaultcore" {
  name                = "privatelink.vaultcore.azure"
  resource_group_name = "test-rg-dnszones"
}

resource "azurerm_key_vault" "example" {
  name                       = "test-kv"
  location                   = data.azurerm_resource_group.resource_group.location
  resource_group_name        = data.azurerm_resource_group.resource_group.name
  sku_name                   = "standard"
  tenant_id                  = data.azurerm_client_config.current.tenant_id
  soft_delete_retention_days = 7
  purge_protection_enabled   = true

  enabled_for_disk_encryption     = false
  enabled_for_template_deployment = false
  enabled_for_deployment          = false

  public_network_access_enabled = false
  enable_rbac_authorization     = true

  network_acls {
    default_action = "Deny"
    bypass         = "AzureServices"
  }

  tags = {
    "source"    = "Terraform"
  }

  lifecycle {
    ignore_changes = [tags]
  }
}

resource "azurerm_private_endpoint" "example" {
  name                = "pep-test"
  location            = data.azurerm_resource_group.resource_group.location
  resource_group_name = data.azurerm_resource_group.resource_group.name
  subnet_id           = data.azurerm_subnet.example.id

  private_service_connection {
    name                           = "Connection-test"
    private_connection_resource_id = azurerm_key_vault.example.id
    subresource_names              = ["Vault"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = data.azurerm_private_dns_zone.vaultcore.name
    private_dns_zone_ids = [data.azurerm_private_dns_zone.vaultcore.id]
  }

  custom_network_interface_name = "test-nic"

   tags = {
    "source"    = "Terraform"
  }

  lifecycle {
    ignore_changes = [tags]
  }

  depends_on = [azurerm_key_vault.example]
}

resource "time_sleep" "name" {
  create_duration = "2m"
  depends_on = [azurerm_private_endpoint.example]
}

resource "azurerm_key_vault_key" "example" {
  name         = "key-test"
  key_vault_id = azurerm_key_vault.example.id
  key_type     = "RSA"
  key_size     = 2048
  key_opts     = ["decrypt", "encrypt", "sign", "unwrapKey", "verify", "wrapKey"]
  depends_on   = [time_sleep.name]

   tags = {
    "source"    = "Terraform"
  }

  lifecycle {
    ignore_changes = [tags]
  }
}

I want to deploy a key or secret with Terraform into a key vault that is created in the same deployment, but I get an error everytime on the first try. When the key vault already exists or if I run the terraform code the 2nd time it works. The error I get is:

Error: checking for presence of existing Key "example-key1" (Key Vault "https://xx-xxx-xxx.vault.azure.net/"): keyvault.BaseClient#GetKey: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Public network access is disabled and request is not from a trusted service nor via an approved private link.\r\nCaller: appid=xxx;oid=xxx;iss=xxx: xx-xxx-xxx;location=westeurope" InnerError={"code":"ForbiddenByConnection"}

with azurerm_key_vault_key.example, on main.tf line 97, in resource "azurerm_key_vault_key" "example": 97: resource "azurerm_key_vault_key" "example" {

Public access must be disabled at all times. I also use the existing vaultcore private dns zone with the private endpoint. We use a hub spoke model so the private dns zone has a link with the vnet hub and the vnet spoke that i'm using is connected to the vnet hub.

I want it to work the first time

I tried putting a wait in the code of 1,2,5 and even 20 minutes, but it also didn't work. I also tried to link the private dns zone also to the vnet spoke i'm using, but that didn't work either.

It doesn't matter if I run it in Visual studio code or through a self-hosted agent that is in the same vnet, I get the error every first time.

The code I use is:

data "azurerm_client_config" "current" {}

data "azurerm_resource_group" "resource_group" {
  name = "test-rg"
}

data "azurerm_subnet" "example" {
  name                 = "test-subnet"
  virtual_network_name = "test-vnet"
  resource_group_name  = "test-vnet-rg"
}

data "azurerm_private_dns_zone" "vaultcore" {
  name                = "privatelink.vaultcore.azure.net"
  resource_group_name = "test-rg-dnszones"
}

resource "azurerm_key_vault" "example" {
  name                       = "test-kv"
  location                   = data.azurerm_resource_group.resource_group.location
  resource_group_name        = data.azurerm_resource_group.resource_group.name
  sku_name                   = "standard"
  tenant_id                  = data.azurerm_client_config.current.tenant_id
  soft_delete_retention_days = 7
  purge_protection_enabled   = true

  enabled_for_disk_encryption     = false
  enabled_for_template_deployment = false
  enabled_for_deployment          = false

  public_network_access_enabled = false
  enable_rbac_authorization     = true

  network_acls {
    default_action = "Deny"
    bypass         = "AzureServices"
  }

  tags = {
    "source"    = "Terraform"
  }

  lifecycle {
    ignore_changes = [tags]
  }
}

resource "azurerm_private_endpoint" "example" {
  name                = "pep-test"
  location            = data.azurerm_resource_group.resource_group.location
  resource_group_name = data.azurerm_resource_group.resource_group.name
  subnet_id           = data.azurerm_subnet.example.id

  private_service_connection {
    name                           = "Connection-test"
    private_connection_resource_id = azurerm_key_vault.example.id
    subresource_names              = ["Vault"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = data.azurerm_private_dns_zone.vaultcore.name
    private_dns_zone_ids = [data.azurerm_private_dns_zone.vaultcore.id]
  }

  custom_network_interface_name = "test-nic"

   tags = {
    "source"    = "Terraform"
  }

  lifecycle {
    ignore_changes = [tags]
  }

  depends_on = [azurerm_key_vault.example]
}

resource "time_sleep" "name" {
  create_duration = "2m"
  depends_on = [azurerm_private_endpoint.example]
}

resource "azurerm_key_vault_key" "example" {
  name         = "key-test"
  key_vault_id = azurerm_key_vault.example.id
  key_type     = "RSA"
  key_size     = 2048
  key_opts     = ["decrypt", "encrypt", "sign", "unwrapKey", "verify", "wrapKey"]
  depends_on   = [time_sleep.name]

   tags = {
    "source"    = "Terraform"
  }

  lifecycle {
    ignore_changes = [tags]
  }
}

• azure
• terraform
• azure-keyvault

Share Improve this question Follow edited Jan 14 at 9:18 user3617809 asked Jan 13 at 16:30 user3617809user3617809 122 bronze badges 5

• please share the code you tried @user3617809 – Vinay B Commented Jan 13 at 16:39
• since you tried using sleep time which in general limited to certain duration of time, we need a task which confirm the existence of private DNS is resolvable before proceeding for this we may need a custom script using null_resource or local-exec to wait as long it completly provisioned @user3617809 – Vinay B Commented Jan 13 at 16:51
• Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. – Community Bot Commented Jan 13 at 23:14
• I added my code in the initial post above. Please let me know if any additional information is needed. Can you maybe the task to check the existence of private dns is resolvable? – user3617809 Commented Jan 14 at 9:20
• did you follow the steps i mentioned @user3617809 – Vinay B Commented Jan 14 at 14:56

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

Deploying key vault key or secret with Terraform using private endpoints in one go.

Since youre using private end point the access to keyvault is limited so inorder to create a key inside a keyvault we need to provide the necessary permissions.

So to overcome the issue add your IP address to your network_acls block in your azurerm_key_vault resource so that it only allows trusted services.

And add the role "Key Vault Crypto Officer" to your user or service principle.

demo configuration:

resource "azurerm_key_vault" "example" {
  name                       = "test-ksvksb"
  location                   = data.azurerm_resource_group.resource_group.location
  resource_group_name        = data.azurerm_resource_group.resource_group.name
  sku_name                   = "standard"
  tenant_id                  = data.azurerm_client_config.current.tenant_id
  soft_delete_retention_days = 7
  purge_protection_enabled   = true

  enabled_for_disk_encryption     = false
  enabled_for_template_deployment = false
  enabled_for_deployment          = false

  public_network_access_enabled = true
  enable_rbac_authorization     = true

   access_policy {
    tenant_id = data.azurerm_client_config.current.tenant_id
    object_id = data.azurerm_client_config.current.object_id
  }

  network_acls {
    default_action = "Deny"
    bypass         = "AzureServices"
    ip_rules       = [Your IP address] #add IP address  here
    virtual_network_subnet_ids = []
  }

  tags = {
    "source" = "Terraform"
  }

  lifecycle {
    ignore_changes = [tags]
  }
}

resource "azurerm_private_endpoint" "example" {
  name                = "pep-test"
  location            = data.azurerm_resource_group.resource_group.location
  resource_group_name = data.azurerm_resource_group.resource_group.name
  subnet_id           = data.azurerm_subnet.example.id

  private_service_connection {
    name                           = "Connection-test"
    private_connection_resource_id = azurerm_key_vault.example.id
    subresource_names             = ["vault"]
    is_manual_connection          = false
  }

  private_dns_zone_group {
    name                 = "default"
    private_dns_zone_ids = [data.azurerm_private_dns_zone.vaultcore.id]
  }

  custom_network_interface_name = "test-nic"

  tags = {
    "source" = "Terraform"
  }

  lifecycle {
    ignore_changes = [tags]
  }
}


resource "azurerm_role_assignment" "key_vault_crypto_officer" {
  scope                = azurerm_key_vault.example.id
  role_definition_name = "Key Vault Crypto Officer"
  principal_id         = data.azurerm_client_config.current.object_id
}

resource "azurerm_key_vault_key" "example" {
  name         = "key-test"
  key_vault_id = azurerm_key_vault.example.id
  key_type     = "RSA"
  key_size     = 2048
  key_opts     = ["decrypt", "encrypt", "sign", "unwrapKey", "verify", "wrapKey"]

  tags = {
    "source" = "Terraform"
  }

  depends_on = [
    azurerm_key_vault.example,
    azurerm_private_endpoint.example,
    azurerm_role_assignment.key_vault_crypto_officer
  ]

  lifecycle {
    ignore_changes = [tags]
  }
}

Deployement:

Refer:

https://learn.microsoft.com/en-us/azure/key-vault/keys/quick-create-terraform#:~:text=This%20article%20focuses%20on%20the%20process%20of%20deploying,Terraform%2C%20you%20create%20configuration%20files%20using%20HCL%20syntax.

Integrate Key Vault with Azure Private Link | Microsoft Learn