amazon web services - How can I apply tolerations to EKS's Add-Ons when using Terraform - Stack Overflow

I'm trying to install Cilium in my EKS cluster, to acomplish that I need to create my cluster's node groups with the following taint:

taints:
   - key: "node.cilium.io/agent-not-ready"
     value: "true"
     effect: "NoExecute"

I also need to create my Add-Ons (VPC CNI, EBS CSI, Kube Proxy and CoreDNS), but because of the taint on the nodes, the Add-Ons are installed with error. I'm using Terraform to create everything.

Update: I was able to create the VPC CNI, Kube Proxy and CoreDNS using the tolerations, as described on the aws eks describe-addon-configuration command. The problem I'm having now is with the AWS EBS CSI Driver. It doesn't support the tolereations scheme I'm trying.

After I run the following command:

aws eks describe-addon-configuration --addon-name aws-ebs-csi-driver --addon-version v1.38.1-eksbuild.2 | jq ".configurationSchema" | jq "fromjson" | jq ".properties.node.properties.tolerations"

The return is:

{
  "default": [
    {
      "effect": "NoExecute",
      "operator": "Exists",
      "tolerationSeconds": 300
    }
  ],
  "description": "Tolerations of the node pod",
  "items": {
    "type": "object"
  },
  "type": "array"
}

My configuration is the following:

configuration_values = jsonencode({
tolerations = [{
  "key" : "node.cilium.io/agent-not-ready",
  "operator": "Equal",
  "value": "true",
  "effect" : "NoExecute"
}]

})

And yet I'm still receiving the following error: InvalidParameterException: ConfigurationValue provided in request is not supported: Json schema validation failed with error: [$.tolerations: is not defined in the schema and the schema does not allow additional properties]

I'm trying to install Cilium in my EKS cluster, to acomplish that I need to create my cluster's node groups with the following taint:

taints:
   - key: "node.cilium.io/agent-not-ready"
     value: "true"
     effect: "NoExecute"

I also need to create my Add-Ons (VPC CNI, EBS CSI, Kube Proxy and CoreDNS), but because of the taint on the nodes, the Add-Ons are installed with error. I'm using Terraform to create everything.

Update: I was able to create the VPC CNI, Kube Proxy and CoreDNS using the tolerations, as described on the aws eks describe-addon-configuration command. The problem I'm having now is with the AWS EBS CSI Driver. It doesn't support the tolereations scheme I'm trying.

After I run the following command:

aws eks describe-addon-configuration --addon-name aws-ebs-csi-driver --addon-version v1.38.1-eksbuild.2 | jq ".configurationSchema" | jq "fromjson" | jq ".properties.node.properties.tolerations"

The return is:

{
  "default": [
    {
      "effect": "NoExecute",
      "operator": "Exists",
      "tolerationSeconds": 300
    }
  ],
  "description": "Tolerations of the node pod",
  "items": {
    "type": "object"
  },
  "type": "array"
}

My configuration is the following:

configuration_values = jsonencode({
tolerations = [{
  "key" : "node.cilium.io/agent-not-ready",
  "operator": "Equal",
  "value": "true",
  "effect" : "NoExecute"
}]

})

And yet I'm still receiving the following error: InvalidParameterException: ConfigurationValue provided in request is not supported: Json schema validation failed with error: [$.tolerations: is not defined in the schema and the schema does not allow additional properties]

• amazon-web-services
• kubernetes
• terraform
• amazon-eks
• cilium

Share Follow edited Feb 4 at 11:27 Pedro Ignacio asked Jan 30 at 14:20 Pedro IgnacioPedro Ignacio 6366 bronze badges 3

• The VPC CNI is like this: resource "aws_eks_addon" "vpc-cni" { cluster_name = var.eks-cluster-name-stage addon_name = "vpc-cni" addon_version = var.vpc-cni-version service_account_role_arn = xxx resolve_conflicts_on_create = "OVERWRITE" resolve_conflicts_on_update = "OVERWRITE" tags = { "eks_addon" = "vpc-cni" } depends_on = [ aws_eks_node_group.xxx, aws_eks_node_group.xxx ] } All other addons are being deployed the same way. – Pedro Ignacio Commented Jan 30 at 14:30
• please update the question with your terraform code not add it in a comment – Chris Doyle Commented Jan 30 at 18:51
• looking at the addons configuration the schema supports passing a list of tollerations aws eks describe-addon-configuration --addon-name vpc-cni --addon-version v1.19.2-eksbuild.1 | jq ".configurationSchema" | jq "fromjson" | jq ".definitions.Tolerations" – Chris Doyle Commented Jan 30 at 19:34

Add a comment  | 

3 Answers 3

Sorted by: Reset to default 2

The terraform documentation has an example on how you can achieve this. In the example the addon is coredns version v1.10.1-eksbuild.1.

Calling the api:

 aws eks describe-addon-configuration \
 --addon-name coredns \
 --addon-version v1.10.1-eksbuild.1 \
 --query "configurationSchema" | jq '. | fromjson'

you can see that tolerations is a property. So to specify the toleration in this example:

resource "aws_eks_addon" "example" {
  cluster_name  = "mycluster"
  addon_name    = "coredns"
  addon_version = "v1.10.1-eksbuild.1"

  configuration_values = jsonencode({
    tolerations = [{
      "key" : "node.cilium.io/agent-not-ready",
      "operator" : "NoExecute"
    }]
  })
}

For the latest versions of these addons:

• aws-ebs-csi-driver: v1.38.1-eksbuild.2
• aws-mountpoint-s3-csi-driver: v1.11.0-eksbuild.1

I had to add this in the Terraform scripts for addon configuration:

configuration_values = jsonencode({
node: {
  tolerations = [
    {
      operator = "Exists"
      effect    = "NoSchedule"
    }
  ]
}

})

This works for the latest versions of EBS CSI driver addons:

• aws-ebs-csi-driver: v1.41.0-eksbuild.1

Passing below code block as input for the https://github.com/terraform-aws-modules/terraform-aws-eks module.

    aws-ebs-csi-driver = {
      addon_version = "v1.41.0-eksbuild.1"
      configuration_values = jsonencode({
        node : {
          tolerations : [
            {
              effect : "NoSchedule",
              key : "<CustomTaintKey>",
              operator : "Equal",
              value : "<CustomTaintValue>"
            }
          ]
        },
        controller : {
          tolerations : [
            {
              effect : "NoSchedule",
              key : "<CustomTaintKey>",
              operator : "Equal",
              value : "<CustomTaintValue>"
            }
          ]
        }
      })