is it possible to create Kubernetes pods, services, replica controllers etc on AWS cloudfromation?

4/8/2020

does AWS cloudformation supports creation of Kubernetes pods, services, replica controllers etc or setting up the EKS clusters and worker nodes and using Kubectl to create the resources are the only way?

-- mygitrepo
amazon-cloudformation
amazon-eks
aws-eks
kubernetes

2 Answers

4/8/2020

You can use CloudFormation to create EKS cluster and worker nodes but you have to use kubectl for any operation on cluster like creating service, pods, deployments etc.....you can’t use CloudFormation for that

-- Jatin
Source: StackOverflow

4/9/2020

Not out of the box, but you can if you use a custom resource type backed by a lambda function in CloudFormation.

The AWS EKS quickstart has an example:

AWSTemplateFormatVersion: "2010-09-09"
Description: "deploy an example workload into an existing kubernetes cluster (qs-1p817r5f9)"
Parameters:
  KubeConfigPath:
    Type: String
  KubeConfigKmsContext:
    Type: String
    Default: "EKSQuickStart"
  KubeClusterName:
    Type: String
  NodeInstanceProfile:
    Type: String
  QSS3BucketName:
    AllowedPattern: ^[0-9a-zA-Z]+([0-9a-zA-Z-]*[0-9a-zA-Z])*$
    ConstraintDescription: Quick Start bucket name can include numbers, lowercase
      letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen
      (-).
    Default: aws-quickstart
    Description: S3 bucket name for the Quick Start assets. This string can include
      numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start
      or end with a hyphen (-).
    Type: String
  QSS3KeyPrefix:
    AllowedPattern: ^[0-9a-zA-Z-/.]*$
    ConstraintDescription: Quick Start key prefix can include numbers, lowercase letters,
      uppercase letters, hyphens (-), dots(.) and forward slash (/).
    Default: quickstart-amazon-eks/
    Description: S3 key prefix for the Quick Start assets. Quick Start key prefix
      can include numbers, lowercase letters, uppercase letters, hyphens (-), dots(.) and
      forward slash (/).
    Type: String
  QSS3BucketRegion:
    Default: 'us-east-1'
    Description: The AWS Region where the Quick Start S3 bucket (QSS3BucketName) is
      hosted. When using your own bucket, you must specify this value.
    Type: String
  LambdaZipsBucketName:
    Description: 'OPTIONAL: Bucket Name where the lambda zip files should be placed,
          if left blank a bucket will be created.'
    Type: String
    Default: ''
  K8sSubnetIds:
    Type: List<AWS::EC2::Subnet::Id>
  VPCID:
    Type: AWS::EC2::VPC::Id
  ControlPlaneSecurityGroup:
    Type: AWS::EC2::SecurityGroup::Id
Conditions:
  CreateLambdaZipsBucket: !Equals
    - !Ref 'LambdaZipsBucketName'
    - ''
  UsingDefaultBucket: !Equals [!Ref QSS3BucketName, 'aws-quickstart']
Resources:
  WorkloadStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: !Sub
        - 'https://${S3Bucket}.s3.${S3Region}.${AWS::URLSuffix}/${QSS3KeyPrefix}templates/example-workload.template.yaml'
        - S3Region: !If [UsingDefaultBucket, !Ref 'AWS::Region', !Ref QSS3BucketRegion]
          S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName]
      Parameters:
        KubeManifestLambdaArn: !GetAtt KubeManifestLambda.Arn
        HelmLambdaArn: !GetAtt HelmLambda.Arn
        KubeConfigPath: !Ref KubeConfigPath
        KubeConfigKmsContext: !Ref KubeConfigKmsContext
        KubeClusterName: !Ref KubeClusterName
        NodeInstanceProfile: !Ref NodeInstanceProfile
  CopyZips:
    Type: Custom::CopyZips
    Properties:
      ServiceToken: !GetAtt 'CopyZipsFunction.Arn'
      DestBucket: !Ref LambdaZipsBucketName
      SourceBucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName]
      Prefix: !Ref 'QSS3KeyPrefix'
      Objects:
        - functions/packages/Helm/lambda.zip
        - functions/packages/DeleteBucketContents/lambda.zip
        - functions/packages/KubeManifest/lambda.zip
        - functions/packages/LambdaEniCleanup/lambda.zip
  VPCLambdaCleanup:
    Type: Custom::LambdaCleanup
    Properties:
      ServiceToken: !GetAtt VPCLambdaCleanupLambdaFunction.Arn
      Region: !Ref "AWS::Region"
      LambdaFunctionNames:
        - !Ref KubeManifestLambda
  VPCLambdaCleanupLambdaFunction:
    DependsOn: CopyZips
    Type: "AWS::Lambda::Function"
    Properties:
      Handler: lambda_function.lambda_handler
      MemorySize: 128
      Role: !GetAtt LambdaCleanUpFunctionRole.Arn
      Runtime: python3.7
      Timeout: 900
      Code:
        S3Bucket: !Ref LambdaZipsBucketName
        S3Key: !Sub '${QSS3KeyPrefix}functions/packages/LambdaEniCleanup/lambda.zip'
  HelmLambda:
    DependsOn: CopyZips
    Type: AWS::Lambda::Function
    Properties:
      Handler: lambda_function.lambda_handler
      MemorySize: 128
      Role: !GetAtt ManifestRole.Arn
      Runtime: python3.6
      Timeout: 900
      Code:
        S3Bucket: !Ref LambdaZipsBucketName
        S3Key: !Sub '${QSS3KeyPrefix}functions/packages/Helm/lambda.zip'
      VpcConfig:
        SecurityGroupIds: [ !Ref EKSLambdaSecurityGroup ]
        SubnetIds: !Ref K8sSubnetIds
  KubeManifestLambda:
    DependsOn: CopyZips
    Type: AWS::Lambda::Function
    Properties:
      Handler: lambda_function.lambda_handler
      MemorySize: 128
      Role: !GetAtt ManifestRole.Arn
      Runtime: python3.6
      Timeout: 900
      Code:
        S3Bucket: !Ref LambdaZipsBucketName
        S3Key: !Sub '${QSS3KeyPrefix}functions/packages/KubeManifest/lambda.zip'
      VpcConfig:
        SecurityGroupIds: [ !Ref EKSLambdaSecurityGroup ]
        SubnetIds: !Ref K8sSubnetIds
  DeleteBucketContentsLambda:
    DependsOn: CopyZips
    Type: AWS::Lambda::Function
    Properties:
      Handler: lambda_function.lambda_handler
      MemorySize: 128
      Role: !GetAtt DeleteBucketContentsRole.Arn
      Runtime: python3.7
      Timeout: 900
      Code:
        S3Bucket: !Ref LambdaZipsBucketName
        S3Key: !Sub '${QSS3KeyPrefix}functions/packages/DeleteBucketContents/lambda.zip'
  CopyZipsFunction:
    Type: AWS::Lambda::Function
    Properties:
      Description: Copies objects from a source S3 bucket to a destination
      Handler: index.handler
      Runtime: python3.7
      Role: !GetAtt CopyZipsRole.Arn
      Timeout: 900
      Code:
        ZipFile: |
          import json
          import logging
          import threading
          import boto3
          import cfnresponse
          def copy_objects(source_bucket, dest_bucket, prefix, objects):
              s3 = boto3.client('s3')
              for o in objects:
                  key = prefix + o
                  copy_source = {
                      'Bucket': source_bucket,
                      'Key': key
                  }
                  print('copy_source: %s' % copy_source)
                  print('dest_bucket = %s'%dest_bucket)
                  print('key = %s' %key)
                  s3.copy_object(CopySource=copy_source, Bucket=dest_bucket,
              Key=key)
          def delete_objects(bucket, prefix, objects):
              s3 = boto3.client('s3')
              objects = {'Objects': [{'Key': prefix + o} for o in objects]}
              s3.delete_objects(Bucket=bucket, Delete=objects)
          def timeout(event, context):
              logging.error('Execution is about to time out, sending failure response to CloudFormation')
              cfnresponse.send(event, context, cfnresponse.FAILED, {}, physical_resource_id)
          def handler(event, context):
              physical_resource_id = None
              if "PhysicalResourceId" in event.keys():
                physical_resource_id = event["PhysicalResourceId"]
              # make sure we send a failure to CloudFormation if the function is going to timeout
              timer = threading.Timer((context.get_remaining_time_in_millis()
              / 1000.00) - 0.5, timeout, args=[event, context])
              timer.start()
              print('Received event: %s' % json.dumps(event))
              status = cfnresponse.SUCCESS
              try:
                  source_bucket = event['ResourceProperties']['SourceBucket']
                  dest_bucket = event['ResourceProperties']['DestBucket']
                  prefix = event['ResourceProperties']['Prefix']
                  objects = event['ResourceProperties']['Objects']
                  if event['RequestType'] == 'Delete':
                      delete_objects(dest_bucket, prefix, objects)
                  else:
                      copy_objects(source_bucket, dest_bucket, prefix, objects)
              except Exception as e:
                  logging.error('Exception: %s' % e, exc_info=True)
                  status = cfnresponse.FAILED
              finally:
                  timer.cancel()
                  cfnresponse.send(event, context, status, {}, physical_resource_id)
  LambdaZipsBucket:
    Type: AWS::S3::Bucket
    Condition: CreateLambdaZipsBucket
  LambdaCleanUpFunctionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: ['sts:AssumeRole']
            Effect: Allow
            Principal:
              Service: [lambda.amazonaws.com]
        Version: '2012-10-17'
      Path: /
      Policies:
        - PolicyName: LambdaRole
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Effect: Allow
                Resource: !Sub "arn:${AWS::Partition}:logs:*:*:*"
              - Action:
                  - 'ec2:*'
                Effect: Allow
                Resource: "*"
  DeleteBucketContentsRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - !Sub 'arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
      Policies:
        - PolicyName: deletebucketcontents
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action: s3:*
                Resource:
                  - !Sub 'arn:${AWS::Partition}:s3:::${LambdaZipsBucketName}/*'
                  - !Sub 'arn:${AWS::Partition}:s3:::${LambdaZipsBucketName}'
  CopyZipsRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - !Su  'arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
      Policies:
        - PolicyName: lambda-copier
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action: s3:GetObject
                Resource: !Sub
                  - 'arn:${AWS::Partition}:s3:::${S3Bucket}/${QSS3KeyPrefix}*'
                  - S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName]
              - Effect: Allow
                Action:
                  - s3:PutObject
                  - s3:DeleteObject
                Resource: !Sub 'arn:${AWS::Partition}:s3:::${LambdaZipsBucketName}/${QSS3KeyPrefix}*'
  ManifestRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: eksStackPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action: s3:GetObject
                Resource: !Sub
                  - "arn:${AWS::Partition}:s3:::${BucketName}/*"
                  - S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName]
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                  - ec2:CreateNetworkInterface
                  - ec2:DescribeNetworkInterfaces
                  - ec2:DeleteNetworkInterface
                Resource:
                  - "*"
              - Action: "kms:decrypt"
                Effect: Allow
                Resource: "*"
              - Action: "s3:GetObject"
                Effect: Allow
                Resource: "*"
  EKSLambdaSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for lambda to communicate with cluster API
      VpcId: !Ref VPCID
  ClusterControlPlaneSecurityGroupIngress:
    Type: AWS::EC2::SecurityGroupIngress
    Properties:
      Description: Allow lambda to communicate with the cluster API Server
      GroupId: !Ref ControlPlaneSecurityGroup
      SourceSecurityGroupId: !Ref EKSLambdaSecurityGroup
      IpProtocol: tcp
      ToPort: 443
      FromPort: 443

It works by creating a lambda function customer resource KubeManifestLambda and HelmLambda that has kubectl and helm installed respectively, both configured with a role that allows them to access the EKS k8s cluster.

Then these custom resources can be used to deploy k8s manifests and helm charts with custom values, like in this example.

 KubeManifestExample:
    Type: "Custom::KubeManifest"
    Version: '1.0'
    Properties:
      # The lambda function that executes the manifest against the cluster. This is created in one of the parent stacks
      ServiceToken: !Ref KubeManifestLambdaArn
      # S3 path to the encrypted config file eg. s3://my-bucket/kube/config.encrypted
      KubeConfigPath: !Ref KubeConfigPath
      # context for KMS to use when decrypting the file
      KubeConfigKmsContext: !Ref KubeConfigKmsContext
      # Kubernetes manifest
      Manifest:
        apiVersion: v1
        kind: ConfigMap
        metadata:
          # If name is not specified it will be automatically generated,
          # and can be retrieved with !GetAtt LogicalID.name
          #
          # name: test
          #
          # if namespace is not specified, "default" namespace will be used
          namespace: kube-system
        data:
          # examples of consuming outputs of the HelmExample resource below's output. Creates an implicit dependency, so
          # this resource will only launch once the HelmExample resource has completed successfully
          ServiceCatalogReleaseName: !Ref HelmExample
          ServiceCatalogKubernetesServiceName: !GetAtt HelmExample.Service0

This even lets you reference other Cloud formation resources such as RDS instances that are created as part of a workload.

-- Dylan
Source: StackOverflow