How to complete setup Kubernetes Cluster on AWS using Kops

Kops on AWS

Thinking Basic requirement

• Installing Cluster ( IAC )

• Authentication ( IAM )

• Authorization ( K8s resources)

• Ingress Controller ( ALB )

• External DNS

Prerequisites before setup cluster

• Kops

• aws cli

• aws-iam-authenticator

• kubectl

What is Kops ?

Kops သည် K for kubernetes၊ ops for Operations ၂ခု ပေါင်းသည့်အခါ Kubernetes Operations ဟုတ် အဓိပ္ပါယ်ရသည်။ အမျိုးမျိုးသော cloud platforms များပေါ်တွင် kubernetes cluster ကို version control system ဖြင့် install ပြုလုပ်ခြင်း၊ manage ပြုလုပ်ခြင်း၊ စသည့် operation များကို ကူညီ အထောက်အပံ ဖြစ်ရန် ရည်ရွယ် ထုတ်လုပ်ထားသည့် CLI tool တစ်ခုလည်း ဖြစ်ပါသည်။ Kops သည် kubernetes project အတူဖြစ်ပေါ်လာပြီး K8s team မှ ထိန်းသိမ်းသည့် tools တစ်ခု ဖြစ်သည်။

What is AWS CLI ?

AWS CLI သည် Developer များ၊ Ops များ အတွက် AWS cloud platform ၏ resources များကို command line မှ တဆင့် အလွယ်တကူ အသုံးပြုနိုင်ရန် အထူးပြုလုပ်ထားသော Amazon Web Services မှ tools တစ်ခုဖြစ်သည်။

What is aws-iam-authenticator ?

Kubernetes Cluster ကို AWS ၏ IAM ကို အသုံးပြု၍ authentication ပြုလုပ်နိုင်ရန် အတွက် ထုတ်လုပ်ထားသော tools တစ်ခုဖြစ်သည်။ aws-iam-authenticator သည် kubernetes special interest groups ( SIG ) ၏ project တစ်ခု ဖြစ်ပြီး ယခု လက်ရှိတွင် Heptio and Amazon EKS OSS မှ ထိန်းသိမ်ထားပါသည်။

What is kubectl ?

Kubectl သည် kubernetes cluster အတွက် client command line interface tool တစ်ခု ဖြစ်သည်။ kubectl သည် kubernetes cluster ကို ဆက်သွယ် ချိတ်ဆက် ရန် $HOME/.kube directory အောက် တွင် ရှိသောက် config file များကို ဖတ်သည်။ environment variable KUBECONFIG ကိုလဲ config file အစား အသုံး ပြုနိုင်သည်။

Let's SETUP

Kops ကို ဒီ link တွင် download ပြုလုပ်နိုင်ပါသည်။

Terminal

$ wget https://github.com/kubernetes/kops/releases/download/1.14.0-alpha.2/kops-linux-amd64
$ chmod +x kops-linux-amd64
$ sudo mv kops-linux-amd64 /usr/local/bin/kops

Kops ကို အသုံးပြုရန် AWS ၏ IAM user တစ်ယောက် လိုအပ်ပါသည်။ ထို IAM user တွင် Kops မှ AWS ၏ resource များကို အသုံးပြုရန် အနည်းဆုံး အောက်ပါ permission များ လိုအပ်ပါသည်။

• AmazonEC2FullAccess

• AmazonRoute53FullAccess

• AmazonS3FullAccess

• IAMFullAccess

• AmazonVPCFullAccess

လိုအပ်သော Permission နှင့် User တစ်ယောက်ဆောက်ပြီးသည့်အခါတွင် အောက်ဖော်ပြပါ Command နဲ့ AWS CLI ကို Setup လုပ်ပေးရမည်။

Terminal

$ aws configure

AWS CLI ကို setup လုပ်ပြီသောအခါ AWS ၏ S3 bucket တစ်ကို kops state file များ သိမ်းရန် အောက်ပါ command များကို အသုံးပြု၍ တည်ဆောက်နိုင်ပါသည်။

Terminal

$ export REGION=<your desire region>
$ export NAME=<your desire cluster name>
$ export KOPS_STATE_STORE=s3://${NAME} #your s3 bucket name
$ aws s3 mb $KOPS_STATE_STORE # create s3 bucket with your cluster name
$ aws s3api put-bucket-versioning --bucket ${NAME}  --versioning-configuration Status=Enabled
$ aws s3api put-bucket-encryption --bucket ${NAME} --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'

AWS S3 တည်ဆောက်ပြီသောအခါ မှာကျွန်တော်တို့ လိုချင်သည့် ပုံစံဖြင့် kubernetes cluster ကို တည်ဆောက်ရန် အဆင်သင့် ဖြစ်နေပြီပဲ ဖြစ်ပါသည်။ Kops ကို အသုံးပြု၍ private နဲ့ public topology ၂ မျိုးဖြင့် cluster ကို တည်ဆောက် နိုင်ပါသည်။

ဒီ scenario မှာတော့ private topology ကျွန်တော် အသုံးပြုသွားမှာ ဖြစ်သည်။ t3.medium instance type နှင့် weave network plugin ကို အသုံးပြုပြီး masters ၃ လုံး၊ nodes ၃လုံးကို အသုံးပြု၍တည်ဆောက်သွားမည်ဖြစ်ပါသည်။

Kops supports the following topologies on AWS

Topology	Value	Description
Public Cluster	public	All masters/nodes will be launched in a public subnet in the VPC
Private Cluster	private	All masters/nodes will be launched in a private subnet in the VPC

ထို.နောက် အောက်ဖော်ပြပါ Command နဲ့ လိုအပ်သော setup များဖြင့် cluster ကိုတည်ဆောက်ပြီး၊ထို cluster ရဲ့ config များကို ပြန်အသုံးပြုနိုင်ရန် mycluster.yml ဖိုင် ထဲကို standard output ဖြင့်ထည့်လိုက်ပါမယ်။

Terminal

$ kops create cluster ${NAME} --state=${KOPS_STATE_STORE} --master-count 3 --master-size t3.medium --node-count 3 --node-size t3.medium --zones us-east-1a,us-east-1b,us-east-1c --topology private --networking weave --ssh-public-key ~/.ssh/id_rsa.pub  -o yaml >> mycluster.yaml

ထို.နောက် mycluster.yml ဖိုင်ကို edit လုပ်ပီး additional policies section တွင် AWS IAM Policy မှ ELB and Route 53 policy များကိုပြင်ဆင်ပြီးနောက် Cluster Authentication အတွက် Line number ၁၀ နှင့် ၁၁ တွင်ရှိသော အောက်ဖော်ပြပါ authentication:aws:{} ကိုထည့်ပေးရန်လိုအပ်ပါသည်။

AWS IAM နှင့် Authentication ကို Integrate မလုပ်ချင်ပါက ယခု Config ကို ပြင်ဆင်စရာမလိုအပ်ပါ။ပြီးတော့ aws-iam-authenticator ကို ထည့်သွင်းပေးရန်မလိုပါ။

mycluster.yaml

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: null
  name: <your cluster name> # no need to touch
spec:
  api:
    loadBalancer:
      type: Public
  authentication:      #need to add this [authenication]
    aws: {}            #need to add this [aws: {}]
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://<your s3 name>/<your cluster name> #no need to touch
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: master-ap-southeast-1a
      name: a
    - instanceGroup: master-ap-southeast-1b
      name: b
    - instanceGroup: master-ap-southeast-1c
      name: c
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: master-ap-southeast-1a
      name: a
    - instanceGroup: master-ap-southeast-1b
      name: b
    - instanceGroup: master-ap-southeast-1c
      name: c
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.14.1
  masterPublicName: api.yourcluster.com  #no need to touch
  networkCIDR: 172.20.0.0/16
  networking:
    weave:
      mtu: 8912
  nonMasqueradeCIDR: 100.64.0.0/10
  additionalPolicies:     #add this additional policy from this line to until line 160
    node: |
      [
        {
          "Effect": "Allow",
          "Action": [
            "acm:DescribeCertificate",
            "acm:ListCertificates",
            "acm:GetCertificate"
          ],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "ec2:AuthorizeSecurityGroupIngress",
            "ec2:CreateSecurityGroup",
            "ec2:CreateTags",
            "ec2:DeleteTags",
            "ec2:DeleteSecurityGroup",
            "ec2:DescribeInstances",
            "ec2:DescribeInstanceStatus",
            "ec2:DescribeSecurityGroups",
            "ec2:DescribeSubnets",
            "ec2:DescribeTags",
            "ec2:DescribeVpcs",
            "ec2:ModifyInstanceAttribute",
            "ec2:ModifyNetworkInterfaceAttribute",
            "ec2:RevokeSecurityGroupIngress"
          ],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "elasticloadbalancing:AddTags",
            "elasticloadbalancing:CreateListener",
            "elasticloadbalancing:CreateLoadBalancer",
            "elasticloadbalancing:CreateRule",
            "elasticloadbalancing:CreateTargetGroup",
            "elasticloadbalancing:DeleteListener",
            "elasticloadbalancing:DeleteLoadBalancer",
            "elasticloadbalancing:DeleteRule",
            "elasticloadbalancing:DeleteTargetGroup",
            "elasticloadbalancing:DeregisterTargets",
            "elasticloadbalancing:DescribeListeners",
            "elasticloadbalancing:DescribeLoadBalancers",
            "elasticloadbalancing:DescribeLoadBalancerAttributes",
            "elasticloadbalancing:DescribeRules",
            "elasticloadbalancing:DescribeSSLPolicies",
            "elasticloadbalancing:DescribeTags",
            "elasticloadbalancing:DescribeTargetGroups",
            "elasticloadbalancing:DescribeTargetGroupAttributes",
            "elasticloadbalancing:DescribeTargetHealth",
            "elasticloadbalancing:ModifyListener",
            "elasticloadbalancing:ModifyLoadBalancerAttributes",
            "elasticloadbalancing:ModifyRule",
            "elasticloadbalancing:ModifyTargetGroup",
            "elasticloadbalancing:ModifyTargetGroupAttributes",
            "elasticloadbalancing:RegisterTargets",
            "elasticloadbalancing:RemoveTags",
            "elasticloadbalancing:SetIpAddressType",
            "elasticloadbalancing:SetSecurityGroups",
            "elasticloadbalancing:SetSubnets",
            "elasticloadbalancing:SetWebACL"
          ],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "iam:GetServerCertificate",
            "iam:ListServerCertificates"
          ],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "waf-regional:GetWebACLForResource",
            "waf-regional:GetWebACL",
            "waf-regional:AssociateWebACL",
            "waf-regional:DisassociateWebACL"
          ],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "tag:GetResources",
            "tag:TagResources"
          ],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "waf:GetWebACL"
          ],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "route53:*"
          ],
          "Resource": "*"
        }
      ]
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 172.20.32.0/19
    name: ap-southeast-1a
    type: Private
    zone: ap-southeast-1a
  - cidr: 172.20.64.0/19
    name: ap-southeast-1b
    type: Private
    zone: ap-southeast-1b
  - cidr: 172.20.96.0/19
    name: ap-southeast-1c
    type: Private
    zone: ap-southeast-1c
  - cidr: 172.20.0.0/22
    name: utility-ap-southeast-1a
    type: Utility
    zone: ap-southeast-1a
  - cidr: 172.20.4.0/22
    name: utility-ap-southeast-1b
    type: Utility
    zone: ap-southeast-1b
  - cidr: 172.20.8.0/22
    name: utility-ap-southeast-1c
    type: Utility
    zone: ap-southeast-1c
  topology:
    dns:
      type: Public
    masters: private
    nodes: private

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: null
  labels:
    kops.k8s.io/cluster: hearu.org
  name: master-ap-southeast-1a
spec:
  image: kope.io/k8s-1.12-debian-stretch-amd64-hvm-ebs-2019-05-13
  machineType: t3.medium
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-ap-southeast-1a
  role: Master
  rootVolumeSize: 60
  rootVolumeType: gp2
  subnets:
  - ap-southeast-1a

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: null
  labels:
    kops.k8s.io/cluster: yourcluster.com # no need to touch
  name: master-ap-southeast-1b
spec:
  image: kope.io/k8s-1.12-debian-stretch-amd64-hvm-ebs-2019-05-13
  machineType: t3.medium
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-ap-southeast-1b
  role: Master
  rootVolumeSize: 60
  rootVolumeType: gp2
  subnets:
  - ap-southeast-1b

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: null
  labels:
    kops.k8s.io/cluster: yourcluster.com #no need to touch
  name: master-ap-southeast-1c
spec:
  image: kope.io/k8s-1.12-debian-stretch-amd64-hvm-ebs-2019-05-13
  machineType: t3.medium
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-ap-southeast-1c
  role: Master
  rootVolumeSize: 60
  rootVolumeType: gp2
  subnets:
  - ap-southeast-1c

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: null
  labels:
    kops.k8s.io/cluster: yourcluster.com #no need to touch
  name: nodes
spec:
  image: kope.io/k8s-1.12-debian-stretch-amd64-hvm-ebs-2019-05-13
  machineType: t3.medium
  maxSize: 3
  minSize: 3
  nodeLabels:
    kops.k8s.io/instancegroup: nodes
  role: Node
  rootVolumeSize: 60
  rootVolumeType: gp2
  subnets:
  - ap-southeast-1a
  - ap-southeast-1b
  - ap-southeast-1c

အားလုံးပြင်ဆင်ပြီးနောက် cluster ကို အောက်ပါ command များဖြင့် bootstrap လုပ်နိုင်ပါပြီ။ Bootstrap လုပ်နေချိန်တွင် မိနစ်အနည်းငယ် စောင့်ရန်လိုအပ်ပါသည်။

Terminal

$ kops create -f mycluster.yaml 
$ kops create secret --name ${NAME} sshpublickey admin -i ~/.ssh/id_rsa.pub
$ kops update cluster ${NAME} --yes
$ kops kops validate cluster # validate cluster is check for your cluster state

Cluster တည်ဆောက်ပြီးဖြစ်သော်လည်း Status ကတော့ valid ဖြစ်သေးမှာမဟုတ်ပါဘူး။ ဘာလို့လဲဆိုတော့ authentication plugin ၏ configuration လိုနေသောကြောင့်ဖြစ်ပါသည်။

json

//K8sAdmin
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "sts:DecodeAuthorizationMessage",
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "sts:*",
            "Resource": "arn:aws:iam::<account id>:role/K8sAdmin"
        }
    ]
}

json

//K8sDeveloper
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::<account id:role/K8sDeveloper"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "sts:GetCallerIdentity",
            "Resource": "*"
        }
    ]
}

အောက်တွင်ဖော်ပြထားသော commands ဖြင့် K8sDeveloper နှင့် K8sAdmin အစရှိသော role နှစ်ခုကို configmap အတွက် တည်ဆောက်သည့်အခါတွင် အပေါ် မှာ json များထွက်လာမည်ဖြစ်ပါသည်။

Example ပြထားခြင်းဖြစ်ပါသည်။

Terminal

$ export ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account')
$ export DEVPOLICY=$(echo -n '{"Version":"2012-10-17","Statement":[{"Sid":"VisualEditor0","Effect":"Allow","Action":["sts:DecodeAuthorizationMessage","sts:GetCallerIdentity"],"Resource":"*"},{"Sid":"VisualEditor1","Effect":"Allow","Action":"sts:*","Resource":"arn:aws:iam::'; echo -n "$ACCOUNT_ID"; echo -n ':role/K8sDeveloper"}]}')
$ aws iam create-role \
  --role-name K8sDeveloper \
  --description "Kubernetes developer role (for AWS IAM Authenticator for AWS)." \
  --assume-role-policy-document "$DEVPOLICY" \
  --output text \
  --query 'Role.Arn'
$ export ADMPOLICY=$(echo -n '{"Version":"2012-10-17","Statement":[{"Sid":"VisualEditor0","Effect":"Allow","Action":["sts:DecodeAuthorizationMessage","sts:GetCallerIdentity"],"Resource":"*"},{"Sid":"VisualEditor1","Effect":"Allow","Action":"sts:*","Resource":"arn:aws:iam::'; echo -n "$ACCOUNT_ID"; echo -n ':role/K8sAdmin"}]}')
$ aws iam create-role \
  --role-name K8sAdmin \
  --description "Kubernetes administrator role (for AWS IAM Authenticator for AWS)." \
  --assume-role-policy-document "$ADMPOLICY" \
  --output text \
  --query 'Role.Arn'

ကျွန်တော်တို. IAM role နှင့် policies များကိုတည်ဆောက်ပြီးနောက် Kubernets resources များတွက် authorization ကို configure ချပေးရန်လိုအပ်ပါသည်။

ထို.နောက် RBAC authorization ကို အသုံးပြုပြီး developer role အတွက် production နှင့် development namespace နှစ်ခုကို configure ချပေးထားပါသည်။

K8sDeveloper role အတွင်းရှိသော user များသည် ထို namespace နှစ်ခုနှင့် resource အချို.သာလျင် အသုံးပြုနိုင်မည်ဖြစ်သည်။

developer-rbac.yaml

# RBAC Policy for developer
# create 2 Roles on for developer and binding roles
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  namespace: development
  name: developer
rules:
- apiGroups: ["*"]
  resources: ["deployments","configmaps","pods","secrets","services","endpoints","persistentvolumeclaims","events","namespaces","statefulsets","replicasets","jobs","cronjobs"]
  verbs: ["create","get","delete","list","update","edit","watch","exec","patch"]
- apiGroups:
  - '*'
  resources:
  - 'pods/exec'
  - 'pods/portforward'
  - 'services/portforward'
  verbs:
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: developer
  namespace: development
roleRef:
  kind: Role
  name: developer
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  namespace: development
  name: k8s.developer
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  namespace: production
  name: developer
rules:
- apiGroups: ["*"]
  resources: ["deployments","configmaps","pods","secrets","services","endpoints","persistentvolumeclaims","events","namespaces","statefulsets","replicasets","jobs","cronjobs"]
  verbs: ["create","get","delete","list","update","edit","watch","exec","patch"]
- apiGroups:
  - '*'
  resources:
  - 'pods/exec'
  - 'pods/portforward'
  - 'services/portforward'
  verbs:
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: developer
  namespace: production
roleRef:
  kind: Role
  name: developer
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  namespace: production
  name: k8s.developer

Terminal

$ kubectl create namespace development
$ kubectl create namespace production
$ kubectl create -f developer-rbac.yaml

aws-iam-authenticator အလုပ်လုပ်ရန် နင့် သတ်မှတ်ထားသည့် user group policy များ အသက်ဝင်ရန် အောက်ပါ configmap ကို create ပြုလုပ်ပေးရန်လိုအပ်ပါသည်။

aws-auth.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kube-system
  name: aws-iam-authenticator
  labels:
    k8s-app: aws-iam-authenticator
data:
  config.yaml: |    
    clusterID: api.mycluster.com 
    server:
      mapRoles:
      - roleARN: arn:aws:iam::<account id>:role/K8sAdmin
        username: kubernetes-admin
        groups:
        - system:masters
      - roleARN: arn:aws:iam::<account id>:role/K8sDeveloper
        username: kubernetes-developer
        groups:
        - k8s.developer

Terminal

$ kubectl create -f aws-auth.yaml

ယခုအခြေအနေတွင် Kubernetes cluster နှင့် authentication/authorization setup ပြုလုပ်ခြင်းမှာ ပြီးမြောက်လုနီးပါးဖြစ်ပါသည်။ Kubernetes client (kubectl) မှ cluster ကို access လုပ်ရန် configure ချဖို. လိုအပ်ပါသည်။ အောက်ဖော်ပြပါ config နှစ်ခုသည် Developer နှင့် Administrator အတွက် ဖြစ်ပါသည်။ ကျွန်တော် ကတော့ Admin config အသုံးပြုပြီး AWS Application LoadBalancer ကိုဆက်လက် setup လုပ်သွားမှာ ဖြစ်ပါသည်။

~/kube/.config (Admin)

# this is for Kubernetes Administrator
# you must change this the value of 'certificate-authority-data' with your ca.
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMwekNDQWJ1Z0F3SUJBZ0lNRmFlcHBua0lEUnRwNkpub01BMEdDU3FHU0liM0RRRUJDd1VBTUJVeEV6QVIKQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13SGhjTk1Ua3dOakV4TURVd05USTRXaGNOTWprd05qRXdNRFV3TlRJNApXakFWTVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBCk1JSUJDZ0tDQVFFQXpmMzlOOUIvaGpUQnViM0tETEhzM1NJUmhVL2pVcUt2RWhTWnpCQngwR0tXNmtONmRUVkkKK3dwUnRoeU9Yd2ZWRHA4RmxRYUFYbWt0UGtGTG9jU2VSelBUWXRod1VBamhSbVNoZFJtcW1yaXBZTlhFdEZYMAp6SkZFS2NRNDJ0TEhYanZZbVc3SGxYc0FDSDRCKy82SEtwVmxGSVdNTDAwb2tTUmEzZVBvdTdWYzlqTkZhd3R0CmFoMThrNStLcHVoL0FyUEs2dmdhcEhLclB2dUNaZG9FWkNtaE4rSVdZTjN6Uk1Wb1A1TEVoTEx5RFFzUzYvN0sKQ3prQ09Pam1QWW01UmxnOVRvb25ZcStwbS9LMXc4V1NLVnZick9VZXF0dXkxUlJYWUxtc0VNVmx0NDRQSS9YTgpEeUlubmtpVmRwc1A4ejAwRjZ5MGEwYXFpT3RGUVVyd3NRSURBUUFCb3lNd0lUQU9CZ05WSFE4QkFmOEVCQU1DCkFRWXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFqUm5ndnJ6SmRMWm8KRFdlbVFaMi83N0xLTmkzTmpRQU9NZUg3WXVpZnlNbTZNdTB2V3F6UU54SDJ1TGhFd0JRN0VocEpYSnFoMmp0dQpldnR6YUF6N2o5MEVKdlN6WHRtSDRQaHd1NFZFMHJWRGVlbUhrMXRZRjhsQ3hVM2xtZFRGZjJ4M29qVnVKNndpCkd3c3RaU0s5V2J5VW53RjNLUXdkMndmd3VUVVpSRGtuSkI2ZGhOcUxNakN3V1M1ekpDUjcvYXFuM0tqb0xvWEgKWkI1VW1UcGd2Sm9rWm0wcUF0ODRNS0hFb2ZCYXJ0M1BUcnplWGI0N1VHRlMrM0h1anVQUTBreW1DT0crK1ZEMwo0K0M5eVlXaTRhYnd6SE9rTmQvMlFKZVoyQjhjbjhlVU5kUGFSSnkweS9keVpSd0NZdXFKSkxZeTVGUTFiQlUwCm96bThnMG01ZWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://api.yourcluster.com
  name: yourcluster.com
contexts:
- context:
    cluster: yourcluster.com
    user: kubernetes-admin
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - api.yourcluster.com
      - -r
      - arn:aws:iam::<account id>:role/K8sAdmin
      command: aws-iam-authenticator
      env: null

~/kube/.config (Developer)

# this is for Kubernetes Developer
# you must change this the value of 'certificate-authority-data' with your ca.
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMwekNDQWJ1Z0F3SUJBZ0lNRmFlcHBua0lEUnRwNkpub01BMEdDU3FHU0liM0RRRUJDd1VBTUJVeEV6QVIKQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13SGhjTk1Ua3dOakV4TURVd05USTRXaGNOTWprd05qRXdNRFV3TlRJNApXakFWTVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBCk1JSUJDZ0tDQVFFQXpmMzlOOUIvaGpUQnViM0tETEhzM1NJUmhVL2pVcUt2RWhTWnpCQngwR0tXNmtONmRUVkkKK3dwUnRoeU9Yd2ZWRHA4RmxRYUFYbWt0UGtGTG9jU2VSelBUWXRod1VBamhSbVNoZFJtcW1yaXBZTlhFdEZYMAp6SkZFS2NRNDJ0TEhYanZZbVc3SGxYc0FDSDRCKy82SEtwVmxGSVdNTDAwb2tTUmEzZVBvdTdWYzlqTkZhd3R0CmFoMThrNStLcHVoL0FyUEs2dmdhcEhLclB2dUNaZG9FWkNtaE4rSVdZTjN6Uk1Wb1A1TEVoTEx5RFFzUzYvN0sKQ3prQ09Pam1QWW01UmxnOVRvb25ZcStwbS9LMXc4V1NLVnZick9VZXF0dXkxUlJYWUxtc0VNVmx0NDRQSS9YTgpEeUlubmtpVmRwc1A4ejAwRjZ5MGEwYXFpT3RGUVVyd3NRSURBUUFCb3lNd0lUQU9CZ05WSFE4QkFmOEVCQU1DCkFRWXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFqUm5ndnJ6SmRMWm8KRFdlbVFaMi83N0xLTmkzTmpRQU9NZUg3WXVpZnlNbTZNdTB2V3F6UU54SDJ1TGhFd0JRN0VocEpYSnFoMmp0dQpldnR6YUF6N2o5MEVKdlN6WHRtSDRQaHd1NFZFMHJWRGVlbUhrMXRZRjhsQ3hVM2xtZFRGZjJ4M29qVnVKNndpCkd3c3RaU0s5V2J5VW53RjNLUXdkMndmd3VUVVpSRGtuSkI2ZGhOcUxNakN3V1M1ekpDUjcvYXFuM0tqb0xvWEgKWkI1VW1UcGd2Sm9rWm0wcUF0ODRNS0hFb2ZCYXJ0M1BUcnplWGI0N1VHRlMrM0h1anVQUTBreW1DT0crK1ZEMwo0K0M5eVlXaTRhYnd6SE9rTmQvMlFKZVoyQjhjbjhlVU5kUGFSSnkweS9keVpSd0NZdXFKSkxZeTVGUTFiQlUwCm96bThnMG01ZWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://api.yourcluster.com
  name: yourcluster.com
contexts:
- context:
    cluster: yourcluster.com
    namespace: development  #you will notiace this. This is means developere default namesapce is development.
    user: kubernetes-developer
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: kubernetes-developer
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - api.yourcluster.com
      - -r
      - arn:aws:iam::<account id>:role/K8sDeveloper
      command: aws-iam-authenticator
      env: null

Terminal

$ wget https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v0.4.0/aws-iam-authenticator_0.4.0_linux_amd64
$ chmod +x aws-iam-authenticator_0.4.0_linux_amd64
$ sudo mv aws-iam-authenticator_0.4.0_linux_amd64 /usr/local/bin/aws-iam-authenticator
$ aws-iam-authenicator version
{"Version":"0.4.0","Commit":"9999cd8047782c94c08e2d24b77dc9488947aa03"}
$ kubectl get nodes # test what you authorized 

ထို့နောက် AWS Application LoadBalancer ကို Kubernetes Ingress Controller နဲ့ integrate ပြုလုပ်ရန် အောက်ဖော်ပြပါအတိုင်း RBAC နှင့် alb-ingress-controller deployment ကို kube-system တွင် create လုပ်ရမည်ဖြစ်ပါသည်။ လိုအပ်သော additional policy များကို cluster bootstrap လုပ်ခဲ့တုန်း ကထည့်သွင်းခဲ့ပြီးဖြစ်ပါသည်။

alb-rbac.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: alb-ingress-controller
  name: alb-ingress-controller
rules:
  - apiGroups:
      - ""
      - extensions
    resources:
      - configmaps
      - endpoints
      - events
      - ingresses
      - ingresses/status
      - services
    verbs:
      - create
      - get
      - list
      - update
      - watch
      - patch
  - apiGroups:
      - ""
      - extensions
    resources:
      - nodes
      - pods
      - secrets
      - services
      - namespaces
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: alb-ingress-controller
  name: alb-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: alb-ingress-controller
subjects:
  - kind: ServiceAccount
    name: alb-ingress
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: alb-ingress-controller
  name: alb-ingress
  namespace: kube-system

alb-ingress-controller.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: alb-ingress-controller
  name: alb-ingress-controller
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alb-ingress-controller
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: alb-ingress-controller
    spec:
      containers:
        - args:
            - --ingress-class=alb
            - --cluster-name=yourcluster.com
          env:
          image: 894847497797.dkr.ecr.us-west-2.amazonaws.com/aws-alb-ingress-controller:v1.0.0
          imagePullPolicy: Always
          name: server
          resources: {}
          terminationMessagePath: /dev/termination-log
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
      serviceAccountName: alb-ingress
      serviceAccount: alb-ingress

Terminal

# setup with your administrator account.
$ kubectl create -f alb-rbac.yaml
$ kubectl create -f alb-ingress-controller.yaml
$ kubectl get pod -n kube-system 

Ingress controller setup လုပ်ပီးနောက် external DNS အတွက် DNS records များကို Ingress create လုပ်လိုက်သည်နှင့် တပြိုင်နက်ထဲ AWS route 53 တွင် automatically update ပြုလုပ်သွားရန် အောက်ဖော်ပြပါ service account, rbac for external dns, deployment များကို ဆက်လက်တည်ဆောက်ရမည်ဖြစ်သည်။

external-dns.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions"] 
  resources: ["ingresses"] 
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: registry.opensource.zalan.do/teapot/external-dns:v0.5.9
        args:
        - --source=service
        - --source=ingress
        - --domain-filter=yourcluster.com # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones
        - --provider=aws
        - --policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization
        - --aws-zone-type=public # only look at public hosted zones (valid values are public, private or no value for both)
        - --registry=txt
        - --txt-owner-id=my-identifier

Terminal

$ kubectl create -f external-dns.yaml

ယခုအခြေအနေတွင်အားလုံးတည်ဆောက်ပြီး ဖြစ်ပါသည်။

တည်ဆောက်ခဲ့သည့် Cluster, IAM authentication and k8s RBAC, ALB ingress and automatic external DNS service များကို စမ်းသပ်ရန်အတွက် အောက်ဖော်ပြပါ 2048 web application game ဖြင့်စမ်းသပ်နိုင်ပါသည်။

Terminal

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.0.0/docs/examples/2048/2048-namespace.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.0.0/docs/examples/2048/2048-deployment.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.0.0/docs/examples/2048/2048-service.yaml
$ kubectl get po,deploy,svc -n 2048-game

2048-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: "2048-ingress"
  namespace: "2048-game"
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
  labels:
    app: 2048-ingress
spec:
  rules:
  - host: yourcluster.com 
    http:
      paths:
      - path: /*
        backend:
          serviceName: "service-2048"
          servicePort: 80

Terminal

$ kubectl create -f 2048-ingress.yaml
$ kubectl get ingress -n 2048-game
$ dig yourcluster.com # see and wait
$ google-chrome-stable yourcluster.com

ယခု ကျွန်တော်တို. လုပ်ပြထားသော tutorial လေးသည် Non-Profit Organization တစ်ခုအတွက် ကျွန်တော်တို. Setup လုပ်ပေးခဲ့သော Production Grade Kops Cluster Setup Use-case မှ ကောက်နုတ်ဖော်ပြထားခြင်းဖြစ်သည့်အတွက် စာဖတ်သူတို.စိတ်ထဲတွင် မရှင်းမလင်းတစ်ခုခုခံစားခဲ့ရရင်ကျွန်တော်တို.ရဲ့ Facebook Group or Page ကိုဆက်သွယ်မေးမြန်းနိုင်ပါသည်။