K
Q

Question

kubernetes and debugging it in general

7/13/2016

I have a kubernetes cluster setup in AWS. We are using the EC2 container registry to store our docker images on. We have the master/minions all setup and everything seems to be working with the cluster.

My spec file is as followed:

apiVersion: v1
kind: Service
metadata:
  name: apim-mysql
  labels:
    app: apim
spec:
  ports:
    # the port that this service should serve on
    - port: 3306
  selector:
    app: apim
    tier: mysql
  clusterIP: None
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: apim-mysql
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: apim
        tier: mysql
    spec:
      imagePullSecrets:
        - name: myregistrykey
      containers:
      - name: mysql
        image: <This points to our EC2 Container Registry and retreives the image>
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        env:
        - name: WSO2_ZIP_FILE
          value: wso2am-1.10.1-SNAPSHOT.zip
        - name: WSO2_VERSION
          value: 1.10.1
        - name: MYSQL_RANDOM_ROOT_PASSWORD
          value: 'yes'
        - name: MYSQL_USER
          value: username
        - name: MYSQL_USER_PASSWD
          value: password
        - name: GET_HOSTS_FROM
          value: dns
          # If your cluster config does not include a dns service, then to
          # instead access environment variables to find service host
          # info, comment out the 'value: dns' line above, and uncomment the
          # line below.
          #value: env
        ports:
        - containerPort: 3306
          name: mysql

What this container does is just sets up a mysql. We will need other nodes within the cluster to connect to this node. Because they will need to use the mysql DB.

I guess my first question is does everything look okay with this spec file? Or does anyone see something wrong?

I do the kubectl create command and it runs successfully:

kubectl create -f mysql.yaml
service "apim-mysql" created
deployment "apim-mysql" created

kubectl get pods shows the pod running:

apim-mysql-545962574-w2qz1        1/1       Running            1          8m

I sometimes when doing kubectl logs get an error showing this:

kubectl logs apim-mysql-545962574-w2qz1
Error from server: dial unix /var/run/docker.sock: no such file or directory

But eventually retrying enough it will go through... if anyone has information on why that occurs it would be great.

When it does work get something like this:

kubectl logs apim-mysql-545962574-w2qz1
Initializing database
2016-07-13T15:51:47.375052Z 0 [Warning] InnoDB: New log files created, LSN=45790
2016-07-13T15:51:52.029915Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2016-07-13T15:51:53.531183Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: b837bd45-4911-11e6-99ba-02420af40208.
2016-07-13T15:51:53.746173Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2016-07-13T15:51:53.746621Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2016-07-13T15:52:19.891437Z 1 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2016-07-13T15:52:19.891705Z 1 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2016-07-13T15:52:19.891733Z 1 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2016-07-13T15:52:19.891778Z 1 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2016-07-13T15:52:19.891831Z 1 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
Database initialized
MySQL init process in progress...
2016-07-13T15:52:34.632188Z 0 [Note] mysqld (mysqld 5.7.13) starting as process 49 ...
2016-07-13T15:52:49.814764Z 0 [Note] InnoDB: PUNCH HOLE support available
2016-07-13T15:52:49.814846Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2016-07-13T15:52:49.814859Z 0 [Note] InnoDB: Uses event mutexes
2016-07-13T15:52:49.814870Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2016-07-13T15:52:49.814928Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
2016-07-13T15:52:49.814932Z 0 [Note] InnoDB: Using Linux native AIO
2016-07-13T15:52:50.243657Z 0 [Note] InnoDB: Number of pools: 1
2016-07-13T15:52:52.175079Z 0 [Note] InnoDB: Using CPU crc32 instructions
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...
MySQL init process in progress...

After a little bit it the pod seems to restart and it will say Initializing database again..

A couple days ago when I ran kubectl logs it would return everything right away and be very fast now it is very slow and doesn't really show anything I have not really changed my spec file at all so I have no idea what is going on. To me it seems like the container isn't getting executed properly when it actually does show some logs... but I unsure.

If anyone has any clue on how to debug this further with some commands I can run it would be very appreciated. I am very stuck at this point and have google'd so much but with no luck.

thanks!

-- Alex Winter

amazon-web-services

containers

docker

kubernetes

mysql

1 Answer

7/17/2016

My experience using kubernetes on aws while getting unhelpful errors as

Error from server: dial unix /var/run/docker.sock: no such file or directory

was resolved by choosing a more hefty aws cluster instance type ... here are relevant env vars

export MASTER_SIZE=t2.medium
export NODE_SIZE=t2.medium
export NUM_NODES=2   #  if not defined aws will auto guess

... also remove mention of resource limiting settings under tag resources until after it runs OK

Following commands are essential ... just leave off mention of namespace if you are not using

kubectl describe svc --namespace=xxx

kubectl get pods --namespace=xxx

kubectl describe pods --namespace=xxx

kubectl describe nodes

Also nice is ability to perform a live edit of a deployment ... first see your deployments .. issue

kubectl get deployments  --namespace=ruptureofthemundaneplane

... output

NAME                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
loudspeed-deployment   1         1         1            1           1h
mongo-deployment       1         1         1            1           1h

so now we know name of deployment to do live edit issue

kubectl edit deployment/mongo-deployment

which will open an edit session in your terminal using default editor where you can change settings at will

I find when troubleshooting a database deployment its handy to also launch an image using below Dockerfile ... this allows you login using exec as per

kubectl exec -ti $(kubectl get pods --namespace=${PROJECT_ID}|grep ${GKE_NODEDEPLOYMENT}|cut -d' ' -f1) --namespace=${PROJECT_ID} -c ${GKE_NGINX} -- bash

where you are free to do interactive database login session (once you install needed client code or put same into below Dockerfile) ... here is matching Dockerfile for this troubleshooting deployment container

FROM ubuntu:16.04

ENV TERM linux
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update
RUN apt-get install -y wget  curl

COPY .bashrc /root/

# ENTRYPOINT ["/bin/bash"]
CMD ["/bin/bash"]


#
# docker build --tag stens_ubuntu .
#
# docker run -d  stens_ubuntu  sleep infinity
#
# docker ps
#
# 
# ... find CONTAINER ID from above and put into something like this
#
# docker exec -ti 3cea1993ed28 bash
#
#

-- Scott Stensland

Source: StackOverflow

KQ

kubernetes and debugging it in general

Similar Questions

1 Answer

K
Q