K
Q

Desired GKE pod not found , google cloud composer

4/1/2019

I am using Google cloud composer ,and created composer environment.Composer environment is ready(has green tick), now I am trying to set variables used in DAG python code using google cloud shell.

command to set variables:

     gcloud composer environments run test-environment \
       --location us-central1 variables -- \
       --set gcp_project xxx-gcp

Exact error message:

  ERROR: (gcloud.composer.environments.run) Desired GKE pod not found. If the environment was recently started, please wait and retry.

I tried following things as part of investigation, but got same error each time. I have created a new environment using UI and not google shell commands. I checked pods in kubernetes engine and all are green , did not see any issue. I verified composer API, Billing kubernetes, all required API's are enabled.

I have 'Editor' role assigned.

added screenshot I saw first time some failures

enter image description here

enter image description here

Error with exit code 1 google troubleshooting guide describe: If the exit code is 1, the container crashed because the application crashed.

-- Tokci
airflow
google-cloud-composer
google-cloud-platform
google-kubernetes-engine
kubernetes

Similar Questions

What are the benefits of enabling mTLS in Istio?
What is the deployment controller sync period for kube-controller-manager?
Horizontal Pod Autoscaler (HPA): Current utilization: <unknown> with custom namespace
Kubernetes python client: authentication problem
Why does kubectl cp command terminates with exit code 126?

1 Answer

4/2/2019

This is a side effect of Composer version 1.6.0 if you are using a google-cloud-sdk that is too old, because it now launches pods in namespaces other than default. The error you see is a result of looking for Kubernetes pods in the default namespace and failing to find them.

To fix this, run gcloud components update. If you cannot yet update, a workaround to execute Airflow commands is to manually SSH to a pod yourself and run airflow. To start, obtain GKE cluster credentials:

$ gcloud container clusters get-credentials $COMPOSER_GKE_CLUSTER_NAME

Once you have the credentials, you should find which namespace the pods are running in (which you can also find using Cloud Console):

$ kubectl get namespaces
NAME                                    STATUS   AGE
composer-1-6-0-airflow-1-9-0-6f89fdb7   Active   17h
default                                 Active   17h
kube-public                             Active   17h
kube-system                             Active   17h

You can then SSH into any scheduler/worker pod, and run commands:

$ kubectl exec \
    --namespace=$NAMESPACE \
    -it airflow-worker-569bc59df5-x6jhl airflow list_dags -r

You can also open a shell if you prefer:

$ kubectl exec \
    --namespace=$NAMESPACE \
    -it airflow-worker-569bc59df5-x6jhl bash

airflow@airflow-worker-569bc59df5-x6jhl:~$ airflow list_dags -r

The failed airflow-database-init-job jobs are unrelated and will not cause problems in your Composer environment.

-- hexacyanide
Source: StackOverflow