K
Q

Sparklyr k8s gClouid - Pods aren't running

2/4/2020

everybody
As the title says, I would be curious to understand why I can't connect via Sparklyr to gCloud clusters using kubernetes.

The steps to configure the system were as follows:

  • project creation on gCloud (free version 300dollars)
  • Cloud SDK installation on MacOS
  • Installation Kubectl binary with curl on MacOS
  • Docker installation
  • By terminal:
  • configure the cluster
    gcloud config set compute/zone us-central1-f gcloud container clusters create spark-on-gke --machine-type n1-standard-2
  • bind the cluster admin to email
    kubectl create clusterrolebinding user-admin-binding --clusterrole=cluster-admin --user=pesca@gmail.com kubectl create clusterrolebinding --clusterrole=cluster-admin --serviceaccount=default:default spark-admin
  • From R connect to the MASTER_IP, importing the public image offered by jluraschi
    remotes::install_github("rstudio/sparklyr”); library(sparklyr) sc <- spark_connect(config = spark_config_kubernetes( "k8s://https://<k8s-ip>", account = "default", image = "docker.io/jluraschi/spark:sparklyr", version = "2.4"))

And the error that appears is: Error from server (NotFound): pods "sparklyr-c27317e4b89" not found

-- pofferbacco
apache-spark
gcloud
kubernetes
r
sparklyr

Similar Questions

How to initialize a master in SparkConf in order to run distributed on a k8s cluster?
Proper Setup of Datadog Log Ingestion on Kubernetes
DNSJava in Kubernetes
What route do service requests pass through Kubernetes?

1 Answer

2/18/2020

Thank you so much for your answer!
At the end of the code you will also find screenshots of the pod configuration made with gcloud. Instead, here are the results from the terminal:

MBP-di-Simone:~ simone$ kubectl get services
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.3.240.1   <none>        443/TCP   13d

MBP-di-Simone:~ simone$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                                     
READY   STATUS    RESTARTS   AGE
default       alpine                                                   1/1     Running   0          13d
kube-system   event-exporter-v0.2.4-5f88c66fb7-67pb6                   2/2     Running   0          13d
kube-system   fluentd-gcp-scaler-59b7b75cd7-mbgxj                      1/1     Running   0          13d
kube-system   fluentd-gcp-v3.2.0-9dlx8                                 2/2     Running   0          5d8h
kube-system   fluentd-gcp-v3.2.0-9w6t2                                 2/2     Running   0          5d8h
kube-system   fluentd-gcp-v3.2.0-dwrlz                                 2/2     Running   0          5d8h
kube-system   heapster-5f6cdd4bd-qmlhb                                 3/3     Running   0          13d
kube-system   kube-dns-79868f54c5-5sqvb                                4/4     Running   0          13d
kube-system   kube-dns-79868f54c5-g9h4q                                4/4     Running   0          13d
kube-system   kube-dns-autoscaler-bb58c6784-9bbcg                      1/1     Running   0          13d
kube-system   kube-proxy-gke-spark-on-gke-default-pool-7fad1be1-2279   1/1     Running   0          13d
kube-system   kube-proxy-gke-spark-on-gke-default-pool-7fad1be1-70hn   1/1     Running   0          13d
kube-system   kube-proxy-gke-spark-on-gke-default-pool-7fad1be1-pnpj   1/1     Running   0          13d
kube-system   l7-default-backend-fd59995cd-8tzjv                       1/1     Running   0          13d
kube-system   metrics-server-v0.3.1-57c75779f-gz776                    2/2     Running   0          13d
kube-system   prometheus-to-sd-ktvbk                                   2/2     Running   0          13d
kube-system   prometheus-to-sd-tmwkw                                   2/2     Running   0          13d
kube-system   prometheus-to-sd-xxx4p                                   2/2     Running   0          13d

MBP-di-Simone:~ simone$ kubectl describe pods [sparklyr-2e62d04d5dd]
Error from server (NotFound): pods "[sparklyr-2e62d04d5dd]" not found
MBP-di-Simone:~ simone$

gCloud Cluster a
gCloud Cluster b
gCloud Cluster permission
gCloud Cluster label

-- pofferbacco
Source: StackOverflow