Zeppelin Notebooks and Interpreters Missing Behind Kubernetes Ingress

4/22/2020

I am running into an issue where the interpreters are missing after enabling the Kubernetes ingress to reach my Zeppelin container.

I'm running Zeppelin 0.8.2 on Kubernetes, and everything works correctly when it is not behind an ingress. I am able to choose a default interpreter in the notebook creation popup, and run paragraphs without issue.

When an ingress is setup in Kubernetes, previously created notebooks and interpreter selection during new notebook creation are missing. I am still able to reach the interpreter settings page, and it appears they are all still there.

Required change made in zeppelin-site.xml to run without the ingress.

<property>
  <name>zeppelin.server.addr</name>
  <value>0.0.0.0</value>
  <description>Server binding address</description>
</property>

By default, was 127.0.0.0

Changes made in shiro.ini based on this forum discussion: https://community.cloudera.com/t5/Support-Questions/How-do-I-recover-missing-Zeppelin-Interpreter-tab-and-its/m-p/160680

sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager 
cacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager 
securityManager.cacheManager = $cacheManager 
securityManager.sessionManager = $sessionManager 
securityManager.sessionManager.globalSessionTimeout = 86400000

Ingress used below with variable names changed

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-redirect: "true"
  name: zeppelin-name
  namespace: namespace_here
spec:
  rules:
  - host: zeppelin.zeppelin-services.my-host.org
    http:
      paths:
      - backend:
          serviceName: zeppelin-name
          servicePort: 8080
        path: /
  tls:
  - hosts:
    - zeppelin.zeppelin-services.my-host.org
    secretName: secret-name
status:
  loadBalancer: {}

What am I missing for Zeppelin to work correctly behind the Kubernetes Ingress, to allow me to see previously created notebooks, and interpreters present in the dropdown menu when creating new notebooks?

-- saturner
apache-zeppelin
kubernetes

1 Answer

4/23/2020

Coworker found the answer. It turns out the information was not reaching the zeppelin page. On inspection, there was a 405 error:

'wss://zeppelin.zeppelin-services.my-host.org/ws' failed: Error during WebSocket handshake: Unexpected response code: 405

He checked the logs on the nginx-ingress pod and found similar failure messages.

Issue was in the ingress yaml used and required an addition to the annotation

metadata:
  annotations:
    nginx.org/websocket-services: zeppelin-name
-- saturner
Source: StackOverflow