Apache Superset on Kubernetes -- properly installing Python dependencies for BigQuery

11/5/2021

I am trying to install Apache Superset with Helm on Kubernetes (GCP). I am updating the 'values.yaml' locally and then while Gcloud Auth'd and connected to K8 (kubectl) running this command:

   helm upgrade superset ~/Documents/superset/ --install -f ~/Documents/superset/values.yaml

My attempt to add PIP dep via 'values.yaml'

Then I run port forward to connect to the K8 Superset via my local browser:

kubectl port-forward service/superset :8088 --namespace default

It assigns a random port that I update in my browser and am able to access Superset with no problems. But I have no dependencies, and I cannot add my Database connection (and test) because when I check my Kubernetes logs I get this error:

Could not load database driver: BigQueryEngineSpec
WARNING:superset.views.base:Could not load database driver: BigQueryEngineSpec 

Conversely, I was actually able to add my JSON key as well as the proper SQLAlchemy forms in the UI when I installed Superset only on Docker, and connect to BigQuery no problem.

But with this K8 install, I did it thru the 'values.yaml'. So, I tried to add the content in

extraConfigs:
  datasources-init.yaml: |
      databases:
      - allow_file_upload: true
        allow_ctas: true
        allow_cvas: true
        database_name: bigquery
        extra: {
        "metadata_params": {},
        "engine_params": {},
        "metadata_cache_timeout": {},
        "schemas_allowed_for_csv_upload": [],
        "credentials_info": {
          "type": "service_account",................
}
}

.... But I don't think it's this above that's the problem, in the yaml. I think it's the dependencies not loading. ? How can I pull down the /app/pythonpath/superset_config.py file from K8 and view it locally using 'kubectl'?

In the Superset Action Log I see this as well action log in Superset UI

How do I properly load my dependencies? Is it because I am changing the 'values.yaml' locally and trying to "update" via helm command? Do I have to also change another config file? Or log into the node or cluster itself? I am trying to follow the official documentation. https://superset.apache.org/docs/installation/running-on-kubernetes

-- PR102012
apache-superset
flask
kubernetes
python

0 Answers