I deploy GitLab-Omnibus on Kubernetes with Helm. All Pods are started. But I have UI page with HTTP code 422 when I press button on reset password.

Repository

https://gitlab.com/charts/gitlab-omnibus

Check GitLab:

$ kubectl exec -it gitlab1-gitlab-565857c8f9-99tdg gitlab-rake gitlab:check
Checking GitLab Shell ...

GitLab Shell version >= 7.1.2 ? ... OK (7.1.2)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... can't check, you have no projects
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK

Access to /gitlab-data/ssh/authorized_keys: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Reply by email is disabled in config/gitlab.yml
Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... can't check, you have no projects
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.3.7)
Git version >= 2.9.5 ? ... yes (2.16.4)
Git user has default SSH configuration? ... yes
Active users: ... 1

Checking GitLab ... Finished

Where is reason? Do you know? I had many problems with deploy gitlab-omnibus, but I fixed all and have just this problem now.

UPDATE

# kubectl logs gitlab1-gitlab-565857c8f9-99tdg
...
==> /var/log/gitlab/prometheus/current <==
2018-06-04_17:31:11.96125 time="2018-06-04T17:31:11Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:267: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:default:default" cannot list nodes at the cluster scope" component="kube_client_runtime" source="kubernetes.go:76"
2018-06-04_17:31:11.96144 time="2018-06-04T17:31:11Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:267: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:default:default" cannot list nodes at the cluster scope" component="kube_client_runtime" source="kubernetes.go:76"
2018-06-04_17:31:12.06446 time="2018-06-04T17:31:12Z" level=error msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:209: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list pods at the cluster scope" component="kube_client_runtime" source="kubernetes.go:76"

==> /var/log/gitlab/gitlab-workhorse/current <==
2018-06-04_17:31:12.18726 gitlab1-gitlab.default:8005 10.233.105.28:38484 - - [2018/06/04:17:31:12 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 10.3.0 (10-3-stable; go1.8.5; linux/amd64)" 0.001

==> /var/log/gitlab/gitlab-rails/production.log <==
Started PUT "/users/password" for 127.0.0.1 at 2018-06-04 17:31:12 +0000
Processing by PasswordsController#update as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "user"=>{"reset_password_token"=>"[FILTERED]", "password"=>"[FILTERED]", "password_confirmation"=>"[FILTERED]"}}
Can't verify CSRF token authenticity
Completed 422 Unprocessable Entity in 2ms (ActiveRecord: 0.0ms)

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"PUT","path":"/users/password","format":"html","controller":"PasswordsController","action":"update","status":422,"error":"ActionController::InvalidAuthenticityToken: ActionController::InvalidAuthenticityToken","duration":3.19,"view":0.0,"db":0.0,"time":"2018-06-04T17:31:12.583Z","params":[{"key":"utf8","value":"✓"},{"key":"_method","value":"put"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"user","value":{"reset_password_token":"[FILTERED]","password":"[FILTERED]","password_confirmation":"[FILTERED]"}}],"remote_ip":"127.0.0.1","user_id":null,"username":null}

==> /var/log/gitlab/gitlab-rails/production.log <==

ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):
  lib/gitlab/middleware/multipart.rb:95:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  lib/gitlab/middleware/go.rb:17:in `call'
  lib/gitlab/etag_caching/middleware.rb:11:in `call'
  lib/gitlab/middleware/read_only/controller.rb:28:in `call'
  lib/gitlab/middleware/read_only.rb:16:in `call'
  lib/gitlab/request_context.rb:18:in `call'
  lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
  lib/gitlab/middleware/release_env.rb:10:in `call'

==> /var/log/gitlab/gitlab-workhorse/current <==
2018-06-04_17:31:12.58832 gitlab.my-domain.com 10.233.78.148:46740 - - [2018/06/04:17:31:12 +0000] "POST /users/password HTTP/1.1" 422 2912 "http://gitlab.my-domain.com/users/password/edit?reset_password_token=A7ypwSFFAFYGhxN1GMdv" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.170 Safari/537.36" 0.055
...

Full log: https://gist.github.com/IvanBiv/f2027c01b1ac676a5c78591bb7689c87

# kubectl exec -t gitlab1-gitlab-565857c8f9-99tdg gitlab-ctl status
run: alertmanager: (pid 1433) 52693s; run: log: (pid 1545) 52692s
run: gitaly: (pid 1330) 52696s; run: log: (pid 1398) 52695s
run: gitlab-monitor: (pid 1385) 52695s; run: log: (pid 1396) 52695s
run: gitlab-workhorse: (pid 1292) 52697s; run: log: (pid 1304) 52697s
run: logrotate: (pid 3173) 2344s; run: log: (pid 1305) 52697s
run: mattermost: (pid 1353) 52696s; run: log: (pid 1424) 52694s
run: node-exporter: (pid 1099) 52729s; run: log: (pid 1425) 52694s
run: postgres-exporter: (pid 1459) 52694s; run: log: (pid 1549) 52692s
run: prometheus: (pid 1407) 52694s; run: log: (pid 1426) 52694s
run: redis-exporter: (pid 1155) 52716s; run: log: (pid 1397) 52695s
run: registry: (pid 1312) 52696s; run: log: (pid 1387) 52695s
run: sidekiq: (pid 824) 52753s; run: log: (pid 1285) 52697s
run: sshd: (pid 46) 53127s; run: log: (pid 45) 53127s
run: unicorn: (pid 790) 52759s; run: log: (pid 1284) 52697s

# kubectl exec -t gitlab1-gitlab-565857c8f9-99tdg  gitlab-rake gitlab:env:info

System information
System:
Current User:    git
Using RVM:    no
Ruby Version:    2.3.7p456
Gem Version:    2.6.14
Bundler Version:1.13.7
Rake Version:    12.3.1
Redis Version:    3.2.11
Git Version:    2.16.4
Sidekiq Version:5.0.5
Go Version:    unknown

GitLab information
Version:    10.8.3
Revision:    564c342
Directory:    /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:    postgresql
URL:        https://gitlab.my-domain.com
HTTP Clone URL:    https://gitlab.my-domain.com/some-group/some-project.git
SSH Clone URL:    git@gitlab.my-domain.com:some-group/some-project.git
Using LDAP:    no
Using Omniauth:    no

GitLab Shell
Version:    7.1.2
Repository storage paths:
- default:     /gitlab-data/git-data/repositories
Hooks:        /opt/gitlab/embedded/service/gitlab-shell/hooks
Git:        /opt/gitlab/embedded/bin/git