Gitlab CI cannot resolve service hostname on kubernetes runner

12/9/2019

I have a GitLab CI pipeline configured to run on Kubernetes runner. Everything worked great until I tried to add services (https://docs.gitlab.com/ee/ci/services/mysql.html) for test job. The service hostname (eg.: mysql) cannot be resolved on kubernetes, resulting into the following error dial tcp: lookup mysql on 10.96.0.10:53: no such host. However, it works on docker runner, but that's just not what I want. Is there any way to

The job definition from .gitlab-ci.yml:

test:
    stage: test
    variables:
        MYSQL_ROOT_PASSWORD: --top-secret--
        MYSQL_DATABASE: --top-secret--
        MYSQL_USER: --top-secret--
        MYSQL_PASSWORD: --top-secret--
    services:
      - mysql:latest
      - nats:latest
    script:
        - ping -c 2 mysql
        - go test -cover -coverprofile=coverage.prof.tmp ./...

Edit:

Logs from runner-jd6sxcl7-project-430-concurrent-0g5bm8 pod show that the services started. There are 4 containers total inside the pod: build,helper,svc-0 (mysql), svc-1 (nats)

svc-0 logs show the mysql service started successfully:

2019-12-09 21:52:07+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2019-12-09 21:52:07+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2019-12-09 21:52:08+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.18-1debian9 started.
2019-12-09 21:52:08+00:00 [Note] [Entrypoint]: Initializing database files
2019-12-09T21:52:08.226747Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2019-12-09T21:52:08.233097Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.18) initializing of server in progress as process 46

svc-1 logs show the nats service started successfully as well:

[1] 2019/12/09 21:52:12.876121 [INF] Starting nats-server version 2.1.2
[1] 2019/12/09 21:52:12.876193 [INF] Git commit [679beda]
[1] 2019/12/09 21:52:12.876393 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2019/12/09 21:52:12.876522 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2019/12/09 21:52:12.876548 [INF] Server id is NCPAQNFKKWPI67DZHSWN5EWOCQSRACFG2FXNGTLMW2NNRBAMLSDY4IYQ
[1] 2019/12/09 21:52:12.876552 [INF] Server is ready
[1] 2019/12/09 21:52:12.876881 [INF] Listening for route connections on 0.0.0.0:6222
-- sveatlo
gitlab-ci
gitlab-ci-runner
kubernetes

1 Answer

12/9/2019

From the documentation link you provided, I see that the mysql hostname should be accessible.

How services are linked to the job

To better understand how the container linking works, read Linking containers together.

To summarize, if you add mysql as service to your application, the image will then be used to create a container that is linked to the job container.

The service container for MySQL will be accessible under the hostname mysql. So, in order to access your database service you have to connect to the host named mysql instead of a socket or localhost. Read more in accessing the services.

Also from the doc

If you don’t specify a service alias, when the job is run, service will be started and you will have access to it from your build container

So can you check if there are errors when the mysql service started.

I provided this as answer since I could not fit this in comment.

-- randominstanceOfLivingThing
Source: StackOverflow