K
Q

"500 Internal Server Error" when pushing docker images using gcloud and a service account with "Owner" permission

2/3/2016

I am trying to push a docker container image to the Google Container Engine registry:

$ sudo gcloud docker push gcr.io/<my_project>/node
The push refers to a repository [gcr.io/<my_project>/node] (len: 1)
24c43271ae0b: Image already exists 
70a0868daa56: Image already exists 
1d86f57ee56d: Image already exists 
a46b473f6491: Image already exists 
a1ac8265769f: Image already exists 
290bb858e1c3: Image already exists 
d6fc4b5b4917: Image already exists 
3842411e5c4c: Image already exists 
7a01cc5f27b1: Image already exists 
dbacfa057b30: Image already exists 
latest: digest: sha256:02be2e66ad2fe022f433e228aa43f32d969433402820035ac8826880dbc325e4 size: 17236
Received unexpected HTTP status: 500 Internal Server Error

I can not make the command verbose more. Neither with:

$ sudo gcloud docker push gcr.io/<my_project>/node --verbosity info

nor with this command that should work:

$ sudo gcloud docker --log-level=info push gcr.io/sigma-cairn-99810/node
usage: gcloud docker  [EXTRA_ARGS ...] [optional flags]
ERROR: (gcloud.docker) unrecognized arguments: --log-level=info

according to the documentation (see EXTRA_ARGS) and --log-level=info is a valid docker option:

SYNOPSIS
    gcloud docker [EXTRA_ARGS ...] [--authorize-only, -a]
        [--docker-host DOCKER_HOST]
        [--server SERVER,[SERVER,...], -s SERVER,[SERVER,...]; default="gcr.io,us.gcr.io,eu.gcr.io,asia.gcr.io,b.gcr.io,bucket.gcr.io,appengine.gcr.io"]
        [GLOBAL-FLAG ...]
...    

POSITIONAL ARGUMENTS
     [EXTRA_ARGS ...]
        Arguments to pass to docker.

I am using the default service account that GCP installs on my container-vm machine instance. I have given it also Owner permissions to all resources in <my_project>.

UPDATE:

Running sudo gsutil ls -bL gs://artifacts.<my_project>.appspot.com I get:

gs://artifacts.<my_project>.appspot.com/ :
    Storage class:          STANDARD
    Location constraint:        US
    Versioning enabled:     None
    Logging configuration:      None
    Website configuration:      None
    CORS configuration:         None
    Lifecycle configuration:    None
    ACL:                []
    Default ACL:            []

If I do the same thing after authenticating with my non-service account, I get both ACL and Default ACL:

ACL:                
  [
    {
      "entity": "project-owners-262451203973",
      "projectTeam": {
        "projectNumber": "262451203973",
        "team": "owners"
      },
      "role": "OWNER"
    },
    {
      "entity": "project-editors-262451203973",
      "projectTeam": {
        "projectNumber": "262451203973",
        "team": "editors"
      },
      "role": "OWNER"
    },
    {
      "entity": "project-viewers-262451203973",
      "projectTeam": {
        "projectNumber": "262451203973",
        "team": "viewers"
      },
      "role": "READER"
    }
  ]
Default ACL:            
  [
    {
      "entity": "project-owners-262451203973",
      "projectTeam": {
        "projectNumber": "262451203973",
        "team": "owners"
      },
      "role": "OWNER"
    },
    {
      "entity": "project-editors-262451203973",
      "projectTeam": {
        "projectNumber": "262451203973",
        "team": "editors"
      },
      "role": "OWNER"
    },
    {
      "entity": "project-viewers-262451203973",
      "projectTeam": {
        "projectNumber": "262451203973",
        "team": "viewers"
      },
      "role": "READER"
    }
  ]
-- Gabriel Petrovay
docker
gcloud
google-container-registry
google-kubernetes-engine

Similar Questions

What does it mean for the application developer to run an application in Kubernetes?
Google Cloud Logging + google-fluentd Dropping Messages
Why autoscale Kubernates if not doing a rolling update?
Containers keep restarting

1 Answer

2/3/2016

Can you run sudo gsutil ls -bL gs://artifacts.<my_project>.appspot.com and see if you can access to the GCS bucket. This will verify permissions for storage for the docker image.

While I think you should have permission by being added to owner, this will verify if you do or not.

As for the EXTRA_ARGS, I think --log-level="info" is only valid for the command docker daemon, docker push does not recognize --log-level="info"

UPDATE

From reviewing the logs again, you are pushing a mostly existing image, as the "image already exist" log entries indicate. On the first new write step it failed. That indicates that the problem seems likely to be that the instance you started with originally only had read only scope.

Can you please run this command and share the output. curl -H "Metadata-Flavor:Google" http://metadata/computeMetadata/v1/instance/service-accounts/default/scopes

We are looking for the scope https://www.googleapis.com/auth/devstorage.read_write.

What might have happened is that the instance was not originally created with this scope, and as the scope on an instance cannot be modified, it maintains only being able to read.

If this is the case, the solution would likely be creating a new instance.

We will file a bug to ensure better messaging is provided in situations like this.

-- k4leung4
Source: StackOverflow