Kubernetes Ambassador ingress docker image rpc timeout

3/11/2021

I’m trying to setup ingress using ambassador for my local cluster and working off this guide here https://kind.sigs.k8s.io/docs/user/ingress

but I receive an rpc and timeout error. Some things i’ve attempted in trying to fix include: Using the Ingress Nginx instead but I received same error, amending the dockerhub path by just having the username and image in the url and deleting and recreating the pods. I've also seen other previous questions and the solutions do not seem to work.

NAMESPACE            NAME                                          READY   STATUS             RESTARTS   AGE
ambassador           ambassador-operator-67668967b8-w28b2          0/1     ImagePullBackOff   0          28m
default              bar-app                                       0/1     ErrImagePull       0          9m19s
default              foo-app                                       0/1     ImagePullBackOff   0          9m19s
kube-system          coredns-74ff55c5b-m7s8r                       1/1     Running            0          38m
kube-system          coredns-74ff55c5b-tgcdg                       1/1     Running            0          38m
kube-system          etcd-kind8-control-plane                      1/1     Running            0          38m
kube-system          kindnet-dch9w                                 1/1     Running            0          37m
kube-system          kindnet-dm5gn                                 1/1     Running            0          38m
kube-system          kindnet-sxxdk                                 1/1     Running            0          37m
kube-system          kube-apiserver-kind8-control-plane            1/1     Running            0          38m
kube-system          kube-controller-manager-kind8-control-plane   1/1     Running            0          38m
kube-system          kube-proxy-n84kf                              1/1     Running            0          38m
kube-system          kube-proxy-twtsf                              1/1     Running            0          37m
kube-system          kube-proxy-zjq6t                              1/1     Running            0          37m
kube-system          kube-scheduler-kind8-control-plane            1/1     Running            0          38m
local-path-storage   local-path-provisioner-78776bfc44-kkrht       1/1     Running            0          38m

This is the error log. kubectl get events --all-namespaces --sort-by='.metadata.creationTimestamp'

kube-system          28m         Normal    Created                   pod/coredns-74ff55c5b-tgcdg                    Created container coredns
kube-system          28m         Normal    Created                   pod/coredns-74ff55c5b-m7s8r                    Created container coredns
kube-system          28m         Normal    Started                   pod/coredns-74ff55c5b-m7s8r                    Started container coredns
local-path-storage   28m         Normal    Started                   pod/local-path-provisioner-78776bfc44-kkrht    Started container local-path-provisioner
local-path-storage   28m         Normal    Created                   pod/local-path-provisioner-78776bfc44-kkrht    Created container local-path-provisioner
local-path-storage   28m         Normal    LeaderElection            endpoints/rancher.io-local-path                local-path-provisioner-78776bfc44-kkrht_c5300431-393d-4ce5-bee6-9fa03b2567e8 became leader
kube-system          28m         Normal    Started                   pod/coredns-74ff55c5b-tgcdg                    Started container coredns
ambassador           20m         Normal    ScalingReplicaSet         deployment/ambassador-operator                 Scaled up replica set ambassador-operator-67668967b8 to 1
ambassador           20m         Normal    SuccessfulCreate          replicaset/ambassador-operator-67668967b8      Created pod: ambassador-operator-67668967b8-w28b2
ambassador           20m         Normal    Scheduled                 pod/ambassador-operator-67668967b8-w28b2       Successfully assigned ambassador/ambassador-operator-67668967b8-w28b2 to kind8-worker
ambassador           15m         Normal    Pulling                   pod/ambassador-operator-67668967b8-w28b2       Pulling image "docker.io/datawire/ambassador-operator:v1.2.9"
ambassador           3s          Warning   Failed                    pod/ambassador-operator-67668967b8-w28b2       Error: ImagePullBackOff
ambassador           5m1s        Normal    BackOff                   pod/ambassador-operator-67668967b8-w28b2       Back-off pulling image "docker.io/datawire/ambassador-operator:v1.2.9"
ambassador           14m         Warning   Failed                    pod/ambassador-operator-67668967b8-w28b2       Error: ErrImagePull
ambassador           19m         Warning   Failed                    pod/ambassador-operator-67668967b8-w28b2       Failed to pull image "docker.io/datawire/ambassador-operator:v1.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/datawire/ambassador-operator:v1.2.9": failed to resolve reference "docker.io/datawire/ambassador-operator:v1.2.9": failed to do request: Head https://registry-1.docker.io/v2/datawire/ambassador-operator/manifests/v1.2.9: dial tcp 18.214.230.110:443: i/o timeout
ambassador           17m         Warning   Failed                    pod/ambassador-operator-67668967b8-w28b2       Failed to pull image "docker.io/datawire/ambassador-operator:v1.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/datawire/ambassador-operator:v1.2.9": failed to resolve reference "docker.io/datawire/ambassador-operator:v1.2.9": failed to do request: Head https://registry-1.docker.io/v2/datawire/ambassador-operator/manifests/v1.2.9: dial tcp 3.211.199.249:443: i/o timeout
ambassador           16m         Warning   Failed                    pod/ambassador-operator-67668967b8-w28b2       Failed to pull image "docker.io/datawire/ambassador-operator:v1.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/datawire/ambassador-operator:v1.2.9": failed to resolve reference "docker.io/datawire/ambassador-operator:v1.2.9": failed to do request: Head https://registry-1.docker.io/v2/datawire/ambassador-operator/manifests/v1.2.9: dial tcp 54.236.165.68:443: i/o timeout
ambassador           14m         Warning   Failed                    pod/ambassador-operator-67668967b8-w28b2       Failed to pull image "docker.io/datawire/ambassador-operator:v1.2.9": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/datawire/ambassador-operator:v1.2.9": failed to resolve reference "docker.io/datawire/ambassador-operator:v1.2.9": failed to do request: Head https://registry-1.docker.io/v2/datawire/ambassador-operator/manifests/v1.2.9: dial tcp 54.236.131.166:443: i/o timeout
default              38s         Normal    Scheduled                 pod/foo-app                                    Successfully assigned default/foo-app to kind8-worker
default              38s         Normal    Scheduled                 pod/bar-app                                    Successfully assigned default/bar-app to kind8-worker
default              37s         Normal    Pulling                   pod/bar-app                                    Pulling image "hashicorp/http-echo:0.2.3"

This is the yaml.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ambassadorinstallations.getambassador.io
spec:
  additionalPrinterColumns:
  - JSONPath: .spec.version
    name: VERSION
    type: string
  - JSONPath: .spec.updateWindow
    name: UPDATE-WINDOW
    type: integer
  - JSONPath: .status.lastCheckTime
    description: Last time checked
    name: LAST-CHECK
    type: string
  - JSONPath: .status.conditions[?(@.type=='Deployed')].status
    description: Indicates if deployment has completed
    name: DEPLOYED
    type: string
  - JSONPath: .status.conditions[?(@.type=='Deployed')].reason
    description: Reason for deployment completed
    name: REASON
    priority: 1
    type: string
  - JSONPath: .status.conditions[?(@.type=='Deployed')].message
    description: Message for deployment completed
    name: MESSAGE
    priority: 1
    type: string
  - JSONPath: .status.deployedRelease.appVersion
    description: Deployed version of Ambassador
    name: DEPLOYED-VERSION
    type: string
  - JSONPath: .status.deployedRelease.flavor
    description: Deployed flavor of Ambassador (OSS or AES)
    name: DEPLOYED-FLAVOR
    type: string
  group: getambassador.io
  names:
    kind: AmbassadorInstallation
    listKind: AmbassadorInstallationList
    plural: ambassadorinstallations
    singular: ambassadorinstallation
  scope: Namespaced
  subresources:
    status: {}
  validation:
    openAPIV3Schema:
      description: AmbassadorInstallation is the Schema for the ambassadorinstallations
        API
      properties:
        apiVersion:
          description: 'APIVersion defines the versioned schema of this representation
            of an object. Servers should convert recognized schemas to the latest
            internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
          type: string
        kind:
          description: 'Kind is a string value representing the REST resource this
            object represents. Servers may infer this from the endpoint the client
            submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
          type: string
        metadata:
          type: object
        spec:
          description: AmbassadorInstallationSpec defines the desired state of AmbassadorInstallation
          properties:
            baseImage:
              description: An (optional) image to use instead of the image specified
                in the Helm chart.
              type: string
            helmRepo:
              description: An (optional) Helm repository.
              type: string
            installOSS:
              description: 'Installs [Ambassador OSS](https://www.getambassador.io/docs/latest/topics/install/install-ambassador-oss/)
                instead of [AES](https://www.getambassador.io/docs/latest/topics/install/).
                Default is false which means it installs AES by default. TODO: 1.
                AES/AOSS is not installed and the user installs using `installOSS:
                true`, then we straightaway install AOSS. 2. AOSS is installed via
                operator and the user sets `installOSS: false`, then we perform the
                migration as    detailed here - https://www.getambassador.io/docs/latest/topics/install/upgrade-to-edge-stack/
                3. AES is installed and the user sets `installOSS: true`, then we
                point users to the docs which gives them    pointers on how to do
                that themselves.'
              type: boolean
            logLevel:
              description: 'An (optional) log level: debug, info...'
              enum:
              - info
              - debug
              - warn
              - warning
              - error
              - critical
              - fatal
              type: string
            updateWindow:
              description: "`updateWindow` is an optional item that will control when
                the updates can take place. This is used to force system updates to
                happen late at night if that’s what the sysadmins want. \n  * There
                can be any number of `updateWindow` entries (separated by commas).
                \ * `Never` turns off automatic updates even if there are other entries
                in the    comma-separated list. `Never` is used by sysadmins to disable
                all updates    during blackout periods by doing a `kubectl apply`
                or using our Edge Policy    Console to set this. * Each `updateWindow`
                is in crontab format (see https://crontab.guru/)   Some examples of
                `updateWindows` are:    - `* 0-6 * * * SUN`: every Sunday, from _0am_
                to _6am_    - `* 5 1 * * *`: every first day of the month, at _5am_
                * The Operator cannot guarantee minute time granularity, so specifying
                \  a minute in the crontab expression can lead to some updates happening
                \  sooner/later than expected."
              type: string
            version:
              description: "We are using SemVer for the version number and it can
                be specified with any level of precision and can optionally end in
                `*`. These are interpreted as: \n * `1.0` = exactly version 1.0 *
                `1.1` = exactly version 1.1 * `1.1.*` = version 1.1 and any bug fix
                versions `1.1.1`, `1.1.2`, `1.1.3`, etc. * `2.*` = version 2.0 and
                any incremental and bug fix versions `2.0`, `2.0.1`,   `2.0.2`, `2.1`,
                `2.2`, `2.2.1`, etc. * `*` = all versions. * `3.0-ea` = version `3.0-ea1`
                and any subsequent EA releases on `3.0`.   Also selects the final
                3.0 once the final GA version is released. * `4.*-ea` = version `4.0-ea1`
                and any subsequent EA release on `4.0`.   Also selects the final GA
                `4.0`. Also selects any incremental and bug   fix versions `4.*` and
                `4.*.*`. Also selects the most recent `4.*` EA release   i.e., if
                `4.0.5` is the last GA version and there is a `4.1-EA3`, then this
                \  selects `4.1-EA3` over the `4.0.5` GA. \n   You can find the reference
                docs about the SemVer syntax accepted   [here](https://github.com/Masterminds/semver#basic-comparisons)."
              type: string
          type: object
        status:
          description: AmbassadorInstallationStatus defines the observed state of
            AmbassadorInstallation
          properties:
            conditions:
              description: List of conditions the installation has experienced.
              items:
                description: AmbInsCondition defines an Ambassador installation condition,
                  as well as the last time there was a transition to this condition..
                properties:
                  lastTransitionTime:
                    format: date-time
                    type: string
                  message:
                    type: string
                  reason:
                    type: string
                  status:
                    type: string
                  type:
                    type: string
                required:
                - status
                - type
                type: object
              type: array
            deployedRelease:
              description: the currently deployed Helm chart
              nullable: true
              properties:
                appVersion:
                  type: string
                flavor:
                  type: string
                manifest:
                  type: string
                name:
                  type: string
                version:
                  type: string
              type: object
            lastCheckTime:
              description: Last time a successful update check was performed.
              format: date-time
              nullable: true
              type: string
          required:
          - conditions
          type: object
      type: object
  version: v2
  versions:
  - name: v2
    served: true
    storage: true
-- manoman687
ambassador
docker
kind
kubernetes

1 Answer

3/11/2021

I figured this out. I'd previously disabled docker bridge0 using this entry under /etc/docker/daemon.json

{
    "iptables": false,
    "bridge": "none"
}

To fixed i simply deleted and restarted docker: systemctl restart docker

-- manoman687
Source: StackOverflow