Good day, i'm newby in kubernetes and try to setup my first environment. I want to following scheme:

• My organization has public IP (x.x.x.x)
• This IP routed to server in private LAN (i.e. 192.168.0.10) with win server + IIS. On IIS i have URL rewrite module and it's act as reverse proxy
• I have kubernetes cluster
• I have some service deployed to k8s
• I want to access this service from the internet with SSL, gained from let's encrypt

I have already setup k8s cluster, deploy traefik (v1.7) ingress and configure them for let's encrypt (setup http->https redirect, setup acme challenge). This works fine - I can observe it from LAN or WAN, and there is no warning about certificate - i see green lock. Now I deploy service (in my case this is graylog). Again - I can observe it from LAN and WAN, but in this case i see warning about certificate (it was issued by TRAEFIK_DEFAULT_CERT). After I saw this I try to search more information and found that I need cert-manager. I deploy cert-manager, create let's encrypt issuer (with ClusterIssuer role), but then I try to issue certificate I get following error (found in challenge description):

Waiting for http-01 challenge propagation: wrong status code '404', expected '200'

My traefik configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-conf
  namespace: kube-system
data:
  traefik.toml: |
    # traefik.toml
    defaultEntryPoints = ["http","https"]
    [entryPoints]
      [entryPoints.http]
      address = ":80"
      [entryPoints.http.redirect]
        regex = "^http://(.*)"
        replacement = "https://$1"
      [entryPoints.https]
      address = ":443"
        [entryPoints.https.tls]
    [acme]
      email = "mymail@example.xyz"
      storage = "/acme/acme.json"
      entryPoint = "https"
      onHostRule = true
        [acme.httpChallenge]
        entryPoint = "http"
    [[acme.domains]]
      main = "my-public-domain.com"

I also try to use wildcard there:

apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-conf
  namespace: kube-system
data:
  traefik.toml: |
    # traefik.toml
    defaultEntryPoints = ["http","https"]
    [entryPoints]
      [entryPoints.http]
      address = ":80"
      [entryPoints.http.redirect]
        regex = "^http://(.*)"
        replacement = "https://$1"
      [entryPoints.https]
      address = ":443"
        [entryPoints.https.tls]
    [acme]
      email = "mymail@example.xyz"
      storage = "/acme/acme.json"
      entryPoint = "https"
      onHostRule = true
        [acme.httpChallenge]
        entryPoint = "http"
    [[acme.domains]]
      main = "*.my-public-domain.com"
      sans = ["my-public-domain.com"]

My cluster issuer:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dev
spec:
  acme:
    email: mymail@example.xyz
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: example-issuer-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: traefik

And my test certificate:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: example-com
  namespace: default
spec:
  secretName: example-com-tls
  renewBefore: 360h # 15d
  commonName: logger.my-public-domain.com
  dnsNames:
  - logger.my-public-domain.com
  issuerRef:
    name: letsencrypt-dev
    kind: ClusterIssuer

I have DNS entries for wildcard domain and can ping it

My certificates stuck in state OrderCreated:

Status:
  Conditions:
    Last Transition Time:  2019-10-08T09:40:30Z
    Message:               Certificate issuance in progress. Temporary certificate issued.
    Reason:                TemporaryCertificate
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  OrderCreated  47m   cert-manager  Created Order resource "example-com-3213698372"

Order stuck in state Created:

Status:
  Challenges:
    Authz URL:  <url>
    Dns Name:   logger.my-public-domain.com
    Issuer Ref:
      Kind:  ClusterIssuer
      Name:  letsencrypt-dev
    Key:     <key>
    Solver:
      http01:
        Ingress:
          Class:  traefik
    Token:        <token>
    Type:         http-01
    URL:          <url>
    Wildcard:     false
  Finalize URL:   <url>
  State:          pending
  URL:            <url>
Events:
  Type    Reason   Age   From          Message
  ----    ------   ----  ----          -------
  Normal  Created  48m   cert-manager  Created Challenge resource "example-com-3213698372-0" for domain "logger.my-public-domain.com"

And, at last, by challenge:

Spec:
  Authz URL:  <url>
  Dns Name:   logger.my-public-domain.com
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-dev
  Key:     <key>
  Solver:
    http01:
      Ingress:
        Class:  traefik
  Token:        <token>
  Type:         http-01
  URL:          <url>
  Wildcard:     false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for http-01 challenge propagation: wrong status code '404', expected '200'
  State:       pending
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Started    55m   cert-manager  Challenge scheduled for processing
  Normal  Presented  55m   cert-manager  Presented challenge using http-01 challenge mechanism

I see that there is 404 response, but I can't understand the reason of it. On my IIS I have following rewrite rules: Let's encrypt bypass - all url matched by .well-known/* rewrited to kubernetes host. http to https - all url not matched by let's encrypt redirected to https sub-domain redirect - all subdomains request rewrited to kubernetes.

In my LAN i have own DNS server, there all domains from my-public-domain.com has mapping to internal addresses, so I can redirect public hostname logger.my-public-domain.com (x.x.x.x) to internal logger.my-public-domain.com (192.168.0.y).

While challenge active I cat see new backend and frontend in traefik dashboard.

Maybe I misunderstand how it should works, but I expected that cert-manager issue certificate with let's encrypt and I can observe my service.