I have a Kubernetes cluster version 1.15.5 in Azure where I have installed cert-manager version v0.14.0

It works fine with automatically issuing lets encrypt certificates against a valid DNS name: MY_DOMAIN pointing to the external IP address of the ingress controller.

I would also like to be able to do this same thing using e.g. certbot. I have tried to run certbot on my cluster with:

kubectl run --generator=run-pod/v1 certbot-shell --rm -i --tty --image certbot/certbot:amd64-latest -- -d MY_DOMAIN --manual --preferred-challenges http certonly

But it fails with:

Create a file containing just this data:

QAPu****-klNq1RBgY

And make it available on your web server at this URL:

http://MY_DOMAIN/.well-known/acme-challenge/QAPu****-klNq1RBgY

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


Press Enter to Continue
Waiting for verification...
Challenge failed for domain MY_DOMAIN
http-01 challenge for MY_DOMAIN
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: MY_DOMAIN
   Type:   unauthorized
   Detail: Invalid response from
   http://MY_DOMAIN/.well-known/acme-challenge/QAPuDTHa****1qlLLOg
   [13.x.x.x]: 404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain

So somehow cert-manager automatically takes care of creating the file during the challenge and making it available at:

http://MY_DOMAIN/.well-known/acme-challenge/QAPu****-klNq1RBgY

But I am not sure how I do that when using certbot or if there some other way to do this??

Based on below suggestions I have tried to install Kube lego (0.1.2) instead (for legacy 1.8 cluster) but seems to fail with:

level=error msg="Error while processing certificate requests: 403 urn:acme:error:unauthorized: Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555. See https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details." context=kubelego 

So I have tried to upgrade to:

https://acme-staging-v02.api.letsencrypt.org/directory

but then I get:

level=error msg="Error while processing certificate requests: Head : unsupported protocol scheme \"\"" context=kubelego

I found:

https://github.com/jetstack/kube-lego/issues/301

So looks like kube-lego cannot be used with ACME version 2 :-(