I am trying to run 3 pods/replicas of a container in which I want one of them to be run as the leader and if that particular pod is stopped or fails, immediately the other non-leader running pods should take the leadership to continue execution that is required from a leader pod. Just to be clear the other pods would be executing but the leader pod once it gains leadership executes some extra code which the other pods will not. Here is the link to the Docs I followed.

Here is a test example I ran with which I am having problems running the actual code that the container is supposed to run.

package main
import (
	"fmt"
	"github.com/operator-framework/operator-lib/leader"
	"sigs.k8s.io/controller-runtime/pkg/manager"
	"sigs.k8s.io/controller-runtime/pkg/client/config"
	"sigs.k8s.io/controller-runtime/pkg/manager/signals"
	"context"
	"time"
	"os"
)

func main() {
	fmt.Println("Program has started successfully")

	cfg, err := config.GetConfig()
	if err != nil {
		os.Exit(1)
	}

	opts := manager.Options{
		LeaderElection:     true,
		LeaderElectionID:   "memcached-operator-lock",
	}
	fmt.Println("Creating a manager")
	mgr, err := manager.New(cfg, opts)
	if err := mgr.Start(signals.SetupSignalHandler()); err != nil {
		fmt.Println(err, "Manager exited non-zero")
		os.Exit(1)
	}
	fmt.Println("Competing to become the leader")
	err = leader.Become(context.TODO(), "memcached-operator-lock")
	if err != nil {
		fmt.Println("Failed to retry for leader lock because : %v ", err)
	}
	fmt.Println("I am the leader, i will pause for 4 minutes")
	time.Sleep(4 * time.Minute)
}

The logs of the leader pod are as follows :

[diamanti@appserv41 leader_test]$ kubectl logs test-rs-qbm7k
Program has started successfully
Creating a manager
I1007 00:14:10.785904       1 leaderelection.go:242] attempting to acquire leader lease  default/memcached-operator-lock...
I1007 00:14:28.184916       1 leaderelection.go:252] successfully acquired lease default/memcached-operator-lock

I want the pod to be able to run till the "I am the leader" print statement and execute the tasks that I want the leader pod would do. Even if I can pass a function somewhere which will allow it to be executed once it gains leadership, is fine with me. Currently, I am making a mistake somewhere and not able to do anything after one of the 3 pods gain leadership.