I'm writing a controller for a k8s CRD.
The job the controller has to do will usually be quick, but could on occasion take a really long time - let's say as much as an hour.
Is that ok for a Reconcile? Or should I move that work out of the controller into a separate pod, and have the controller monitor the process of that pod?
I see no reason why the reconcile loop couldn't take as long as you need.
Technically speaking a reconcile is just getting a copy of a resource i.e. an HTTP Get or an event if you're using the Watch API, followed by a change to the resource e.g updating the resource Status fields i.e an HTTP PUT/POST.
The only caveat is making sure the resource version you have is still the latest one when trying to change it. Including resource versions in your request should solve this problem.
More info here: https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions