I have been trying to switch my application to use Google Cloud logging (aka Stackdriver). I have built a custom Logback appender which uses the Google Java Cloud Logging Library.
When I run my application in the IDE it logs correctly to Google Cloud Logging. When I run it inside a Kubernetes container on GKE, I get an Authentication Error. However in both scenarios I am use the same GCP Service Account.
I have even gone to the length of outputting the Key from the Google credentials object to prove they are the same in each environment. Which they are.
Does anyone have any ideas of what I might be missing? Is there any sort of context related settings that might cause authentication to fail from inside the container? Anyone got any ideas about what to try next?
Update: Here is the stack trace:
java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at com.google.cloud.logging.LoggingImpl.flush(LoggingImpl.java:579) at at com.google.cloud.logging.LoggingImpl.write(LoggingImpl.java:560) at at uk.co.processflows.platform.util.StackdriverAppender.append(StackdriverAppender.kt:135) at at uk.co.processflows.platform.util.StackdriverAppender.append(StackdriverAppender.kt:18) at at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84) at at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51) at at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270) at at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257) at at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421) at at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383) at at ch.qos.logback.classic.Logger.error(Logger.java:538) at at uk.co.processflows.platform.pushnotification.PushNotificationClient.forward(PushNotificationClient.kt:58) at at uk.co.processflows.platform.push.PushMessagingConnectionHandlerImpl.broadcast(PushMessagingConnectionHandlerImpl.kt:257) at at uk.co.processflows.platform.push.PushMessagingConnectionHandlerImpl.broadcastMessageToTenant(PushMessagingConnectionHandlerImpl.kt:153) at at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at at java.base/java.lang.reflect.Method.invoke(Method.java:567) at at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50) at at com.sun.proxy.$Proxy69.broadcastMessageToTenant(Unknown Source) at at uk.co.processflows.platform.workflow.TaskLockControllerImpl$lockTask$2.run(TaskLockControllerImpl.kt:94) at at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at at java.base/java.lang.Thread.run(Thread.java:835) Caused by: java.util.concurrent.ExecutionException: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:552) at at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:431) at at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:97) at at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:68) at at com.google.cloud.logging.LoggingImpl.flush(LoggingImpl.java:577) at ... 25 common frames omitted Caused by: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at com.google.cloud.logging.spi.v2.GrpcLoggingRpc$2.apply(GrpcLoggingRpc.java:190) at at com.google.cloud.logging.spi.v2.GrpcLoggingRpc$2.apply(GrpcLoggingRpc.java:184) at at com.google.api.core.ApiFutures$GaxFunctionToGuavaFunction.apply(ApiFutures.java:204) at at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:206) at at com.google.common.util.concurrent.AbstractCatchingFuture$CatchingFuture.doFallback(AbstractCatchingFuture.java:194) at at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:107) at at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) at at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) at at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) at at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95) at at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77) at at com.google.api.gax.rpc.BatchedFuture.setException(BatchedFuture.java:55) at at com.google.api.gax.rpc.BatchedRequestIssuer.sendResult(BatchedRequestIssuer.java:84) at at com.google.api.gax.rpc.BatchExecutor$1.onFailure(BatchExecutor.java:98) at at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) at at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1056) at at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) at at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) at at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) at at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:179) at at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:135) at at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:117) at at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) at at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) at at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) at at com.google.api.core.AbstractApiFuture$InternalSettableFuture.setException(AbstractApiFuture.java:95) at at com.google.api.core.AbstractApiFuture.setException(AbstractApiFuture.java:77) at at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97) at at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68) at at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1056) at at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) at at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) at at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) at at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:515) at at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:490) at at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:699) at at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397) at at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459) at at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) at at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546) at at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467) at at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584) at at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at ... 3 common frames omitted Caused by: com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:73) at at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72) at at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60) at ... 30 common frames omitted Caused by: io.grpc.StatusRuntimeException: UNAUTHENTICATED: Credentials should use fail() instead of throwing exceptions at at io.grpc.Status.asRuntimeException(Status.java:532) at ... 22 common frames omitted Caused by: java.lang.NoSuchMethodError: io.grpc.MethodDescriptor.getServiceName()Ljava/lang/String; at at io.grpc.auth.GoogleAuthLibraryCallCredentials.serviceUri(GoogleAuthLibraryCallCredentials.java:162) at at io.grpc.auth.GoogleAuthLibraryCallCredentials.applyRequestMetadata(GoogleAuthLibraryCallCredentials.java:103) at at io.grpc.CallCredentials2.applyRequestMetadata(CallCredentials2.java:58) at at io.grpc.internal.CallCredentialsApplyingTransportFactory$CallCredentialsApplyingTransport.newStream(CallCredentialsApplyingTransportFactory.java:107) at at io.grpc.internal.ForwardingConnectionClientTransport.newStream(ForwardingConnectionClientTransport.java:49) at at io.grpc.internal.InternalSubchannel$CallTracingTransport.newStream(InternalSubchannel.java:690) at at io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:245) at at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1.start(CensusTracingModule.java:392) at at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1.start(CensusStatsModule.java:694) at at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:32) at at com.google.api.gax.grpc.GrpcHeaderInterceptor$1.start(GrpcHeaderInterceptor.java:94) at at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:310) at at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:282) at at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:191) at at com.google.api.gax.grpc.GrpcDirectCallable.futureCall(GrpcDirectCallable.java:58) at at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:64) at at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:86) at at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63) at at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41) at at com.google.api.gax.tracing.TracedBatchingCallable.futureCall(TracedBatchingCallable.java:82) at at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126) at at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87) at at com.google.api.gax.rpc.BatchExecutor.processBatch(BatchExecutor.java:82) at at com.google.api.gax.rpc.BatchExecutor.processBatch(BatchExecutor.java:53) at at com.google.api.gax.batching.ThresholdBatcher.pushCurrentBatch(ThresholdBatcher.java:233) at at com.google.api.gax.batching.ThresholdBatcher$1.run(ThresholdBatcher.java:76) at ... 6 common frames omitted
Update 2:
I have just spotted deep inside the caused by hiarchy of the stack trace there is this message.
Caused by: java.lang.NoSuchMethodError: io.grpc.MethodDescriptor.getServiceName()Ljava/lang/String; at at io.grpc.auth.GoogleAuthLibraryCallCredentials.serviceUri(GoogleAuthLibraryCallCredentials.java:162)
That doesn't sound much like an authentication problem.
It turns out there was a dependancy compatibility issue with the GRPC jars. Our application uses GRPC as does apparently the Google Could Logging library. Changing our app to use the older version of GRPC that the Google cloud logging library was built against fixed the issue.
It appears when running in the container the jar files are loaded in a different order than running locally. Hence why this only happened inside Kubernetes.
Given your issue with using the library, you might want to write your logs directly. This doc on creating structured logs might be helpful.
Update I did not see the stack trace when I originally answered this question, and it leads me to be believe your issue is not with authentication at all. My answer may be relevant to those seeing an authentication issue with GCP APIs in GKE, so I'll leave the original here.
Original If you're getting an authentication error, it's likely you're not using the service account key correctly. You can read this tutorial to learn about how it should be done. In short, you need to export the service account key as a JSON file and mount it as a secret.