We are getting the following error

"One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric.: timeSeries[0]"

when we tried to create container level metric in GKE. We are creating custom metric to update app_queue length for every second. It works well under normal case. But when the pods scales beyond 100 - we receive this error for some of the API calls and it is very high in number.

Question: Is the any quota or setting we are missing? or some best practice of creating custom metrics?

Stack trace:

      File "/usr/local/lib/python3.6/dist-packages/Base-1.0-py3.6.egg/base/custom_metrics.py", line 63, in create_custom_metric
        client.create_time_series(project_name, [series])
      File "/usr/local/lib/python3.6/dist-packages/google/cloud/monitoring_v3/gapic/metric_service_client.py", line 775, in create_time_series
        request, retry=retry, timeout=timeout, metadata=metadata)
      File "/usr/local/lib/python3.6/dist-packages/google/api_core/gapic_v1/method.py", line 139, in __call__
        return wrapped_func(*args, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
        on_error=on_error,
      File "/usr/local/lib/python3.6/dist-packages/google/api_core/retry.py", line 177, in retry_target
        return target()
      File "/usr/local/lib/python3.6/dist-packages/google/api_core/timeout.py", line 206, in func_with_timeout
        return func(*args, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/google/api_core/grpc_helpers.py", line 61, in error_remapped_callable
        six.raise_from(exceptions.from_grpc_error(exc), exc)
      File "<string>", line 3, in raise_from
    google.api_core.exceptions.InvalidArgument: 400 One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric.: timeSeries[0]