How to Fix: GCP Pub/Sub Message Delivery Failed

The Scenario

It’s 3 PM on a Wednesday, and your data pipeline starts dropping events. Your application publishes messages to a Pub/Sub topic, but the subscriber (a Cloud Run service) stops acknowledging them. You see the DEADLINE_EXCEEDED error in the subscriber logs, and the subscription’s unacked_message_count metric starts climbing.

Symptoms

Subscriber logs show DEADLINE_EXCEEDED errors when pulling messages.
The pubsub.googleapis.com/subscription/num_undelivered_messages metric spikes.
Messages are not being processed, and the backlog grows.

Root Cause

The subscriber is taking longer than the default 10-second acknowledgement deadline to process each message. Pub/Sub redelivers the message, but the subscriber is still busy, creating a cycle of timeouts and retries. This is often caused by a downstream dependency (e.g., a database or API) that is slow or unavailable.

Resolution (Step-by-Step)

Check the subscriber’s processing time.

gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=<your-service> AND jsonPayload.message=~\"DEADLINE_EXCEEDED\""

Look for logs that show how long the message handler takes.

Increase the acknowledgement deadline on the subscription.
```
gcloud pubsub subscriptions update <your-subscription> --ack-deadline=60
```
This gives the subscriber up to 60 seconds to process a message.
Implement asynchronous processing. If the handler can be made non-blocking, modify your subscriber code to acknowledge the message immediately after receiving it, then process it in the background. This prevents the deadline from expiring.
If using a pull subscriber, increase the max_outstanding_messages and max_outstanding_bytes settings. For example, in the Go client library:
```
sub.ReceiveSettings.MaxOutstandingMessages = 100
sub.ReceiveSettings.MaxOutstandingBytes = 1e8
```

Why This Sometimes Doesn’t Work

Simply increasing the ack deadline can mask the real problem—a slow downstream dependency. If the database is under heavy load, a longer deadline just delays the failure. You must also fix the bottleneck (e.g., add connection pooling, increase DB instance size, or add caching). Otherwise, the subscriber will eventually hit the maximum 600-second deadline.

Verification

Run the following to confirm the backlog is draining:

gcloud pubsub subscriptions describe <your-subscription> --format="get(pushConfig.pushEndpoint,ackDeadlineSeconds)"

Then check the num_undelivered_messages metric in Cloud Monitoring. It should trend downward after the fix.

Common Follow-up Questions

Q: What if I see RESOURCE_EXHAUSTED instead of DEADLINE_EXCEEDED? A: That means the subscriber’s pull request is being throttled. Increase the flow control settings (e.g., max_outstanding_messages) and ensure your subscriber is not making too many concurrent pull requests.

Q: Can I set the ack deadline to 600 seconds for all subscriptions? A: No. The maximum is 600 seconds, but a long deadline delays failure detection. Use it only as a temporary fix while you optimize the handler.

Q: Does the error affect ordered delivery? A: Yes. If you have message ordering enabled, a DEADLINE_EXCEEDED on one message will block all subsequent messages in that ordering key until it’s processed or the lease expires.

Prevent This in the Future

Proactive monitoring of subscriber processing latency and backlog depth would catch this class of issue before it impacts production. Teams use Better Stack to set up real-time alerts on Pub/Sub metrics like num_undelivered_messages and subscriber processing time, so you get notified when the backlog starts growing. Set up uptime monitoring and alerts with Better Stack