How to Fix: AWS DynamoDB ProvisionedThroughputExceededException

The Scenario

You’re running a production microservice that writes time-series sensor data to a DynamoDB table. During a Black Friday event, traffic spikes 5x, and suddenly your service starts logging ProvisionedThroughputExceededException errors. The application latency goes through the roof, and downstream consumers start timing out.

Symptoms

Application logs show: ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded.
CloudWatch alarms trigger for ThrottledRequests > 0 on the DynamoDB table.
Requests to the table return HTTP 400 errors, and your retry logic is exhausting its backoff window.

Root Cause

DynamoDB enforces a hard limit on read/write capacity units (RCUs/WCUs) per partition. When a single partition (often due to a “hot key”) exceeds its allocated throughput, or when the total table capacity is exhausted, DynamoDB rejects requests with this exception. The root cause is usually uneven access patterns (e.g., all writes hitting one partition key) or insufficient provisioned capacity for the traffic burst.

Resolution (Step-by-Step)

Identify the hot partition
Run CloudWatch Logs Insights on your DynamoDB table logs (enable dynamodb:DescribeTable and dynamodb:GetRecords logging first if not already):

fields @timestamp, @message
| filter @message like /ProvisionedThroughputExceededException/
| stats count() by partitionKey
| sort count() desc
| limit 10

Check current table metrics
In the AWS Console, go to DynamoDB > Your Table > Metrics. Look at ConsumedWriteCapacityUnits vs ProvisionedWriteCapacityUnits. If consumed consistently hits provisioned, you need more capacity.
Increase provisioned capacity (short-term fix)
Use the AWS CLI to bump capacity immediately:
```
aws dynamodb update-table \
    --table-name sensor-data \
    --provisioned-throughput ReadCapacityUnits=500,WriteCapacityUnits=1000
```
Wait for the table to transition to ACTIVE (usually 1–2 minutes). Re-run your load test.

Implement exponential backoff in client code
If you’re using the AWS SDK, enable retries with jitter. Example for Python (boto3):

import boto3
from botocore.config import Config

config = Config(
    retries={'max_attempts': 10, 'mode': 'adaptive'}
)
client = boto3.client('dynamodb', config=config)

Redesign partition keys to avoid hot spots
Add a suffix (e.g., partitionKey_randomSuffix) to distribute writes across partitions. For time-series data, use a composite key like deviceId#YYYY-MM-DD-HH and ensure you’re writing to many unique keys.

Why This Sometimes Doesn’t Work

Simply increasing provisioned throughput may not fix the issue if a single partition key is the bottleneck. DynamoDB allocates throughput per partition (roughly 3000 RCU or 1000 WCU per partition). If all traffic hits one key (e.g., deviceId=popular_sensor), that partition caps out, and you’ll still see throttling even with high table-level capacity. The real fix is to shard the hot key (e.g., append a random number to the partition key) or switch to on-demand capacity mode for unpredictable workloads.

Verification

Run a test write loop and check for throttling:

aws cloudwatch get-metric-statistics \
    --namespace AWS/DynamoDB \
    --metric-name ThrottledRequests \
    --dimensions Name=TableName,Value=sensor-data \
    --start-time $(date -u -d '5 minutes ago' +%Y-%m-%dT%H:%M:%SZ) \
    --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
    --period 60 \
    --statistics Sum

Expected output: Sum should be 0 for the last 5 minutes.

Common Follow-up Questions

Q: Should I switch to on-demand capacity mode?
A: Yes, if your traffic is unpredictable or bursts are common. On-demand auto-scales instantly, but costs more per request. For steady-state workloads, provisioned with auto-scaling is cheaper.

Q: How do I find which partition key is hot?
A: Enable DynamoDB Accelerator (DAX) or use CloudWatch Contributor Insights to see top partition keys by throttled requests. You can also query AWS/DynamoDB metrics with the PartitionKey dimension.

Q: Do retries with exponential backoff always work?
A: No. If the bottleneck is a single hot partition, retries will keep hitting the same partition and failing. You must fix the key design first.

Prevent This in the Future

To catch throttling before it impacts users, set up proactive monitoring on DynamoDB’s ThrottledRequests metric with a low threshold (e.g., > 0 for 1 minute). Better Stack can aggregate these CloudWatch alarms, trigger on-call notifications, and correlate them with deployment events for faster root cause analysis.
Set up uptime monitoring and alerts with Better Stack