How to Fix: GCP IAM Permission Denied Service Account

The Scenario

You’re deploying a microservice via Cloud Run that needs to read from a Cloud Storage bucket, or perhaps a Terraform apply is failing during a Friday afternoon release. The service account attached to the resource (e.g., my-service-account@<project-id>.iam.gserviceaccount.com) keeps getting slapped with a 403 when it tries to access a GCP API or resource.

Symptoms

• Error 403: The caller does not have permission in Cloud Run logs.
• Permission 'storage.objects.get' denied on resource when running a Cloud Function.
• Terraform plan/apply fails with Error 403: googleapi: Error 403: Required 'compute.instances.get' permission for 'projects/<project>/zones/<zone>/instances/<instance>'.

Root Cause

The service account lacks the necessary IAM role(s) on the specific resource or project. GCP IAM is resource-hierarchical (org → folder → project → resource), and permissions are not inherited automatically if the role is missing at the right level. Common culprits: assigning roles at the project level when the resource is in a different project, or forgetting to grant the role on the specific resource itself (e.g., a Cloud Storage bucket).

Resolution (Step-by-Step)

1. Identify the exact service account and resource. Check the error message for the service account email and the resource name.

   # Example: Error mentions service account and bucket
   # service-account: my-app-sa@my-project.iam.gserviceaccount.com
   # resource: my-bucket

2. Grant the required role at the appropriate level. Use gcloud or the GCP Console. For resource-level permissions (e.g., a bucket), use the --condition flag or resource-specific commands.

   # Grant Storage Object Viewer on a specific bucket
   gcloud storage buckets add-iam-policy-binding gs://my-bucket \
     --member="serviceAccount:my-app-sa@my-project.iam.gserviceaccount.com" \
     --role="roles/storage.objectViewer"

   For project-level permissions (e.g., Compute Engine):

   gcloud projects add-iam-policy-binding my-project \
     --member="serviceAccount:my-app-sa@my-project.iam.gserviceaccount.com" \
     --role="roles/compute.viewer"

3. If the resource is in a different project, ensure the service account has the roles/iam.serviceAccountTokenCreator role on itself (for impersonation) and the correct role on the target project’s resource. Cross-project access requires explicit grants.

Why This Sometimes Doesn’t Work

The most common gotcha is IAM propagation delay — it can take up to 2-3 minutes (sometimes 7+ minutes in rare cases) for policy changes to fully propagate across GCP’s backend. If you test immediately after adding the role, you might still see the 403. Also, check for organization policies that might be blocking the specific permission, even if the IAM role is correct. For example, a constraint like constraints/iam.allowedPolicyMemberDomains can silently deny access.

Verification

Test the permission directly using gcloud with the service account’s credentials:

# Impersonate the service account and test access
gcloud auth activate-service-account my-app-sa@my-project.iam.gserviceaccount.com --key-file=key.json
gsutil ls gs://my-bucket/

If it works, the issue is resolved. If not, wait 2 minutes and retry.

Common Follow-up Questions

Q: I added the role, but the error persists. What now? A: Check if the role was added at the correct level. A project-level role won’t grant access to a resource in a different project. Also verify the service account email is exactly correct — watch for typos.

Q: Can I grant permissions to a service account across projects? A: Yes. You grant the role on the target project’s resource, and the member is the service account email from the source project. The service account must also have the iam.serviceAccounts.actAs permission on itself.

Q: How do I find which permission is missing? A: The error message usually contains the exact permission name (e.g., storage.objects.get). Search the GCP IAM roles list for a role that includes that permission.

Prevent This in the Future

The fastest way to catch IAM misconfigurations before they hit production is to monitor Cloud Audit Logs for google.iam.admin.v1 permission denied events. Set up proactive alerting on these logs so your team gets notified immediately when a service account is blocked. Set up uptime monitoring and alerts with Better Stack