Why this matters: When you enable encryption at rest for Amazon RDS or Aurora using AWS KMS, the database instance relies on the KMS service to decrypt its storage volumes on the fly. If the IAM identity or the RDS service-linked role loses its cryptographic permissions, the database may enter an inaccessible state or fail to perform essential operations like snapshots or cross-region replication, leading to silent production failures.

Consider a hypothetical pipeline that utilizes an Amazon Aurora PostgreSQL cluster. You have enforced encryption at rest using a Customer Managed Key (CMK). The infrastructure is deployed via Terraform, and you have strictly scoped IAM roles for the application services connecting to the database. Everything passes CI, but once deployed, the application encounters persistent "Access Denied" errors, and the database instance frequently reports performance degradation during automated snapshot intervals.

Symptoms

You observe the following in your monitoring tools:

  • Application logs report connection timed out or authentication failed errors, even though credentials are correct.
  • The RDS console displays a status of "Available," but attempts to create a manual snapshot fail with an AccessDeniedException.
  • CloudWatch Metrics show a spike in KMSKeyError or similar diagnostic events.

The false lead here is to assume a network or VPC security group issue. You might spend hours checking Security Groups or NACLs, only to find that the network path is perfectly clear. The culprit is almost always a mismatch between the KMS Key Policy and the IAM role permissions required to perform the kms:Decrypt and kms:GenerateDataKey operations.

black cables
Photo by Victor Barrios on Unsplash

1. The KMS Key Policy is Too Restrictive

The most common failure pattern occurs when the KMS Key Policy does not explicitly grant the RDS service-linked role (or the specific IAM role interacting with the database) the necessary permissions to perform cryptographic operations.

How to confirm:
Check the Key Policy in the AWS KMS console or via the CLI:

aws kms get-key-policy \ --key-id <your-kms-key-id> \ --policy-name default

How to fix:
Ensure your Key Policy allows the RDS service to use the key. A standard policy block should look like this:

{ "Sid": "AllowRDSUse", "Effect": "Allow", "Principal": { "Service": "rds.amazonaws.com" }, "Action": [ "kms:Encrypt", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*", "kms:DescribeKey" ], "Resource": "*" }

2. Missing Permissions for the IAM User/Role

Even if the Key Policy is open, your application or the user attempting to access the database must have an IAM policy that allows them to use the key.

How to confirm:
Check the IAM policy attached to the principal executing the connection:

aws iam simulate-principal-policy \ --policy-source-arn <your-iam-role-arn> \ --action-names kms:Decrypt kms:GenerateDataKey \ --resource-arns <your-kms-key-arn>

How to fix:
Attach an inline or managed policy to the IAM role that allows the kms:Decrypt action for your specific key ARN. Avoid using Resource: "*".

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:GenerateDataKey" ], "Resource": "<your-kms-key-arn>" } ] }

3. KMS Key Scheduled for Deletion

If you recently performed a cleanup of "unused" keys, you might have inadvertently scheduled the key used by your RDS instance for deletion.

How to confirm:
Check the key status:

aws kms describe-key --key-id <your-kms-key-id>

Look for the KeyState field. If it is PendingDeletion, the database will lose access to the key material required to decrypt its storage.

How to fix:
Cancel the deletion immediately if the waiting period has not expired:

aws kms cancel-key-deletion --key-id <your-kms-key-id>

4. Cross-Account KMS Access Failures

In complex architectures, the RDS instance might reside in one account while the KMS key resides in a centralized security account. This requires a two-way trust: the KMS key policy must trust the RDS account, and the RDS account's IAM role must have permission to use the key.

How to confirm:
Check if the Key Policy allows access from the external account ID:

{ "Sid": "AllowExternalAccountAccess", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<external-account-id>:root" }, "Action": "kms:*", "Resource": "*" }

How to fix:
Update the Key Policy to include the ARN of the IAM role in the remote account, or the root user of that account, then ensure the remote IAM role has the corresponding policy described in point #2.

a colorful toy on a table
Photo by Shubham Dhage on Unsplash

Preventing it

  • Infrastructure as Code (IaC) Validation: Use tflint or checkov to scan your Terraform templates for overly permissive or missing KMS key policies before deployment.
  • Service-Linked Roles: Never attempt to manually modify or delete the AWSServiceRoleForRDS service-linked role. It is managed by AWS and required for RDS to interact with KMS.
  • CloudTrail Auditing: Set up an EventBridge rule to alert you whenever a kms:DisableKey or kms:ScheduleKeyDeletion event occurs.
  • IAM Access Analyzer: Enable IAM Access Analyzer to identify if your KMS key policies allow access from outside your account, which helps in identifying accidental misconfigurations in multi-account environments.

Clean up

If you created resources for testing, ensure you clean them up to avoid ongoing costs:

  1. Delete the RDS/Aurora instance: aws rds delete-db-instance --db-instance-identifier <your-instance-id> --skip-final-snapshot (Use --final-db-snapshot-identifier if you wish to retain data).
  2. Disable/Delete the KMS Key: If the key is no longer needed, schedule its deletion: aws kms schedule-key-deletion --key-id <your-kms-key-id> --pending-window-in-days 7.
  3. Remove IAM Policies: Delete any temporary IAM roles or policies created for testing the access scenarios.

Cover photo by Kedibone Isaac Makhumisane on Unsplash.

This article was drafted with AI assistance and is intended for technical review and fact-checking before publication.