AWS DevOps Agent, Part 3: I Attacked My Own Web Server. Here's What the AI Found in the VPC Flow Logs.

Part 3 of 4. Overview · Setup · EC2 · Network · RDS

Part 2 was an operations problem. This one is a security problem: a public web server gets scanned and flooded. Can AWS DevOps Agent work out who did it, how, and what to fix, using only VPC Flow Logs and CloudWatch?

The setup

Lab architecture (redrawn from the AWS DevOps Agent workshop)

• Lab3-WebServer: a public EC2 instance
• VPC Flow Logs going to CloudWatch Logs (/aws/vpc/flowlogs/Lab3-networksec)
• Lab3-UnusualConnections: an alarm on NetworkPacketsIn with a sum above 500 per 5 minutes

From my own laptop I ran the workshop's simulate-attack.sh against the server's public IP:

chmod +x simulate-attack.sh ./simulate-attack.sh <WEB_SERVER_PUBLIC_IP> 

The script runs four phases over about 2 minutes:

1. Port scan of 15 common ports
2. Web enumeration of admin and login pages
3. SSH brute force with common usernames
4. Connection flood of 450+ HTTP requests

About 5 minutes later, Lab3-UnusualConnections went to In alarm.

Starting the investigation

• Details: Suspicious network traffic detected from external source - need to analyze attack patterns and assess security posture
• Starting point: CloudWatch alarm Lab3-UnusualConnections triggered due to high incoming network traffic. Investigate source, attack methodology, and recommend security improvements.

The workshop says to expect 8–12 minutes, because querying Flow Logs takes longer than reading metrics.

The verdict: Severity high

The summary had three layers, and that structure is what I'd want from a human analyst as well:

• Root cause: security group sg-… exposes TCP 22/80/443/8080 to 0.0.0.0/0. This was baked into the CloudFormation template at stack creation… never tightened since. CloudTrail confirmed there were no later manual changes.
• Direct trigger: because port 80 was open to everyone, an external IP (my laptop) ran a recon port scan and then a Layer-7 HTTP flood on TCP/80: 2,764 accepted packets, about 80% of inbound traffic. It also found a similar flood from a different IP on 30 September, three days before my test and before I had run anything, so someone else had found the same open host.
• Contributing cause: the NetworkPacketsIn alarm also fires on benign AWS/S3 return traffic to ephemeral ports.

It described the overall pattern as "opportunistic, rotating-IP internet scanning/flooding of an exposed public host, not a single persistent actor, and not a control-plane compromise."

The false-positive finding

I only asked about the latest alarm. The agent went back through every firing of Lab3-UnusualConnections since 30 September and broke each one down:

3 of the 4 'recurring spike' windows were dominated by AWS service IP ranges (52.217.x, 52.216.x, 16.182.x) sending large packet bursts to high ephemeral destination ports, the signature of legitimate S3/download return traffic, not an attack.

In other words, the alarm is mostly noise. A plain packet-count threshold can't tell an aws s3 cp apart from a flood. If an on-call engineer was paged for all four of those, they would soon start ignoring it, and that's how a real attack gets missed.

Findings and evidence

The findings list shows how the agent labels its own reasoning:

• 🔴 root cause: the world-open security group since stack creation
• 🟠 cause: open ports enable unsolicited scans; benign S3 traffic triggers the alarm
• ⚪ hypothesis, ruled out: "Security group modified after creation to open SSH/HTTP/HTTPS". CloudTrail showed it was never modified.
• 🔵 symptom: recurring short spikes since 30 September
• ⚪ gap: GuardDuty access denied by SCP
• ⚪ observations: no misuse of instance-role credentials, VPC Flow Logs confirm the flood and scan, no control-plane changes
• ⚪ topology: instance, VPC, subnet, security group, flow log ID and log group, all mapped

Keeping the ruled-out hypotheses in the list is useful. In a postmortem, what it wasn't matters as much as what it was.

Mitigation plan, with guardrails

The plan: remove the 0.0.0.0/0 rules for 22 and 8080 completely, and scope 80/443 to trusted clients or an ALB+WAF, then reconcile the CloudFormation stack so the fix lasts.

The pre-validate steps are careful:

• Save the current rule set as the rollback baseline
• Confirm VPC Flow Logs are active so you can see the effect of each removal
• Check recent ACCEPTed flows on each port so you don't cut off a legitimate client
• Check that the instance is reachable through SSM Session Manager before removing port 22, so you don't lock yourself out

One more detail: the apply steps that revoke security group rules were flagged "Automated execution of this removal is denied by the safe-deployment deletion-operation policy, must be run manually by an operator." The agent can propose destructive changes, but it won't run them itself. That's the right default.

Follow-up chat

I worked through the workshop's suggested questions in chat.

"What specific ports were targeted?" It returned the full scan list (1433, 139, 21, 5432, 3306, 9200, 23, 53, 135, 3389, 8080, 443), then pointed out that the attack that actually succeeded was only on port 80 (2,764 packets) and port 22 (57 packets, "too small to be a brute-force attempt"). It explicitly ruled out SYN flood and SSH brute force as the attack type.

"How can I prevent similar attacks?" Lock down the security group, put WAF with rate limiting in front of 80/443, enable GuardDuty (which was blocked by an SCP in this account), and fix the alarm's false-positive rate.

"Should I implement AWS WAF?" It said yes, and then added the catch: "WAF attaches to CloudFront, ALB, or API Gateway, not directly to an EC2 instance… That's a bigger architectural change than just 'turning on WAF'." It suggested a path (ALB → WAF rate-based rule → security group only from the ALB → managed rule groups) and asked for confirmation before building anything.

Takeaways

• 🕵️ It turned Flow Logs into a clear attack story: the source IP, the phases, which ports were only probed and which were actually hit
• 🔕 It found an alarm-quality problem I hadn't asked about. 3 of 4 firings were S3 noise.
• 🧪 It kept ruled-out hypotheses visible, which is useful for audits and postmortems
• 🛑 Destructive changes are blocked from auto-execution by policy
• 🧱 It noticed the misconfiguration came from the IaC template, and the fix targets the template

Finally, in Part 4: an RDS MySQL instance that runs out of connections, and a case where the agent's chat answers contradicted its own mitigation plan.