The scenario is fictional (RFC 5737 IPs). Detection only; no attack steps.
You found the intrusion, killed the process, changed the password. Done? Attackers plan to be locked out. Within minutes of getting in they build a second and third way back.
What you'll learn
- The two classic persistence spots: SSH keys and cron
- Baseline diffs that find them in minutes
- How to tell admin changes from attacker changes
- Why you preserve before you clean
The story at a glance
| Time | What happened |
|---|---|
| 02:20 | Root login succeeds after brute force |
| 02:26 | An unknown key appears in root's authorized_keys |
| 02:27 | A cron entry runs every minute |
| Later | Password reset and process kill don't help |
Keys: diff against a saved baseline
find / -name authorized_keys -exec sha256sum {} \; | sort > /var/lib/keys_base.txt # normal times find / -name authorized_keys -exec sha256sum {} \; | sort > /var/lib/keys_now.txt # investigation diff /var/lib/keys_base.txt /var/lib/keys_now.txt In my sandbox test only root's file showed up in the diff.
Cron: new files and fetch-and-execute
find /etc/cron.d /var/spool/cron -type f -exec sha256sum {} \; | sort grep -REl 'curl[^|]*\|[[:space:]]*(ba)?sh' /etc/cron.d /etc/crontab /var/spool/cron The grep flagged the fake malicious cron and ignored a legit backup job. Legit tools sometimes use curl | bash, so check whether the destination is yours.
Normal vs. attack
| Normal | Attack | |
|---|---|---|
| Changed by | config management (ansible), uid 0 | www-data (uid 33) with euid 0 |
| Path | change ticket exists | from the web server process |
| When | maintenance window | minutes after intrusion |
Catch it live: auditd (from the book, not sandbox-tested)
-w /root/.ssh/authorized_keys -p wa -k ssh_key_change -w /etc/crontab -p wa -k cron_persist -w /etc/cron.d/ -p wa -k cron_persist -w /var/spool/cron/ -p wa -k cron_persist -w /etc/systemd/system/ -p wa -k systemd_persist Preserve first, clean later
Copy files, hash, record stat times and who changed them. Only then remove the key line and the cron file.
Takeaways
Assume redundancy: keys, cron, systemd, profile.d, other users' ~/.ssh. Save the baseline before you need it.
Next: SSH that skipped the bastion.
Adapted from chapters 26 and 9 of my Japanese book on server defense.
