Security Incident Response

In one minute

• Security incidents follow the same incident command structure as outages, with three differences: there is an adversary who adapts, evidence must be preserved, and legal and disclosure obligations start a clock.
• The classic lifecycle (NIST SP 800-61 Rev. 2): prepare; detect and analyse; contain, eradicate and recover; learn. Rev. 3 (2025) re-maps these onto the six Cybersecurity Framework 2.0 functions. Containment comes before full understanding, but not at the cost of destroying evidence.
• For leaked credentials, the most common modern incident: revoke and rotate first, then work out what the credential could do and what it actually did, from audit logs.
• Assume the attacker may still be inside and watching your channels. Coordinate on a separate channel, and be careful about tipping them off before containment is ready.
• Agents add new paths: prompt injection that exfiltrates data, leaked agent tokens, and agents that commit secrets. The same playbook applies, with agent identities and tool-call logs as evidence.

How it works

What makes security incidents different

In an outage, the system is your adversary only by accident. In a security incident, a person is actively trying to keep access, hide their tracks and expand their reach. That changes three things. Evidence: logs, disk images and memory may be needed to understand scope and for legal reasons, so you capture them before rebuilding machines. Secrecy: until containment is ready, alerting the attacker can make them destroy evidence or dig in deeper. Obligations: data protection laws and contracts often require notifying regulators or customers within fixed deadlines (GDPR's is 72 hours after the controller becomes aware of a personal data breach, where feasible; a processor must tell the controller without undue delay and has no separate 72-hour clock, per GDPR Article 33), so legal and communications join early.

The lifecycle

Prepare. Logging that would let you answer "who did what, when" (cloud audit logs, authentication logs, database access logs), retained for months. A response plan with roles and contacts, including legal. Break-glass accounts. Practised drills.

Detect and analyse. Alerts from secret scanning, anomaly detection (logins from new countries, unusual API calls), customer or researcher reports. Confirm it is real, assess severity, declare. Most alerts are not crises: triage weighs the data at risk, the trust relationships the attacker could use, which controls still hold, and whether the activity looks commodity or targeted (Building Secure and Reliable Systems, ch. 17). Name an incident commander who accepts the role, as in any other incident.

Contain. Short-term: revoke credentials, isolate hosts (network quarantine rather than shutdown, to keep memory evidence), block attacker IPs, disable compromised accounts. Long-term: patch the hole, harden. The pull towards fixing at once is strong, but locking accounts or wiping machines before you know the scope tells the attacker you are there, and they may hide footholds or destroy data. Google's security responders therefore finish enough investigation first, unless an imminent, clearly identified risk justifies shutting things down and revealing the response (Building Secure and Reliable Systems, ch. 17).

Eradicate and recover. Remove persistence (new accounts, keys, scheduled tasks, backdoors the attacker created), rebuild from known-good images, restore data, monitor closely for return.

Learn. Blameless review, as for any incident, plus the specific question: how did they get in, and how would we have detected it sooner?

NIST published SP 800-61 Rev. 3 in April 2025, superseding the 2012 revision. It drops the four-phase loop as the organising model and uses the six CSF 2.0 functions instead (Govern, Identify, Protect, Detect, Respond, Recover). Detect, Respond and Recover are the incident response itself; Govern, Identify and Protect are the wider risk management that supports it, and lessons learned are meant to flow into improvement continuously instead of waiting for the end (NIST SP 800-61r3). The activities in this lesson are the same; only the packaging changed.

The leaked credential playbook

A secret in a public repository, a log file, a screenshot, or an agent's context sent to the wrong place. Steps:

1. Revoke or rotate immediately. Assume it was used. Automated scanners find public secrets within minutes.
2. Determine capability: what could this credential do? Its IAM policy, scopes and network restrictions.
3. Determine use: what did it do? Query audit logs for every action by this credential since the earliest time it could have leaked, separating known legitimate sources (your servers' IPs, CI) from unknown ones.
4. Scope impact: which resources were read, changed or created; was data copied out; were new credentials or accounts created (persistence)?
5. Remove persistence and fix the leak path (secret scanning in pre-commit and CI, secrets in a manager, short-lived credentials).

The same shape everywhere

A security incident is the trust-but-verify and partial-failure shapes under adversarial pressure: you cannot trust the system's own view of itself, so you rely on independent records, and you assume the worst until evidence narrows it down. Fraud teams, auditors and physical security follow the same pattern: contain, preserve evidence, scope, remediate, report.

Down the stack

Diagram

flowchart TD
  D["Detect: secret scanner alert"] --> R["Revoke and rotate key now"]
  R --> C["Capability: what could it do?"]
  C --> U["Use: audit logs since leak time"]
  U --> S["Separate known sources from unknown"]
  S --> I["Impact: resources read, changed, created"]
  I --> P["Remove persistence: new keys, users, roles"]
  P --> L["Legal and comms decide notification"]
  L --> F["Fix leak path, learn"]

Worked example

At 14:05, GitHub secret scanning alerts that an AWS access key was pushed to a public repository at 13:52 in a demo project. The key belongs to a CI user with read access to two S3 buckets and permission to create IAM access keys (an over-broad permission left from an old migration).

14:08: the key is deactivated. 14:10: an incident is declared with a security lead as incident commander, a separate channel, and legal informed. 14:15 to 14:50: CloudTrail is queried for all events by the key ID since 13:52. Results: 214 events from the CI system's known NAT IPs (legitimate builds); 37 events from an unknown IP starting at 13:58, six minutes after the push. Those include ListBuckets, GetObject on 1,240 objects in the exports bucket, and one CreateAccessKey for a different IAM user at 14:01.

The created key is persistence: the attacker had a second key that survived the first revocation. It is deactivated at 14:52; queries on it show 12 more calls, all reads, ending at 14:40. The exported objects included a customer CSV with names and emails for about 30,000 users, so legal begins the notification assessment within the 72-hour window. Fixes: remove iam:CreateAccessKey from CI, replace long-lived keys with short-lived OIDC federation for CI, pre-commit secret scanning, and an alert on any CreateAccessKey call.

In the wild

• Cloudflare, 2023: credentials missed in a rotation. After the October 2023 Okta breach, Cloudflare rotated credentials but skipped four (one service token and three service accounts) that it wrongly believed were unused. An attacker used them from 14 November, reached Jira, Confluence and Bitbucket, installed the Sliver C2 framework and downloaded 76 repositories before detection on 23 November. The "Code Red" response then rotated more than 5,000 production credentials and reimaged every machine in its global network. Lesson: a rotation is only as good as your inventory of what a credential touches (Cloudflare).
• Okta, 2023: session tokens inside support files. Customers had uploaded HAR files to support cases, and those files contained live session tokens. An attacker who got into a support service account used them to hijack sessions at 5 customers; files from 134 customers were accessed. Detection took about 14 days, partly because viewing a file through a case writes a different log event than opening it from the Files tab. Lesson: audit logs have to record every access path to the evidence you care about (Okta).
• CircleCI, 2022-23: a stolen session beats 2FA. Malware stole an engineer's already 2FA-authenticated SSO session cookie on 16 December 2022; data was taken on 22 December and discovery came on 29 December through a customer report of odd GitHub OAuth activity. On 4 January 2023 CircleCI told every customer to rotate all secrets, cut production access to a small group and added step-up authentication. Lesson: revoke sessions as well as passwords, and tell customers early what to rotate (CircleCI).

Trade-offs

Pitfalls in production

• Rotating the leaked key but not checking what it created. Fix: always search for new credentials, users and roles created by the compromised identity.
• Discussing the incident in channels the attacker may read. Fix: separate, out-of-band channel for response, ideally clean infrastructure prepared in advance, and care with tools that auto-fetch links.
• Rebuilding machines before capturing evidence. Fix: snapshot disks and capture memory, or isolate first.
• Audit logs with short retention or none on data access. Fix: enable data-access logging for sensitive stores, retain for months.
• Late legal involvement. Fix: legal and communications in the response plan from declaration.

Leadership lens

• Make credentials short-lived and scoped by default, especially for CI and agents; long-lived keys are the most common incident root.
• Turn on secret scanning in pre-commit, CI and the code host, with automatic revocation where providers support it.
• Rehearse the leaked credential playbook quarterly, including the audit log queries, so they are ready when needed.
• Include agent identities and tool-call logs in security monitoring; agents are new principals with new leak paths.

Check yourself

1. Why search for actions like CreateAccessKey after a key leak?

   Show answer

   Attackers create new credentials to keep access after the original is revoked; finding and revoking them is part of containment.

2. Why might you isolate a compromised host rather than shut it down?

   Show answer

   Shutting down loses memory evidence such as running processes, connections and keys; isolation stops network activity while preserving it.

3. Why do short-lived credentials reduce incident impact?

   Show answer

   A leaked token expires quickly on its own, shrinking the window in which an attacker can use it.

4. Why coordinate on a separate channel during a security incident?

   Show answer

   The attacker may have access to normal systems, including chat or email, and could watch the response and adapt.

Sources and further study

Primary

• Cloudflare: Thanksgiving 2023 security incident: a detailed first-party account of detection, a credential rotation gap and a full rebuild.
• GDPR Article 33: the 72-hour notification duty for controllers and the separate duty on processors.

Systems docs

• AWS IAM: disabling permissions for temporary credentials: session lifetimes and how to cut off a stolen session.
• AWS IAM identifiers: what the AKIA and ASIA prefixes mean.

Books

• Building Secure and Reliable Systems, Adkins et al.: chapter 15 (Investigating Systems), chapter 16 (Disaster Planning), chapter 17 (Crisis Management, with the operational security advice used above) and chapter 18 (Recovery and Aftermath).
• Site Reliability Engineering, Beyer et al.: chapter 14 (Managing Incidents), the incident command structure that a security response reuses.
• The Site Reliability Workbook: chapter 9 (Incident Response), with worked cases of declaring early and handing off command.

Other curricula

• Google SRE Workbook: Incident Response: frames security response as the same command structure as an outage, which matches this lesson's starting point; it does not cover evidence handling.

Free resources

• NIST SP 800-61 Rev. 3: Incident Response Recommendations: the standard framework, updated in 2025.
• AWS Security Incident Response Guide: cloud-specific preparation, investigation and containment.
• About secret scanning (GitHub Docs): detecting leaked tokens and partner auto-revocation.
• Secrets Management Cheat Sheet (OWASP): storing, rotating and auditing secrets.
• Incident Handler's Handbook (SANS): a practical, step-by-step response guide.
• PagerDuty Incident Response: incident command practices, including security incidents.