Testing Microsoft Sentinel detection coverage - Case AWS
SIEM users should consider performing testing to validate that their detections actually work, especially if they are doing actual detection engineering (instead of just relying on vendor-provided rules).
Building a testing methodology is a very organization specific task. My goal for this article is to demonstrate some basic principles and one specific example for detection testing in Microsoft Sentinel.
If we compare the SIEM world to generic testing methodologies, we could come up with the following taxonomy:
Integration testing can be done to validate that a SIEM integration is receiving data. In Sentinel terminology this would mean validating the Data Connector health and data integrity in Log Analytics table.
Unit testing can be done for verifying that an individual detection or use case is operating optimally, focusing on a single Analytics Rule or a selection of related rules.
End-to-end testing can be done to verify the functionality of all detections, automations and data integrity related to a SIEM integration.
Since we are basically borrowing terminology from another field (software development), I imagine there will be other opinions on these as well.
There would also be a lot to talk about testing methodologies when it comes to SIEM and detection engineering in general, or Microsoft Sentinel in particular. I see many opportunties for future articles around this topic.
In order to get at least something published before my summer holidays, I am now focusing in one end-to-end testing scenario: validating Amazon Web Services CloudTrail coverage in Sentinel, using a public testing tool that can simulate multiple known attack patterns against AWS.
AWS & Sentinel configuration
My test environment is a multi-account AWS environment integrated to Microsoft Sentinel:
Testing target is an individual AWS account that is a member of an AWS Organization. The testing account is disposable and created just for running this testing sprint.
CloudTrail logging is collected in an Organization Trail. The trail is configured to collect only Management Write type events (a common CloudTrail setup for SIEM integrations). Sentinel recieves this data with the S3 Data Connector.
GuardDuty and VPC flow logs are disabled in the testing target, as I only want to see CloudTrail based detection coverage in this scenario.
In Sentinel I have enabled all the Microsoft provided rules that target the Amazon Web Services data source as can be seen in the screenshot. This also includes GuardDuty and VPC based rules, but as those data sources are not being ingested it does not really make a difference here.
Testing tool (Stratus)
For the testing tool I chose Stratus Red Team.
Stratus has a pretty good AWS adversary simulation coverage. Also as a big benefit for me Stratus does not require existing infrastructure, as it deploys the test resources with Terraform when running the tests.
You can see all tests available currently in Stratus here: https://stratus-red-team.cloud/attack-techniques/list/
For this testing sprint I chose to limit the number of tests to cover two ATT&CK Tactics, Credential Access and Defense Evasion:
+--------------------------------------------------------+----------------------+
| TECHNIQUE NAME | MITRE ATT&CK TACTIC |
+--------------------------------------------------------+----------------------+
| Retrieve EC2 Password Data | Credential Access |
| Steal EC2 Instance Credentials | Credential Access |
| Retrieve a High Number of Secrets Manager secrets | Credential Access |
| Retrieve And Decrypt SSM Parameters | Credential Access |
| Delete CloudTrail Trail | Defense Evasion |
| Disable CloudTrail Logging Through Event Selectors | Defense Evasion |
| CloudTrail Logs Impairment Through S3 Lifecycle Rule | Defense Evasion |
| Stop CloudTrail Trail | Defense Evasion |
| Attempt to Leave the AWS Organization | Defense Evasion |
| Remove VPC Flow Logs | Defense Evasion |
+--------------------------------------------------------+----------------------+
Running the tests
To get easily reportable test results, I decided to run the Stratus tests one-by-one, making sure each test has enough time to complete the end-to-end scenario:
Perform the test (“Detonating” in Stratus terminology).
Wait for the activities to be logged in CloudTrail.
Wait for the Sentinel data connector polling interval to run.
Wait for the scheduled Analytics Rules to run.
Running the Stratus tests is simple. You need AWS CLI configured and authenticated on the test machine. There are some things to understand about Stratus, so be sure to read the documentation, but in practice the tests are very simple to run. Example test detonation and results below:
~ stratus detonate aws.credential-access.ec2-get-password-data
2022/06/08 14:18:44 Checking your authentication against AWS
2022/06/08 14:18:44 Warming up aws.credential-access.ec2-get-password-data
2022/06/08 14:18:44 Initializing Terraform to spin up technique prerequisites
2022/06/08 14:18:49 Applying Terraform to spin up technique prerequisites
2022/06/08 14:18:58 Running ec2:GetPasswordData on 30 random instance IDs
Collecting the results
For each test I recorded the related events in CloudTrail, verified that the event is found in Log Analytics and checked if any of the built-in Analytics Rules detected the activity.
Some initial comments for a couple of suprising findings were also recorded as soon as discovered. Further analysis will be done later, detections that do not work as expected will be fixed as part of normal detection engineering processes. Some things will end up as not beight detectable at all without too high false positive penalties.
I kept the reporting pretty simple for now, as you can see:
For this article I was mainly focusing in building this methodology and testing practices, so some individual tests do not have comments or development plans. But as you can see, it looks like there is some work to do for improving AWS detection coverage in this environment.
Conclusions
Building a testing program for your Security Operations can be a good way to improve your detection coverage. It sounds like a thing we could take for granted, but in reality it is not as common as could be expected.
There are a few things that need to be considered if you want to start a similar testing process:
Select the testing scope carefully. For example organise testing sprints that have a clear frame (such as a single ATT&CK Tactic).
It is better to run single tests in series instead of multiple in parallel. Some testing tools have a “fire all tests” option, but it is easier to verify the results when running individual tests.
Make sure you can separate the test actions from other log data (for example by identifying the User Agent of the testing tool).
I am looking forward to continuing this topic by running the rest of the AWS tests for all ATT&CK Tactics, and also to evaluate how to further develop the testing process itself.
One practical development item here is how to integrate testing into detection engineering; when a new rule is being developed, how to design the unit test at the same time. Stay tuned.