Security alert fatigue and fine-tuning to combat it

Alert fatigue is a real problem in IT security. This can happen at the worst time, when analysts check their tools and see another event, or even another 50-100 events, after the one they just checked. They click through the events looking for the smallest reason they can find to dismiss the event and not have to escalate or investigate the problem further.

They’ve been through this before, can see where the real problems are, and just want to get rid of these events and get on with doing other work. Unfortunately, as many analysts know, an innocent looking event could be the clue to something more serious. Every event must be thoroughly investigated to make sure there is no evidence of error.

When going through alerts multiple times, the fact that they can be very similar is a big part of alert fatigue. Another part of the cause is false positives. Analysts can have difficulty maintaining vigilance when most of the events they go through are false positives. New technologies have emerged that claim they can reduce the number of false positives. While they may, or may not, be effective at ingesting alerts and identifying true positives, this only adds to the analysts’ workload by creating another tool to log in and receive alerts.

A Tripwire article describes alert fatigue as a combination of too many false positives, as well as a reason to increase your organization’s security awareness. Another CSO article points out that there are a large number of organizations dealing with too many false positives that overload their analysts. Do you agree? Let’s look at some steps you can take to combat this.

Tuning

This seems obvious, but is often overlooked. Tuning is a combination of reducing false positives, working with alerts, and correlating events and trends to ensure greater accuracy. Each of these helps the analyst refine the alerts being investigated. Tuning should be a balanced approach that reduces the number of unnecessary events received and ensures there are no blind spots that an attacker can exploit to go undetected.

Which alerts are of interest?

Eliminating blocked attacks helps the analyst pay more attention to potential incidents that were not stopped. Once this is done, the next issue of importance is: Which alerts are of interest? Determining this requires a bit of research. It should be determined from what affects you the most, to what might be a threat but may or may not be worth investigating. That involves knowing:

– where the sensitive information is located

– how it can be accessed and how it should be accessed (two very different things)

– who has access

– what traffic is normal on the network

– what should be on the endpoints, (the security baseline for the endpoints)

– and many other variables

This part is by far the longest part of the tuning process and don’t think the information is static. Once you have gathered this information, policies, standards and procedures should be established for when a department or vendor changes the way they interact with the network and endpoints. Keep in mind, however, that even if you manage to get those standards out, you will most likely be the last to know what changes were made and why. Being adaptable to change is key. This is the baseline for the network.

The baseline defines what is “normal.”

The baseline gives us what is normal. However, one should not be alerted to something just because it deviates from the baseline. There are many events that will create a deviation that are not malicious in nature. The goal is to create alerts that are malicious and are causing a deviation.

You may also want to limit the focus to targeted attacks. An example of this might be scanning activity. If you are scanning endpoints with a vulnerability scanner, you should exclude it from alerts.

Threat landscape

The most difficult challenge is determining which external attacks are targeting a particular industry, or even the organization itself.

Determining which alerts are valuable requires knowing what is worth investigating. To do that, you need to know what the threat landscape is for in an industry and an organization specifically. Once you have that information, you can get a good sense of what to look out for. If you take your own threat landscape and apply it to the “known good” baseline, you will have a better idea of what alerts you need, what alerts would be useful, and how to configure those alerts for your organization. This, like baselining, is not a static goal and must be kept up to date on new threats and information. Again, adaptability is key to success.

Fine-tuning false positives

Okay, now that you have a list of alerts that the analyst will check and watch for what matters to the organization. However, it’s not over yet. Technology will need to be fine-tuned to detect false positives, adapt to changes in the network, and stay on top of the ever-changing threat landscape. Analysts are a great source of information on this. They can tell you what triggered that new false positive, what might be the new trend they are seeing, etc. But remember when narrowing down a false positive to make sure that only false positives are eliminated. If you refine too much in one direction, you may be creating a blind spot.

Once you have the alerts where you want them, and the analysts have a stable (but not overwhelming) number of alerts, you may want to start looking for opportunities to see how to expand the network and take a look at what else might be useful to look at. It is advisable not to overload analysts.

A good guideline is to implement new alerts in a test environment while making sure that it is the user who is the only one monitoring the new alert. That way, you can fine-tune the rule before you start acting on it. Doing it this way can help ensure that a rule is delivered that can be trusted immediately. It should be kept in mind that the goal of optimization is to protect the organization. This can only be achieved when alerts can be investigated in a timely manner and analysts can review events without doubting the validity of the alerts or being overwhelmed by the volume of alerts.

Management support for security analysts

What if you already have a list of alerts that analysts are seeing? Well, the best approach is to make sure you have management approval to implement an action plan to review and replace existing alerts using the above process.

That will help in the long run, but there can be problems with this. Management may require certain things to be looked at because of historical issues. Alerts may want to be set up for specific circumstances. As most know, nothing can happen in the ideal way. However, by working together with management, you will be able to create a list of alerts and criteria that will work in everyone’s favor.

The support of the management team is important. They are the ones who weigh the requirements assigned to them against the associated risk to determine best practices. Improving visibility while reducing the risk of alert fatigue is good for the organization and management. As such, it is not difficult to find common ground and will ensure that objectives are not missed during the tuning process.

Conclusion

Redborder, with the goal of protecting organizations, takes an active role in technology. But it should not be forgotten that it is necessary to empower analysts by providing them with well-defined alerts. This will increase the tool’s capabilities in protection and help prevent alert fatigue.

For easy reference, here is an easy-to-review checklist that will help you in the tuning process:

1 Check the queue of events that analysts go through each day.

2 Remove events that are blocked from the queue, but analyze them for trends and possible targets and weaknesses within the organization.

3 Determine baselines in the network and endpoints.

4 Determine the threat landscape and relevant alerts.

5 Work with analysts to rule out false positives.

6 Determine which new alerts could provide additional value to the organization and deploy them only once they are tuned to an appropriate level.

7 Remain vigilant and repeat steps 3 through 7.