Welcome to Volume 5, Issue 7 of The Internet Security Conference Newsletter, Insight. Insight provides commentaries and educational columns, authored by some of the best minds in the security community.
TISC is about sharing clue. So is this newsletter. We promise to provide something useful each issue. If we don't, flame me. If you like the issue, let us know!
Enjoy, and be safe,
Dave
Logging and auditing information helps security administrators determine if security policy is implemented correctly, and if systems have been or are under attack. Remarkably, gathering and integrating event-related data from security systems into a single repository so security analysts can correlate and analyze them remains a difficult and human-intensive task.
Automating the processes of security information collection; normalization of diverse security event-related information into a common format; and analysis is the Holy Grail the industry calls Security Information Management, SIM. In today's column, Anton Chuvakin examines one aspect of SIM: event correlation.
Interested in SIM? Visit my May 5, 2003 blog entry, What is Security Information Management?, and Anton's earlier column, Cross Platform Security Analysis.
Happy Reading!
In the deep and somewhat muddy sea of security marketing terms, correlation appears to be the current pack leader, closely chased by intrusion prevention. In this paper we will try to cast an objective look at what security correlation really is and how it helps to improve the organization's security posture.
A recent security spending survey by Information Security Magazine indicates that deployment rates of many security technologies will soar in the next three years. All the above devices, whether aimed at prevention or detection, generate huge volumes of audit data. Firewalls and other devices logging network connection information are especially guilty of producing vast oceans of data. Many diverse data formats and representations are used for those log files and audit trails. Also, a percentage of events generated by network IDS and IPS are false alarms and do not map to real threats. To further confuse the issue, different devices might report on the same things happening on the network, but in a different way, with no apparent way of figuring the truth of their relationship.
There is a definite need for a consistent analysis framework to identify various threats, prioritize them and learn their impact on the target organization. This must be done as fast as possible (preferably in real- time) for attack identification. It is also important that analysis be performed also over the long term for threat trending and risk analysis. For a detailed summary of security data analysis challenges see my earlier column.
SIM (Security Information Management) is the "discipline" that will solve the correlation security event data challenge. Correlation is defined as "establishing or finding relationships between entities". However, the good security-specific definition is lacking.
In security, event correlation may be defined as improving threat identification and assessment process by looking not only at individual events, but at their sets, bound by some common parameter.
Security-specific correlation can be loosely categorized as rule-based or statistical (algorithmic). A rule-based correlation engine has some pre- existing knowledge of the attack (the rule) and from this is able to define what has actually been detected in precise terms. Such attack knowledge is used to relate events and analyze them together in a common context.
Statistical correlation does not employ any pre-existing knowledge of the malicious activity, but instead relies upon the knowledge (and recognition) of normal activities, which has been accumulated over time. Ongoing events are then rated by a built-in algorithm and may also be compared to the accumulated activity patterns, to distinguish normal from abnormal (suspicious).
This distinction among correlation types is somewhat similar to signature vs. anomaly IDS, and takes a SIM function as a kind of meta-IDS, operating on a higher-level data: log records as opposed to packet (streams). Combined, both correlation methods can help to sift through the large volume of diverse data and identify high severity threats.
Rule-based correlation engines apply scenarios that an known attack must follow to detect exactly this attack. Such scenarios might be encoded in the form of a sequence of events (first this, then that...), therefore some action(s) must trigger the detection process.
Rule-based correlation deals with states, conditions, timeouts and actions. Let us define those important terms. A state is a well-defined logical or operational mode that the correlation rule might be in. A state may contain various conditions, such as matching incoming events by the source IP address, protocol, port, event type, producing security device type, username and other data components of the event. Although data components vary from device to device, a SIM solution typically normalizes many data component formats using a generic or uniform event schema. This schema must be rich enough to normalize data without incurring any information loss
A timeout defines how long the correlation engine will be in a certain state, for a given rule. If the correlation engine must maintain a lot of rules in waiting state in memory, this resource might be exhausted. Thus, rule timeouts plan an important role in correlation performance. A transition is an event when one rule state is switched to another one. For a complicated rule, many transitions are possible. An action is response event, initiated when all the rule conditions are met. Various actions may result from rules, such as user notification, alarm escalation, configuration changes or automatic incident case investigation.
The correlation engine is able to track various states and switch from state to state, depending on conditions and incoming events. The correlation engine receives events in real time from the alarm-generating security devices it monitors, and applies the relevant correlation rules to the event flow, taking into account any data filtering (e.g., log records that are not considered relevant).
Correlation rules may be applied to incoming events in real-time (as they arrive) or to the historical events stored in the database. In the latter case, the rules are used as a form of data mining or analytics, which helps uncover concealed activity, e.g., threats such as slow port scans or low level activity from Trojan or exploitation software operating on a compromised host. Such rules may be run periodically, for incident identification, or in the course of the investigation of suspicious activity, for seeking out the prior occurrences of similar (and thus possibly related) activity.
Unlike real-time rules, which become useless if the incidence of false alarms is too high (as is the case for poorly designed and administered signature-based IDSs), database rules can tolerate a certain level of false alarms for the purpose of drastically reducing false negatives. This is due to the fact that real-time rules usually feed the alarm notification system, while database rule-based correlation will be launched by the analyst during security incident the investigation. As long as the rule-based analytics will uncover a hidden threat, which is impossible to discover otherwise, an analyst might be able to tolerate a certain level of false alarms.
Statistical correlation uses special numeric algorithms to calculate threat levels incurred by the security relevant events on various IT assets. Such correlation looks for deviations from normal event levels and other routine activities. Risk levels may be computed from the incoming events and subsequently tracked in real time or historically, so that deviations become apparent. Algorithmic correlation may leverage event categorization to compute the threat levels specific to various attack types, such as the threat of a denial of service or worm/virus attack, and track them over time.
Detecting threats using statistical correlation does not require any pre- existing knowledge of the attack to be detected. Statistical methods may, however, be used to detect threats on pre-defined activity thresholds. Such thresholds may be configured based on the experiences monitoring the environment. For example, if normal level of specific reconnaissance activity (e.g., port scans) is exceeded for a prolonged period of time, the alarm might be generated by the system.
Correlation may also use various parameters for enterprise assets to skew the statistical algorithm for higher accuracy detection. Some of them are defined by system users (such as the affected asset value to the organization) or are automatically computed from other available event context data (such as vulnerability scanning results or measure of normal user activity on the asset). That allows us to define broader context for transpiring security events and thus help understand how they contribute to the organization's risk posture.
Both types of correlation have inherent challenges, which can fortunately be mitigated by combining both methods to create coherent correlation coverage, leading to quality threat identification and ranking.
First, can we assume that the attacker will follow a scenario, which can be caught by the rule-based correlation system? Unlike the network IDS system that needs a specific signature with detailed knowledge of the attack, a correlation system rule may cover the broad range of malicious activities, especially if intelligent security event categorization is utilized. It may be done without going into the specifics of a particular IDS's signatures. For example, rules may be written to look for certain activities that usually accompany the system compromise, such as backdoor communication or the installation of hacker tools on a compromised system. An attacker is hard-pressed to avoid installing rootkits, RATS and other code if he intends to use the compromised machine for his purposes. Extensive research using deception networks or honeynets allows us to learn more and more of the attackers' patterns of behavior and to encode them as correlation rules, available out of the box.
Second, can multiple rules cause the number of false positives to actually increase instead of decrease? Indeed, deploying many rules without any regard to the environment might generate false alarms.
However, it is much easier to understand and tune the SIM correlation rules than intricate binary matching patterns. The latter requires in- depth understanding of the attack network packets, memory corruption issues and specifics of the exploitation techniques. On the other hand, tuning the correlation rule involves changing the timeouts and adding or removing conditions. Overall, in case of correlation rules, one may also define response actions with higher confidence, since one can bind the rules to a specific asset or group of assets.
Third, rule-based correlation is relatively intensive computationally. However, using highly optimized correlation engines and intelligently applying filters to limit the flow of events allows gaining maximum advantage of the rule-based correlation. Additionally, many rules can be combined together so that the correlation engine does not have to keep many similar events in memory. It also makes sense to apply more specific correlation rules to a large number of assets, where false positives flood might endanger the security, and to apply wider and more generic rules to critical assets, where an occasional false alarm is better than missing a single important alert. This way all the suspicious activities directed against a small group of critical assets will be detected.
Fourth, statistical correlation may not pick up anomalous activity if it is performed at low enough levels, essentially merging with the normal. Hiding attack patterns under volumes and volumes of similar normal activity might deceive the statistical correlation system. Similarly, a single occurrence of an attack might not impact the statistical profile enough to be noticed. However, careful baselining of the environment and then using statistical methods to track the deviations from such baseline might allow detecting some of attacks that are "stealthy". Also, rule- based correlation efficiency compensates for those rare events and enables their detection, even if algorithmic correlation misses them.
SIM products leveraging advanced correlation techniques and intelligent alert categorization may become indispensable. It certainly seems that, as enterprises deploy more and more security point solutions, appliances and devices, human correlation alone will become increasingly difficult, and inevitably, impractical. The situation we find ourselves in today is one where "best of breed" is a practical necessity, but this typically means that we have many security devices, and each one only addresses certain aspects of the overall security services. Thus, we need to integrate the interpretation of security events under some common umbrella. Security Information Management solutions are promising alternatives to the human correlation we rely on so heavily today.
Anton Chuvakin, Ph.D., GCIA, GCIH (www.chuvakin.org) is a Senior Security Analyst with a major security information management company. His areas of infosec expertise include intrusion detection, UNIX security, forensics, honeypots, etc. In his spare time he maintains his security portal, www.info-secure.org.