Originally published by Interop This Week, reposted with permission.

To date, deployment of Intrusion Detection Systems (IDS) has been a tumultuous and often unrewarding experience for network administrators. Difficult to configure, even for advanced security technicians, and overly susceptible to positive and negative false alarms, ID systems are shut or dummied down. The irony here is obvious: today’s ID system itself is too intrusive for many production environments.

It’s easy to fault ID systems, to conclude “too complicated, too fragile, too prone to error, unnecessary overhead…”, and to shelve the technology. Yet if we made this choice 20 years ago, we wouldn’t have a global, dynamic, adaptive routing system for the Internet. Routing configuration was and remains an “expletive deleted”. Over two decades of deployment, we’ve overcome many of the early hurdles and have built a more reliable and scalable routing system than anyone imagined, even 10 years ago. Every new technology needs time to mature from innovation to mainstream. Anyone familiar with the history of telecommunications appreciates that change in telephony is measured in decades. But the Internet is overly influenced by Moore’s Law.  Somewhat like the capacity of integrated circuits, we expect Internet technology to prove and improve itself every 18 months.

In Nanotechnology without Genies, Lyle Burkhead’s chapter There is no Moore's Law for software offers a crystal clear insight into why applying Moore’s Law is inappropriate to software as a whole, and in my opinion, to ID systems in particular. IC’s specifically and hardware in general improve because we begin “with the idea of a circuit, and implement this idea on successively smaller and faster substrates”. Software innovations, from search engines to Intrusion Detection, are successive refinements of abstract ideas into an execution of processes that emulate intelligence.

Intrusion Detection is like chess, or a game of network cat-and-mouse. ID software to date commonly analyzes the actions of an attacker in more or less linear terms: “this stream of packets matches a stream known to be a smurf, SYN, or other known attack signatures.” Signature-based ID systems are adequate to deal with misuse intrusions, but can’t deal with out-of-the-box thinkers who pen-test, audit, or attack networks, purposely thinking non-linearly with the expectation of ultimately discovering code, policy, and logic flaws. They also can’t adequately deal with anomalous behaviors and resulting intrusions, e.g., the disgruntled insider who abuses authorized access, the unwitting user who is victimized by a worm, the server that is back-doored.

Some of the most recent generation of ID systems are tackling anomaly detection head-on. Systems from the likes of x,y,z … [DMP1] and others expect that by observing normal behavior of users on networks, they can assign values of “normal” to metrics, and compare future behavior and traffic against what is asserted to be normal. This is only one approach to what is commonly understood to be the kind of “fuzzy” thinking needed to deal with the difficulty problem of distinguishing anomalous from normal behavior: neural networks, machine learning, and even mimicking of the biological immune systems.

The idea that ID software and appliances could accurately identify abuse or a previously unknown attack is heady enough, and a significant improvement over deployed ID systems. But innovation from the IDS newcomers promises to go beyond this. Several companies have already considered the value of the accumulated histogram of network activity that’s used to distinguish normal from anomaly as e-evidence and e-intelligence, for example. They’ve considered the kinds of dynamic and proactive mechanisms that are easily implemented with such sophisticated advanced warning systems, such as automated traffic filtering at firewalls and denial of service countermeasures, as well.

These are very promising and encouraging steps forward. But as revolutionary as the innovation appears to be, implementation will still be evolutionary, and while certainly not at a pace equal to Moore’s Law, ID technology is heading in the right direction at a pace that’s still very remarkable and exciting.