[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: IDS: need info
FAQ: See http://www.ticm.com/kb/faq/idsfaq.html
IDS: See http://www-rnks.informatik.tu-cottbus.de/~sobirey/ids.html
HELP: Having problems.. Then email questions to ids-owner@uow.edu.au
NOTE: You MUST remove this line from reply messages as it will be filtered.
SPAM: DO NOT send unsolicted mail to this list.
USUB: email "unsubscribe ids" to majordomo@uow.edu.au
---------------------------------------------------------------------------
>I realize this has been covered in other threads so I'll be brief. I don't
>think there is a 'best' IDS. In reading the Netranger documentation, Cisco
>fails to mention the number of concurrent connections Netranger 2.2.0 can
>monitor. The Hivermute ARMS IDS from Hiverworld can concurrently monitor
>85,000 connections. To my knowledge this is far more than any competing IDS
>product on the market.
What an odd basis for comparison!!
IDS need to be compared on a lot of different parameters, not
just how many connections they can track, packets they can
grab per second, records they can store, filters they run,
how high up the stack they track, etc!!! You're not doing
anyone (customers _or_ vendors) a favor by encouraging such
simplistic measurements. ack when I used to sell firewalls,
I was disgusted by the measures some vendors came up with, to
distort the apparent effectiveness of their products. Let's
not do the same thing with IDS!
Let me give some specifics:
THE BOX
1) What kind of _box_ was the IDS running on? Most network-based
ID systems will perform radically differently on a 100Mhz
pentium with 16MB of RAM compared to a 450Mhz machine with
128MB of RAM. So, when you're talking about limits, there
are _hard_ limits (table sizes, etc) and there are practical
limits (memory bandwidth, available RAM, etc). It's the
practical ones that usually bite you first!!
For example, NFR doesn't have any hard limits in its IP
state tracking code. Does that mean I can say NFR can
simultaneously track infinite numbers of connections?
No, I'll say "it can track as many as your underlying
hardware can handle - the most we've seen was 100,000+
on a saturated FDDI using a 450Mhz machine with 256MB
of RAM"
If I were an unethical vendor cooking a benchmark, I
could create 10,000,000 TCPs going past a system,
using just empty handshakes with a low data rate. Then
I'd claim I could track more connections than anyone
else.*
PACKET CAPTURE
2) Packet capture rates don't mean much unless you also take
into account what the product _does_ with the packets
once it's captured them! If all it does is count them
and discard them, a product can handle very high data
rates. If it's reassembling the IP up the stack to
layer 7, and doing IP checksumming, etc, etc, then
it may be able to capture fewer packets but it's
actually doing something with them. But, of course
a lot of that may depend on item #1 above ("THE BOX")
If I were an unethical vendor cooking a benchmark,
I'd make sure my product didn't report packet drops,
if it had them, or I'd count packets I saw go by but
wasn't able to process.*
RECORD STORAGE
3) In most IDS that allow record storage at a certain point
you will run up against query speed and storage array
speed. It's not sufficient to record packets since you
have to eventually look at them. :) Unless you use
very expensive hardware, searching a 100Gb RAID array
for one packet is going to take a little while. :)
With most IDS the amount and type of stuff you retain
is semi tuneable. If I were an unethical vendor cooking
a benchmark, I'd turn all that off, and distort my results.*
FILTERS/SIGNATURES
4) In ID systems that have programmable logic or signatures,
the number that you run may or may not mean anything.
I'm concerned that the IDS market may get like the
virus scanning market - "we're better because we detect
10,000,000 viruses" -- the question is how many
useful error/misuse/anomaly detection test points are
deployed. That also has an impact on performance (see
"THE BOX" above). It's more than just numbers that
matter. For example, one of our filters does protocol
verification on how a client does FTP - it generates
alarms if it sees anything that doesn't fit "normal"
FTP use. As a result it catches a couple buffer
overruns that may not even exist, as well as FTP
bouncing, and who knows what all else? Is that one
filter or 10? Or 100?
The bottom line for the customer is: does it detect
stuff? Don't get lost in easily manipulated numbers.
If I were an unethical vendor cooking a benchmark,
I'd include filters to detect zillions of useless
ancient attacks (SMTP DEBUG, WIZ) and maybe even
things that didn't exist (Hex 06A in an FTP login!)
until I had 10,000,000 "detection rules".*
UP THE STACK
5) How much analysis does the IDS actually do? This is going
to be a _really_ interesting area in the future, especially
for network IDS. For example, if the IDS doesn't do
defragmenting and IP reassembly into valid streams,
it's vulnerable to all kinds of simple masking attacks.
(like fragrouter, and cthulhu3.0 for examples) If the
IDS just treats each packet as a single event it can
be _very_ fast but at the expense of rigorousness. If
that's what you want, as the customer, that's great,
but will the vendor tell you? This is going to be
especially interesting when the router/switch makers
realize IDS is a "product market" and start rushing
things into their firmware. :) Let's imagine you
have a router which is trying to do reassembly and
defragging on a saturated FDDI. Oops, suddenly you
need to add a few MB of RAM (like maybe 128?) to
the router. Or pass it off to someplace else, eating
your bandwidth. Or just search each packet and be
trivially spoofed but market yourself as an IDS.
If I were an unethical vendor I'd build a product
that had 2 modes: up-the-stack processing (the good
stuff) and per-packet processing (the useless stuff)
and I'd have the device switch modes under extreme
load. Then it'd show as able to do amazingly rigorous
searches at amazing speed but it'd be bogus.* (Unlike
some of my other "unethical vendor" examples, this
one doesn't appear to have been implemented yet)
Anyhow, I give you two words: caveat emptor.
Right now, the industry hasn't yet figured out how to measure
these things. Some folks have been trying. There were some
interesting studies run by Data Communications (Dave Newman
et al) last year. But even their well-thought-out study had
a few flaws. For example, raw performance measures slight
the products that actually do up-the-stack processing,
as do large numbers of connections. Its like the old days
when folks used to measure a firewall's security by the
amount of peak bandwidth you could push through it.
Exciting times are ahead! :)
mjr.
(* I'm not. We don't cook benchmarks at NFR. Your mileage may vary.)
--
Marcus J. Ranum, CEO, Network Flight Recorder, Inc.
work - http://www.nfr.net
home - http://www.clark.net/pub/mjr