#6) Finding the Needle in the Haystack

So you have a network forensics solution that is able to capture and store terabytes of packets?  Presumably, this represents days of network traffic.  Tell me, how long will it take to find the needle in the haystack?  No, don’t rely on the product’s marketing literature.  Those figures are usually contrived to represent a perfect and best case scenario.  Rather, give it a real-world test and see how the numbers work for your situation.

For example, assuming you have a few days of recorded packets from a 10 Gig network, how long will it take to find the number of times an obscure IP address traversed your network in the last 72 hour?  In terms of time, if your solution takes just as long (or longer) to find the packets as it took to capture, then is it really a solution?  More importantly, if you are an incident responder, would you find this response time acceptable?

One of my favorite explanations about Big Data goes something like this:

Big Data is about strategy. It is about being able to process a colossal amount of data within a desirable time frame.

I couldn’t agree more — especially when it comes to capturing terabytes, even petabytes, of network traffic. Packet capture is the epitome of Big Data. By definition it is high velocity, high volume, and high variety. Therefore, finding the “needle in the haystack” is a colossal challenge that requires a solid strategy.

We recently had a large enterprise customer place one of our packet capture probes along side their legacy solution.  They wanted to compare the relative performance between the two “big data” solutions by applying the use case mentioned above.  The incumbent took over 16 hours to complete the query.  In contrast, our solution took less than a minute to complete the same query — while capturing at 10Gbps.

Yep, most certainly, not all Big Data solutions are the same.  It’s definitely about strategy. How good is your vendor’s Big Data strategy?  A quick way to find out is to apply a “needle in the haystack” test.

#5) Give me PCAP or give me death!

Yes, the title is a play on Patrick Henry’s famous words, “Give me liberty, or give me death!”, given at the Virginia Convention prior to the Revolutionary War.

But seriously, this seems to be a creed declared by a significant number of end-users.  If vendors would only listen to their customers, the network and security engineers, who deal with trouble shooting, incident response, malware tracking on a day-to-day basis, then their product would have a chance for wider acceptance.  What the customer ultimately wants is the data in PCAP format; not some proprietary format that limits them to a set of expensive (and sometimes useless) tools.

The beauty about PCAP is that it is THE standard format for packet data.  It’s simple.  It’s open.  And it keeps the data portable. This enables the customer to slice and dice packets with their tool of choice.  Furthermore, if an app doesn’t exist for their use case, then it’s fairly simple to quickly cobble something together from the plethora of PCAP-enabled tools and languages available in the open-source community.  Two great examples are Bro and Suricata.  Both are incredibly powerful and flexible tools that can be used to process PCAP files.

It’s no wonder why many organizations opt for building their own network forensics (packet capture) solution instead of buying commercial-off-the-shelf.  It’s not that they mind paying for the solution, it’s that they resent being cuffed to a proprietary format.  This is how potentially valuable security data is lost and left for dead.  Instead, they prefer the liberty of PCAP, a truly portable format, that allows valuable data to be shared and utilized in a wider scope.

Beware of any vendor that insists on using a solution incorporating a proprietary format for storing and processing packets.  To do so is the kiss of death.

#4) Beware of the one-trick pony

The term “one-trick pony” historically refers to a circus pony which has been taught to perform one trick. Today, the term refers to something that is limited to performing one skill or capability.

A good example of this is a packet capture solution that achieves 10Gbps but only under certain conditions; i.e., fixed UDP packets of 1500 bytes. Or, fixed UDP packets of 64 bytes. You get the idea. The point is that a packet capture appliance designed to perform well under certain fixed conditions is basically a one-trick pony; not high-performance solution.

This blog entry is a corollary to entry #2. When a evaluating the performance of a packet capture solution, it is imperative to conduct a battery of tests that represent traffic of variable and extreme conditions. Not just one simple test consisting of a stream of packets at a fixed rate, packet size, and protocol type.