It’s been a very busy few weeks. Today, I’m on the train headed for NYC so I have some spare time to make a blog entry.
This is a continuation of a short blog series about the four C’s of high-performance write-to-disk; namely: capture, controller, cores, and cache. Today it is about controller acceleration; specifically, RAID controller.
Design Rule #2: Controller Acceleration
To support 10Gbps write-to-disk capture, a RAID configuration must be able to write ~1.25 GB/s at a sustained rate. On average, one can achieve between 75-100MB/s per drive, depending of the type of drive. In general, a 16-drive RAID volume will provide the bandwidth needed. This is especially the case if the application is writing packets in the industry standard PCAP format, where each packet requires additional metadata such as time-stamp and packet length.
Managing a 16-drive RAID volume with software RAID implementation may be possible; however, for a few hundred dollars more a hardware RAID controller is the way to go. With SW RAID, performance will vary depending on system load as it contends for CPU cycles to manage, coalesce, and dispatch I/O operations. In some cases, it could impede processor cache performance and drastically reduce application performance.
If your application is analyzing and/or indexing packets as it is writing to disk, then you’ll definitely want to dedicate as much CPU cycles as possible to the application. At 10Gbps, the system load will be significant enough to degrade SW-based RAID performance. A HW RAID controller, on the other hand, completely offloads the management of I/O buffers and spindles, resulting in higher throughput and more CPU cycles for your application. If you are serious about write-to-disk performance, than I suggest that you consider a RAID controller to achieve maximum I/O performance.
One of our favorite controllers is the Adaptec RAID 5805 controller (The 51645 is pictured above). I give credit to Ed Jamison at MBX for introducing us to this controller. For the price, this board is rock-solid with plenty of processing power and bandwidth to support high bandwidth and IOPS at sustained rates. Most Linux distributions provide native support for the 5805 with no special build required.
4 Responses to “Achieving 10Gbps Write-to-disk Performance (Part 3)”
If it’s OK, I’m going to ask you a few more questions. I’m trying to build a test system capable of writing 10gbps in our lab (to test high latency transfers at that speed), so this is very relevant to me.
How many controllers do you need to have a sustained 10gbps transfer rate? Your pictures clearly show two of them loaded into the box — is that enough for burst, or sustained transfer rate?
And these are standard off-the-shelf SATA drives? SAS drives? Or do you place SSD’s in the array?
Our design includes two controllers and we use Seagate drives with SAS interfaces. That’s more than enough to handle 10Gbps at sustained rates. In fact, with our multi-threaded software, we’re seeing 18Gbps sustained; and 20Gbps bursts.
Can’t wait for the part 4! Keep it coming….
Thanks. I will post part 4 next week. I will also show video clips to underscore the power of multi-threaded software.