By W. Curtis PrestonThere are easy, cheap things you can do to optimize your enterprise backup system and have a dramatic impact. This column will discuss how to figure out what may be slowing down your backup system, how to fix it and will give you some hints as to when to throw in the towel. You will learn backup best practices, and when the best thing to do is to make a major upgrade.
The first and most important thing that you must do to be able to improve your enterprise backup system is to be armed with information. You need to know solid answers to the following questions:
- How much data are you backing up in a full? (You need this number for each client you are backing up, as well as the aggregate number.)
- How much data are you backing up in an incremental? (You also need these for each client.)
The answer to these questions are best found by querying your backup system. It may take some time if you've never used that part of your data backup system, so you may need to get some support (from your backup software vendor, or perhaps a user-based support community). But if this critical information can't be obtained from your backup software, it's time to move on to a different backup software package. Symantec NetBackups' bpimagelist and EMC NetWorker's mminfo commands are examples of how to obtain this information.
The next question you need an answer to is:
- What is the network interface for your backup system and how is it configured (e.g., TOE, jumbo frames)?
This question is best answered by either the system or network administrator. The answer you want to hear is "10 GbE offload NIC with jumbo frames," but it's probably not the answer you're going to hear. 10 GbE is obviously the most recent topology and offers incredible benefits to your backup system. The most obvious benefit is that you'll have a network that is faster than your tape drive. Without a network that is faster than your target, it is impossible to properly design your data backup system. Therefore, you will need to change your target or network. If you are unable to upgrade your network, the next most obvious step would be to move to some type of disk-based backup system, as disk is much more forgiving of slow networks than tape is. But if it's possible to upgrade your backup server's NIC, that's a whole lot easier and cheaper than completely rearchitecting your backup system.
An offload NIC offloads some or all of the TCP processing from your host CPU. If it offloads all of the processing it is called a TOE card, for TCP Offload Ethernet card. If it offloads some functions but not others, it is just called an offload card.
Manufacturers of both types of cards would be glad to explain why their approach is better than the other, but suffice it to say that either is better than having neither. Finally there are jumbo frames. The standard Maximum Transfer Unit (MTU) of Ethernet is 1,500 bytes and this value was decided decades ago. The argument for Jumbo Frames is that today's network speeds are so fast that making a frame every 1,500 bytes requires too many CPU interrupts; a "jumbo" frame size of 9,000 bytes is more appropriate. If your NIC and network support Jumbo Frames, your CPU can be interrupted six times less frequently, increasing the effective speed of your interface.
If you're stuck on 100 MbE, then you really have to upgrade both your network infrastructure and NIC to have any kind of decent performance. There's no reason to buy a GbE NIC, though, even if your network doesn't support 10 GbE yet. Buy a 10 GbE NIC and have it step down to 1 GbE. Then you'll be the first one to take advantage of 10 GbE once they upgrade the network. If you're using 1 GbE, and your network can support 10 GbE, upgrading your backup server's NIC to 10 GbE should be the first thing on your priority list.
You should also ask yourself:
- What are the I/O throughput capabilities of your backup server?
This question is a little difficult to answer using specifications; it is best to answer via testing where you take other things out of the picture. The details on how to do this are way beyond the scope of this column, but a basic suggestion would be to use tools like iostat in Linux and Unix, and iometer in Windows. How fast can you move I/O through this server? Are you limited by the backplane or your I/O bus? If you are, there's not much you can do other than to upgrade the server on which you're running your backups.
Next, find out:
- What is the native transfer rate of your tape drive?
- What compression ratio are you getting on your data?
These questions are about finding out how fast your tape drive should be running. So look at the tapes marked full by your backup software and see how much data you typically fit on a tape before it says it's full. If you consistently fit 600 GB on a 400 GB tape, then you're getting 1.5:1 compression. Multiply that number by the vendor's rated throughput for your drive, and you've got your target throughput number for your tape drive.
Then you need to do whatever you can to supply a rate of data to that tape drive that is close to its target throughput rate. If your target rate is 150 MBps and you only have a 1 GbE NIC, you see why we spent so much time on talking about the network. Techniques such as multiplexing (good for backups, can hurt restores) and disk staging (good for backups, doesn't help restores) are the things you should explore to get your backups to go fast enough to keep that tape drive happy.
Incremental backups almost always go too slow to keep a tape drive happy, so they will most certainly need to be staged to disk. Full backups can either be staged to disk or multiplexed (if your backup software supports it). Once you've made one tape drive happy, see if you can do the same for two or three or more. Don't back up to more tape drives than you can stream. Even if you're upgraded to 10 GbE and you're pumping 800 MBps of backups into your backup server, that's only enough to keep four LTO-5 tape drives streaming at full speed. If you can't keep one tape drive happy, though, adding more tape drives to the mix will only make things worse.
- What is the fastest speed at which you will need to restore data?
Finally, you need to investigate whether your backup system is capable of performing the fastest restores that it is required to make. Look at the largest and fastest restore that you'll ever need to do and test to see if your system is capable of performing that restore. For example, make sure that any multiplexing that you're using hasn't had a negative impact on restore speed.
Although getting the answers to these questions may take some work, you'll be well on your way to learning some valuable backup best practices, and solving some common problems with your enterprise backup system.
About this author: W. Curtis Preston (a.k.a. "Mr. Backup"), executive editor and independent backup expert, has been singularly focused on data backup and recovery for more than 15 years. From starting as a backup admin at a $35 billion dollar credit card company to being one of the most sought-after consultants, writers and speakers in this space, it's hard to find someone more focused on recovering lost data. He is the webmaster of BackupCentral.com, the author of hundreds of articles, and the books "Backup and Recovery" and "Using SANs and NAS."
This was first published in June 2010