Data deduplication tutorial

For backup/storage administrators looking minimize the time it takes to back up their data, the best option is often to use a post-process method. This has the advantage of backing up data faster, reducing the backup window. The disadvantage of this method is that additional storage space is consumed. Backup data is sent to a temporary holding area in order to speed the backup process. Once that completes, the data is reexamined for duplicates, with duplicate data removed (some post systems start deduping before the whole backup is complete, so they may not require as much storage on the target).

An alternative to deduplicating data after a backup is to perform deduplication inline as data is being sent to the backup device. The advantage with this method is that no extra space is required. Another advantage is that once the data is deduplicated and stored, the process is done, and backup data may be replicated to offsite storage. With post-processing deduplication, data must be written to storage, then deduplicated at a later time, and then replicated to offsite storage. As a result, the time to complete the entire backup process -- including replicating to offsite systems -- can be longer than systems that deduplicate inline.

Data deduplication can dramatically decrease the amount of disk space required for backup data, while retaining the significant performance improvements that disk-based backup devices have over tape. Thus, disk-based backup targets, whether they are NAS devices or VTLs, allow these systems to deliver high service-level objectives, while remaining cost competitive with tape-based systems.

Product	Simpana 8	DDX	Avamar	DL 4000	SIR	VLS	Diligent
Vendor	Comm-Vault	Data Domain	EMC Corp.	EMC Corp.	FalconStor Software	HP Co.	IBM Corp.
Deployment type	Backup software	VTL w/ storage	Backup software	VTL appl. w/ storage	VTL w/ or wo storage	VTL appl. w/ storage	VTL appl. w/ or wo storage
Dedupe cost	Add-on	Included	Included	Add-on	Add-on	Add-on	Included
When Dedupe	Inline	Inline	Inline	Inline and post process	Post process	Post process	Inline
Dedupe location	Distributed	Target	Source	Target	Target	Target	Target
Chunk size	Variable	Variable	Variable	Variable	Variable	Variable	Variable
Access method	-	-	Hardware dependent	-	-	-	-
NAS (NFS/CIFS)	Yes	Yes	-	No	No	No	No
FC primary storage	No	No	-	No	No	No	No
FC tape storage (VTL)	No	Yes	-	Yes	Yes	Yes	Yes
iSCSI primary storage	No	No	-	No	No	No	No
iSCSI tape storage (VTL)	Yes	Yes	-	Yes	Yes	Yes	Yes

Product	HydraStor DataRedux	FAS DeDupe	Enterprise Archive	DXi	S2100 w/ DeltaStor	VTL Prime	PureDisk
Vendor	NEC Corp.	NetApp	Permabit	Quantum Corp.	Sepaton Inc.	Sun Inc.	Symantec Corp.
Deployment type	Secondary storage	Primary storage	Secondary Storage	VTL appl. w/ storage	VTL w/ or wo storage	VTL appl. w/ storage	Backup software
Dedupe Cost	Add-on	Included (No cost license)	Included	Add-on	Add-on	Add-on	Included
When Dedupe	Inline	Post Process	Inline	Both (Inline and post process)	Post Process	Post Process	Inline
Dedupe location	Target	Target	Target	Target	Target	Target	Source
Chunk Size	Variable	4 KB block	Variable	Variable	Variable	Variable	Variable
Access Method	-	-	-	-	-	-	Hardware dependent
NAS (NFS/CIFS)	Yes	Yes	Yes	Yes	No	No	-
FC primary storage	No	Yes	No	No	No	No	-
FC tape storage (VTL)	No	No	No	Yes	Yes	Yes	-
iSCSI primary storage	No	Yes	No	No	No	No	-
iSCSI tape storage (VTL)	No	No	No	Yes	Yes	Yes	-

It is likely that over time, data deduplication will become a service and be offered as a feature in conjunction with multiple product types and deployment scenarios. Until this time, you must carefully evaluate their cost, performance and data retention goals prior to choosing a data deduplication product that will deliver the optimal benefits in their particular environment, or test the product carefully in your environment before you buy it.