Data De-duplication is a popular and popular storage technology that optimizes storage capacity.It eliminates redundant data by removing duplicate data from the dataset, leaving only one copy.As shown in the figure below.This technology can greatly reduce the need for physical storage space to meet the growing demand for data storage.Dedupe technology can bring many practical benefits, mainly including the following aspects:
| (1) | Meet ROI(Return On Investment)/TCO(Total Cost of Ownership) requirements; |
| (2) | The rapid growth of data can be effectively controlled; |
| (3) | Increase effective storage space and improve storage efficiency; |
| (4) | Save the total storage cost and management cost; |
| (5) | Save the network bandwidth of data transmission; |
| (6) | Save operation and maintenance costs such as space, power supply and cooling. |
Dedupe technology is widely used in data backup and archiving systems, because there are a lot of duplicate data after multiple backups of data, which is very suitable for this technology.In fact, dedupe technology can be used in many situations, including online data, near-line data, and offline data storage systems. It can be implemented in file systems, volume managers, NAS, and sans.Dedupe can also be used for data disaster recovery, data transmission and synchronization, as a data compression technology can be used for data packaging.Dedupe technology can help many applications reduce data storage, save network bandwidth, improve storage efficiency, reduce the backup window, and save costs.
Dedupe has two main dimensions: deduplocation ratios and performance.Dedupe performance depends on the specific implementation technology, while Dedupe rate is determined by the characteristics of the data itself and application patterns, as shown in the table below.Storage vendors currently report deduplication rates ranging from 20:1 to 500:1.
| High deduplication rate | Low deduplication rate |
| Data created by the user | Data from the natural world |
| Data low rate of change | Data high rate of change |
| Reference data, inactive data | Active data |
| Low data change rate application | High data change rate application |
| Full data backup | Incremental data backup |
| Data long-term storage | Data short-term storage |
| Wide range of data applications | Small range of data applications |
| Continuous data business processing | General data business processing |
| Small data segmentation | Big data segmentation |
| Elongate data segmentation | Fixed length data segmentation |
| Data content perceived | Data content unknown |
| Time data deduplication | Spatial data deduplication |
Dedupe Implementation Points
Various factors should be considered when developing or applying Dedupe technology, as these factors directly affect its performance and effectiveness.
| (1) | What | What data are de-weighted? |
| (2) | When | When will the weight be eliminated? |
| (3) | Where | Where is the weight elimination? |
| (4) | How | How to reduce the weight? |
Dedupe Key Technology
Deduplication process of storage system in general is this: first of all the data file is divided into a set of data, for each block of data to calculate the fingerprint, and then based on fingerprint Hash search keywords, matching indicates the data for the duplicate data blocks, only stores data block index number, otherwise it means the data block is the only piece of a new, storage of data block and create relevant meta information.Thus, a physical file in the storage system corresponds to a logical representation of a set of FP metadata.When reading the file, first read the logical file, then according to the FP sequence, take out the corresponding data block from the storage system, restore the copy of the physical file.It can be seen from the above process that the key technologies of Dedupe mainly include file data block segmentation, data block fingerprint calculation and data block retrieval.
(1) File data block segmentation
(2) Data block fingerprint calculation
(3) Data block retrieval
To find these recommend models to start your Network Packet Deduplication:
Mylinking™ Network Packet Broker(NPB) ML-NPB-6400 48*10GE SFP+ plus 4*40GE/100GE QSFP28, Max 880Gbps
Mylinking™ Network Packet Broker(NPB) ML-NPB-5660 6*40GE/100GE QSFP28 plus 48*10GE/25GE SFP28, Max 1.8Tbps
Mylinking™ Network Packet Broker(NPB) ML-NPB-5060 48*10GE SFP+ plus 2*40GE QSFP, Max 560Gbps
Mylinking™ Network Packet Broker(NPB) ML-NPB-4860 48*10GE SFP+, Max 480Gbps, Function Plus
Mylinking™ Network Packet Broker(NPB) ML-NPB-4810 48*10GE SFP+, Max 480Gbps
Mylinking™ Network Packet Broker(NPB) ML-NPB-2410P 24*10GE SFP+, Max 240Gbps, DPI Function
Mylinking™ Network Packet Broker(NPB) ML-NPB-6400
48*10GE SFP+ plus 4*40GE/100GE QSFP28, Max 880Gbps
Post time: Oct-18-2022
