{'fraction_non_alphanumeric': 0.023809523809523808, 'fraction_numerical': 0.005357142857142857, 'mean_word_length': 4.98220640569395, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
NF-Dedupe: A novel no-fingerprint deduplication scheme for flash-based SSDs

NAND flash-based Solid State Drives (SSDs) have been widely deployed in data centers of cloud computing due to their high performance compared with hard disks, while the limited lifespan of flash memory makes SSDs not very suitable for write-intensive applications. Deduplication is an effective method used to reduce the write traffic of applications thus can be used to extend the lifespan of SSDs. However, traditional deduplication schemes rely on the time-consuming fingerprint computing process to find duplicated data, which may impair the write performance of SSDs. Accordingly, Pre-hashing was proposed to reduce the chances of fingerprint computing thus improving the performance of SSDs with deduplication, but at the cost of degrading deduplication rate. In this paper, we propose NF-Dedupe, a new deduplication scheme that needs no fingerprint computing for flash-based SSDs. NF-Dedupe determines whether a write page is duplicated or not by comparing the write page with its potential duplicated page read from underlying flash chips byte by byte, rather than relying on the comparison of fingerprints. As flash memory is known for its high parallelism and low read latency, reading a page from flash chip and comparing two pages byte by byte introduce lower overhead than the fingerprint computing does. We evaluate the NF-Dedupe via trace-driven simulations. Experimental results have shown that NF-Dedupe outperforms the other approaches and can achieve the deduplication rate ranging from 5.3% to 29.9% and the write latency is improved by a factor of up to 21% with an average of 12%.