Duplicate File Detective

Duplicate File Detective v2.2.0.0 Released

2008-03-27T20:34:00.002-04:00

Duplicate File Detective v2.2.0.0 has been released, and it provides a number of feature enhancements and bug fixes including:

Support for SHA256 and SHA512 hashing algorithms
Support for encoding hash / checksum as base64 (in addition to hex)
Greatly improved SmartMark system support
Numerous improvements to the built-in file hash calculation tool
Improved scan filtering capabilities
A broad range of performance enhancements
Experimental support for low-priority I/O on Windows Vista and Server 2008
A handful of bug fixes and other improvements

Please take a moment to review the complete release notes, and then download v2.2 when ready. This is a free upgrade for all v2.x license owners.

Duplicate File Detective - The Movie!

2007-12-02T21:30:00.000-05:00

We've recently published a Duplicate File Detective v2.0 screencast which provides a quick, scenario driven walk-through of how the product works. The scenario shown is one of finding duplicate images in a user's "Pictures" folder, and it briefly demonstrates:

Use of duplicate file search filtering
The integrated image preview and properties docking panels
Use of our advanced SmartMark technology to "mark" a range of duplicate files
Use of the Duplicate Result Manager feature to safely delete marked duplicate files to the Windows recycle bin

Please take a moment to watch the screencast. Any and all comments are most welcome (just email us).

Finding Duplicate Files with Content Hashing

2007-11-24T11:24:00.001-05:00

Duplicate File Detective provides the ability to specify precisely how duplicate file matches are identified. Specific combinations of file matching options each have their own set of trade-offs. For example, matching files based solely upon file name and size is very quick, but will obviously miss files that have duplicate contents but different file names (and may also incorrectly identify files with different file contents, but the same names and sizes, as duplicates).

For this reason, extending duplicate file searches beyond a quick exploratory run will usually require comparison of file contents. In Duplicate File Detective, file contents can be matched with or without regard to other file attributes such as name or modification date.

Duplicate File Detective compares file contents through a process known as file hashing. File hashing can be defined as the generation of a smaller, unique key value (a "digital fingerprint") from the (much larger) contents of a given file.

Duplicate File Detective can hash file contents using a variety of cryptographic algorithms, including CRC32, ADLER32, MD5, and SHA1. The first two generate 32-bit file hashes, while the latter two generate 128-bit and 160-bit file hashes, respectively. Generally speaking, the stronger (larger) the generated hash values, the more accurate the file content comparison. In fact, the chance that two different files could produce the same 128-bit or 160-bit digital fingerprint is incredibly small.

Duplicate File Detective allows you to mix and match file content matching techniques to suit your specific requirements, but the built-in Project Wizard creates a reasonable set of defaults based upon your general objectives:

Quick duplicate file scan - This project type matches duplicate files by name and size alone. It executes very quickly because no file contents are analyzed, and provides a fair degree of accuracy.
Checksum duplicate file scan - Matches files by name, size, and CRC content hash. Also fairly quick, but works to ensure that files with the same name and size also have the same contents.
Strong checksum duplicate file scan - Matches files by contents alone (via strong MD5 hashing). Allows for the isolation of duplicate files regardless of other attributes (such as name and / or modification date).

In Duplicate File Detective v2.x, file content hashing can be further re-enforced with byte-for-byte content matching. Byte-for-byte matching can help provide that final degree of confidence that two duplicate files are truly, 100% identical at the byte level.