Remove Duplicate Files

Duplicate File Detective v3.5 Released

noreply@blogger.com (Mark Richards) — Thu, 18 Feb 2010 16:51:00 +0000

We're proud to announce the release of Duplicate File Detective v3.5, available for immediate download from our product website.

New in v3.5:

- New "folder view" docking panel
- Search path protection
- File extension researcher tool
- Exclusion of protected file types during search
- Symbolic and hard link creation support
- Hard link identification during duplicate search
- New "group count" and "group size" report columns
- Improvements to search logging
- Many usability and performance enhancements
- Various bug fixes

For additional details on this release, please see the product release notes.

Duplicate File Detective v3.0.1.69 Released

noreply@blogger.com (Mark Richards) — Thu, 17 Dec 2009 17:58:00 +0000

Duplicate File Detective v3.0.1.69 is now available for download.

This is a maintenance release, and is a free upgrade for all existing v3.0 license holders. It resolves a handful of bugs and provides minor feature enhancements. Full release notes here.

Actual vs. Allocated File Sizes

noreply@blogger.com (Mark Richards) — Tue, 08 Dec 2009 21:38:00 +0000

Did you know that Duplicate File Detective is capable of showing both actual and allocated space for the duplicate files it finds? Here's a screen shot (click to enlarge):

The "allocated" column shown above is hidden by default, but it's simple to make it visible. Just click the View tab in the ribbon bar, then click the Column Chooser button. This will cause the Column Chooser pop-up window to appear, and you can drag-and-drop columns into the report view in any position you like.

Another way to show this report column is to right-click on the duplicate search report header and choose it from the Columns pop-up menu.

So - what's the difference between allocated (also known as "size on disk") and actual disk space? To find out, please check out this blog article on disk-space-manager.com.

Create Shortcuts to Moved (or Removed) Duplicate Files

noreply@blogger.com (Mark Richards) — Mon, 30 Nov 2009 15:18:00 +0000

Duplicate File Detective v3.0 contains a large number of new features that are the direct result of user feedback.

One of the most important of these is the ability to create shell shortcuts when moving or removing (deleting) duplicate files. The idea here is quite simple - if you have (for example) a group of four duplicate files, it should be possible to move or delete three of them and create shell shortcuts in their place - those shell shortcuts will then resolve (or point) to the first unmarked file in the same duplicate group.

You can also create shell shortcuts when archiving duplicate files (e.g. adding them to a zip file), as long as you elect to remove the original duplicate files as they are added to the archive (the original file must no longer exist in order for its replacement with a shell shortcut to make sense).

Let's take a look at how these options are presented within Duplicate File Detective v3.0. Please click the screen shot below for a slightly larger image.

When creating shell shortcuts to moved or deleted duplicate files, the prerequisites are as follows:

Duplicate files must be moved or deleted to make room for their replacement with shell shortcuts.
At least one member of each duplicate group being processed must be unmarked. This is the target file to which duplicate shortcuts will resolve.
If any of the above requirements are not met, shell shortcut creation will not be attempted for that duplicate group.

The benefit of this approach is clear - if you're working to dedupe a large document repository, for example, you can archive a series of duplicate files into a single large zip file and still allow users to access those files through (newly created) shell shortcuts.

In future releases of Duplicate File Detective, we'll be expanding upon this concept even further. Please stay tuned for details.

Calculating File Hashes (Checksums) in Duplicate File Detective v3

noreply@blogger.com (Mark Richards) — Wed, 25 Nov 2009 15:31:00 +0000

Today I'd like to briefly illustrate some of the improvements made to the file hash calculator tool in Duplicate File Detective v3.0.

First, and perhaps most importantly, the file hash calculator has been re-designed to allow for the generation of multiple file hash types for multiple file paths - all in a single pass.

As you can see in the adjacent screen shot, it is now possible to generate CRC32, ADLER32, MD5, SHA1, SHA256, and SHA512 file checksums with only a single read of each target file.

The output of the file hash / checksum operations has also been re-engineered to display results in a hierarchical (tree like) fashion. For each file hashed, a series of child nodes show the hash value for each type (e.g. CRC32, MD5, etc.) selected in the area above.

It is also possible to copy file hash results to the Windows clipboard, in which case they are "flattened" into a series of plain-text lines.

Here's a quick sample of what the hash results shown above look like on the Windows clipboard (folder names and hash values are truncated for easier viewing in this limited space):

C:\MoveTest\Sample Pictures\Lighthouse.jpg (CRC32): A3A2B75C
C:\MoveTest\Sample Pictures\Lighthouse.jpg (MD5): 8969288F4245120E7C3870287CCE...
C:\MoveTest\Sample Pictures\Lighthouse.jpg (Adler32): A6EA1A41
C:\MoveTest\Sample Pictures\Lighthouse.jpg (SHA1): 1B4605B0E20CECCF91AA278D10E...
C:\MoveTest\Sample Pictures\Lighthouse.jpg (SHA256): FF86372CE43519D675B8D8D29...
C:\MoveTest\Sample Pictures\Lighthouse.jpg (SHA512): 9BDD0C215A9BE94F6F677F8AD...
C:\MoveTest\Sample Pictures\Hydrangeas.jpg (CRC32): 03632F46
C:\MoveTest\Sample Pictures\Hydrangeas.jpg (MD5): BDF3BF1DA3405725BE763540D66...
[...]

The actual file hash / checksum generation process is extremely fast and efficient, and can also be canceled if needed.

Finally, it is also now possible to right-click any number of selected files within the main Duplicate File Detective result report and elect to generate hash values for them. This combination makes it very easy to generate and compare hash values of arbitrary strength (e.g. bit length) as needed.

Duplicate File Detective 3 is Released

noreply@blogger.com (Mark Richards) — Tue, 24 Nov 2009 15:52:00 +0000

Key Metric Software is proud to announce the release of Duplicate File Detective v3.0, a major upgrade to our professional-grade duplicate file management solution.

Duplicate File Detective v3.0 provides a number of new features that focus on extending the scalability and reach of the product. New features include full Unicode support, new duplicate file processing options (such as the ability to replace moved or deleted duplicates with shell shortcuts), duplicate search logging, support for Windows 7 and Server 2008, and scalability improvements that allow the product to report upon (and process) millions of duplicate file entries in a single session.

- More Info on What's New in v3
- Full Release Notes
- Download Duplicate File Detective v3.0 Now

We're extremely proud of this release and, as always, welcome your thoughts and feedback.

Duplicate File Detective v2.2.0.0 Released

noreply@blogger.com (Mark Richards) — Fri, 28 Mar 2008 00:34:00 +0000

Duplicate File Detective v2.2.0.0 has been released, and it provides a number of feature enhancements and bug fixes including:

Support for SHA256 and SHA512 hashing algorithms
Support for encoding hash / checksum as base64 (in addition to hex)
Greatly improved SmartMark system support
Numerous improvements to the built-in file hash calculation tool
Improved scan filtering capabilities
A broad range of performance enhancements
Experimental support for low-priority I/O on Windows Vista and Server 2008
A handful of bug fixes and other improvements

Please take a moment to review the complete release notes, and then download v2.2 when ready. This is a free upgrade for all v2.x license owners.

Duplicate File Detective - The Movie!

noreply@blogger.com (Mark Richards) — Mon, 03 Dec 2007 02:30:00 +0000

We've recently published a Duplicate File Detective v2.0 screencast which provides a quick, scenario driven walk-through of how the product works. The scenario shown is one of finding duplicate images in a user's "Pictures" folder, and it briefly demonstrates:

Use of duplicate file search filtering
The integrated image preview and properties docking panels
Use of our advanced SmartMark technology to "mark" a range of duplicate files
Use of the Duplicate Result Manager feature to safely delete marked duplicate files to the Windows recycle bin

Please take a moment to watch the screencast. Any and all comments are most welcome (just email us).

Finding Duplicate Files with Content Hashing

noreply@blogger.com (Mark Richards) — Sat, 24 Nov 2007 16:24:00 +0000

Duplicate File Detective provides the ability to specify precisely how duplicate file matches are identified. Specific combinations of file matching options each have their own set of trade-offs. For example, matching files based solely upon file name and size is very quick, but will obviously miss files that have duplicate contents but different file names (and may also incorrectly identify files with different file contents, but the same names and sizes, as duplicates).

For this reason, extending duplicate file searches beyond a quick exploratory run will usually require comparison of file contents. In Duplicate File Detective, file contents can be matched with or without regard to other file attributes such as name or modification date.

Duplicate File Detective compares file contents through a process known as file hashing. File hashing can be defined as the generation of a smaller, unique key value (a "digital fingerprint") from the (much larger) contents of a given file.

Duplicate File Detective can hash file contents using a variety of cryptographic algorithms, including CRC32, ADLER32, MD5, and SHA1. The first two generate 32-bit file hashes, while the latter two generate 128-bit and 160-bit file hashes, respectively. Generally speaking, the stronger (larger) the generated hash values, the more accurate the file content comparison. In fact, the chance that two different files could produce the same 128-bit or 160-bit digital fingerprint is incredibly small.

Duplicate File Detective allows you to mix and match file content matching techniques to suit your specific requirements, but the built-in Project Wizard creates a reasonable set of defaults based upon your general objectives:

Quick duplicate file scan - This project type matches duplicate files by name and size alone. It executes very quickly because no file contents are analyzed, and provides a fair degree of accuracy.
Checksum duplicate file scan - Matches files by name, size, and CRC content hash. Also fairly quick, but works to ensure that files with the same name and size also have the same contents.
Strong checksum duplicate file scan - Matches files by contents alone (via strong MD5 hashing). Allows for the isolation of duplicate files regardless of other attributes (such as name and / or modification date).

In Duplicate File Detective v2.x, file content hashing can be further re-enforced with byte-for-byte content matching. Byte-for-byte matching can help provide that final degree of confidence that two duplicate files are truly, 100% identical at the byte level.