Step 1: Learn why duplicate files are a problem
There are many problems associated with duplicate files:
- Unnecessary and redundant.
- Take up valuable storage space.
- May contain sensitive information and are stored in an unmanaged location.
- Important changes made to the original document will not be made to the duplicate rendering the contents invalid and obsolete.
- Legal holds and retention rules may not be applied to duplicates creating a liability.
- Search results are muddied by multiple returns of the same documents.
- Data analytics and metrics can be skewed by an overabundance of the same information.
Finding all of your duplicates and getting them under control can be a huge undertaking. But it is possible, and well worth doing.
Step 2: Find the duplicate files
A popular technique that we use to find duplicate files is through hashing. The process of hashing generates a hash code for each file. A hash code is a fixed length code that represents a binary source of any length.
Here is an example:
| 9466c304e74cbe6ebb7abf2eab0cf903 |
If any file has the exact same hash code as another file, we can be 100% sure that the files being compared are exact duplicates.
Step 3: Generate hash codes
A hash code is generated by passing any sized string (the contents of a file) to a hash function that returns the hash value or code. The hash function treats the string as bytes to generate the code.
Step 4: Find the same files, even by a different name
The hash of a file only applies to the actual contents. In the search results shown below you will see that 4 of the files all have the exact same hash code, even though 1 has a different name, 1 has a different extension, and one is an exact copy.
The changed document from above only had one period removed. The slightest change to a document will result in the generation of an entirely different hash code.
Step 5: Visualize the duplicates
By hashing all of your data and recording it in an index, you can quickly visualize and begin to understand the amount of duplicates that are present in your organization. It is not uncommon to find out that over half of the files in any given dataset are exact duplicates.
Next Steps:
Learn how to find similar files in our video Discovery Desktop – Functions and Settings
Advanced Articles:
For current Shinydocs customers, visit our Help Desk for access to complete product manuals, how-to and troubleshooting articles, use cases and instructional videos.