Perceptual hashes can be used to deduplicate sets of images. Below we provide two examples (one simple, one larger scale).
For most use cases, we recommend using PHash with
with 0.2 as the distance threshold as in the example below. You may wish to adjust
this threshold up or down based on your tolerance for false negatives / positives.
In practice, deduplicating in memory on your machine by the methods below may be impractical. For larger-scale applications, you may wish to use tools like FAISS, Annoy, or databases with functionality for querying based on distance such as MemSQL.
For the supported hashers, below are our recommended thresholds with expected false positive rates of <1%.
In this example, we download a ZIP file containing 18 images. One of the images is duplicated twice and another image is duplicated once.
import os import glob import zipfile import urllib.request import tabulate import pandas as pd from perception import tools, hashers urllib.request.urlretrieve( "https://thorn-perception.s3.amazonaws.com/thorn-perceptual-deduplication-example.zip", "thorn-perceptual-deduplication-example.zip" ) with zipfile.ZipFile('thorn-perceptual-deduplication-example.zip') as f: f.extractall('.') filepaths = glob.glob('thorn-perceptual-deduplication-example/*.jpg') duplicate_pairs = tools.deduplicate(files=filepaths, hashers=[(hashers.PHash(hash_size=16), 0.2)]) print(tabulate.tabulate(pd.DataFrame(duplicate_pairs), showindex=False, headers=['file1', 'file2'], tablefmt='rst')) # Now we can do whatever we want with the duplicates. We could just delete # the first entry in each pair or manually verify the pairs to ensure they # are, in fact duplicates.
In the example below, we use the Caltech 256 Categories dataset. Like most other public image datasets, it contains a handful of duplicates in some categories.
The code below will:
- Download the dataset
- Group all the filepaths by category (the dataset is provided in folders)
- Within each group, find duplicates using PHash. We will compare not just the original images, but also the 8 isometric transformations for each image.
import os import tarfile from glob import glob import urllib.request import tqdm from perception import hashers, tools urllib.request.urlretrieve( "http://www.vision.caltech.edu/Image_Datasets/Caltech256/256_ObjectCategories.tar", "256_ObjectCategories.tar" ) with tarfile.open('256_ObjectCategories.tar') as tfile: tfile.extractall() files = glob('256_ObjectCategories/**/*.jpg') # To reduce the number of pairwise comparisons, # we can deduplicate within each image category # (i.e., we don't need to compare images of # butterflies with images of chess boards). filepath_group = [ ( filepath, os.path.normpath(filepath).split(os.sep)[-2] ) for filepath in files ] groups = list(set([group for _, group in filepath_group])) # We consider any pair of images with a PHash distance of < 0.2 as # as a duplicate. comparison_hashers = [(hashers.PHash(hash_size=16), 0.2)] duplicate_pairs =  for current_group in groups: current_filepaths = [ filepath for filepath, group in filepath_group if group == current_group ] current_duplicate_pairs = tools.deduplicate( files=current_filepaths, hashers=comparison_hashers, isometric=True, progress=tqdm.tqdm ) duplicate_pairs.extend(current_duplicate_pairs) # Now we can do whatever we want with the duplicates. We could just delete # the first entry in each pair or manually verify the pairs to ensure they # are, in fact duplicates.
Video deduplication requires more thought depending on your tolerance for false positives and how important temporal relationships are. Below is one example approach for deduplicating a group of videos by taking frames from each video that are sufficiently different from each other (to avoid keeping too many) and then using them all to find pairs of videos that have matching frames.
import urllib.request import zipfile import glob import tqdm import perception.hashers import perception.tools hasher = perception.hashers.PHash(hash_size=16) # Download some example videos. urllib.request.urlretrieve( "https://thorn-perception.s3.amazonaws.com/thorn-perceptual-video-deduplication-example.zip", "thorn-perceptual-video-deduplication-example.zip" ) with zipfile.ZipFile('thorn-perceptual-video-deduplication-example.zip') as f: f.extractall('.') # Set a threshold for matching frames within videos and across videos. intravideo_threshold = 0.1 intervideo_threshold = 0.2 files = glob.glob('thorn-perceptual-video-deduplication-example/*.m4v') + \ glob.glob('thorn-perceptual-video-deduplication-example/*.gif') hashes =  for f in tqdm.tqdm(files): previous = None for frame, _, _ in perception.hashers.tools.read_video(f, frames_per_second=1): current, quality = hasher.compute_with_quality(frame, hash_format='vector') if quality < 90 or ( previous is not None and hasher.compute_distance(current, previous) < intravideo_threshold ): # This frame is either low quality or too similar to the previous frame continue previous = current hashes.append((f, current)) duplicates = perception.tools.deduplicate_hashes( hashes=hashes, threshold=intervideo_threshold, hasher=hasher )