Wanted: Software to Find Duplicate Image Files on Hard Drive

I am looking for software that can eliminate dulplicate image files on
my hard drive. I would like this software to find all duplicates so I
can eliminate them before backing up the image files to a DVD.
Ideally, the software should be able to identify identical image files
even if they have different file names [but the same image/pixels]. I
would imagine that such software exists.

Any recommendations? -- Thanks!

HKEK wrote:
I am looking for software that can eliminate dulplicate image files on
my hard drive. I would like this software to find all duplicates so I
can eliminate them before backing up the image files to a DVD.
Ideally, the software should be able to identify identical image files
even if they have different file names [but the same image/pixels]. I
would imagine that such software exists.

I don't know of any pre-existing program but the core algorithm shouldn't
take more than maybe 10 lines in any decend scripting language like e.g.
Perl. Because literal comparison of file content is quite expensive I would
compute and store some checksum, e.g. MD5, and do a binary comparision only
if the checksums of 2 files match.

Add another 20-30 lines for standard overhead and user interaction and you
got a nice little script. I may actually do it if I feel bored this weekend.

jue

On Sat, 8 Dec 2007 19:18:07 -0800 (PST), HKEK
wrote:

I am looking for software that can eliminate dulplicate image files on
my hard drive.
Any recommendations? -- Thanks!

DeDuper is a small freeware program that finds exact duplicates, but
is limited to jpg and bmp. It's quite fast.

http://www.niftyprograms.com/deduper.htm

VisiPics is another freeware program that finds exact dups and near
dups, depending on settings used. It can compare jpg, bmp, tiff, png,
etc. Not quite as fast as DeDuper but more flexible.

http://www.download.com/VisiPics/300...-10712877.html
jimbok

jimbok wrote:
On Sat, 8 Dec 2007 19:18:07 -0800 (PST), HKEK
wrote:

I am looking for software that can eliminate dulplicate image files
on my hard drive.
Any recommendations? -- Thanks!

DeDuper is a small freeware program that finds exact duplicates, but
is limited to jpg and bmp. It's quite fast.

http://www.niftyprograms.com/deduper.htm

VisiPics is another freeware program that finds exact dups and near
dups, depending on settings used. It can compare jpg, bmp, tiff, png,
etc. Not quite as fast as DeDuper but more flexible.

http://www.download.com/VisiPics/300...-10712877.html
jimbok

There is a program called Windows which does that very well if you know the
names of the files which may be duplicated. You can get it from Microsoft.
Aren't they in Seattle or somewhere?

DP

Dennis Pogson wrote:

There is a program called Windows which does that very well if you
know the names of the files which may be duplicated. You can get it
from Microsoft. Aren't they in Seattle or somewhere?

DP

No, they are in my dustbin. I use debian (much more reliable).

--
Neil
reverse ra and delete l
Linux user 335851

Try d'peg http://www.somewareonthe.net/index_dpeg.htm
I've used this to sort through thousands of pictures and it works like a
charm. It's extremely powerful and will find dups that have been
cropped, rotated or color corrected


HKEK wrote:
I am looking for software that can eliminate dulplicate image files on
my hard drive. I would like this software to find all duplicates so I
can eliminate them before backing up the image files to a DVD.
Ideally, the software should be able to identify identical image files
even if they have different file names [but the same image/pixels]. I
would imagine that such software exists.

Any recommendations? -- Thanks!

On Sun, 09 Dec 2007 12:34:28 GMT, Ender
wrote:

Try d'peg http://www.somewareonthe.net/index_dpeg.htm
I've used this to sort through thousands of pictures and it works like a
charm. It's extremely powerful and will find dups that have been
cropped, rotated or color corrected

Surely they're not duplicates in that case?

--
John Bean

John Bean wrote:

On Sun, 09 Dec 2007 12:34:28 GMT, Ender
wrote:

Try d'peg http://www.somewareonthe.net/index_dpeg.htm
I've used this to sort through thousands of pictures and it works like a
charm. It's extremely powerful and will find dups that have been
cropped, rotated or color corrected

Surely they're not duplicates in that case?


I've seen many politicians reverse themselves any number of times, yet
they remain exactly identical to their previous contents and quality.




mike

On Sun, 09 Dec 2007 04:17:22 GMT
"Jürgen Exner" wrote:

HKEK wrote:
I am looking for software that can eliminate dulplicate image files
on my hard drive. I would like this software to find all duplicates
so I can eliminate them before backing up the image files to a DVD.
Ideally, the software should be able to identify identical image
files even if they have different file names [but the same
image/pixels]. I would imagine that such software exists.

I don't know of any pre-existing program but the core algorithm
shouldn't take more than maybe 10 lines in any decend scripting
language like e.g. Perl. Because literal comparison of file content
is quite expensive I would compute and store some checksum, e.g. MD5,
and do a binary comparision only if the checksums of 2 files match.

Add another 20-30 lines for standard overhead and user interaction
and you got a nice little script. I may actually do it if I feel
bored this weekend.

Yup. Humans are all natural born puzzle solvers. What is it that
makes so many of us think we can't write simple software like this?

My first whack would be a hybrid of perl and bash. Stuff this in
a file called "finddups":

#!/usr/bin/perl
while () {
chop;
@tokens = split / +/, $_;
if (exists($files{$tokens[0]})) {
print "$tokens[1] is a duplicate of $files{$tokens[0]}\n";
} else {
$files{$tokens[0]} = $tokens[1];
}
}

Then do something like this:

find / -name \*.jpg -exec md5sum {} \;| ./finddups

Adjust the find as needed if you also have .JPG, .jpeg, .JPEG, or
whatever. It'll take a long time because md5sum is expensive, but
it will identify all the dups on your disk.

Gee, that was fun! Now all it needs is to bring the find and the
MD5 stuff inside the perl script and add a nice Tk GUI. I could
sell it to people who think they can't program and get rich! :-)

Paul Allen

On Dec 9, 8:02 am, Jake Orsen wrote:
On Sun, 09 Dec 2007 06:26:15 GMT, jimbok wrote:
On Sat, 8 Dec 2007 19:18:07 -0800 (PST), HKEK
wrote:

I am looking for software that can eliminate dulplicate image files on
my hard drive.
Any recommendations? -- Thanks!

DeDuper is a small freeware program that finds exact duplicates, but
is limited to jpg and bmp. It's quite fast.

http://www.niftyprograms.com/deduper.htm

VisiPics is another freeware program that finds exact dups and near
dups, depending on settings used. It can compare jpg, bmp, tiff, png,
etc. Not quite as fast as DeDuper but more flexible.

http://www.download.com/VisiPics/300...-10712877.html
jimbok

Here's anotherhttp://www.bolidesoft.com/imagecomparer.html

Here's an excellent freeware one that was originally designed for use with
making tiled images from many smaller imageshttp://www.keronsoft.com/dupdetector.html

There's hundreds of such utilities out there. The only ones that don't know
about these are the resident-trolls in this newsgroup who have zero experience
with real photography. They become more and more obvious the more that you know
about real photography.- Hide quoted text -

- Show quoted text -

That one is a bit scary. I don't want to eliminate "similar" shots.