image recovery

Last month, I accidently formatted a card that I had not copied. At
the time, I was stupidly, and against my own advice I liberally offer
to this forum, penny-pinching on storage ... I was carefully in-camera
editing images and taking new ones. This leads to nasty fragmentation
problems, but hey, I'm an expert! I know what I am doing! Hubris!
Oh, the pain!

Now, last February, two people were telling me that I was flat out
wrong to say that automatic recovery tools were not going to work in
situations like this. So hey, maybe they were right and my previous
experience with these tools was an aberration. "Zero Assumptions" and
the like to the rescue!

HA HA HA!! Of the ~240 images that were on this card at the time it
was formatted, I could not find a tool that would recover them all.
Not one. One tool returned 74. Another got me about 163. Similar
results for others. And most hilariously, of the supposed files, only
about _half_ were actually usable. Naturally, I was not surprised at
all at this result -- since I know just how hard this problem is,
having done it a few times in the past (FAT, ext2, etc).

A lesson I didn't need to learn again: never let a bunch of ignorant
wannabes tell you what reality is; I bit-copied the card to my laptop
for later analysis.

Later was this past weekend. Here is what I have done so far:

I carefully examined (via tiffdump and a hex editor) the structure of
the Canon .CR2 file and made a small program that reliablity detects
them. This turns out to be especially simple, since Canon -- bless
them -- store the link to the raw data TIFF IFD at the front of the
file,and this IFD has a fixed structure, so figuring out total file
length etc is easy without a full TIFF decoder etc.

This detector was placed into a simple scanner that sweeps the bit-copy
of the card, looking for contiguously allocated files. Inside of
minute or so it pulls out 183 such files -- every single one of them
decodable. These 183 are removed from consideration and the process
repeated and another 30 otherwise perfect files are obtained: 213 of
~240.

Not bad for a few hours of work. However, the fragmentation problem
now rears its ugly head for the remaining 34 TIFF headers.
Fortunately, though, it's easy to discover the _last_ cluster of a
file, because a .CR2 files (or at least the ones I have) all end with
"FF D9" (JPEG "end-of-image"), and it turns out I am very lucky in that
for 27 of these 34, the CR2 file is unfragmented prior to the raw data:
I have a 1536x1024 JPEG of the image. The job is just to select about
200 clusters from about (average) 1000. Order is implicit by the
nature of always increasing FAT allocation. However, a major problem
is that these clusters are all random data, virtually no way to tell
one way or another if the cluster has image data or not. And writing
some manual tool is, like, so third-world, don't you think? So how to
do it?

Heh heh heh ... lets just say that if one makes "zero assumptions" one
obtains "zero results". I'm virtually certain I'll be able to extract
the remaining 27 images (maybe even more) with the help of a modified
dcraw and some image correlations. On your knees, pilgrim: the
Internet is God.

wrote:
Last month, I accidently formatted a card that I had not copied. At
the time, I was stupidly, and against my own advice I liberally offer
to this forum, penny-pinching on storage ... I was carefully in-camera
editing images and taking new ones. This leads to nasty fragmentation
problems, but hey, I'm an expert! I know what I am doing! Hubris!
Oh, the pain!

Now, last February, two people were telling me that I was flat out
wrong to say that automatic recovery tools were not going to work in
situations like this. So hey, maybe they were right and my previous
experience with these tools was an aberration. "Zero Assumptions" and
the like to the rescue!

HA HA HA!! Of the ~240 images that were on this card at the time it
was formatted, I could not find a tool that would recover them all.
Not one. One tool returned 74. Another got me about 163. Similar
results for others. And most hilariously, of the supposed files, only
about _half_ were actually usable. Naturally, I was not surprised at
all at this result -- since I know just how hard this problem is,
having done it a few times in the past (FAT, ext2, etc).

A lesson I didn't need to learn again: never let a bunch of ignorant
wannabes tell you what reality is; I bit-copied the card to my laptop
for later analysis.

Later was this past weekend. Here is what I have done so far:

I carefully examined (via tiffdump and a hex editor) the structure of
the Canon .CR2 file and made a small program that reliablity detects
them.

snipped

It goes to show you that a little specialized knowledge can go a long
way. Most people would have recovered what was reasonably recoverable
(existing tools) and sacrificed the rest in the name of expediency.

Of those that you recovered ... any worth recovering?

Cheers,
Alan

--
-- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm
-- r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm
-- [SI] gallery & rulz: http://www.pbase.com/shootin
-- e-meil: Remove FreeLunch.

Alan Browne wrote:

Of those that you recovered ... any worth recovering?

Fortunately, yes. And I would have stopped at 213 -- "low hanging
fruit" -- but for the fact that I know there are at least two more good
images in there (or at least they looked good in-camera). Naturally,
the end product here is not only the instant images of uncertain
quality, but at least the ability to reproduce the feat in the future
without a fuss ... and maybe this thread will be the announcement of
the "eawckyegcy data recovery service", where the corporate mascot will
be the Gila Monster.

wrote in message
oups.com...
Alan Browne wrote:

Of those that you recovered ... any worth recovering?

Fortunately, yes. And I would have stopped at 213 -- "low hanging
fruit" -- but for the fact that I know there are at least two more good
images in there (or at least they looked good in-camera). Naturally,
the end product here is not only the instant images of uncertain
quality, but at least the ability to reproduce the feat in the future
without a fuss ... and maybe this thread will be the announcement of
the "eawckyegcy data recovery service", where the corporate mascot will
be the Gila Monster.


My Nikon D200 only formats the part of the card with out photos. I don't
know if it a feature of the camera or a feature I set on the camera., but it
has saved me several times from loosing photos. I have owned several digital
cameras since they first arrived on the market, and this is the first camera
that has such a feature.