raw files are HUGE

On Wed, 11 Apr 2007 23:14:45 +0100, John Bean wrote:

You may be right, but couldn't thumbnails be available in
uncompressed form even though the main image has been compressed?

The thumnail (and bigger preview) in most raw files is a
JPEG anyhow, so it's even more compressed than the raw data
;-)

Note that I'm didn't state how thumbnails are or aren't stored. I
just suggested that there shouldn't be a good reason why thumbnails
*must* be compressed. I'd think that the cameras that store a
normal jpeg image along with the raw files (some cameras using a
single file to store both, others saving two separate files) might
well want to compress the jpegs, but there wouldn't be as good a
reason to force the compression of the much smaller thumbnails.


Many raw files are a variant of TIFF, so the compression of
the image data is independent of the thumbnail(s) stored
elsewhere in the file. That's why a simple registry patch
for Windows to tell it that a NEF or a DNG is just a TIFF in
disguise results in Explorer showing thumbs for those files
despite having no idea what the raw data inside the file
actually represents, compressed or not.

Correct, Explorer doesn't have a "preview" option for NEF files,
but does when they're renamed to .TIFF, and then it shows a very
small thumbnail. That doesn't necessarily indicate that TIFF is in
any way similar to NEF, ORF or other raw files, just that other
non-raw components may be stored similarly in TIFF and raw files.
Don't some cameras create much larger TIFF files than raw files?
That would indicate that one isn't quite the other in disguise.

In article , John Bean
wrote:

On Wed, 11 Apr 2007 17:38:08 -0400, ASAAR
wrote:

On Wed, 11 Apr 2007 13:55:34 -0700, Ken Lucke wrote:

It's just not worth it - IMO, if you're going to be shooting (AND
keeping, including /keeping/ /track/ /of/) that many files, the slight
savings in disk space is not really *even* going to be a factor.

Besides, if they're compressed, you're not going to have a thumbnail
preview available to quickly look at them in the Finder or the Windows
browser (all you're going to have is the icon for whatever file type
they are), so you're going to have to decommpress them /just/ to see
what they are when trying to find one, if you don't use management
tools like Aperture or Lightroom - and you /can't/ use them if you are
using compressed files.

You may be right, but couldn't thumbnails be available in
uncompressed form even though the main image has been compressed?

The thumnail (and bigger preview) in most raw files is a
JPEG anyhow, so it's even more compressed than the raw data
;-)

Many raw files are a variant of TIFF, so the compression of
the image data is independent of the thumbnail(s) stored
elsewhere in the file. That's why a simple registry patch
for Windows to tell it that a NEF or a DNG is just a TIFF in
disguise results in Explorer showing thumbs for those files
despite having no idea what the raw data inside the file
actually represents, compressed or not.

I don't think you understand. If you compress ANY file into a .zip,
..sit, .rar, .sitx, or whatever, the file then becomes that file type,
until uncompressed again. Any emmbedded preview, thumbail, or other
data is all compressed right along with the rest of the data, and thus
is not accessible to the OS to display. Unless you specifically
generate (taking the extra time, CPU cycles, and knowledge to do so) a
custom icon/thumbnail for each and every file, it shows up in the
operating system as the generic icon for that type of compressed file.

--
You need only reflect that one of the best ways to get yourself a
reputation as a dangerous citizen these days is to go about repeating
the very phrases which our founding fathers used in the struggle for
independence.
-- Charles A. Beard

On 2007-04-11, ASAAR wrote:
Correct, Explorer doesn't have a "preview" option for NEF files,
but does when they're renamed to .TIFF, and then it shows a very
small thumbnail. That doesn't necessarily indicate that TIFF is in
any way similar to NEF, ORF or other raw files, just that other
non-raw components may be stored similarly in TIFF and raw files.

DNG files use the TIFF structure to encode the information. If NEF
files can be seen as previews when renamed to .TIFF, then they likely
follow the same kind of structure.

When converting MRW files to DNG files, the Adobe DNG converter (with
the default settings) produces TIFF files containing a uncompressed
thumbnail image, a small JPEG sub-file, along with the actual RAW data
as another sub-file. A program that knows how to handle JPEG-in-TIFF
files may even be able to view the embeded picture in addition to the
thumbnail, but a raw converter would be needed to access the full data.

Don't some cameras create much larger TIFF files than raw files?
That would indicate that one isn't quite the other in disguise.

A TIFF image file contains red+green+blue components for each pixel,
totalling 3 or 6 bytes per pixel (depending on if the data is stored as
8 or 16 bit data). This data may be compressed with LZW, but that
wouldn't likely produce more than a 25% reduction in size.

On the other hand, raw files contain only one of the red, green, or blue
components per pixel, and is usually packed as 2 values in 3 bytes for
12-bit sensor data. Adobe DNG files may be further compressed
losslessly internally with JPEG-LS, which achieves about a further 40%
size reduction compared to the packed data size.

FWIW, the compression used internally in DNG files is significnatly more
effective than any other external compressor I've tried on either my
original MRW files or uncompressed DNG files, and I've tried a quite few
(zip, 7zip, rar, rzip, and bzip2).

--
Bruce Guenter http://untroubled.org/

On Wed, 11 Apr 2007 16:01:06 -0700, Ken Lucke
wrote:
I don't think you understand. If you compress ANY file into a .zip,
.sit, .rar, .sitx, or whatever, [snip]


Sorry, cross purposes. I thought you were referring to
compression of the raw image in the file rather than
external compression of the file itself. I agree with your
point.

--
John Bean

On Apr 12, 1:08 am, Bruce Guenter wrote:
[snip]
DNG files use the TIFF structure to encode the information. If NEF
files can be seen as previews when renamed to .TIFF, then they likely
follow the same kind of structure.
[snip]

Correct. In fact, NEFs and DNGs are so similar that you can do a lot
of examination of NEFs using the dng_validate.exe tool from the DNG
SDK.

--
Barry Pearson
http://www.barrypearson.co.uk/photography/

On Wed, 11 Apr 2007 14:06:31 -0700, Ken Lucke
wrote:

In article , Ken Lucke
wrote:
within minutes of walking in the door, thus giving me the ability to
store 300,000 MORE raw files than I was capable of doing a few minutes

Oops, sorry, I think I slipped a couple of decimal points there. Try
30,000,000 more.

You slipped again, and in the wrong direction.

322,122,547,200 bytes storage (300 GB - [yes I know there's some slight
overhead lost]) / 10240 byte files (average) == 31,457,280 files

1024 bytes is 1 KB (Kilobyte) so that calculation is for 10 KB files,
not 10 MB files.

A 300 GB disk holds approximately (allowing for overhead etc.):
300 1 GB files (duh? :-)
3,000 100 MB files
30,000 10 MB files

jc

--

"The nice thing about a mare is you get to ride a lot
of different horses without having to own that many."
~ Eileen Morgan of The Mare's Nest, PA

In article , JC Dill
wrote:

On Wed, 11 Apr 2007 14:06:31 -0700, Ken Lucke
wrote:

In article , Ken Lucke
wrote:
within minutes of walking in the door, thus giving me the ability to
store 300,000 MORE raw files than I was capable of doing a few minutes

Oops, sorry, I think I slipped a couple of decimal points there. Try
30,000,000 more.

You slipped again, and in the wrong direction.

322,122,547,200 bytes storage (300 GB - [yes I know there's some slight
overhead lost]) / 10240 byte files (average) == 31,457,280 files

1024 bytes is 1 KB (Kilobyte) so that calculation is for 10 KB files,
not 10 MB files.

A 300 GB disk holds approximately (allowing for overhead etc.):
300 1 GB files (duh? :-)
3,000 100 MB files
30,000 10 MB files


Yeah, you're right. I've been having a bad week, my brain is not
functioning properly (that's what happens when virtually your entire
life gets wiped out by identity theft).

--
You need only reflect that one of the best ways to get yourself a
reputation as a dangerous citizen these days is to go about repeating
the very phrases which our founding fathers used in the struggle for
independence.
-- Charles A. Beard