A short study on digicam's fixed jpeg compression ratio

Hi all,

I have recently purchased Nytech ND 4020 digital camera (aka Konica KD
4000). I have shot about 500 images now and the camera seems to work
fine. When moving the image files from memory card to PC I noticed that
every image shot with best quality is of same file size, 1 492 992
bytes. I would understand this if the images were uncompressed. Since
they are JPEG comperssed the file size should vary depending on the
image: image with lots of detail should be larger than an image with
less detail.

The manual says that best image quality setting uses fixed 8:1
compression ratio. At first this seems to be correct, since the
compressed file is 1458 kBs and uncompressed image data in RAM is 11 664
kBs (2304x1728, 4 Mpixels). I started to wonder how does the camera
implement this "fixed ratio jpeg compression"? I have never heard about
that kind of JPEG compression before.

First step towards the answer was when I rotated the images using a
software that has lossless JPEG rotation. The rotation decreased file
size significantly. It happened only once: when I kept rotating the file
the size did not continue to decrease. Second step was to edit file's
Exif data. It did the same: file was shrunk. The files that were rotated
or Exif edited were between 414 kB and 947 kB compared to original 1458 kB.

At this point I opened the files with hex editor and studied JPEG/EXIF
specs. The specs told that every file should start with tag FFD8 and end
with tag FFD9. Every rotated and Exif edited file followed this rule
since they were saved by a software on my PC. But the unmodified images
straight from the camera did not! The unmodified files start with FFD8
and then follows the data. At some point comes FFD9 tag but the file
does not end there. There are lots of data after the end tag and it
looks like "uninitialized memory" when browsed with hex editor.

Conclusion: the camera uses standard JPEG compression (like all other
digicams?) and only imitates 8:1 fixed ratio compression by inflating
the file to match 8:1 compression result. Why? Why does it fill my sD
cards with unused data? I guess that it simplifies the user experiment:
a regular user is happy when he gets fixed image size and can predict
when the card will fill up. And it might have something to do with
software development: it might be easier to write the software so that
every time you shoot it reserves fixed 1458 kB from the card and then
dumps image data into it starting from the beginning. Because the end
tag is misplaced the file does not follow JPEG spec and is actually not
a JPEG file. Happily enough all modern software seem to read and open
these files.

Is there someone who owns Nytech ND 4020 or Konica ND 4000 that could
confirm this behavior? How about other Nytech/Konica models? How about
digicams from other vendors, do they use same kind of "bogus fixed ratio
compression"?

Well, maybe I should spend more time shooting than analyzing the binary
files...

--
Heikki@Finland

"Heikki Siltala" wrote in message
...
[]
Conclusion: the camera uses standard JPEG compression (like all other
digicams?) and only imitates 8:1 fixed ratio compression by inflating
the file to match 8:1 compression result. Why? Why does it fill my sD
cards with unused data? I guess that it simplifies the user experiment:
a regular user is happy when he gets fixed image size and can predict
when the card will fill up. And it might have something to do with
software development: it might be easier to write the software so that
every time you shoot it reserves fixed 1458 kB from the card and then
dumps image data into it starting from the beginning. Because the end
tag is misplaced the file does not follow JPEG spec and is actually not
a JPEG file. Happily enough all modern software seem to read and open
these files.

Heikki, that sounds like a fair conclusion to me - pad the files so that
they are always the same size! Not clever. Not clever, at all!

Just an aside - what size is a blank floppy disk? Let me check.
1,457,664 bytes is the "available" space when formatting a disk. I just
though for one minute it might be padded out to the size of a floppy disk!

What a horrible little camera (software-wise).

David

"Heikki Siltala" wrote in message
...
[]
Conclusion: the camera uses standard JPEG compression (like all other
digicams?) and only imitates 8:1 fixed ratio compression by inflating
the file to match 8:1 compression result. Why? Why does it fill my sD
cards with unused data? I guess that it simplifies the user experiment:
a regular user is happy when he gets fixed image size and can predict
when the card will fill up. And it might have something to do with
software development: it might be easier to write the software so that
every time you shoot it reserves fixed 1458 kB from the card and then
dumps image data into it starting from the beginning. Because the end
tag is misplaced the file does not follow JPEG spec and is actually not
a JPEG file. Happily enough all modern software seem to read and open
these files.

Heikki, that sounds like a fair conclusion to me - pad the files so that
they are always the same size! Not clever. Not clever, at all!

Just an aside - what size is a blank floppy disk? Let me check.
1,457,664 bytes is the "available" space when formatting a disk. I just
though for one minute it might be padded out to the size of a floppy disk!

What a horrible little camera (software-wise).

David

"Heikki Siltala" wrote in message
...
digicams?) and only imitates 8:1 fixed ratio compression by inflating
the file to match 8:1 compression result. Why? Why does it fill my sD
cards with unused data? I guess that it simplifies the user experiment:


good job, very thourough analysis:


Based on what you report, I think the program allocates for the maximum size
in a variable, then write to that variable...I think the FFD9 acts like a
string terminator. I don't think it is padding or adding, it sounds like
really sloppy programming, where they allocate space for something, but do
not check to see the size of what they are writing to that variable
(memory).

for instance you get this behaviour when you write to a char[n] but write
less than n characters to it. I find it hard to believe they would make
such a amatuer mistake...maybe I am misunderstanding your results....

Bob

"Heikki Siltala" wrote in message
...
digicams?) and only imitates 8:1 fixed ratio compression by inflating
the file to match 8:1 compression result. Why? Why does it fill my sD
cards with unused data? I guess that it simplifies the user experiment:


good job, very thourough analysis:


Based on what you report, I think the program allocates for the maximum size
in a variable, then write to that variable...I think the FFD9 acts like a
string terminator. I don't think it is padding or adding, it sounds like
really sloppy programming, where they allocate space for something, but do
not check to see the size of what they are writing to that variable
(memory).

for instance you get this behaviour when you write to a char[n] but write
less than n characters to it. I find it hard to believe they would make
such a amatuer mistake...maybe I am misunderstanding your results....

Bob

Heikki Siltala writes:

Because the end
tag is misplaced the file does not follow JPEG spec and is actually not
a JPEG file.

Are you sure? It sounds like the file does follow the JPEG spec for all
the data between the start and end markers, and the markers are correct.
The only anomaly is that there's garbage data after the end marker. This
may not violate the JPEG spec if it just specifies what happens between
markers. It's illegal only if the spec explicitly says "there shall be
no data in the file following the end marker".

Is there someone who owns Nytech ND 4020 or Konica ND 4000 that could
confirm this behavior? How about other Nytech/Konica models? How about
digicams from other vendors, do they use same kind of "bogus fixed ratio
compression"?

I have several Canon digicams. The file size is different every frame.
The best quality (superfine) setting is about 7:1 compression ratio.

Dave

Heikki Siltala writes:

Because the end
tag is misplaced the file does not follow JPEG spec and is actually not
a JPEG file.

Are you sure? It sounds like the file does follow the JPEG spec for all
the data between the start and end markers, and the markers are correct.
The only anomaly is that there's garbage data after the end marker. This
may not violate the JPEG spec if it just specifies what happens between
markers. It's illegal only if the spec explicitly says "there shall be
no data in the file following the end marker".

Is there someone who owns Nytech ND 4020 or Konica ND 4000 that could
confirm this behavior? How about other Nytech/Konica models? How about
digicams from other vendors, do they use same kind of "bogus fixed ratio
compression"?

I have several Canon digicams. The file size is different every frame.
The best quality (superfine) setting is about 7:1 compression ratio.

Dave

Heikki Siltala writes:

Because the end
tag is misplaced the file does not follow JPEG spec and is actually not
a JPEG file.

Are you sure? It sounds like the file does follow the JPEG spec for all
the data between the start and end markers, and the markers are correct.
The only anomaly is that there's garbage data after the end marker. This
may not violate the JPEG spec if it just specifies what happens between
markers. It's illegal only if the spec explicitly says "there shall be
no data in the file following the end marker".

Is there someone who owns Nytech ND 4020 or Konica ND 4000 that could
confirm this behavior? How about other Nytech/Konica models? How about
digicams from other vendors, do they use same kind of "bogus fixed ratio
compression"?

I have several Canon digicams. The file size is different every frame.
The best quality (superfine) setting is about 7:1 compression ratio.

Dave

Dave Martindale wrote:
Are you sure? It sounds like the file does follow the JPEG spec for all
the data between the start and end markers, and the markers are correct.
The only anomaly is that there's garbage data after the end marker. This
may not violate the JPEG spec if it just specifies what happens between
markers. It's illegal only if the spec explicitly says "there shall be
no data in the file following the end marker".

I studied the specs again. The camera seems to output Exif 2.2 files so
I focused on Exif 2.2 spec "JEITA CP-3451 Exchangeable image file format
for digital still cameras: Exif Version 2.2", available at
http://www.kodak.com/global/plugins/...fStandard2.pdf.


Your point is well made since I found no explicit information like
"there shall be no data in the file following the end marker". Still,
Page 11 Figure 6 describes "Basic Structure of Compressed Data Files"
and indicates that the file structure is

SOI,APP1(,APP2),DQT,DHT(,DRI),SOF,SOS,Compressed Data,EOI

where SOI is the start marker (FFD8) and EOI is the end marker (FFD9).
Ih the figure the file ends right after the end marker. I would
interpret this so that there can be nothing after the end marker. At
this light I would say the camera doesn't output valid Exif 2.2 files
because the file continues after the end marker.

I also studied from "JEITA CP-3461 Design rule for Camera File system
DCF Version 2.0" available at http://tsc.jeita.or.jp/avs/data/cp3461.pdf
but it doesn't give information on the issue.

I examined the files output by the camera a little bit further. When I
take photos with different resolution and compression ratio settings
there is always at least some unnecessary data after the end marker. So
the issue is not dependent on the resolution and compression ratio used.

--
H

Heikki Siltala writes:

Your point is well made since I found no explicit information like
"there shall be no data in the file following the end marker". Still,
Page 11 Figure 6 describes "Basic Structure of Compressed Data Files"
and indicates that the file structure is

SOI,APP1(,APP2),DQT,DHT(,DRI),SOF,SOS,Compressed Data,EOI

where SOI is the start marker (FFD8) and EOI is the end marker (FFD9).
Ih the figure the file ends right after the end marker. I would
interpret this so that there can be nothing after the end marker. At
this light I would say the camera doesn't output valid Exif 2.2 files
because the file continues after the end marker.

Generally, figures in standards are intended to illustrate what the
standard document is describing, and make it easier to understand. They
may illustrate a typical file structure, but not necessarily one that is
absolutely required. It's the words of the standard that specify what
is and is not allowed.

I'd interpret what you describe as an illustration showing that typical
JPEG files don't have any data after the end marker. But unless the
standard says explicitly that extra trailing data is not allowed, it
does not violate the standard. It is stupid, and wastes space, and
indicates poorly-written software, but it's not "illegal".

Dave