Could you actually see photos made from RAW files?

Eric Stevens wrote:
On Tue, 02 Jun 2009 00:19:18 +1000, Bob Larter
wrote:

Eric Stevens wrote:
On Sat, 30 May 2009 21:32:27 -0800, (Floyd L.
Davidson) wrote:

Floyd has failed to point out that at this point he has omitted a
great deal of previous text. I know he has been around more than long
enough to know that this is not the right thing to do.

Eric Stevens wrote:
Floyd, I suspect you have been smoking something which is not good for
you. Subject to statistical error limitations, there is a one to one
correspondence between the source image and the RAW file. One can be
converted to the other using the rules inherent in the camera's
software.
What do you mean by "the source image"?

That which is projected onto the sensor by the lens.

That's what's encoded in the RAW file. It's a losslessly compressed dump
of the contents of the image sensor, without any interpretation.

Not so. The sensor stores volts. The RAW file maps this in 12 or 14
bit patterns. Some transformation has clearly taken place. Nor is it
necessarily compressed and, if compression has taken place it need not
be lossless.

It would perhaps have been preferable if Bob had used
"interpolation" rather than "interpretation", but in
fact he is technically correct anyway.

What you describe is encoding of the data, not
interpretation or interpolation.

He's talking about the *process* of converting from the RAW image to the
RGB image that you see on your screen, which includes Bayer
deconvolution. As he says, there is no 1:1 relationship between a pixel
("sensel") on the image sensor & a pixel on the RGB image that you see
on your screen.

My original proposition was that there is a one to one correspondence
between the source image (that which is projected onto the sensor by
the lens) and data which is stored in the RAW file. Its not possible
that any single RAW file could be produced by more than one sensor
image.

What isn't possible is to use the recorded data in the
RAW file to precisely reproduce an image which is
exactly the same as the one which was projected onto the
sensor. That is indeed simply because there were
multiple possible images which could produce the same
raw data.

--
Floyd L. Davidson http://www.apaflo.com/floyd_davidson
Ukpeagvik (Barrow, Alaska)

Eric Stevens wrote:
On Tue, 02 Jun 2009 00:28:02 +1000, Bob Larter
wrote:

Eric Stevens wrote:
On Sun, 31 May 2009 16:11:00 -0800, (Floyd L.
[...]
I've been talking about the RAW file from the beginning. So too were
you at that time. Remember "Floyd, I suspect you have been smoking
something which is not good for you. Subject to statistical error
limitations, ... The data in the RAW file can't be restructured to
make a different image without changing the data."

This is incorrect. As Floyd has said, the same RAW data can result in an
infinite number of final images. For example, one can process it with a
huge range of WB values, or change the EV correction by plus or minus a
couple of stops, set the black level to anything you like, etc, etc.
None of these changes require changes to the RAW data, merely the
interpretation of it.

Its incorrect, as you (or somebody) chopped out the important part. I
originally wrote:

"Floyd, I suspect you have been smoking something which is not
good for you. Subject to statistical error limitations, there is a
one to one correspondence between the source image and the
RAW file. One can be converted to the other using the rules
inherent in the camera's software. The data in the RAW file can't
be restructured to make a different image without changing the
data."

We were then talking about the "source image" - that which was
projected onto the sensor by the lens - and not that infinite range of
images which could be created downstream by the manipulation of the
RAW data.

The raw data can produce multiple images. That is due
to the interpolation that you continue to reference, and
is the reason I continue to discuss the image produced
by interpolation of the raw data.

The exact same reasoning can be applied earlier in the
data flow where it is true there are multiple different
images which could be projected onto the sensor that
will generate precisely the same raw data.

--
Floyd L. Davidson http://www.apaflo.com/floyd_davidson
Ukpeagvik (Barrow, Alaska)

Eric Stevens wrote:
On Tue, 02 Jun 2009 02:59:51 +1000, Bob Larter
wrote:

Eric Stevens wrote:
Nope. What comes out of the sensor is not what is saved in the RAW
file. There is a transformation involved.

What transformation is that? TTBOMK, the only transformation is the A2D
conversion. And that lack of transformations is, after all, the whole
point of the RAW file format in the first place.

AFAIK the contents of the RAW file (ignoring metadata etc) is a fully
detailed map of the sensor data but is not just a 'dump' of the sensor
data. At the very least the quantized sensor voltage has to be
transformed into 12 or 14 bit data. Various corrections may be applied
in the process.

That technically is data encoding, not any sort of
transform that changes the image. Generally the only
adjustment is the ISO sensitivity (analog gain between
the sensor and the ADC).

As noted in another message Nikon apparently has done
some work on white balance adjustment, but while they
hinted at it in describing the D2x, they have never
given any details, and to my knowledge have never
mentioned it again in regard to later models. Whatever,
it is suspected that they arranged the sensor read
channels in a way that caused each color to be sent to
specific analog amplifiers (otherwise used for ISO
adjustment), and then made minor gain adjustments to
accomplish White Balance.

Not exactly earth shaking; but regardless there has never
been any public release of information about exactly what
they did or didn't do.

--
Floyd L. Davidson http://www.apaflo.com/floyd_davidson
Ukpeagvik (Barrow, Alaska)

Eric Stevens wrote:
I've got to plead guilty to that. In a moment of brain fade I got
sucked into what Floyd was trying to talk about rather than the
original topic which I was trying to talk about. Rereading all this
below I can see that I had become more than somewhat confused.

Lets be more honest about it Eric. I didn't confuse you
at all, simply because you started out in total
confusion. It's finally beginning to sink in, to the
point where even you can see it.

I'm talking in the hypothetical sense of being able to derive from the
RAW data the light pattern which fell on the sensor to create the RAW
data in the first place. In the case of RAW data which has not been
messed around in some way there is only the one sensor image which
will correspond.

That last sentence is not valid, and is a major source
for your confusion. As nospam said, that doesn't make
sense.

--
Floyd L. Davidson http://www.apaflo.com/floyd_davidson
Ukpeagvik (Barrow, Alaska)

Eric Stevens wrote:

With all these changes to its form, the original data has clearly been
transformed. This does not mean that its meaning has been changed.
Merely that its form has been changed.

Lets be more technically astute. The data is *not* being
"transformed". What you are referring to is called encoding.
The data values are unchanged in theory. In practice of course
there are various distortions that accompany any given type of
encoding, plus there are environmental factors such as noise.

But the point is that no change to the data is intended to be
an "adjustment" to the image.

Your turn now. Have a look at what I am trying to say about the
meaning of transformation. 1/4 = 0.25

Good example. One number is encoded as a fraction, the
other as a decimal value. They are *not* different, and
hence the image they relate to has not been changed.

--
Floyd L. Davidson http://www.apaflo.com/floyd_davidson
Ukpeagvik (Barrow, Alaska)

Eric Stevens wrote:
On Mon, 01 Jun 2009 02:45:35 -0800, (Floyd L.
Davidson) wrote:

I have changed the silly
things you've written, and I changed
the meaning of a single sentence. Please blame
me for what you write!

Do you see what happens when you delete text without indicating the
fact?

I see that you are dishonest. And disgusting.

And lets be clear that I have *never* edited any quotes of
your articles in a way that would change the meaning of what
you said.

Again though, it's time that you learned that proper
Usenet netiquette is to trim the quoted text to only that
require for context. I do that. You don't.

Agian though, its time that you learned that proper Usenet netiquette
is to indicate where you have trimmed the quoted text. I do that. You
don't.

That has never been a "norm" on Usenet. (Besides, it is clear
from this article that you actually do not do what you claimed
to!)

I agree about the interpolation. As I explained in another article, I
boobed on that. The question has never been just one of interpolation
or no interpolation. Its been about the transformation of the image
which falls on the sensor to the data which is recorded in the RAW
file. One image gives one set of data. The one set of data can only
give the one sensor image (except for the Nikon RAW files using a
lossy compression).

Many images could give the exact same data. You simply do not
understand how the data is processed and what the work flow is.

Note that "sensor data", in the context of this
discussion, would be the analog data directly read from
the sensor ....

... what analog data?

The sensor is an analog device.

Not really. It's counting photons.

The sensor is an analog device by definition. Look up
what an analog device is and stop making up your own
definitions.

Neither of us has used it in that context, and since you
have repeatedly confused various parts of the data flow
it is absolutely essential to differentiate the analog
sensor data from the digital data output of the ADC.

In many contexts the sensor, the ISO amplifiers, and the
ADC are all considered "the sensor" in order to simplify
a discussion that really does not involve them other than
as a unit. This is clearly not one of those discussions.

Thats all part of the firmware which transforms the sensor data in the
data of the RAW file.

That is *not* part of the firmware. Firmware, other
than setting the ISO gain, has no part in any of that
other than turning it on and off.

Again, look up the definition of firmware and stop using
you own.

What do you make of:

. Image quality: NEF (RAW ) + JPEG
. NEF (RAW) recording: Lossless compressed or Compressed
. Image size: S or M

That sounds like a change in the way raw data from the sensors have
been interpreted and saved to the RAW file.

It looks like the specifications from the User Manual to
me. Where's the change?

I don't know. All I know is that the text above has been cut and
pasted from the Nikon site
http://nikoneurope-en.custhelp.com/a...ail/a_id/25962
describing "D300 Firmware Update 1.10 Windows". Clearly the downloaded
software has something to do with the transformation of the sensor
data to the data of the RAW file.

That is not clear at all. It says nothing to indicate
that. If all you know about it is that it is from that
particular URL... why would you cite it to support a
false claim that it says nothing about?

The original argument was over whether or not it was possible to work
back from the RAW data and arrive at more than one sensor image. I

A nonsense idea, but...

said that subject to statistical limitations there is only one sensor
image corresponding with the RAW data of the file. This raised the
question of the transformation of the sensor data to the data of the
RAW file. One aspect of the argument is whether there is any
transformation and I say there is. Then we got sidetracked onto
whether or not the transformation is all hardware, and I said that
software may be involved. That's why the question of the firmware
upgrade of the D300 entered the picture. The point of this is that the
RAW file data may not just be dump of the sensor data.

But the fact is that essentially it is just that, a dump of the
sensor data. Squirm all you like...

All of this is irrelevant to the original question which was centered
around whether the transformation from sensor image to RAW file data
was fully reversible. In most cases it is in which case you can show

In no case is it reversible.

that only the one sensor image corresponds to any one RAW file data

It is easy to show that there are multiple images on the
sensor that could produce exactly the same data set.

set. I described this as being a 'one to one' relationship. This is

And that has been thoroughly covered in other articles,
because it is simply false.

where the current furore started. A moment's brain fade let me be
dragged into questions of interpolation but I was clearly wrong. I'm
going to ignore that issue.

If you ignore every part of this discussion where you've
been wrong, we have nothing left to discuss except
perhaps the spelling of your name.

--
Floyd L. Davidson http://www.apaflo.com/floyd_davidson
Ukpeagvik (Barrow, Alaska)

In article , Floyd L. Davidson
wrote:

unless you upscale or downscale, the number of input sensels will be
the same as the number of output pixels.

That is not true.

The Nikon D3, as one example, is specified as having a
12.87 Mega pixel sensor. The images it generates are
specified to be 4256x2832, which works out to 12.05 Mega
pixels. Dave Coffin's dcraw program generates a
4284x2844 (12.18Mp) image.

so he keeps the border pixels. that doesn't negate what i said.

On Mon, 01 Jun 2009 18:37:19 -0800, (Floyd L.
Davidson) wrote:

Eric Stevens wrote:

With all these changes to its form, the original data has clearly been
transformed. This does not mean that its meaning has been changed.
Merely that its form has been changed.

Lets be more technically astute. The data is *not* being
"transformed". What you are referring to is called encoding.
The data values are unchanged in theory. In practice of course
there are various distortions that accompany any given type of
encoding, plus there are environmental factors such as noise.

But the point is that no change to the data is intended to be
an "adjustment" to the image.

Your turn now. Have a look at what I am trying to say about the
meaning of transformation. 1/4 = 0.25

Good example. One number is encoded as a fraction, the
other as a decimal value. They are *not* different, and
hence the image they relate to has not been changed.

This is why we have been arguing. See the definitions of
Transformation at http://www.thefreedictionary.com/transformation

In the particular context the ones that apply a

3. Mathematics
a. Replacement of the variables in an algebraic expression by
their values in terms of another set of variables.
b. A mapping of one space onto another or onto itself.
4. Linguistics
a. A rule that systematically converts one syntactic form or
form of a sentence into another.
b. A construction or sentence derived by such a rule; a
transform.

It is the systematic conversion which is at the heart of it all.



Eric Stevens

On Mon, 01 Jun 2009 18:27:05 -0800, (Floyd L.
Davidson) wrote:

Eric Stevens wrote:
I've got to plead guilty to that. In a moment of brain fade I got
sucked into what Floyd was trying to talk about rather than the
original topic which I was trying to talk about. Rereading all this
below I can see that I had become more than somewhat confused.

Lets be more honest about it Eric. I didn't confuse you
at all, simply because you started out in total
confusion. It's finally beginning to sink in, to the
point where even you can see it.

I'm talking in the hypothetical sense of being able to derive from the
RAW data the light pattern which fell on the sensor to create the RAW
data in the first place. In the case of RAW data which has not been
messed around in some way there is only the one sensor image which
will correspond.

That last sentence is not valid, and is a major source
for your confusion. As nospam said, that doesn't make
sense.

Are you really saying that a given RAW data file can be created by
more than one image?



Eric Stevens

On Mon, 01 Jun 2009 18:53:15 -0800, (Floyd L.
Davidson) wrote:

Eric Stevens wrote:
On Mon, 01 Jun 2009 02:45:35 -0800, (Floyd L.
Davidson) wrote:

I have changed the silly
things you've written, and I changed
the meaning of a single sentence. Please blame
me for what you write!

Do you see what happens when you delete text without indicating the
fact?

I see that you are dishonest. And disgusting.

I'm trying to get through to others what is wrong with what you have
been doing to me. The only difference is that I confined myself to
deleting individual words from the one paragraph and then I pointed
out what I had done. You - you have been deleting whole paragraphs so
as to completely change the sense of what we have been arguing about.
You then compound matters by not even indicating where or that you
have snipped. To finish it off, you start preaching about netiquette!

And lets be clear that I have *never* edited any quotes of
your articles in a way that would change the meaning of what
you said.

Bull****. And, as you have already said, you have been round Usenet
more than long enough to know better.

Again though, it's time that you learned that proper
Usenet netiquette is to trim the quoted text to only that
require for context. I do that. You don't.

Agian though, its time that you learned that proper Usenet netiquette
is to indicate where you have trimmed the quoted text. I do that. You
don't.

That has never been a "norm" on Usenet.

Bull****. It was the norm when you hung around sci.archaeology. I
agree it is not common here but people don't normally snip out
numerous segments of the article to which they have been responding in
the way that you have been doing it.

(Besides, it is clear
from this article that you actually do not do what you claimed
to!)

I agree about the interpolation. As I explained in another article, I
boobed on that. The question has never been just one of interpolation
or no interpolation. Its been about the transformation of the image
which falls on the sensor to the data which is recorded in the RAW
file. One image gives one set of data. The one set of data can only
give the one sensor image (except for the Nikon RAW files using a
lossy compression).

Many images could give the exact same data. You simply do not
understand how the data is processed and what the work flow is.

Let's get this clear. Are you saying that many different images
projected onto the sensor can result in the same RAW data file?

Note that "sensor data", in the context of this
discussion, would be the analog data directly read from
the sensor ....

... what analog data?

The sensor is an analog device.

Not really. It's counting photons.

The sensor is an analog device by definition. Look up
what an analog device is and stop making up your own
definitions.

I know what an analog device is. The wells in the sensor count
electrons as a proxy for photons. You can't have fractional electrons.
You can't have fractional photons. You must have integer values. It
cannot be an an analog device.

Neither of us has used it in that context, and since you
have repeatedly confused various parts of the data flow
it is absolutely essential to differentiate the analog
sensor data from the digital data output of the ADC.

In many contexts the sensor, the ISO amplifiers, and the
ADC are all considered "the sensor" in order to simplify
a discussion that really does not involve them other than
as a unit. This is clearly not one of those discussions.

Thats all part of the firmware which transforms the sensor data in the
data of the RAW file.

That is *not* part of the firmware. Firmware, other
than setting the ISO gain, has no part in any of that
other than turning it on and off.

You are referring to 'firmware' as though it was 'hardware'. Yet Nikon
can program the camera to behave differently so some software/firmware
must be involved.

Again, look up the definition of firmware and stop using
you own.

What do you make of:

. Image quality: NEF (RAW ) + JPEG
. NEF (RAW) recording: Lossless compressed or Compressed
. Image size: S or M

That sounds like a change in the way raw data from the sensors have
been interpreted and saved to the RAW file.

It looks like the specifications from the User Manual to
me. Where's the change?

I don't know. All I know is that the text above has been cut and
pasted from the Nikon site
http://nikoneurope-en.custhelp.com/a...ail/a_id/25962
describing "D300 Firmware Update 1.10 Windows". Clearly the downloaded
software has something to do with the transformation of the sensor
data to the data of the RAW file.

That is not clear at all. It says nothing to indicate
that. If all you know about it is that it is from that
particular URL... why would you cite it to support a
false claim that it says nothing about?

The original argument was over whether or not it was possible to work
back from the RAW data and arrive at more than one sensor image. I

A nonsense idea, but...

.... but that's where we came in. You wrote:

"That's what a camera raw file (the so called RAW format) is... a
pile of parts that you can build an image from, and while the
photographer may have had one specific image in mind when that
pile of data was saved, it can be restructured to make a lot of
different images too."

.... and I responded:

"Floyd, I suspect you have been smoking something which is not
good for you. Subject to statistical error limitations, there is a
one to one correspondence between the source image and the
RAW file. One can be converted to the other using the rules
inherent in the camera's software. The data in the RAW file can't
be restructured to make a different image without changing the
data."

Basically what I said was that what I just quoted you as saying is a
'nonsense idea'.

said that subject to statistical limitations there is only one sensor
image corresponding with the RAW data of the file. This raised the
question of the transformation of the sensor data to the data of the
RAW file. One aspect of the argument is whether there is any
transformation and I say there is. Then we got sidetracked onto
whether or not the transformation is all hardware, and I said that
software may be involved. That's why the question of the firmware
upgrade of the D300 entered the picture. The point of this is that the
RAW file data may not just be dump of the sensor data.

But the fact is that essentially it is just that, a dump of the
sensor data. Squirm all you like...

Unless you can explain how a particular electronic charge from the
sensor can be saved _in_its_original form_ on a Compact Flash card
then you have to accept that the data from the sensor is transformed
before it is saved.

All of this is irrelevant to the original question which was centered
around whether the transformation from sensor image to RAW file data
was fully reversible. In most cases it is in which case you can show

In no case is it reversible.

You can track back through the logic and determine the original state
of the sensor which gave rise to the particular Raw data file. Mind
you, You would have to know Nikon's original transformation algorithm
before you could work it backwards. I wouldn't like to have to do it.

that only the one sensor image corresponds to any one RAW file data

It is easy to show that there are multiple images on the
sensor that could produce exactly the same data set.

I invite you to do so.

set. I described this as being a 'one to one' relationship. This is

And that has been thoroughly covered in other articles,
because it is simply false.

where the current furore started. A moment's brain fade let me be
dragged into questions of interpolation but I was clearly wrong. I'm
going to ignore that issue.

If you ignore every part of this discussion where you've
been wrong, we have nothing left to discuss except
perhaps the spelling of your name.

Well that's a good wriggle but if you read back above you will find
there are many points you have to answer. I would particularly like a
clear explanation of how there can be multiple images on the sensor
that could produce exactly the same data set. After all, this is where
we came in.



Eric Stevens