"Raw" file issues?

wrote:

Reichman, et al, say they don't want "trade secrets" revealed, but they
are woefully clueless in that their demands for documentation of these
files is exactly that.

That's nonsense. No one is asking Nikon for the process by which white
balance is arrived at, only the conclusion. If that could lead to
reverse engineering their process, then the fact that you can get it
using Nikon's software would lead to the exact same eventuality.

--
Jeremy |

wrote:
[snip]
I can't comment on Nikon stuff, since I'm a Canon sort of person.

Canon 10D: a bizarre CIFF, for which they published the specifications
for the "superstructure" of this file. It's a kind of weird TIFF; you
can, in fact, TIFFize a CIFF. There are two images in this file: the
JPEG and an strangely encoded raw sensor dump. That Dave Coffin
managed to figure it out speaks for his abilities as a Reverse
Engineer.
[snip]

By "weird TIFF", do you mean "weird TIFF 6.0" or "weird TIFF/EP"? Is
it, in fact, "standard TIFF/EP"? You may know the following, but I
suspect many people don't:

1. *TIFF 6.0* is owned by Adobe, and was owned by Aldus before that.

When you set your camera to TIFF, that really means TIFF 6.0. Raw
processors normally have TIFF 6.0 as an *output* option. Photo-editors
normally read and/or write TIFF 6.0 as options. (You can actually use
TIFF 6.0 instead of PSD for Photoshop - make sure you ask for layers to
be preserved). If a magazine buys one of your photographs in TIFF form,
they mean TIFF 6.0.

TIFF 6.0 images have the full colour information for each pixel,
(except for monochrome images, of course). TIFF 6.0 is not suitable for
Raw files, because it has no tags defined for holding sensor data, and
there are many other metadata tags that it would need also.

2. *TIFF/EP* ("Electronic Photography") is owned by ISO, and costs 150
euros. It was developed from TIFF 6.0, and much of it is copied
directly from TIFF 6.0.

It has those extra tags for sensor data, (look for tags starting CFA -
Color Filter Array). It also has a lot of extra metadata tags. Camera
manufacturers typically (but not necessarily) start with TIFF/EP when
designing their Raw formats, because it has many of the tags they need.
Indeed, one manufacturer originally used the .TIF extension for its Raw
files.

Like TIFF 6.0, TIFF/EP has very many options. And, of course, although
it is a standard, there is no constraint on manufacturers to conform
precisely to it. So resultant Raw formats proliferate, and there are
lots of unnecessary differences.

3. *DNG*, like TIFF/EP, is a development of TIFF 6.0, but I prefer to
think of it as a development of TIFF/EP. It has a few tags that TIFF/EP
doesn't have, to cater for technology innovations since TIFF/EP was
defined, and for control purposes, such as the 2 version tags that
define the range of DNG versions that a particular file is compatible
with.

DNG cuts through all the unnecessary options in TIFF/EP, eliminates
some of the more complicated structural options, and makes some tags
mandatory. So it ensures that the image data can always be extracted,
along with sufficient extra information that Raw processors can handle
the image data without needing to know about the camera. (The file is
self-contained). But it also defines how camera manufacturers can
safely store any "secret sauce" in a DNGPrivateData tag defined for the
purpose. "Safely" means that anyone else can re-write a DNG file while
preserving the private data, because although its contents are private,
its format is well-structured.

4. To summarise: TIFF 6.0 is what people tend to mean when they say
"TIFF" without qualification. TIFF/EP is TIFF 6.0 plus a lot of the
stuff needed to make Raw files. DNG is TIFF/EP brought up to date and
made fit for purpose. (Now I'll duck for cover!)

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/

DoN. Nichols wrote:
[snip]
But -- I don't see how the situation with RAW files is any worse
than these limitations from the film days. And in some ways, it is a
*lot* better. If you process a RAW file with the wrong software, the
worst that will happen is that you won't get anything useful. The
original RAW file can be saved until you get the *right* software.

I think the analogy with film can be taken too far. For example, DNG
could be more usefully compared with the developed negative, rather
than the undeveloped negative. Then we would be talking about
standardisation of film sizes to enable the negative to be processed
"anywhere" and by lots of enlargers and scanning and printing
equipment.

For asset management systems that handle many Raw formats, or decades
in the future when we want to process old Raw files, it may be hard to
"get the *right* software".

Sometimes we should just ask "what do we, as photographers or users of
photographs, want?" We surely wouldn't want lots of variants of small
negatives, nearly but not quite 35mm / 2 x 8 sprockets / 24mm x 36mm
image size. That is what we have with Raw formats at the moment.

(I'll pass over the fact that the DNG logo has 2 x 7 sprockets, not 2 x
8!)

Note that I prefer to do my image processing on unix computers,
which are *not* supported by either Kodak or Cannon (or any of the other
digital SLR makers, AFIK). For this reason, I am quite pleased to have
Dave Coffin's "dcraw" software available.
[snip]

You've just made the case for OpenRAW , and even DNG. Dave Coffin's
work is valuable, but it should never have been necessary. You should
be able to obtain a variety of Unix-based converters, etc, which were
based on proper specifications.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/

wrote:
In message ,
"Ryadia@Home" wrote:


RichA wrote:


No, you don't.

The only way a TIFF can carry all of the original RAW data is if the
TIFF has the RAW values, non-demosaiced, non-white-balanced, and the
black pixels as well, at their original values, or multiplied by an
integer.


Bloody sensationalist bull **** is all that comes out of that site.

Douglas


The interesting part of this thread is lack the of photography and
concentration on technology. This is the first time in the history of
photography that a photographer has access to anything other than the
image they record. Good, bad or indifferent, that image is what so many
photographers spent so long learning how to capture... In the camera.

So now they take the picture which formally got processed in the dark,
with chemicals and conditions someone else produced and if the colour
was a bit off or the shadows a bit dense, there was always the others in
the bracket to choose from.

Remove the chemical, temperature and processing time from this
description and add using proprietary software, the image was
'developed' and what precisely is so different about digital photography
that drives this demand for total disclosure of the process?

When you didn't like the images from a patented process of developing
Kodachrome, you simply changed to a different film/chemical combination.
Same here. Nothing at all to make you stay with Nikon, Canon, Olympus or
whatever. A lot of noise going on here about nothing.

--
Douglas...
It's traditional, painter's use it, Rembrandt used it.
Now you can put your photos on it too!
http://www.canvasphotos.com.au

On Fri, 27 May 2005 20:00:11 +1000, "Ryadia@Home"
wrote:

snippage

When you didn't like the images from a patented process of developing
Kodachrome, you simply changed to a different film/chemical combination.
Same here. Nothing at all to make you stay with Nikon, Canon, Olympus or
whatever. A lot of noise going on here about nothing.

I guess the point some are trying to make is that with film, if you didn't
like how one flim/developing combination worked, you could pick a different
one but still use the same equipment (camera/lens/falsh etc.) The
difference here, is if you don't like what one camera company are doing,
you've (generally) got to replace the whole kit. It's a far more drastic
change than switching film brands.


Regards,
Graham Holden (g-holden AT dircon DOT co DOT uk)
--
There are 10 types of people in the world;
those that understand binary and those that don't.

John Francis wrote:
[snip]
So - I've registered a SourceForge project for an open source
RAW-to-DNG converter. I'd be interested in hearing from C++
programmers who could contribute to the effort.
[snip]

I see it. I'll keep an eye on progress.

I don't think I have the up-to-date skills, or the platform, to help
directly. But I would certainly like to trial a Windows version, if
there ever is one, at an early stage.

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/

Ryadia@Home wrote:
[snip]
Remove the chemical, temperature and processing time from this
description and add using proprietary software, the image was
'developed' and what precisely is so different about digital photography
that drives this demand for total disclosure of the process?
[snip]

A problem with this thread is that, because the film and digital
workflows don't have a one-to-one correspondence, people can make
different analogies and come to different conclusions. For example:

1. If we think of a common Raw format as similar to an *undeveloped*
negative, then we might discuss the degree to which development
specifications are open so that many can use them. For example, C-22,
C-41, E-6, K-14. Then we can talk about the advantages of being able to
choose one's service supplier, and perhaps even one's chemical
supplier.

I think it is better to discuss how open the *interfaces* are, rather
than how open the *processes* are. It makes sense for me to ask for a
film to be developed using C-41, but that doesn't mean I need know
about C-41, and indeed the service I use may not have all the details.

Surely we have benefitted from some common specifications in this area?

2. If we think of a common Raw format as similar to a *developed*
negative, then we are more likely to talk about the advantages of
standard film sizes. These make it easier for us to use the same
enlarger, or the same scanner, or slide projector, for many different
makes of film from many different cameras. And we can use many
different services to get others to do this if we don't want to do it
ourselves.

We have certainly benefitted from common specifications in this area.

3. Once we are in the digital realm, we benefit from common file
formats such as TIFF 6.0 and JPEG. A very large set of interchange
benefits have grown up around these standards. I think a common Raw
format is analogous to TIFF 6.0 and JPEG, *not* because they fit into
the workflow at the same place, (they don't), but because they
illustrate what happens when we get recognised common image formats at
various points. Things open up to the benefit of all of us.

I was interested to note that Photobase sell a "Convert to DNG"
service. If they make money from this, it suggests there can be a very
real cost to photographers of the proliferation of Raw formats. I don't
criticise Photobase for offering this service, but it is a pity that it
could ever be necessary.

If we step back from details of the digital workflow, and look at
complex multi-vendor systems of all kinds, we can see many cases where
we get benefits from designing a clean "architecture", with good
specifications for the interfaces between the components. In fact, much
of modern life is build on these principles. Railways: wheel guage;
loading guage; coupling specification; platform height. Computers:
common interfaces all over the place. Bicycles. Guns & bullets. (Hm!)
Credit card specifications.

Why is Raw thought to be so different from just about everything else
in the digital image value chain? Consider:

When I've I've taken some pictures, I put the memory card into a card
reader. The configuration of the pins and the size of the card is
specified by a standard. So is the exact nature of the voltage changes
of the pins as the data is read from the card. The card reader is
plugged into a USB port. USB is a standard specification, of course. My
laptop doesn't have a USB 2 port, so I use a PC Card with USB 2 ports.
The PC Card interface, including pins and voltage changes, is another
standard, of course.

If I shoot TIFF 6.0 or JPEG, they are determined by available
specifications, one a de-facto standard, the other an industry
standard. A magazine might ask for a TIFF 6.0 image. Or if I publish to
the web, a JPEG image will be identified by an HTML page, with the
presentation determined by a style sheet. HTML 4.01 Strict and CSS2 are
"recommendations" (equivalent to standards) of W3C. I upload them to
the website using the standard FTP protocol, and viewers download them
using the standard HTTP protocol.

This systematic coupling of innovative hardware, firmware, and software
components via layers of standards and agreed specifications under
version control is the normal way our complex multi-vendor world works.
But ... I don't shoot TIFF or JPEG. I shoot Raw. Are we supposed to
believe that these principles can't work for Raw formats? That if a
magazine wants my Raw image, I shouldn't give them a standard format,
but I should give them the proprietary specification so that they can
obtain software to extract the image? Of course not! It is obvious that
the same principles can and should apply to Raw formats too.

Raw is not different *in principle* from all the other components. It
simply hasn't yet reached the same level of maturity. Those of us who
realise its immaturity need to be encouraging it to grow up. We musn't
let it believe that it can remain adolescent for ever. We certainly
shouldn't be making excuses for it!

Photographers, and users of photographs, need future Raw formats to
conform to an agreed specification, which will become at least a
de-facto standard. Then we need a way for older Raw files to be
convertable satisfactorily to the agreed format, so that they can be
handled in the same way without risk of being neglected. The primary
reason for obtaining the specifications for those older Raw formats
should be to enable high-quality converters to be developed. NOT so
that all packages that ever handle Raw files recognise all those Raw
formats for ever. Expecting (say) a new Raw processing package launched
in 5 years time to support 100s of Raw formats is just plain silly! Why
should it have to recognise more than 1?

--
Barry Pearson
http://www.barry.pearson.name/photography/
http://www.birdsandanimals.info/

On Thu, 26 May 2005 22:08:27 +0000 (UTC), Ben Rosengart
wrote:

I have had to help computer users who changed computers and suddenly
couldn't open their old files any more. It isn't pretty. Those
who pooh-pooh these kinds of problems have probably never had to
deal with them. Or maybe the whole concept of planning for the
long term is alien to some people.

Out of interest, having lived through all the formats from 7" floppies
on down, could you be more specific as to what problems these people
encountered?

My copy of Word that come in a 2004 version of Office XP still opens
an ASCII text file format, a standard that dates back to 1963.

Today, the latest version of Photoshop still opens the HPGL/GCG files,
the oldest graphics format I can think of. Dating back to the early
80's or before. Postscript is nearly as old, and again, I can still
open those.

These are between 25 and 40 years old, and we can still open them. I
don't think people need to worry *quite* so much.

Finding a 35mm slide projector for under $150 is however becoming a
problem. And that format is still being created today.

--
Owamanga!
http://www.pbase.com/owamanga

On Fri, 27 May 2005 01:20:05 +0000 (UTC), (John
Francis) wrote:

In article ,
Alan Browne wrote:
RichA wrote:

http://www.luminous-landscape.com/essays/raw-flaw.shtml

Let's all do our part!!

Well, I've started on my part.

Currently the Adobe DNG converter is pretty much the only choice
if you want to convert your RAW files to DNG. That's not a lot
of help for folks on a Unix/Linux platform, or for the putative
geek 25 years down the road trying to compile on his new platform.

So - I've registered a SourceForge project for an open source
RAW-to-DNG converter. I'd be interested in hearing from C++
programmers who could contribute to the effort.

C++ ? Are you unix peeps still using that?

g

--
Owamanga!
http://www.pbase.com/owamanga

Barry Pearson wrote:

(I'll pass over the fact that the DNG logo has 2 x 7 sprockets, not 2 x
8!)

Cropped Digital Negative.

;-)

--
-- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm
-- r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm
-- [SI] gallery & rulz: http://www.pbase.com/shootin
-- e-meil: Remove FreeLunch.