Low-light high-ISO torture-test for Panasonic FZ5

http://pbase.com/waltdnes/concert20060820

Start off with a free concert on a cloudy evening... and then night
falls. Panasonic's high-ISO images are (in)famous. I took a series of
2048x1536 images at ISO 400, with shutter forced to 1/100th or 1/80th
sec. The images were brightened and then binned down to 682x512. I've
posted jpegs of both the originals and the worked-on versions.

One nice side-effect of ImageMagick's "-sigmoidal-contrast" option is
that it gets rid of the reddish pallor on under-exposed shots. Where
the under-exposure wasn't too gross, I didn't brighten to a specific
brightness. Instead, I brightened to the point where flesh tones look
close to natural.

The first page has hardly any difference. But as it gets dark, the
differences really show.

--
Walter Dnes; my email address is *ALMOST* like
Delete the "z" to get my real address. If that gets blocked, follow
the instructions at the end of the 550 message.

Walter Dnes (delete the 'z' to get my real address) wrote:
http://pbase.com/waltdnes/concert20060820
The images were brightened and then binned down to 682x512. I've
posted jpegs of both the originals and the worked-on versions.

Could you detail the imagemagick processing (command line?)
that you used?

BugBear

On Wed, 23 Aug 2006 12:40:29 +0100, bugbear, wrote:
Walter Dnes (delete the 'z' to get my real address) wrote:
http://pbase.com/waltdnes/concert20060820
The images were brightened and then binned down to 682x512. I've
posted jpegs of both the originals and the worked-on versions.

Could you detail the imagemagick processing (command line?)
that you used?


Sure... I used the "-sigmoidal-contrast" option with "convert" to do
the brightening. If you want to dig into the guts of things, see IM
Examples, Sigmoidal Non-linearity Contrast
http://www.cit.gu.edu.au/~anthony/gr...lor/#sigmoidal for
the mathematical formula, and graph plots to see how it works.

I took a few shortcuts...

- the output filename is the input file name, with an "a" prefixed to
it e.g. "p1010465.tif" becomes "ap1010465.tif"

- The second part of the parameter is always zero, which causes
maximum brightness boost for brightness values near zero. This is the
desired behaviour for brightening dark images. The first part of the
parameter is a float number usually around 3 or 4, except in the extreme
cases of p1010465.tif where I used 25, and p1010466.tif where I used 30.
The higher the number, the more brightening. Just like the case in film
photography, a lot of brightening causes graininess. Laws of physics
don't change. The commandline in both posix (linux, unix, *BSD, etc)
and Windows versions looks like so...

convert "p1010465.tif" -depth 8 -sigmoidal-contrast "25,0%" "ap1010465.tif"

Quotes around the filenames are optional, unless you have filenames
with embedded spaces etc, which makes quotes mandatory. Rather than
type this out by hand every time, I set up a short script (under linux),
which I call "logg". The script looks like so...

#!/bin/bash
convert "${1}" -depth 8 -sigmoidal-contrast "${2},0%" "a${1}"

I don't have a Windows machine handy to test this. I believe the
equivalant Windows script would be logg.bat or logg.cmd, like so...

convert "%1" -depth 8 -sigmoidal-contrast "%2,0%%" "a%1"

....and both posix and Windows versions are invoked like so...

logg p1010465.tif 20

For the binning from 2048x1536 down to 682x512 (factor of 3), the
manual process would be to first crop to 2046x1536 (whole number
multiples of 3) and then use convert's "-filter box -resize" option...

convert -depth 8 source_file.tif -crop 2046x1536 dummy.tif
convert -depth 8 dummy.tif -filter box -resize 682x512 output_file.tif

With the help of ImageMagick's "identify" command, I've managed to set
up a linux script, which I call binn. It will take a randomly sized
input file, calculate correct sizes, crop to a whole number multiple,
and resize by a desired factor, like so...

binn input_file output_file n

....where "n" is a small positive integer to divide the image size by.
This is probably impossible as a bat or cmd file. As always, be careful
when accepting script files from strangers, especially when the script
contains backtick expandos. Here is my ~/bin/binn script...

#!/bin/bash

calcgeometry() {
x=`echo ${3} | sed "s/x.*$//"`
y=`echo ${3} | sed "s/^.*x//"`
newx=$((${x} / ${bin_size}))
checkx=$((${newx} * ${bin_size}))
if [[ ${checkx} -ne ${x} ]]; then
crop_flag="YES"
fi
newy=$((${y} / ${bin_size}))
checky=$((${newy} * ${bin_size}))
if [[ ${checky} -ne ${y} ]]; then
crop_flag="YES"
fi
newgeometry="${newx}x${newy}"
}

source_file=${1}
output_file=${2}
bin_size=${3}
crop_flag="NO"
calcgeometry `identify ${1}`

if [[ ${crop_flag} = "YES" ]]; then
cropgeometry="${checkx}x${checky}+0+0"
convert -depth 8 ${1} -crop ${cropgeometry} dummy.tif
source_file="dummy.tif"
fi

convert -depth 8 ${source_file} -filter box -resize ${newgeometry} ${output_file}

if [[ -a dummy.tif ]]; then
rm dummy.tif
fi

--
Walter Dnes; my email address is *ALMOST* like
Delete the "z" to get my real address. If that gets blocked, follow
the instructions at the end of the 550 message.

Walter Dnes (delete the 'z' to get my real address) wrote:
For the binning from 2048x1536 down to 682x512 (factor of 3), the
manual process would be to first crop to 2046x1536 (whole number
multiples of 3) and then use convert's "-filter box -resize" option...

convert -depth 8 source_file.tif -crop 2046x1536 dummy.tif
convert -depth 8 dummy.tif -filter box -resize 682x512 output_file.tif

Thanks for all the detail; (I am an ImageMagick uyser myself).

Why do you want to "box filter" AKA nearest neighbour the image? I'd have thought
the "best" sampling possible would be desired.

This is clearly deliberate on your part, since
you're going to substantial effort to do it.

BugBear

On Thu, 24 Aug 2006 09:59:25 +0100, bugbear, wrote:

Why do you want to "box filter" AKA nearest neighbour the image? I'd
have thought the "best" sampling possible would be desired.

This is clearly deliberate on your part, since you're going to
substantial effort to do it.

It's called "binning". For technical details, see websites...
http://www.noao.edu/outreach/aop/glossary/binning.html
http://www.andor.com/library/digital_cameras/?app=320

The "executive summary" explanation of binning goes like so...

- take an NxN rectangle of pixels from the source image and average
them out to one "super-pixel" in the output image

- the signal is linear, and is proportional to the number of pixels
used; i.e. signal is multiplied by N^2

- random noise is (by definition) random, and is proportional to the
square root of the number of pixels used; i.e. noise is multiplied
by N

- therefore SNR (Signal to Noise Ratio) has gone up by N^2 / N = N

Binning a 2048x1536 pixel image down to 682x512 effectively cleans up
the signal as if ISO was 1/3rd of the actual value... *WITHOUT LOSING
THE ADDITIONAL BRIGHTNESS OF THE HIGHER ISO*!!! E.g. my ISO 400 shots,
after being binned in 3x3 grids, have a noise level that you would
expect from ISO 133 shots. TANSTAAFL (There Ain't No Such Thing As A
Free Lunch). You're trading off pixels for improved SNR at higher ISO
levels.

There have been a lot of rants on this forum about the "megapixel
madness" inflicted on consumers by digital camera marketing droids. A
lot of people here would gladly trade off some pixels for higher SNR.
Binning allows you to do so after the fact if you take oversized photos
and are willing to reduce them.

--
Walter Dnes; my email address is *ALMOST* like
Delete the "z" to get my real address. If that gets blocked, follow
the instructions at the end of the 550 message.

"Walter Dnes (delete the 'z' to get my real address)"
wrote in message
...
SNIP
Binning a 2048x1536 pixel image down to 682x512 effectively cleans
up the signal as if ISO was 1/3rd of the actual value... *WITHOUT
LOSING THE ADDITIONAL BRIGHTNESS OF THE HIGHER ISO*!!!

Yes, noise reduction is one benefit of binning/averaging (it could
also be used to add all low level signal in the box), but you also
introduce aliasing artifacts. It depends on the image content if it
produces visibly annoying artifacts, but the artifacts are e.g.
visible in the strings of the standing bass.

I doubt you would have gotten very different results with the default
Lanczos or the Sinc filter, except for cleaner aliasing behaviour. And
because those filters have a larger (weighted) support than Box, noise
reduction is also a characteristic of them.

Although it by-passes the filters, the '-adapted-resize' function may
also deliver the desired results.

--
Bart

"Walter Dnes (delete the 'z' to get my real address)"
wrote:

- therefore SNR (Signal to Noise Ratio) has gone up by N^2 / N = N

Binning a 2048x1536 pixel image down to 682x512 effectively cleans up
the signal as if ISO was 1/3rd of the actual value... *WITHOUT LOSING
THE ADDITIONAL BRIGHTNESS OF THE HIGHER ISO*!!! E.g. my ISO 400 shots,
after being binned in 3x3 grids, have a noise level that you would
expect from ISO 133 shots. TANSTAAFL (There Ain't No Such Thing As A
Free Lunch). You're trading off pixels for improved SNR at higher ISO
levels.

You _may_ be underestimating the ISO gain from binning.

I think doubling the ISO increases noise by a factor of 1.414, not 2.0.

Doubling the ISO for the same size pictures reduces the photons counted by a
factor of two, but increases the noise by only a factor of 1.414, the square
root thereof.

In other words, binning 4 pixels together gets you ISO 100 noise from an ISO
400 exposure.

I think. Maybe.

David J. Littleboy
Tokyo, Japan

On Fri, 25 Aug 2006 19:15:31 +0900, David J. Littleboy, wrote:

You _may_ be underestimating the ISO gain from binning.

I think doubling the ISO increases noise by a factor of 1.414, not 2.0.

Doubling the ISO for the same size pictures reduces the photons counted by a
factor of two, but increases the noise by only a factor of 1.414, the square
root thereof.

In other words, binning 4 pixels together gets you ISO 100 noise from an ISO
400 exposure.

I think. Maybe.

It's more complex than that in real life. If you can force the
histogram over to the right up to, *BUT NOT PAST* the right-hand edge,
you get the maximum number of tonal values. Beyond that, you get faster
shutter or more DOF, but no more tonal info. Explained better on
webpage http://www.luminous-landscape.com/tu...se-right.shtml
Most people here are probably fimiliar with the concept, but it's
included for the benefit of any newbies reading this post.

Life's a series of trade-offs. You want...
- the fastest shutter speed (least amount of motion blur)
- the smallest aperture (most depth-of-field)
- the largest sensor and the lowest ISO (least amount of noise)
- the histogram almost to the right edge (for the most tonal info)
- a camera+lens at a price that doesn't destroy the average budget

Select any 4 g.

--
Walter Dnes; my email address is *ALMOST* like
Delete the "z" to get my real address. If that gets blocked, follow
the instructions at the end of the 550 message.

"Walter Dnes (delete the 'z' to get my real address)"
wrote:
On Fri, 25 Aug 2006 19:15:31 +0900, David J. Littleboy,
wrote:

You _may_ be underestimating the ISO gain from binning.

I think doubling the ISO increases noise by a factor of 1.414, not 2.0.

Doubling the ISO for the same size pictures reduces the photons counted
by a
factor of two, but increases the noise by only a factor of 1.414, the
square
root thereof.

In other words, binning 4 pixels together gets you ISO 100 noise from an
ISO
400 exposure.

I think. Maybe.

It's more complex than that in real life. If you can force the
histogram over to the right up to, *BUT NOT PAST* the right-hand edge,
you get the maximum number of tonal values.

Oops: you're responding to something other than what you quotedg.

David J. Littleboy
Tokyo, Japan

On Sun, 27 Aug 2006 15:59:01 +0900, David J. Littleboy, wrote:

Oops: you're responding to something other than what you quotedg.

You're right; I did drift off on a tangent. My point was that under
certain conditions, being able to shoot at higher ISO speed (and clean
up noise after-the-fact) has major benefits. At other times it's not
worth it.

--
Walter Dnes; my email address is *ALMOST* like
Delete the "z" to get my real address. If that gets blocked, follow
the instructions at the end of the 550 message.