Organizing working images, archiving all images, what approach to take?

I'm curious to know how others are dealing with long term storage of
digital images? I'll lay out some of the factors as I see them.

I'm using a 5D, shooting raw most of the time.
Primary software is Photoshop CS2 and Canon's Digital Photo Professional
I tend to use 4gb memory

There are a lot of large files to grapple with.
I want to keep all images 'forever'.
I need to find archived images.
I would like to view thumbnails of archived images.
There are many images that are very similiar.
Some of the images are chosen to process further.
The images that get attention end up as jpg or tiff.

My current system is approx like so:
Return from a shooting session with 2 4gb chips of images.
Copy each chip to separate folder on workstation's hdd.
Folder names reflect date and subject/location (2008-01-18 Death Valley)
Create dvd of each folder, label by date and subject/location.
Select some raw images to work on, move them to working dir.
I usually retain the numeric tag that the camera assigns to an image
through the processing stage of an image, until it gets published. This
helps me keep track of which raw image was the source.
Eventually when workstation hdd are crammed, have to delete older
folders containing raw images.

It's an ok system. Issues I see a
DVD media has limited lifespan so eventually all will turn to dust.
DVD software/hardware still seem very flakey to me; I'm never sure a
copy won't be corrupted or somehow unreadable.
Even if I make 2 copies of images, if one fails other might fail for the
same reason.
Once on dvd it's hard to view thumbnails, and thumbnails are important
when sifting older images for variations on a theme.
It'd be ideal if thumbnails were available on my workstation for all
archived images.
Tried using external hdds for storage but still questions re durability
etc, aside from cost.

I'm a programmer so I may be able to write something that could help
with whatever process I end up using over time, if there isn't already
such software. But I'm very interested to hear what others are doing
with image management.