Foveon: The Clever Image Sensor That Has Failed to Catch On
Back in the early 2000s, Sigma — a company usually known in the photography community as a lens manufacturer — released a new type of camera sensor named the Foveon X3. This sensor technology was first patented in 1999 by a company called Foveon, Inc. (later acquired in 2008 by Sigma) and featured an innovative color perception technology: a 3-layer stack of photosensitive diodes.
Back to Basics: How Your Camera Captures Colors
Silicon is widely known as a semiconductor material and is used as such in many electronic circuits. For photographers, however, the main interest in silicon is that it is a photosensitive material.
To be more specific, silicon absorbs light for any wavelength comprised between 400nm and 1100nm. The good news is that this range covers the visible light spectrum (400nm to 750nm, roughly) and even a bit of the near-infrared spectrum. The bad news is that silicon by itself does not discriminate between the color of the light and only accumulates any photon in the 400-1100nm range.
A first step in improving the image quality is to add a filter to remove the infrared light (between 750nm and 1100nm) before it reaches the sensor. The resulting system constitutes a monochrome camera sensor (providing black and white images).
From there on, the most common way to gather some color information is to use a color filter array (CFA). Usually, this filter is organized following a Bayer pattern (from the name of a Kodak engineer) with red, green, green, and blue filters.
Thanks to these three filters, one can recover the color information in the image. However, the process comes at the expense of some image resolution. For a given “red” pixel, one has to make an interpolation to find the missing green and blue information. Many interpolation algorithms exist, but the process always results in some form of color artifact and image softening.
Color filter arrays, in addition to being inaccurate, also absorb many photons in the process of collecting some color information. The absorption varies depending on the filter choice and image content, but it is generally admitted that the light flux is divided by a factor of 3 before reaching the sensor. That’s 1.5 stops of light.
The Clever Idea Behind Foveon Technology
It happens that, depending on its color, a given photon travels a specific distance in silicon before being absorbed. This “travel distance” is related to the energy of the photon, which depends on its wavelength. Short wavelength photons (blue light, UV light, etc.) are more energetic than longer wavelength ones (red light, infrared light, etc.) and thus are absorbed after a shorter journey in silicon.
Since absorption depth in silicon provides an estimate of the photon wavelength, the measure of the absorption depth provides theoretically the color information. The clever idea of Foveon sensors is to use “depth filters” instead of side-by-side color filters.
On paper, at least, removing any color filter should increase the light flux falling on the sensor and remove the drawbacks associated with color filters.
Back when Foveon sensors first reached the market in 2002 in the Sigma SD9 DSLR, most digital cameras were far from 10MPx resolution. The Canon EOS-1D had 4.2MPx and the Nikon D100 had 6MPx. Since color filter arrays require digital interpolation, for a given number of pixels, the color filter array offered a massive sharpness upgrade.
Sigma actually tried to market this difference in image quality by claiming that the SD9 camera had a 3.43MP output image but “10.3 million effective pixels,” meaning that their pixels were performing 3 times better. Several academic papers have indeed found that Foveon sensors offered a higher perceived resolution thanks to the lack of interpolation and of color artifact around sharp edges. From this perspective, Foveon sensors provide a clear and measurable benefit to the user.
One last point claimed by Sigma is that Foveon sensors outperform color filter sensors when it comes to color accuracy. The claim is explained by the lack of color information interpolation. Indeed, for a given Foveon pixel, the red, green, and blue data are recorded in the same place, contrary to a color filter array.
On paper at least, the Foveon sensor has it all: better light flux means better low-light performance, better color accuracy, and better sharpness of the image. But paper specs never quite tell the full story…
What Went Wrong With the Brilliant Foveon Idea?
The real question from there on really boils down to this: since the idea behind Foveon sensors is quite clever, why aren’t we all using Foveon sensors?
Resolution
Some of the interest in the Foveon technology vanished with the evolution of CMOS sensors as the dramatic increase in pixel density has reduced one of the main selling points of Foveon sensors. Sure, it’s not really an apples-to-apples comparison since pixel counts only contribute to a fraction of the final image quality (along with the lens, the amount of light available, etc). Since even smartphones can deliver at least 20MPx images, Foveon is less attractive than in the early 2000s.
Color
The color advantage of the Foveon sensor is theoretically undeniable but offers only little practical benefit. Outside of color-calibrated monitors and laboratory environments, the difference in color accuracy is small enough to go unnoticed by most casual photographers. It certainly is more subtle than the effect of white balance mistakes or creative Instagram filters.
Noise
Here is potentially the most debatable point of all. On paper, Foveon sensors should collect about 3 times more light, which should translate in a 1.7 times improvement in signal to noise ratio. That is, assuming the shot noise is dominant. However, it turns out that in practical comparisons, the low-light performance of Foveon sensors falls short. Even Foveon General Manager Shri Ramaswami admitted it in a 2014 interview:
“One of the limitations of Foveon’s approach is that image noise is higher than in conventional sensors,” he said. “This is probably due in part to inefficiencies within the sensor architecture itself — perhaps some light is lost to internal structures that separate the layers — and also in part to the processing that has to be done to produce pure colors from the rather mixed signals that the chip actually captures.”
CMOS sensors and Foveon sensors share a similar design issue: some of the light leaks from one pixel to the next. In CMOS sensors, this crosstalk problem is solved using vertical isolation structures between the photodiodes.
In Foveon sensors, however, it is not possible to apply the same trick (otherwise, you would only get the blue color information). Thus, Foveon sensors tend to suffer from photon leakage from one photodiode layer to the next.
Image Processing: The Hidden Problem Behind Unusual Sensors
A drawback that is common to all “unusual” color systems (be it Fuji X-Trans, RGBW filters, or Foveon) is that the image processing is also affected by the choice. Even though Bayer filters might not be the best way to capture colors, they are around for several decades and are absolutely dominant in the sensor market. As a consequence, the incentive is strong for image processing researchers and companies to fine-tune all algorithms for this type of sensor.
De-noising, for instance, is usually done with some assumptions about the statistics regarding the collected photons that cannot be directly applied to Foveon sensors. Sadly, given the share of Foveon sensors used, most image processing companies don’t have any incentive to support Foveon files, not to mention developing image processing pipelines dedicated to such sensors.
The trend is only getting worse with machine-learning-based algorithms. To put it bluntly, machine-learning algorithms are only as good as the data used to train them (both in terms of quantity and quality). Since there are thousand times more images available from Bayer filter arrays than from Foveon, the former sensors will benefit from better algorithms than the latter.
Other Marketing Considerations
Lastly, we need to address the elephant in the room. Photography equipment doesn’t always sell for technical reasons. Pricing, marketing, lens ecosystems, branding — there are many factors in the success of a technology.
Sigma has tried to provide a sensor different from anything else, but the company had to fight two battles at the same time. The first one was to prove the image quality of Foveon sensors. The second one was to prove that, as a camera manufacturer, their cameras outperformed Canon or Nikon ones.
Foveon could have been Sigma’s secret weapon in the competitive camera industry. But, the performance difference between traditional and Foveon sensors is not clear cut. Not clear enough, at least, to attract new photographers to the Sigma brand.
Will the Foveon Sensor Even Return?
Sigma has started developing a full-frame Foveon sensor, and the project is in an early prototype stage according to Sigma’s latest statement. It seems likely that the project could remain stuck in the prototype phase and never reach the market. At best, from this calendar, it seems unrealistic to expect a mass-produced sensor before 2024.
As a company, Sigma has a good and well-deserved reputation for lens manufacturing and would like to extend its business further into cameras and sensors. However, a camera is far more than just a sensor. The real challenge for Sigma might very well be the lack of high-quality Image Processing. Sigma even had to create a plugin for Adobe Photoshop in order to get Foveon files opened.
While it isn’t impossible that Sigma releases a new camera with a full-frame Foveon sensor, there is still a long way to go in selling the technology outside of the existing Foveon community, let alone making it an industry standard.
About the author: Timothee Cognard is an optical expert and photographer based in Paris, France.
