Biologically Inspired Computer Vision (2015)

Part I
Fundamentals

Chapter 3
Retinal Processing: From Biology to Models and Applications

David Alleysson and Nathalie Guyader

3.1 Introduction

To perceive the surrounding world, its beauty and subtlety, we constantly move our eyes, and, our visual system does an incredible job, processing the visual signal on different levels. Even though vision is the most frequently studied sense, we still do not fully understand how the neuronal circuitry of our visual system builds a coherent percept of the world. However, several models have been proposed to mimic some parts and some of the ways in which our visual system works.

Biologically inspired computer vision is not solely based on scientific results in the field of biology alone; various behavioral responses obtained through psychophysical experiments provide much needed data to propose models and to test their efficiency. Contrast sensitivity functions (CSF), which represent perceived luminance contrast as a function of spatial or temporal frequencies, were measured before the discovery of the neural network underlying them (see Section 3.3). Similarly, trichromacy, the fact that color vision lies in a tridimensional space, was described before the discovery of the three types of cones in the retina. In most cases, psychophysics precedes biology (See Chapter 5). To a certain extent, psychophysics remains biologically inspired, because everybody agrees that human visual behavior is “rooted” and emerges with the neural circuitry of the visual system. Often, it is simpler to find a model of human behavior instead of the biological structure and function of the neurons that support it.

However, the relationship between behavior and the underlying biology might be difficult to find, as is illustrated by the following three examples. The first one concerns the debate on random cone wiring to post-receptoral cells in color vision as discussed in Refs [1] and [2]. The second is about the fact that our visual world appears stable, whereas our eyes move continuously; and more especially, the neurons and mechanism involved in this processing are not known. The third concerns the neurophysiological explanation of the measured contrast sensitivity function. We still wonder whether the processing responsible for this perception occurs in the retina, in the transmission between retina and cortex, or in cortical areas.

Despite these examples, knowing exactly at which level a particular phenomenon is coded in our visual system is not essential to propose models of vision. The aim of models is to reproduce the functionalities of human vision to develop signal-processing algorithms that mimic the visual system functions. The limit to the imitation of nature by artificial elements is not a barrier to this if we are able to reproduce behavior. However, we must remain careful that a model, even one that works perfectly for behavior, is not necessarily an exact description of what happens in the visual system.

With the main objective being the description of biologically inspired computer vision, this chapter mixes results that come from psychophysics with results stemming from biology. We focus on the processes carried out at the retinal stage and we introduce some pathways between retinal biology and physiology, which are useful for computer vision and which result from psychophysical experiments and computational modeling. Because such pathways must encompass several different disciplines, we are not able to provide an exhaustive description and we choose to present some aspects. However, this introduction describes all the factors that should be taken together to understand the issue of computer vision.

The first part of the chapter is a short description of the anatomy and physiology of the retina. Useful links are given to the reader who wishes to obtain further information. The second part summarizes different types of retinal models. We present a panorama of these models and discuss their performance and usefulness both to explain vision and to explain their use in computer vision. The last part is dedicated to some examples of the utilization of models of vision in the context of digital color camera processing. This last part allows us to illustrate our purpose through real applications.

3.2 Anatomy and Physiology of the Retina

3.2.1 Overview of the Retina

Because vision is the most widely studied sense, a vast body of research in biology, physiology, medicine, neuroanatomy, psychophysics as well as computational modeling provide data to better understand this sense. This chapter goes through the basics of retinal functioning rather than providing a detailed description. The aim is to help a “vision researcher” to understand roughly how the retina works and how to model it for applications in computer vision. If one wants to know more about retina anatomy and physiology, he/she might want to read two free online books: (Webvision, The Organization of the Retina and Visual System by Helga Kolb and colleagues (http://webvision.med.utah.edu/) and Eye, brain and vision by David Hubel (http://hubel.med.harvard.edu/book/bcontex.htm).

The first detailed description of retinal anatomy was provided by Ramon y Cajal about a century ago [3]. Since then, with technological improvements, detailed knowledge about the organization of the retina and the visual system has been developed. A schematic representation of the retina is shown in Figure 3.1. This diagram is a simplified representation because the retina is much more complex and contains more types of cells. Basically, the retinal layers are organized into columns of cells, from the “input” cells, the photoreceptors, rods, and cones, to the “output” cells, the retinal ganglion cells, that have their synapses directly connected to the thalamus (lateral geniculate nucleus (LGN)), and from there to other areas, mainly the primary visual cortex, also called V1. In the retina, layers of cells are inversed compared to the transmission of light, with the photoreceptors located at the back of the retina. Two main layers of cells are generally described in the literature: (i) the outer plexiform layer (OPL), the connection between photoreceptors, horizontal cells, and bipolar cells and (ii) the inner plexiform layer (IPL), the connection between the bipolar, amacrine, and ganglion cells.

Figure 3.1 Schematic representation of the organization into columns of retinal cells. Light is transmitted through the retina from the ganglion cells to the photoreceptors. The neural signal is transmitted in the opposite direction from photoreceptors, their axons located at the outer plexiform layer (OPL), to ganglion cells, their dendritic field located in the inner plexiform layer (IPL), passing through bipolar cells (adapted from Ref. [4]).

The photoreceptors convert the light, that is, the incoming photons, into an electrical signal through membrane potential; this corresponds to transduction, a phenomenon that exists for all the sensory cells. The electrical signal is transmitted to bipolar and ganglion cells. The electrical signal is integrated within ganglion cells and transformed into nerve spikes. Nerve spikes are a time-coded digital form of an electrical signal. This is used to transmit nervous system information over long distances, in this case through the optic nerve and into the visual cortical areas of the brain.

3.2.2 Photoreceptors

The retina contains an area without any photoreceptors where ganglion axons connect the retina through the LGN forming the optic nerve, to the primary visual cortex (back of the brain, occipital lobe). The center of the retina, called the fovea, is a particular area of the retina with the highest density of photoreceptors; it is located close to the center of the retina (the eye's optical axis) and corresponds to central vision. The area of the visual field that is gazed at is directly projected onto this central area. There are two main types of photoreceptors, which are named according to their physical shapes: rods and cones. The fovea only contains cones (Figure 3.2), the photoreceptors which are sensitive to color information. Three types of cones might be distinguished for their sensitivity to three different wavelengths: S-cones, sensitive to short wavelengths that respond to bluish color, M-cones, sensitive to medium wavelengths that correspond to greenish-yellowish color, and L-cones, sensitive to long wavelengths that correspond to reddish color. These photoreceptors are activated during day-vision (high light intensity). The number of cones drastically decreases with the eccentricity onto the retinal surface. On the periphery, the number of cones is very small compared to the number of rods. These latter photoreceptors are responsible for scotopic vision (vision under dim light). Not only are photoreceptors sensitive to light but a freshly discovered ganglion cell with light sensitivity through melanopsin expression is also responsible for circadian rhythm accommodation and pupil aperture [5].

Figure 3.2 Number of photoreceptors (rods and cones) at the retina surface (in square millimeters) as a function of eccentricity (distance from the fovea center, located at 0). The blind spot corresponds to the optic nerve (an area without photoreceptor). Redrawn from Ref. [6]

The arrangement of photoreceptors on the retina surface has huge consequences on our visual perception. Our visual acuity is maximal for the region of the visual field gazed at (central vision), whereas it decreases rapidly for the surrounding areas (peripheral vision). It is important to note that the decrease in our visual acuity in peripheral vision is only partly due to retina photoreceptor distribution. In fact, other physiological and anatomical explanations might be given (see Section 3.2.3). Moreover, although we do not feel it in our everyday life, only our central vision is colored and peripheral vision is achromatic. Another particularity of cone arrangement is that the placing of L, M, and S cones is random in the retina (see Figure 3.5(b)) and differs from one individual to another [7].

Photoreceptors dynamically adapt their responses to various levels of illumination [8]. This property allows human vision to function and to be efficient for several decades of light intensity. In fact, we perceive the world and are able to recognize a scene on a sunny day as well as in twilight.

3.2.3 Outer and Inner Plexiform Layers (OPL and IPL)

At this stage, it is important to introduce the notion of receptive field. The receptive field of a cell is necessary to explain the functioning of retinal cells, and more generally, visual cells. This notion was introduced by Hartline for retinal ganglion cells [9]. According to Hartline's definition, the receptive field of a retinal ganglion cell corresponds to a restricted region of the visual field where light elicits response of the cell. This notion was extended to other retinal cells as well as other cortical or sensory cells. The receptive fields of visual cells correspond to a small area of the visual field. For example, the receptive field of a photoreceptor is a cone-shaped volume with the different directions in which light elicits a response from the photoreceptor [10]. However, most of the time, the receptive field of a visual cell is schematized in two dimensions corresponding to a visual stimulus located at a fixed distance from the participant.

The OPL corresponds to the connection between photoreceptors, bipolar cells, and horizontal cells. This connection is responsible for the center/surround receptive fields displayed by bipolar cells. Two types of bipolar cells can be distinguished: on-center and off-center. The receptive fields of retinal bipolar cells correspond to two concentric circles with different luminance contrast polarities. On-center cells respond when a light spot falls inside the central circle surrounded by a dark background and off-center cells respond to a dark stimulus surrounded by light [11]. The interaction between center and surround of the receptive fields causes contour enhancement, which can be easily implemented in image processing with a spatial high-pass filter.

The IPL corresponds to the connection between bipolar, ganglion, and amacrine cells. Ganglion cells still have center/surround receptive fields. Retinal ganglion cells vary significantly in terms of their size, connection, and response. As the number of photoreceptors decreases with retinal eccentricity, the size of the receptive fields of bipolar and ganglion cells increases. At the retinal center, there is the so-called one-to-one connection where one ganglion cell connects to one bipolar cell and to one photoreceptor ensuring maximal resolution around the optical axis [12]. However, at larger eccentricities, one bipolar cell connects up to several hundreds of photoreceptors.

There are essentially three types of anatomically and functionally distinct ganglion cells that project differentially to the LGN [13]. The most numerous type (around 80%) in the fovea and near fovea are the midget ganglion cells, so called because of their small size. They are the cells that connect one-to-one with photoreceptors in the fovea center. Their responses are sustained and slow, they are sensitive to high contrast, high spatial frequency as well as L-M color opponency. They project their axons to the parvocellular part of the LGN. In contrast, parasol ganglion cells are large and connect several bipolars and several photoreceptors even in the fovea. Their responses are transient and fast. They are known to be sensitive to low contrast and carry low spatial frequency, and are color blind. They also participate in night vision by integrating rod responses. They project their axons in the magnocellular pathway of the LGN. A third kind, called bistratified ganglion cells have dendrites which form two layers in the IPL. They are known to carry S-(L+M) color opponency and project in the koniocellular pathway of the LGN. Note that a new kind of ganglion cell containing melanopsin pigment that makes them directly sensitive to light has been recently discovered [5].

3.2.4 Summary

Most retinal ganglion cell axons are directly connected to the superior colliculi and the cortical visual areas passing through the lateral geniculate nuclei. The role of the retina is to preprocess the visual signal for the brain. Then the signal is successively processed by several cortical areas to bring a coherent percept of the world.

The main functions of the retina include the following:

· converting photons into neural signals that can be transmitted through neurons of our visual system;

· adapting its response to the mean luminance of the scene but also to the local luminance contrast;

· decomposing the input signal: low spatial frequencies are transmitted first to the cortex, immediately followed by high spatial frequencies and opponent color information.

In the next section, we present how models of retina simulate these different functions.

3.3 Models of Vision

3.3.1 Overview of the Retina Models

Models of the acquisition and preprocessing done by the retina mainly focus on modeling how retinal cells encode visual information. More specifically, these models address the following questions: are we able to reproduce the measured response of retinal cells using simple identified elements from biology? Or are we able to explain some sensation that we have when we perceive certain visual stimuli? These two questions are answered by two main types of model, the models that are based on the physiology of retinal cells and those that are based on the description of a stimulus.

A fundamental model was proposed by Hodgkin and Huxley [14]. It explains the dynamics of neural conduction, describing the conductance along the neuron's membrane with a differential equation. Mead and Mahowald [15] were the first to propose an equivalent electrical model of early visual processing and implement it in a silicon circuit. The advantage of defining an equivalent electrical model is double. First, an electrical model of analog signal and digital image processing algorithms might be translated into differential equations, offering a direct correspondence between retina functioning and signal/image processing. Such an approach allows different retinal cells to be simulated that have different connections to other cells (gap-junction, chemical synapse) with a slightly different electrical model. Finally, all these models are integrated into a network corresponding to the whole retina. Second, it is easy to infer the behavior of such a heterogeneous network of cells as long as the network follows certain rules [16].

In the following section, we briefly describe some models of the retina based on the Hodgkin and Huxley model of neural transmission and its equivalent electrical system.

Then, we introduce a different type of model based on information theory. Its goal is to preprocess the input signal into an efficient representation of the natural scene. Consequently, these models are based on so-called ecological vision because the preprocessing of the retina optimizes the representation of information coming from the visual scene.

We finish this part on retinal models by discussing a more general approach to neural modeling. Such models build a geometric space of neural activities and then infer the behavior of the network using the geometric rules stated at the neural scale. Neurogeometry, a term initially used by Jean Petitot, is a very persuasive example of such a category of model. Focusing on color vision, Riemannian geometry (e.g., it generalizes geometric spaces with a fixed metric by allowing a positionwise variation of their metrical properties) has been used extensively to explain color discrimination or differences from the LMS cone excitation space. We describe how these line-element models have tried to put human color discrimination capabilities in the context of Riemannian geometry.

3.3.2 Biological Models

3.3.2.1 Cellular and Molecular Models

A model at the cellular and molecular scale was proposed by van Hateren [17]. His model of the outer retina has been validated against the measured signal on the horizontal cells to wide field, spectrally white stimuli. Horizontal cells are known to change their sensitivities and control their bandwidth with background intensity. The response is nonlinear and the nonlinear function changes for background ranging from 1 to 1000 trolands.¹ All these properties are taken into account in the van Hateren model. He proposed a three-cascade feedback loop representing successively the calcium concentration of the cone outer pigment, the membrane voltage of the cone inner pigment, and the signal modulation of the horizontal cells to cone signals. His model shows that cone responses are the major factor regulating sensitivity in the retina. In a second version, he proposes to include pigment bleaching in the cone outer segment to better simulate cone response [18].

Van Hateren implements his model using an autoregressive moving-average filter that allows the processing to be modified in real time following the adaptation parameter (or the state at time of the static nonlinearity part of the model) and fast calculus [17, 18]. He also manages to give all the constants involved in the model and implementation in Fortran and Matlab² which make the model very useful. The model is principally suited to simulating cone responses, but can be used in digital image processing such as high dynamic range processing [19, 20].

Even by focusing on cellular and molecular modeling, van Hateren falls on a more general kind of modeling that does not necessary follow biology closely. These general models use two kinds of building blocks, a linear (L) multicomponent filtering and a componentwise nonlinearity (N). The blocks are then considered in cascade, in what is called LNL or NLN models, to consider either the case of two linear operators L sandwiched the nonlinear N or vice versa. The NLN model (also called Hammerstein–Wiener) is less tractable than the LNL model [21, 22]. These models can be understood as a decomposition of the general nonlinear Volterra system [23, 24]. Because nonlinearity is componentwise, the only mixing between components (either spatial, spectral, or temporal) is through a linear function. These models have been found to be representative of the mechanism of gain control in the retina [25, 26] but may potentially be more interesting to model color vision [26–29].

Cellular and molecular network models of the retina are the most promising because they are on a perfect scale to fit electrophysiological measurement and the theory of the dynamic on heterogeneous neural network. However, they are difficult because they reach the limit of our knowledge about the optimization of complex systems.

3.3.2.2 Network Models

Models that simulate the way the retina works are numerous — from detailed models of a specific physiological phenomenon to models of the whole retina. In this chapter, we wanted to emphasize two interesting models that take into account the different aspects of retina function. These models naturally make simplifications regarding the complexity of retinal cells but provide interesting results by replicating experiments either on particular response cells (e.g., the ganglion cells of cats) or on more global perceptual phenomena (contrast sensitivity function). An implementation of two of these models is provided through executable software that can be freely downloaded:

· Virtual retina from Wohrer and Kornprobst [30]: http://www-sop.inria.fr/neuromathcomp/public/software/virtualretina/

· The retina model originally proposed by Hérault and Durette[31] and improved and implemented by Benoit et al. [32] http://docs.opencv.org/3.0-beta/modules/bioinspired/doc/retina/index.html

The model proposed by Hérault [4] summarizes the main functions of the retina and it is compared to an image preprocessor. His simplified model of retinal circuits is based on the electrical properties of the cellular membrane that provides a linear spatiotemporal model of retinal cell processing. Beaudot et al. [33, 34] first proposed a basic electrical model equivalent to a series of cells including a membrane's leak current and connected by gap junctions. This model provides the fundamental equations of retina functions to finally model the retina circuitry through a spatiotemporal filter (see Figure 3.3 for its transfer function and Figure 3.4 for simulation on an image).

Figure 3.3 Simulation of the complexity of the spatiotemporal response of the retina in Hérault's model. (a) Temporal evolution of the modulus of the spatial transfer function (only in one dimension ) for a square input . When light is switched on, the spatial transfer function is low-pass at the beginning and becomes band-pass (black lines). Blue and red lines show the temporal profile for respectively low spatial frequency (magnocellular pathway) and high spatial frequency (parvocellular pathway). When the light is switched off, there is a rebound for low spatial frequency. (b) The corresponding temporal profiles for high spatial frequencies (HF-X, parvo) and low spatial frequencies (LF-X magno). The profile for Y cells is also shown (adapted with permission from Ref. [4]).

Figure 3.4 Simulation of Hérault's model on an image that is considered steady. Each image shows respectively the response of cones, horizontal cells, bipolar cells ON and OFF, midget cells (Parvo), parasol cells (Magno X), and cat's Y cells (Magno Y) in response to the original image.

It simulates the main function of the retina with a luminance adaptation as well as a luminance contrast mechanism that results in contrast enhancement. Finally, owing to the nonseparability of the spatial and temporal dimensions, the low spatial frequencies of the signal are transmitted first and later refined by the high spatial frequencies. This phenomenon reproduces the coarse to fine processing that occurs in the visual system [35]. The retina model has been integrated into a real-time C/C++ optimized program [32]. In addition, these authors proposed, with the retina model, a model of the cortical cells to show the advantages of bioinspired models to develop efficient modules for low-level image processing.

Wohrer and Kornprobst [30] also proposed a bioinspired model of the retina and proposed a simulation software, called Virtual Retina. The model is biologically inspired and offers a large-scale simulation (up to 100,000 neurons) in reasonable processing time. Their model includes a linear model of filtering in the OPL, a shunting feedback at the level of bipolar cells accounting for contrast gain control, and a spike generation process modeling ganglion cells. Their model reproduces several experimental measurements from single ganglion cells such as cat X and Y cells (like the data from Ref. [36]).

3.3.2.3 Parallel and Descriptive Models

Werblin and Roska [37] based their model on the identification of several different ways of processing by cells in the retina. The characteristics of the cells were investigated by electrophysiology. They considered the retina as a highly parallel processing system, providing several maps of information built by the network of different cells. The advantage of this approach is to provide images or image sequences of the different functions of the retina.

Similarly, Gollisch and Meister [38] have recently proposed the possibility that the retina could be embedded with high-level functions. They show elegantly how these functions could be implemented by the retinal network.

Another class of model has been called computational models of vision [39, 40]. They aim to model the scale-invariant processing of the visual system by integrating the spatiotemporal filtering of the retina into pyramidal image processing.

In conclusion to this part, no one knows to what degree of sophistication the retina is built, but what we do know is that almost 98% of the ganglion cells that have their dendrites connecting the retina to the LGN have been identified robustly. Nevertheless, their role in our everyday vision and perception has not yet been fully understood.

3.3.3 Information Models

Describing a scene viewed by a human observer requires consideration of every point of the scene as reflecting light. For a complex scene, this light field represents a huge amount of information. It is therefore not possible for a brain to encode every individual scene in that way to enable useful information to be retrieved. From a biological perspective, the retina has hundreds of millions of photoreceptors but the optic nerve that conducts information from the retina to the brain comprises only tens of millions of fibers. These arguments have led to the idea that a compression of information occurs in the visual system [41, 42].

From a computational point of view, this compression is similar to the compression of sound in mp3 or image in jpeg file format. A transform of the input data is done on a basis that reduces redundancy (the choice of the basis is given by statistical analysis of the expected input data). The representation of information onto this basis allows the dimension of the representation to be reduced.

From a biological point of view, Olshausen and Field [43, 44] found that learning a sparse, redundancy-reduced code from a set of natural images gives a basis that resembles the receptive fields of neurons in the visual system V1 (see Chapter 4). This work has been further extended [45, 46], specifically in the temporal [47, 48] and color domains with the correspondence between opponent color coding by neurons and statistical analysis of natural colored images [49–52]. A particular treatment of the theory focused on retinal processing [53–56] (see Chapter 14).

Ecology of vision, by stating that the visual system is adapted to the natural visual environment, gives a simple rule and efficient methods for testing and implementing the principle. Either principal component analysis (PCA), independent component analysis (ICA), or any sparsity methods applied to patches of natural color image sequences are used. However, there is no evidence that inside the retina or along the pathway from retina to cortex, there are neurons that implement such an analysis. Actually, information theory is a black box, describing the relationship between what enters the box and what exits the box well. But it does not give any rule of what is inside the box: in the case of vision, the processing of visual information by neurons.

In color vision, considering the sampling of color signals by the cone mosaic in the retina prevents the application of redundancy reduction [57, 58]. The idea is based on the fact that the statistical analysis performed in redundancy reduction implicitly supposes that the image is stationary in space (statistical analysis is done over patches of the image and should be coherent from one patch to another). But, because of the random arrangement of cones in the retina mosaic, even if the natural scene is stationary, the signal sampled is not.

Moreover, redundancy reduction models seek a unique model of information representation. However, because the cone mosaic differs from one individual to one another, it is likely that the image representation also changes from one individual to another [59]. It is thus possible that, on average, redundancy reduction offers a good model to explain the transformation from incoming light to neural representation into the cortex, but there is no evidence that it applies identically for each individual.

3.3.4 Geometry Models

Geometry models have been extensively developed for color vision and object location in 3D space. The reason is certainly that, as Riemann [60] argues, these two aspects of vision need a more general space than the Euclidean space. For him, they are the only two examples of a continuous manifold in three dimensions that cannot be well represented in Euclidean geometry. Apparently, this motivated his research on what is now called Riemannian geometry. The major distinction between Euclidean and Riemannian geometry is that, in the latter, the notion of distance or norm (length of a segment defined by two points) is local and depends on position in space. Conversely, in Euclidean geometry the norm is fixed all over space.

Soon after Riemann, Von Helmholtz [61] published the first Riemannian model for color discrimination. Before explaining this model in detail, let us consider the evolution of psychophysics at that time. It was shown by Weber that a variation of the sensation of weight depends on the weight considered. This law is denoted by , where is a constant. Intuitively speaking, if you have a weight in your hand, let us say, kilogram, then you feel a difference in weight if you add another weight such that the difference in weight exceeds a constant multiplying . The same kind of law was found for light by Bouguer [62]. Formalizing the relationship between physical space and psychological space, Fechner [63] proposed that the law of sensation be given by , which is one possible solution of Weber's law because the derivative of Fechner's law gives .³ It should be noticed that [65] proposes that the law of sensation is given by a power law, , where is real.⁴ Because the relation is linear in log–log space whereas Fechner's law is linear in log space. This could be interpreted as adding a compressive law before reaching the space of sensation.

Helmholtz's idea was to provide a model of color discrimination based on Fechner's law embedded in a three-dimensional Riemannian space representing color vision space. The line element is therefore given by

3.1

where , , and represent the excitation of putative color mechanism in the red, green, and blue parts of our color vision.⁵

It is, to some extent, a direct extension of Fechner's law in three-dimensional space as shown by Alleysson and Méary [66]. It is also a Riemannian space because the norm depends on the location of the space, given by the denominator values of , , and . And, locally (i.e., for fixed , , and values) the norm is the square norm. This class of discrimination model has been called a line-element. Many extensions of Helmholtz's model have been given (see, e.g., [26, 67–70]).

Despite these theoretical models, the prediction of color discrimination was not very accurate. MacAdam [71] proposes a direct measure of the color difference in CIExy space for a fixed luminance. His well known MacAdam's ellipses served as a basis measure for discrimination and the CIE proposed two nonlinear spaces CIE-LUV, CIE-Lab to take into account those discrimination curves. It has been shown that MacAdam ellipses could be modeled on a Riemannian space [72] although the corresponding curvatures become negative or positive following their position on the color space.

Recently, geometry has been used in a more general context than the color metric. It has been used to model the processing of assembly in neurons in the visual system in a field called neurogeometry [73, 74]. Using geometry rules based on the orientation function modeling the primary visual cortex V1, Jean Petitot was able to show the spatial link of neural activities that Kaniza's figure illusion elicits. The approach of neurogeometry is very elegant and fruitful and will probably be generalized to the whole brain. However, it is still challenging when considering color vision and the sampling of color information with retinal cone mosaic, which differs in arrangement and proportion from one individual to one another [66].

3.4 Application to Digital Photography

The main application of retina models is in the processing of raw images from digital cameras. In digital cameras, the image of the scene is captured by a single sensor covered with an array of color filters to allow for color acquisition. A particular filter is attached to each pixel of the sensor. The main color filter array (CFA) used today in the digital camera is called the Bayer CFA from the name of its inventor [75]. This filter array is made up of three colors (either RGB or CMY) and is a periodic replication of a subpattern of size over the surface of the sensor. Consequently, one of the three colors is represented twice in the array (usually 1R, 2G, and 1B), see Figure 3.5(a). In comparison, Figure 3.5(b) represents a simulation of the random arrangement of L, M, and S-cones in the human retina.

Figure 3.5 (a) Bayer color filter array (CFA) (b) Simulation of the arrangement of the L-cone, M-cone, and S-cone in the human retina (Redrawn from Ref. [7])

At the output of the sensor, the image is digitalized into 8, 12, or more bits. However, the image is not a color image yet because (i) there is only a single color per pixel, (ii) color space is camera based, given by the spectral sensitivity of the camera, (iii) white may not be white following the tint of the light source, (iv) tones could appear unnatural because of nonlinearity in the processing of photometric data. This is the purpose of the digital processor inside the camera—to convert raw image coming from the sensor to a nicely viewed color image. Some of the papers and books that review this are Refs. [76–78].

Classically, the image processing pipeline is based on the following sequential operation: image acquisition by the sensor, some corrections on raw data (dead pixels, blooming, white pixels, noise reduction, etc.)demosaicing to retrieve three color information per pixel, white balancing, and tone mapping. In the following section, we will describe these operations and make references to the literature. We are not being exhaustive in the description, but we focus on studies that use models of vision to state their methods directly or indirectly.

3.4.1 Color Demosaicing

Color demosaicing is an inverse problem [79] that estimates color intensity at various positions that are not acquired by the sensor. During the initial stages, demosaicing was investigated as an interpolation problem using a copy of neighboring pixels to fill in missing pixels or as a bilinear interpolation which was easily done by analog electronic elements such as delay, adder, and so on However, the emergence of digitalization of images at the output of the sensor has allowed more complex demosaicing.

The specificity of color image with inter-channel correlation has been taken into account and almost all methods propose interpolation of color difference instead of color channel directly [80, 81] to take advantage of the correlation between color channels. Another specificity of images that has been taken into account is the fact that images are two dimensional, made on flat surfaces as well as those with contours. By identifying the type of area present in a local neighborhood, it is possible to interpolate along edges rather than across them. This property of color image has been extensively used for demosaicing [80, 82]. Conjointly, some correction methods have been proposed [80–82] to improve the quality of the demosaiced image.

Another approach developed originally by Alleysson et al. [83, 84] and extended by Dubois [85] consists in modeling the expectation of the spatial frequency spectrum of an image acquired through a mosaic. In the case of the Bayer CFA of Figure 3.5, the spatial frequency spectrum shows nine zones of energy: one is in the center, corresponding to luminance (or achromatic information), whereas the remaining eight are on the edge of the spectrum, corresponding to chrominance (or color information at constant luminance) (See Figure 3.6 for an illustration. The demosaicing consists then in estimating (adaptively to image content or not) luminance and chrominance components, by selecting information corresponding either to luminance or chrominance in the spatial frequency spectrum.

Figure 3.6 (a) Color image acquired through the Bayer CFA. (b) The corresponding modulus of the spatial frequency spectrum , showing spectrum of luminance (R+2G+B) information in the center and spectrum of chromatic information (R-B, R-2G+B) in the borders.

This approach to demosaicing has been shown to be very fruitful. First, it is a linear method which can easily be optimized for a particular application or for any periodic CFA. Lu et al. [86] have shown the theoretical equivalence of frequency selection with alternating projection on convex sets (given by pixels and spatial frequency elements of the image) proposed by Gunturk et al. [81], whereas [87] showed the equivalence with a variational approach to dematricing. This model of information representation in a CFA image is also strongly related to the computational model of vision, but includes color. Nevertheless, the random nature of the cone mosaic in the human retina and the difference in arrangement between individuals makes the analogy difficult to establish formally. More generally, the role of eye movement in the reconstruction of color is still to be discovered and could be part of the demosaicing in human vision.

3.4.2 Color Constancy – Chromatic Adaptation – White Balance – Tone Mapping

The retina and visual system show a large extent of adaptation [8, 88], allowing human vision to be accurate even if the dynamic range of the scene is wide. Also, the perceived color of objects is very stable despite a large difference in their reflective light due to change in illumination. Think of the spectrum of a banana under daylight or indoor incandescent light, both of which are quite different, but the banana still appears yellow under both type of lighting. The so-called color constancy phenomenon is a consequence of chromatic adaptation that happens in the retina and visual system as illustrated by Land's retinex theory [89]. A large amount of literature has been produced on retinex theory and its application to image processing.

The principal effect on image quality of not implementing chromatic adaptation in a digital camera can be seen because white on objects or surfaces no longer appears white.

From a computational point of view, chromatic adaptation is often based on the von Kries model which stipulates that adaptation is a gain control in the responses of LMS cones. Roughly, the von Kries model of chromatic adaptation stipulates as follows: , where is the vector containing the preadapted image, is the vector containing the postadapted image, is the unitary matrix () for the transformation from image in the camera color space to image in the LMS color space and is the adaptation transform given by

3.2

where the index represents adaptation under illuminant 1 and index adpatation under illuminant 2. , , and stand for the cone excitation space.

By studying the corresponding colors (colors that are perceived as identical under different light sources), many authors have tested for which LMS space the von Kries model is the most efficient [90–92]. Often, chromatic adaptation is limited to white balancing, and the so-called gray world assumption is made [93].

Tone mapping refers to the modification of the tone values in an image to match the dynamic range between sensor and display (i.e., 12 bits for cameras and 8 bits for display). This operation is particularly challenging when several dynamic ranges of the image are considered from acquisition to restitution. Generally, digital color sensors respond linearly to light intensity but result in a poor-looking image when displayed or printed. Display and printing have intrinsic nonlinearity and the process undergone in our visual system is also nonlinear. Tone mapping operators could be static and global, meaning that a unique nonlinear function is applied identically to all the pixels in the image. It could also be dynamic and local depending on the local content of the scene [89, 94]. Dynamic local tone mapping operators could, in principle, include all adaptive operations such as chromatic adaptation and white balance.

There are several examples of the use of retinal models for tone mapping [94–98].

3.5 Conclusion

In the past, computer vision was mainly concerned with the physical properties of light and how it can be sensed by artificial material. The final user, the human for example, when addressing photography, was not considered, and it was usual to think that all the problems of image rendering and interpreting were due to a poor image-acquisition system. This way of thinking was principally carried forth by the idea that the retina did not do much processing and that the brain was too complex to be modeled.

This view must change today. First, the retina processes the visual information sophisticatedly as explained in this chapter. And second, parts of neural circuitry of the visual system are well known and efficiently modeled. However, a few problems still remain owing to some misunderstandings. These misunderstandings lie in the knowledge that we have about the mechanisms involved when an observer decides the quality or interprets an image. All the efforts have been made in finding, in natural scenes, a universal model of our perception without considering that each individual has its own biological substrate.

Such an approach has been very fruitful because whatever the object of the visual system is, for adapting to the physical input world optimizing transduction or for compensating inter-individual differences, the mechanisms of the visual system are always interesting to be discovered. However, neglecting the specificity of each individual and how this specificity is compensated to enable us to share a common vision is certainly a major drawback of contemporary models of perception.

For computer color vision, we usually use the colorimetry reference observer CIE1924) and CIE1931xy as the reference for color space perception and to develop all computer vision algorithms (reproduction on media, detection, identification, etc.). However, this reference observer for colorimetry is an average of the perception of several observers and hence is not suited to the perception of a particular individual and might in the same way not be suited to a particular application.

Color mosaic sampling, which differs from one individual to another, is a typical example of the compensation mechanism that occurs in our visual system to shape our perception [57–59]. This might be illustrated by the fact that in some circumstances, discrimination does not correspond to appearance [99, 100]. Discrimination is the ability of a system to quantify a difference between incoming signals. As such, it has a quantitative behavior. Appearance, on the other hand, is a qualitative measure because it is not necessarily quantifiable and because it says something directly and not through a few numbers. This is surprising because we usually design a boundary between categories with the locus of best discrimination. In many application systems, we measure the system's responses to a set of physical variables, optimized the processing and the representation for discrimination purpose between different categories of stimuli and we expect it will improve our decision in between multiple categories of actions.

Actually, the problem of ignoring compensations for inter-individual differences in color vision could be considered as noise. For all ecological models of vision, noise comes from the physical world and very few consider internal noise generated by neural processing. In color vision, we usually consider luminance and color to be subjected to the same independent, identically distributed noise. However, the fact that human cone mosaic is so different between observers will certainly generate a differential noise in luminance and chrominance among observers.

The problem of how individual biological differences in humans allow common vision remains to be solved. It is a challenge to explain how we share so much agreement in vision despite our very different biological material. To bring some explanations would certainly improve our knowledge of the functioning of the visual system and its underlying model. It would certainly provide an efficient way to reproduce natural vision in biologically inspired computer vision.

References

1. 1. Solomon, S.G. and Lennie, P. (2007) The machinery of colour vision. Nat. Rev. Neurosci., 8, 276–286.

2. 2. Buzás, P. et al. (2006) Specificity of M and L cone inputs to receptive fields in the parvocellular pathway: random wiring with functional bias. J. Neurosci., 26, 11148–11161.

3. 3. Piccolino, M., Strettoi, E., and Laurenzi, E. (1989) Santiago Ramon y Cajal, the retina and the neuron theory. Doc. Ophthalmol., 71, 123–141.

4. 4. Hérault, J. (2010) Vision: Images, Signals and Neural Networks-Models of Neural Processing in Visual Perception, World Scientific Publishing.

5. 5. Dacey, D.M. et al. (2005) Melanopsin-expressing ganglion cells in primate retina signal colour and irradiance and project to the LGN. Nature, 433, 749–754.

6. 6. Osterberg, G. (1935) Topography of the Layer of Rods and Cones in the Human Retina, Nyt Nordisk Forlag.

7. 7. Roorda, A. and Williams, D.R. (1999) The arrangement of the three cone classes in the living human eye. Nature, 397, 520–522.

8. 8. Clark, D.A. et al. (2013) Dynamical adaptation in photoreceptors. PLoS Comput. Biol., 9, e1003289.

9. 9. Hartline, H.K. (1938) The response of single optic nerve fibers of the vertebrate eye to illumination of the retina. Am. J. Physiol., 121, 400–415.

10.10. Alonso, J.-M. and Chen, Y. (2009) Receptive field. Scholarpedia, 4, 5393.

11.11. Rodieck, R. (1998) The First Steps in Seeing, Sinauer Associates, Inc.

12.12. McMahon, M.J. et al. (2000) Fine structure of parvocellular receptive fields in the primate fovea revealed by laser interferometry. J. Neurosci., 20, 2043–2053.

13.13. Dacey, D.M. et al. (2003) Fireworks in the primate retina: in vitro photodynamics reveals diverse LGN-projecting ganglion cell types. Neuron, 37, 15–27.

14.14. Hodgkin, A.L. and Huxley, A.F. (1952) A quantitative description of membrane current and its application to conduction and excitation in nerve. J. Physiol., 117, 500.

15.15. Mead, C.A. and Mahowald, M.A. (1988) A silicon model of early visual processing. Neural Netw., 1, 91–97.

16.16. Rojas, R. (1996) Neural Networks: A Systematic Introduction, Springer-Verlag.

17.17. van Hateren, H. (2005) A cellular and molecular model of response kinetics and adaptation in primate cones and horizontal cells. J. Vis., 5, 331–347.

18.18. Van Hateren, J. and Snippe, H. (2007) Simulating human cones from mid-mesopic up to high-photopic luminances. J. Vis., 7.

19.19. Van Hateren, J. (2006) Encoding of high dynamic range video with a model of human cones. ACM Trans. Graph. (TOG), 25, 1380–1399.

20.20. Keil, M.S. and Vitrià, J. (2007) Pushing it to the limit: adaptation with dynamically switching gain control. EURASIP J. Appl. Signal Process., 2007, 117.

21.21. Crama, P. and Schoukens, J. (2004) Hammerstein–wiener system estimator initialization. Automatica, 40, 1543–1550.

22.22. Billings, S. and Fakhouri, S. (1982) Identification of systems containing linear dynamic and static nonlinear elements. Automatica, 18, 15–26.

23.23. Benardete, E.A. and Victor, J.D. (1994) An extension of the M-sequence technique for the analysis of multi-input nonlinear systems, in Advanced Methods of Physiological System Modeling, Springer-Verlag, pp. 87–110.

24.24. Volterra, V. (2005) Theory of Functionals and of Integral and Integro-differential Equations, Dover Books on Mathematics Series, Dover Publications.

25.25. Shapley, R. and Victor, J. (1979) Nonlinear spatial summation and the contrast gain control of cat retinal ganglion cells. J. Physiol., 290, 141–161.

26.26. Alleysson, D. and Hérault, J. (2001) Variability in color discrimination data explained by a generic model with nonlinear and adaptive processing. Color Res. Appl., 26, S225–S229.

27.27. Webster, M.A. and Mollon, J. (1991) Changes in colour appearance following post-receptoral adaptation. Nature, 349, 235–238.

28.28. Zaidi, Q. and Shapiro, A.G. (1993) Adaptive orthogonalization of opponent-color signals. Biol. Cybern., 69, 415–428.

29.29. Chichilnisky, E. and Wandell, B.A. (1999) Trichromatic opponent color classification. Vision Res., 39, 3444–3458.

30.30. Wohrer, A. and Kornprobst, P. (2009) Virtual retina: a biological retina model and simulator, with contrast gain control. J. Comput. Neurosci., 26, 219–249.

31.31. Hérault, J. and Durette, B. (2007) Modeling visual perception for image processing, in Computational and Ambient Intelligence, Springer-Verlag, pp. 662–675.

32.32. Benoit, A. et al. (2010) Using human visual system modeling for bio-inspired low level image processing. Comput. Vision Image Underst., 114, 758–773.

33.33. Beaudot, W., Palagi, P., and Hérault, J. (1993) Realistic simulation tool for early visual processing including space, time and colour data, in New Trends in Neural Computation, Springer-Verlag, pp. 370–375.

34.34. Beaudot, W.H. (1996) Adaptive spatiotemporal filtering by a neuromorphic model of the vertebrate retina. Proceedings IEEE International Conference on Image Processing, Vol. 1, pp. 427–430.

35.35. Buser, P. and Imbert, M. (1987) Neurophysiologie Fonctionnelle Tome 4: Vision, Collection Méthodes, Hermann.

36.36. Enroth-Cugell, C. and Robson, J.G. (1966) The contrast sensitivity of retinal ganglion cells of the cat. J. Physiol., 187, 517–552.

37.37. Werblin, F.S. and Roska, B.M. (2004) Parallel visual processing: a tutorial of retinal function. Int. J. Bifurcation Chaos, 14, 843–852.

38.38. Gollisch, T. and Meister, M. (2010) Eye smarter than scientists believed: neural computations in circuits of the retina. Neuron, 65, 150–164.

39.39. Landy, M.S. and Movshon, J.A. (1991) Computational Models of Visual Processsing, The MIT Press.

40.40. Heeger, D.J., Simoncelli, E.P., and Movshon, J.A. (1996) Computational models of visual processing. Proc. Natl. Acad. Sci. U.S.A., 23, 623–627.

41.41. Barlow, H.B. (2012) Possible principles underlying the transformation of sensory messages, in Sensory Communication, (ed. W.A. Rosenblith), The MIT Press, 2012. MIT Press Scholarship Online, 2013. DOI: 10.7551/mitpress/9780262518420.003.0013.

42.42. Field, D.J. (1987) Relations between the statistics of natural images and the response properties of cortical cells. J. Opt. Soc. Am. A, 4, 2379–2394.

43.43. Olshausen, B.A. and Field, D.J. (1996) Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607.

44.44. Olshausen, B.A. and Field, D.J. (2004) Sparse coding of sensory inputs. Curr. Opin. Neurobiol., 14, 481–487.

45.45. Bell, A.J. and Sejnowski, T.J. (1997) The “independent components” of natural scenes are edge filters. Vision Res., 37, 3327–3338.

46.46. Atick, J.J. (1992) Could information theory provide an ecological theory of sensory processing? Netw. Comput. Neural Syst., 3, 213–251.

47.47. van Hateren, J.H. and Ruderman, D.L. (1998) Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex. Proc. R. Soc. London, Ser. B: Biol. Sci., 265, 2315–2320.

48.48. Dong, D.W. and Atick, J.J. (1995) Statistics of natural time-varying images. Netw. Comput. Neural Syst., 6, 345–358.

49.49. Buchsbaum, G. and Gottschalk, A. (1983) Trichromacy, opponent colours coding and optimum colour information transmission in the retina. Proc. R. Soc. London, Ser. B.: Biol. Sci., 220, 89–113.

50.50. Ruderman, D.L., Cronin, T.W., and Chiao, C.-C. (1998) Statistics of cone responses to natural images: implications for visual coding. J. Opt. Soc. Am. A, 15, 2036–2045.

51.51. Tailor, D.R., Finkel, L.H., and Buchsbaum, G. (2000) Color-opponent receptive fields derived from independent component analysis of natural images. Vision Res., 40, 2671–2676.

52.52. Lee, T.-W., Wachtler, T., and Sejnowski, T.J. (2002) Color opponency is an efficient representation of spectral properties in natural scenes. Vision Res., 42, 2095–2103.

53.53. Atick, J.J. and Redlich, A.N. (1992) What does the retina know about natural scenes? Neural Comput., 4, 196–210.

54.54. Atick, J.J. and Redlich, A.N. (1990) Towards a theory of early visual processing. Neural Comput., 2, 308–320.

55.55. Atick, J.J., Li, Z., and Redlich, A.N. (1992) Understanding retinal color coding from first principles. Neural Comput., 4, 559–572.

56.56. Doi, E. and Lewicki, M.S. (2007) A theory of retinal population coding. Adv. Neural Inf. Process. Syst., 19, 353.

57.57. Alleysson, D. and Süsstrunk, S. (2004) Spatio-chromatic ICA of a mosaiced color image, in Independent Component Analysis and Blind Signal Separation, Springer-Verlag, pp. 946–953.

58.58. Alleysson, D. and Süsstrunk, S. (2004) Spatio-chromatic PCA of a mosaiced color image. Conference on Colour in Graphics, Imaging, and Vision, Number 1 in 2004, Society for Imaging Science and Technology, pp. 311–314.

59.59. Alleysson, D. (2010) Spatially coherent colour image reconstruction from a trichromatic mosaic with random arrangement of chromatic samples. Ophthalmic Physiol. Opt., 30, 492–502.

60.60. Riemann, B. (1854) Ueber die hypothesen, welche der geometrie zu grunde liegen, Habilitationsschrift, Abhandlungen der Königlichen Gesellschaft der Wissenschaften zu Göttingen, vol. 13, http://www.maths.tcd.ie/pub/HistMath/People/Riemann/Geom/Geom.pdf; Translation English by Clifford, W. (1854) On the hypothesis which lie at the basis of Geometry. Nature, 1873, http://192.43.228.178/classics/Riemann/WKCGeom.pdf. Trad. Française par Houel J. Sur les hypothèses qui servent de fondement à la géométrie http://ddata.over-blog.com/xxxyyy/0/31/89/29/Fusion-58/F58007.pdf.

61.61. Von Helmholtz, H. (1866) Handbuch der physiologischen Optik: mit 213 in den Text eingedruckten Holzschnitten und 11 Tafeln, vol. 9, Voss.

62.62. Bouguer, P. (1729) Essai d'optique, sur la gradation de la lumiere, Claude Jombert.

63.63. Fechner, G.T. (1907) Elemente der psychophysik, vol. 2, Breitkopf & Härtel.

64.64. Wiener, N. (1921) A new theory of measurement: a study in the logic of mathematics. Proc. London Math. Soc., 2, 181–205.

65.65. Stevens, S.S. (1957) On the psychophysical law. Psychol. Rev., 64, 153.

66.66. Alleysson, D. and Méary, D. (2012) Neurogeometry of color vision. J. Physiol. Paris, 106, 284–296.

67.67. Schrödinger, E. (1920) Grundlinien einer theorie der farbenmetrik im tagessehen. Ann. Phys., 368, 481–520.

68.68. Vos, J. and Walraven, P. (1972) An analytical description of the line element in the zone-fluctuation model of colour vision–I. Basic concepts. Vision Res., 12, 1327–1344.

69.69. Stiles, W. (1946) A modified helmholtz line-element in brightness-colour space. Proc. Phys. Soc. London, 58, 41.

70.70. Koenderink, J., Van de Grind, W., and Bouman, M. (1972) Opponent color coding: a mechanistic model and a new metric for color space. Kybernetik, 10, 78–98.

71.71. MacAdam, D.L. (1942) Visual sensitivities to color differences in daylight. J. Opt. Soc. Am. A, 32, 247–273.

72.72. Kohei, T., Chao, J., and Lenz, R. (2010) On curvature of color spaces and its implications. Conference on Colour in Graphics, Imaging, and Vision, vol. 2010, Society for Imaging Science and Technology, pp. 393–398.

73.73. Petitot, J. (2003) The neurogeometry of pinwheels as a sub-riemannian contact structure. J. Physiol. Paris, 97, 265–309.

74.74. Citti, G. and Sarti, A. (2010) Neuromathematics of vision. AMC, 10, 12.

75.75. Bayer, B. (1976) Color imaging array. US Patent 3,971,065.

76.76. Ramanath, R. et al. (2005) Color image processing pipeline. IEEE Signal Process. Mag., 22, 34–43.

77.77. Li, X., Gunturk, B., and Zhang, L. (2008) Image demosaicing: a systematic survey, Electronic Imaging 2008, International Society for Optics and Photonics, pp. 68221J–68221J-15.

78.78. Lukac, R. (2012) Single-Sensor Imaging: Methods and Applications for Digital Cameras, CRC Press.

79.79. Ribes, A. and Schmitt, F. (2008) Linear inverse problems in imaging. IEEE Signal Process. Mag., 25, 84–99.

80.80. Hirakawa, K. and Parks, T.W. (2005) Adaptive homogeneity-directed demosaicing algorithm. IEEE Trans. Image Process., 14, 360–369.

81.81. Gunturk, B.K., Altunbasak, Y., and Mersereau, R.M. (2002) Color plane interpolation using alternating projections. IEEE Trans. Image Process., 11, 997–1013.

82.82. Kimmel, R. (1999) Demosaicing: image reconstruction from color CCD samples. IEEE Trans. Image Process., 8, 1221–1228.

83.83. Alleysson, D., Süsstrunk, S., and Hérault, J. (2002) Color demosaicing by estimating luminance and opponent chromatic signals in the fourier domain. Color and Imaging Conference, Number 1 in 2002, Society for Imaging Science and Technology, pp. 331–336.

84.84. Alleysson, D., Susstrunk, S., and Hérault, J. (2005) Linear demosaicing inspired by the human visual system. IEEE Trans. Image Process., 14, 439–449.

85.85. Dubois, E. (2005) Frequency-domain methods for demosaicking of bayer-sampled color images. IEEE Signal Process Lett., 12, 847.

86.86. Lu, Y.M., Karzand, M., and Vetterli, M. (2010) Demosaicking by alternating projections: theory and fast one-step implementation. IEEE Trans. Image Process., 19, 2085–2098.

87.87. Condat, L. (2009) A generic variational approach for demosaicking from an arbitrary color filter array. 2009 16th IEEE International Conference on Image Processing (ICIP), IEEE, pp. 1625–1628.

88.88. Shapley, R. and Enroth-Cugell, C. (1984) Visual adaptation and retinal gain controls. Prog. Retin. Res., 3, 263–346.

89.89. Land, E.H. (1983) Recent advances in retinex theory and some implications for cortical computations: color vision and the natural image. Proc. Natl. Acad. Sci. U.S.A., 80, 5163.

90.90. Lam, K. (1985) Metamerism and colour constancy. PhD thesis. University of Bradford.

91.91. Finlayson, G.D. et al. (2012) Spectral sharpening by spherical sampling. J. Opt. Soc. Am. A, 29, 1199–1210.

92.92. Finlayson, G.D. and Süsstrunk, S. (2000) Performance of a chromatic adaptation transform based on spectral sharpening. Color and Imaging Conference, vol. 2000, Society for Imaging Science and Technology, pp. 49–55.

93.93. Van De Weijer, J. and Gevers, T. (2005) Color constancy based on the grey-edge hypothesis. IEEE International Conference on Image Processing, 2005. ICIP 2005, vol. 2, IEEE, pp. II–722.

94.94. Meylan, L., Alleysson, D., and Süsstrunk, S. (2007) Model of retinal local adaptation for the tone mapping of color filter array images. J. Opt. Soc. Am. A, 24, 2807–2816.

95.95. Mantiuk, R., Myszkowski, K., and Seidel, H.-P. (2006) A perceptual framework for contrast processing of high dynamic range images. ACM Trans. Appl. Percept. (TAP), 3, 286–308.

96.96. Ferwerda, J.A. et al. (1996) A model of visual adaptation for realistic image synthesis. Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, ACM, pp. 249–258.

97.97. Reinhard, E. and Devlin, K. (2005) Dynamic range reduction inspired by photoreceptor physiology. IEEE Trans. Visual Comput. Graphics, 11, 13–24.

98.98. Larson, G.W., Rushmeier, H., and Piatko, C. (1997) A visibility matching tone reproduction operator for high dynamic range scenes. IEEE Trans. Visual Comput. Graphics, 3, 291–306.

99.99. Bachy, R. et al. (2012) Hue discrimination, unique hues and naming. J. Opt. Soc. Am. A, 29, A60–A68.

100. 100. Danilova, M. and Mollon, J. (2012) Foveal color perception: minimal thresholds at a boundary between perceptual categories. Vision Res., 62, 162–172.

¹ Troland is a unit of conventional retinal illuminance corrected for pupil size.² Implementation details are available at https://sites.google.com/site/jhvanhateren/home/software-van-hateren-snippe-2007³ [64] states that in order to transform between Weber and Fechner, one would have to make the JNDs infinitesimally small which makes no sense in this context.⁴ The exponent depends on the type of experiment either protetic (measured) or metatic (sensation). In Reference [65] .⁵ Today we would probably use , , and instead of , , and to handle the LMS cone excitation space.