RidgeRun Auto exposure/Auto white balance library for DM368 and DM365

From RidgeRun Developer Connection
Revision as of 15:42, 28 June 2011 by Mmadrigal (Talk | contribs)

Jump to: navigation, search


CMOS or CCD sensor video capture quality can be enhance with image processing, like auto white balance (AWB) and auto exposure algorithms (AE):


Some camera sensors don't include auto white balance and/or auto exposure processing, so RidgeRun offers a library with AE and AWB algorithms called librraew. This library was initially developed for the DM365 platform. The DM365 video processing front end (VPFE) has an H3A module designed to support control loops for auto focus, auto white balance and auto exposure by collecting statistics about the imaging/video data. There are two blocks in this module:

  • Auto focus engine
  • Auto exposure and auto white balance engine


  • Accumulation of the sum of squared pixels per color.
  • Minimum and maximum pixels values in each window per color.

The AE/AWB engine can be configured to use up to 36 horizontal windows with sum + {sum of squares or min+max} output or up to 56 horizontal windows with sum output. The AE/AWB engine can also be configure to use up to 128 vertical windows. The width and height for the windows is programmable.

Currently, librraew has testing has focused on Aptina CMOS sensor mt9p031, but if you provide the appropriate functions for the library, it can works with any sensor. The implementation is a plain C library and can be re-used with and integrated with any application. RidgeRun uses ipiped (see below) for testing and demonstration.


Auto white balance

When an image of a scene is captured by a digital camera sensor, the sensor response at each pixel depends on the scene illumination. Depending of the illumination, a distinct color cast appears over the captured scene. This effect appears in the captured image due to the color temperature of the light. If a white object is illuminated with a low color temperature light source, the object in the captured image will be reddish. Similarly, when the white object is illuminated with a high color temperature light source, the object in the captured image will be bluish. The human eye compensates for color cast automatically through a characteristic known as color constancy, allowing the colors to be independent of the illumination. Auto white balance tries to simulate the color constancy for images capture.

Many AWB algorithms follow a two-stage process:

  • Illumination estimation: can be done explicitly by choosing from a known set of possible illuminations or implicitly with assumptions about the effect of such illuminations. The algorithms implemented in librraew use the implicitly estimation.
  • Image color correction: is achieved through an independent gain adjustment of the three color signals. Commonly only the blue and red gains are adjusted.

The librraew supports several AWB algorithms:

Gray World
White Patch
The White Patch algorithm assumes that the maximum response in an image is caused by a perfect reflector, that represents the color of the illumination. White balancing attempts to equalize the maximum value of the three channels to produce a white patch. This algorithm must be used on scenes that ensure the captured image does not include saturated pixels.
White Patch 2
This variation of the White Patch algorithm aims to resolve the problems with the saturated pixels. White Patch 2 ses an average of local maximums instead of an absolute maximum. The results can change depending of the number of sample windows used in processing the image.

Auto exposure

One of the main problems affecting image quality, leading to unpleasant pictures, comes from improper exposure to light. The exposure is the amount of light that reaches the image sensor. Exposure determines the lightness or darkness of the resulting image. If too much light strikes the image sensor, the image will be overexposed, washed out, and faded. If too little light reaches the camera sensor produces an underexposed image, dark and lacking in details especially in shadow areas. Auto exposure (AE) algorithms adjust the captured image in an attempt to reproduce the most important regions (according to contextual or perceptive criteria) with a level of brightness, more or less in the middle of the possible range.

Auto exposure algorithms involves three processes:

  • Light metering: generally accomplished using the camera sensor itself or an external device as exposure detector.
  • Scene analysis: brightness metering methods used to estimate the scene illumination according to image metrics. Using this value, calculates the brightness adjustments to have the best exposure.
  • Image brightness correction: ensures that the correct amount of light reaches the image sensor, illumination and shutter time parameters are adjusted. The CMOS image sensor parameter is often called exposure time. The exposure time is defined as the amount of time that the sensor integrates light. In other words, it determines how long the sensor photo diodes array is exposed to light.



Center weighted


Implementation Details

Three applications are used to support the auto exposure and auto white-balance (AEW) adjustments in the RidgeRun's SDK:

  • librraew, a library that includes auto white balance and auto exposure algorithms.

Running Ipiped

Ipiped must run in background. If you are using RidgeRun's SDK and enabled ipiped it will be setup to start automatically when the system boots.

 ipiped &


Running Ipipe-client






To quit the interactive console you can use quit or exit.

In order to know the commands that are available run:

ipipe-client help

or get into the interactive console and execute help.

This shows a description of each command, as follows:

Command                         Description

set-debug               Enable/Disable debug messages.
init-aew                Initialize AEW algorithms.
stop-aew                End AEW algorithm.
shell                   Execute a shell command(shell_cmd) using interactive console.
ping                    Show if ipipe-daemon is alive.
quit                    Quit from the interactive console.
exit                    Exit from the interactive console.
get-sensor              Show the sensor that is being used.

If you want more detailed information about a command execute:

ipipe-client help <command> 

Controlling librraew with ipipe


Command: init-aew
Description: Initialize AEW algorithms                                  
        WB: white balance algorithm, the options are:
                G -for gray world algorithm
                W -for retinex algorithm
                W2 -for variant of retinex algorithm
                N -for none
        AE: auto exposure algorithm, the options are
                EC -for electronic centric
                N -for none
        G: gain type, the options are:
                S -to use the sensor gain
                D -to use the digital
        EM: exposure metering method, the options are:
                P -for partial metering that take into account the light
                information of a portion in the center and the rest of
                the frame is ignored. The size  of the center depends of
                of the parameter center_percentage
                C -for center weighted metering that take into account
                the light information coming from the entire frame with
                emphasis placed on the center area
                A -for average metering that take into account the light
                information from the entire frame without weighting
                SG -for segmented metering that divides the frame
                on 6 pieces and weighting them to avoid backlighting
        T: wait time in us, specifies the time between
                algorithm adjustments, max value=1s=1000000us
        fps: minimum frame rate
        seg: frame segmentation factor, each frame is segmented into
                regions, this factor represents the percentage of the
                maximum number of possible regions
        width: captured video/image horizontal size
        height: captured video/image vertical size
        center_percentage: defines the percentage of the image width
                and height to be used as the center size

Also you can stop automatic adjustments with the command stop-aew



Using librraew with other software rather than ipiped

librraew is a plain C library and can be re-used and integrated with any custom application. Please contact RidgeRun for the documentation of the librraew API.

Using the demo version of librraew

You can request a demo version of librraew in order to test the auto-white balance and auto-exposure algorithms before you decide to buy it. This library will allow you to use all the features that comes with the full version of the librraew but with the following limitations:

  • The algorithm will be darken the image periodically.
  • After a while the algorithm will stop working and the image capturing will be done with the last values calculated by the library. In order to test the library again you will need to restart the algorithm.

Basic example of use

In this section it will be shown a basic example of use of the ipiped and the librraew library. Once your board has started run the following commands:

ipipe-client run-config-script dm365_mt9p031_config
ipipe-client init-aew G EC S C 200000 30 50 640 480 50

The first command will chain the ipiped using an existing script. The second one will start the auto-white balance and the auto-exposure algorithms for an image size of 640x480 pixels and a minimum frame rate of 30fps (See Controlling librraew with ipipe section for more details).

Once you have the ipiped and the AEW algorithms running you can run any image/video capture GStreamer pipeline to test it.

As an example it was used the pipeline shown below which captures 30 images from the camera module and encodes them to JPEG format in order to get 30 different JPEG images.

gst-launch -e v4l2src always-copy=false num-buffers=30 chain-ipipe=false ! video/x-raw-yuv,format=\(fourcc\)UYVY, \
width=640, height=480 !  dmaienc_jpeg ! queue !  multifilesink location=image%0d.jpeg

The images 1 and 2 show the difference between an capture made with and without the AEW algorithm, respectively.

center Figure 1. Image taken using the AEW algorithm.


  1. Battiato, G. Messina, and A. Castorina. Exposure correction for imaging devices: an overview. In Single-Sensor Imaging: Methods and Applications for Digital Cameras, chapter 12. Rastislav Lukac, October 2008.
  2. Lee J.S., Jung Y.Y, Kim B.S., and Ko S.J. An advanced video camera system with robust af, ae, and awb control. IEEE Transactions on Consumer Electronics, 47:694–699, August 2001
  3. Edmund Y. Lam. Combining gray world and retinex theory for automatic white balance in digital photography. Consumer Electronics, 2005. (ISCE 2005). Proceedings of the Ninth International Symposium on, pages 134–139, June 2005.
  4. Edmund Y. Lam and George S. K. Fung. Automatic white balancing in digital photography. In Single-Sensor Imaging: Methods and Applications for Digital Cameras. Taylor & Francis Group, LLC, 2009.
  5. Nitin Sampat, Shyam Venkataraman, Thomas Yeh, and Robert L. Kremens. System implications of implementing auto-exposure on consumer digital cameras. Proc. SPIE. Sensors, Cameras, and Applications for Digital Photography, 3650:100–107, March 1999