Small Vision System User Manual SRI Small Vision System User’s Manual Software version 2.1b February 2001 Kurt Konolige and David Beymer SRI International konolige@ai.sri.com http://www.ai.sri.
Small Vision System User Manual 2 1 Introduction ______________________________________________________ 4 1.1 Implementations of the Stereo Engine ____________________________________ 5 1.2 The Small Vision System_______________________________________________ 6 1.3 Hardware and Software Requirements ___________________________________ 7 1.3.1 1.3.2 Analog Framegrabbers ___________________________________________________ 7 Digital Framegrabbers ___________________________________________________ 7 1.
Small Vision System User Manual 3 3.5 Filtering ___________________________________________________________ 35 3.6 Performance ________________________________________________________ 37 4 Calibration ______________________________________________________ 38 4.1 Calibration Procedure ________________________________________________ 39 4.1.1 4.1.
Small Vision System User Manual 4 1 Introduction The SRI Stereo Engine is an efficient realization of an area correlation algorithm for computing range from stereo images. Figure 1 shows the results of running the algorithm on a typical scene. The image on the top is the left image of an original stereo pair, while the one on the bottom is a disparity image computed from the stereo pair.
Small Vision System User Manual 1.1 5 Implementations of the Stereo Engine The Stereo Engine exists in several implementations, including embedded, low-power systems and general purpose microcomputers. The embedded systems, or Small Vision Modules (SVMs), contain DSPs or other standalone processors, and produce digital range information. They are meant for end applications where size, cost, and power limitations are critical.
Small Vision System User Manual 1.2 6 The Small Vision System The Small Vision System (SVS) is meant to be an accessible development environment for experimenting with applications for stereo processing. It consists of a library of functions for performing stereo correlation. Figure 1-2 shows the relationship between the SVS library and PC hardware. Images come in via a pair of aligned video cameras, called a stereo head.
Small Vision System User Manual 1.3 Hardware and Software Requirements The SVS libraries exist for most Unix systems, as well as MS Windows 95/98/2000/NT; that is, on the most common computer platforms available. We have spent considerable effort in optimizing SVS for PCs using the MMX instruction set (Pentium MMX and Pentium II, III), and it will perform best on these platforms, using either Linux or MS Windows. Performance information is in Section 3.6. 1.3.
Small Vision System User Manual 1.4 The SVS Distribution The SVS distribution can reside in any directory; normally, it is placed in c:\svs (Windows 95/NT systems) or /usr/local/svs or a user’s home directory (Unix systems). Here is the directory structure of the SVS distribution. svs readme update docs smallv.pdf bin smallv(.exe) stframe(.exe) stfr(.exe) stcap(.exe) XXXgrab.dll,.lib svs.lib libsvs.so fltkdll.dll libfltk.so data seq-cal.ssi seq.ssi *-L/R/C.bmp *.ini samples src svs.h image_io.
Small Vision System User Manual 9 2 Getting started with smallv The smallv program is a standalone application that exercises the SVS library. It is a GUI interface to the stereo programs, and in addition can load and save stereo image sequences. The smallv program is a useful tool for initial development of a stereo application, and can also be used to check out and adjust a stereo camera setup. The smallv program is in the bin/ directory.
Small Vision System User Manual 2.1 Inputting Live Stereo Video The SVS libraries provide support for live video as stereo input. To input video, you must do the following steps. 1. Decide on a stereo head and framegrabber. 2. Install the framegrabber, following instructions that come with the framegrabber or the stereo head (STH-V3 or MEGA-D). 3. Copy the appropriate framegrabber DLL to bin\svsgrab.dll (MS Windows), or bin/libcap.so (Unix); see Section 2.1.2). 4.
Small Vision System User Manual 3. Use a single framegrabber, and a stereo head that interlaces two video signals onto a single video stream. The STH-V3 from Videre Design (www.videredesign.com) is one such stereo head. 4. Use a digital stereo head, the MEGA-D (STH-MD1) from Videre Design. This stereo head outputs a digital signal on the 1394 bus, and any OHCI (Open Host Controller Interface) card can be used to input the video.
Small Vision System User Manual /dev/fg1 (if there are two). Check the device names for your installed devices, and make soft links to the SVS names, e.g., for Matrox Meteor cards, use the command ln -s /dev/mmetgrab0 /dev/fg0 The framegrabber interface used by smallv under MS Windows or Linux is indicated in the message window at startup. There are some limitations in framegrabber drivers that should be noted.
Small Vision System User Manual It is important for stereo processing to have the left camera image appear as the left image in the smallv program. Once the video input is displayed, you can check this by pointing the cameras along your line of sight. The right camera appears on your right side, and the right image on the smallv display should show this image. You can cover one camera with your hand, and observe which displayed image goes dark.
Small Vision System User Manual Combination sampling modes are available, e.g., “x4 bin+dec” samples the image down to ¼ size in horizontal and vertical directions, by decimating by 2 and then binning by 2. 2.1.7 Image Source The source for stereo images can be either a memory buffer or a live video stream. The Source drop list lets you choose between these, or to stop any input. Buffer input is discussed in Section 2.1.9. 2.1.
Small Vision System User Manual 15 Figure 2-5 Two 320x240 subwindows (bottom) of a 640x480 image (top). desirable for minimizing motion blur, and it may be more convenient to increase gain while exposure is not at a maximum. The values of exposure, gain, brightness, and contrast are all represented as a percent. The colorized version of the MEGA-D digital camera can input color images, and the color balance can be adjusted manually using the red/blue differential gain.
Small Vision System User Manual 16 The placement of subwindows is controlled by the vertical (Y) and horizontal (X) offset controls in the Subimage box of the Video Parameters dialog window. These parameters can be changed in real time, enabling electronic panning of the live image. 2.1.11 Vergence When in subwindow mode, the two cameras in a stereo rig generally will have the same X and Y offsets, so that they keep the parallel line-of-sight characteristic of the stereo rig.
Small Vision System User Manual 17 2.1.12 Color As of Version 2.1, SVS supports color input and display. Besides the two monochrome left/right stereo channels, there is a third color channel, with images in RGB 24-bit format. The color channel does not participate in stereo processing, but it can be useful in applications that combine color and stereo information, for example, object tracking.
Small Vision System User Manual 2.2 18 Storing, Saving, and Loading Stereo Data smallv provides a basic facility for loading and saving stereo data streams. The file load and store functions are part of the SVS library, and their source code is included. smallv exercises these functions, and provides a memory buffer for storing live stereo video. In smallv, the buffer always holds the left and right input stereo images. 2.2.
Small Vision System User Manual 2.3 19 Display smallv displays two images in its display area. The left display is always the left input image. Input images are displayed in grayscale, unless color information is present: in this case, the left image will be shown in color. The right display can be either the right input image, or the results of processing. Processing results are always displayed in “greenscale”, using shades of green.
Small Vision System User Manual 2.4 20 Stereo Processing and Parameters In smallv, stereo processing takes place in conjunction with the input of stereo images. The basic cycle is: get stereo pair -> process pair -> display pair The input is either from live video or the buffer (Sections 2.1 and 2.1.9). In freeze mode, the same pair is processed continuously, so adjustments can be made in stereo parameters. 2.4.
Small Vision System User Manual 21 Figure 2-7 Original stereo pair (top) and rectified pair (bottom). 2.4.4 Disparity Search Range Even with stereo rectification, it may not be possible to match every object in the scene, because the horopter is not large enough. In this case, the horopter can be enlarged by changing the number of disparities searched by the stereo process. This search range can vary from 8 to 80 pixels. Larger search ranges enlarge the horopter, but not in a linear fashion, i.e.
Small Vision System User Manual 22 because there is some fixed overhead in processing the images. Obviously, the smallest search range necessary for the application is the best choice. Disparities are interpolated to 1/16 pixel, so a search range of 16 means that there are 256 integral disparity values, ranging from 0 (no disparity) to 255 (maximum disparity of 15 15/16 pixels). The search range is selected using the Disparities drop list in the Parameters area.
Small Vision System User Manual 2.5 23 Filtering Stereo processing will generally contain incorrect matches. There are two major sources for these errors: lack of sufficient image texture for a good match, and ambiguity in matching when the correlation window straddles a depth boundary in the image. The SVS stereo processing has two filters to identify these mismatches: a confidence measure for textureless areas, and a left/right check for depth boundaries.
Small Vision System User Manual 2.6 Saving and Restoring Parameters All of the parameters that control the operation of the SVS Stereo Engine can be saved to a file for later use. Parameter files can be loaded and saved using the File menu: Load Param File and Store Param File. The file data/megad-75.ini contains a sample file for a 7.5 mm lens on the MEGA-D stereo rig. It serves as an example of the settings available through parameter files.
Small Vision System User Manual Rx -0.008051 Ry -0.003771 Rz -0.000458 [left camera] pwidth 1280 pheight 960 dpx 0.007500 dpy 0.007500 sx 1.000000 Cx 582.260123 Cy 506.081223 f 7.798704 kappa1 0.002983 kappa2 -0.000040 proj # rotation between left and right cameras # number of pixels in the camera # pixel spacing, mm # aspect ratio # camera center, pixels # focal length, mm # radial distortion parameters # projection matrix: from left camera 3D coords # to left rectified coordinates 1.041674e+003 6.
Small Vision System User Manual 26 3 Stereo Geometry Stereo algorithms compute range information to objects by using triangulation. Two images at different viewpoints see the object at different positions: the image difference is called disparity. This section discusses the basic equations that govern the relationship between disparity and range.
Small Vision System User Manual 3.1 27 Disparity The figure below displays stereo geometry. Two images of the same object are taken from different viewpoints. The distance between the viewpoints is called the baseline (b). The focal length of the lenses is f. The horizontal distance from the image center to the object image is dl for the left image, and dr for the right image. Figure 3-1. Definition of disparity: offset of the image location of an object.
Small Vision System User Manual Figure 3-2. Inverse relationship between disparity and range. This plot is for a focal length of 6.3 mm, a baseline of 80 mm, and a pixel width of 14 mm.
Small Vision System User Manual 3.2 Horopter Stereo algorithms typically search only a window of disparities, e.g., 16 or 32 disparities. In this case, the range of objects that they can successfully determine is restricted to some interval. The horopter is the 3D volume that is covered by the search range of the stereo algorithm. The horopter depends on the camera parameters and stereo baseline, the disparity search range, and the X offset. Figure 3-3 shows a typical horopter.
Small Vision System User Manual Disparity: 5 4 3 2 1 0 -1 -2 -3 Figure 3-5. Planes of constant disparity for verged stereo cameras. A search range of 5 pixels can cover different horopters, depending on how the search is offset between the cameras. The location and size of the horopter depends on the application. If an object falls outside the horopter, then its true disparity will not be found, and instead it will get some random distribution of disparities.
Small Vision System User Manual 31 to arbitrary resolutions. These first three options change the camera geometry, and thus have a corresponding effect on the range resolution, which decreases (see below). The only way to increase the horopter size and maintain range resolution is to increase the disparity search window size, which leads to more computation.
Small Vision System User Manual 3.3 32 Range Resolution Often it's important to know the minimal change in range that stereo can differentiate, that is, the range resolution of the method. Give the discussion of stereo geometry above, it's easy to see that that range resolution is a function of the range itself. At closer ranges, the resolution is much better than farther ranges. Range resolution is governed by the following equation.
Small Vision System User Manual 3.4 33 Area Correlation Window Stereo analysis is the process of measuring range to an object based on a comparison of the object projection on two or more images. The fundamental problem in stereo analysis is finding corresponding elements between the images. Once the match is made, the range to the object can be computed using the image geometry. Area correlation compares small patches, or windows, among images using correlation.
Small Vision System User Manual Figure 3-7 Effects of the area correlation window size. At top is the original left intensity image. The greenscale images show windows of 7x7, 9x9, 11x11, and 13x13 windows (clockwise from upper left).
Small Vision System User Manual 3.5 Filtering Like most vision algorithms, the results of stereo processing can contain errors. In the case of stereo, these errors result from noisy video signals, and from the difficulty of matching untextured or regularly textured image areas. Figure 3-8 shows a typical disparity image produced by the SRI algorithm. Higher disparities (closer objects) are indicated by brighter green (or white, if this paper is printed without color).
Small Vision System User Manual 36 left/right check can be implemented efficiently by storing enough information when doing the original disparity correlation. In practice, the combination of an interest operator and left/right check has proven to be the most effective at eliminating bad matches.
Small Vision System User Manual 3.6 Performance Using standard PC hardware, running either MS Windows 95/98/2000/NT or Linux, the SVS can compute stereo range in real time. Table 3-1 gives some typical timings for a 500 MHz Pentium III processor. Because the Stereo Engine has a very small memory footprint, the timings scale almost linearly with increasing processor speed.
Small Vision System User Manual 38 4 Calibration Most stereo camera setups differ from an ideal setup in which the cameras are perfect pinhole imagers and are aligned precisely parallel. The divergence from ideal causes problems in the quality of the stereo match since epipolar lines are not horizontal. In addition, if the camera calibration is unknown, one does not know how to interpret the stereo disparities in terms of range to an object.
Small Vision System User Manual 4.1 Calibration Procedure An automatic calibration procedure using five image pairs of a planar calibration target is included as part of the smallv program. Given the image pairs of the calibration object, the system automatically locates corner features in the target, fits a model of the target to the images, and finally produces an estimate for the left and right camera intrinsics, the stereo head extrinsics, and the rectification matrices P0 , P1 , H 0 , and H1 .
Small Vision System User Manual Figure 4-4 Smallv calibration dialog window. 7. 4.1.2 c) Computes the rectification matrices from the joint calibration in (b). When the calibration is finished, the parameter listing at the bottom is updated, and you have a couple additional options in the lower button bar: save writes the parameter file to disk and OK exits the calibration dialog and transfers the new parameters to the main smallv window.
Small Vision System User Manual 41 The imager width is the number of sels (sensor elements) in each line of the imager. Again, this can be found in the imager specifications. If these are not available, just use the image width as it comes from the framegrabber, e.g., for NTSC video it is 640. The framegrabber width is the width of the image output by the framegrabber, in pixels. Typically it will be an analog NTSC signal, which is 640 pixels.
Small Vision System User Manual 42 5 API Reference To add stereo processing to your own programs, you call functions in the Stereo Engine library. These functions are available in svs.dll (Windows 95/98/2000/NT) or libsvs.so (Unix systems). The header file is src/svs.h. The current version of the library is 2.1. A simple example of the use of these functions is in the sample program samples/stframe.c.
Small Vision System User Manual 5.1 Stereo Parameter Structure The parameter structure svsSP holds information about the whole stereo process, from camera characteristics, the images input from the cameras, and the parameters involved in controlling the stereo process. Almost every function in the API refers to a parameter structure. By default, there is a global parameter structure svsParams in svs.dll that is used whenever the parameter structure is not specified in a function.
Small Vision System User Manual and double double double double double double 44 Z along the line of sight */ Tx; /* [mm] */ Ty; /* [mm] */ Tz; /* [mm] */ Rx; /* [rad] Yaw */ Ry; /* [rad] Pitch */ Rz; /* [rad] Roll */ unsigned long reserved; } svsSP; typedef struct { /* Intrinsic parameters */ int pwidth; /* [pix] Width of frame grabber's image */ int pheight; /* [pix] Height of frame grabber's image */ double dpx; /* [mm/pix] X dimension of pixel in frame grabber */ double dpy; /* [mm/pix] Y dimension
Small Vision System User Manual 5.2 Acquiring images from the video capture board The SVS library includes functions for interfacing to various video capture boards. These functions are written for each board, to present a standard interface. Several global variables are used to determine the parameters of video capture. 5.2.1 Video Capture Interface Libraries Video is captured through a framegrabber, either analog or digital.
Small Vision System User Manual 46 void svsVideoClose(void) Closes the connection to the video capture device(s). Usually not needed, since the connection will be closed when a program exits. However, it's nice to do this, and some drivers under MS Windows are less forgiving. 5.2.4 Capture Device Format Parameters These global parameters dictate the basic format for analog framegrabber devices. They should be set before calling the svsVideoOpen function. The default formats are for the STH-V3 system.
Small Vision System User Manual 47 All of the frame size parameters should be set after opening the stereo head with svsVideoOpen, and before returning a video stream with svsVideoStartContinuous. Setting the size parameters during streaming video will have no effect, until the streaming is stopped and svsVideoStartContinuous is called again. The exceptions are the offset and vergence of a subwindow, and color information, which can be changed during video streaming.
Small Vision System User Manual 48 If max_decimation is 1, then the stereo cameras don’t support explicit decimation. A value of 2 means the cameras can decimate by 2 horizontally and vertically, and a value of 4 means they can decimate by 2 or 4. For example, the MEGA-D has a decimation value of 4, which means that it supports image sizes of 1288 x 960, 640 x 480, and 320 x 240. Note that most analog framegrabbers support implicit subsampling, by setting the frame size.
Small Vision System User Manual 49 and manual gain. The shift from auto to manual is controlled by the value set on gain or exposure. Similarly, some color cameras can automatically balance the image colors (autowhite), and some cameras allow manual setting of color balance (manualwhite). Values of 1 indicate the capability is available on the cameras. int color If set to 1, then color information is returned by calls to svsVideoGetImage, if the stereo head supports it.
Small Vision System User Manual 50
Small Vision System User Manual 5.3 51 Continuous Capture In continuous capture mode, the capture device sends images to main memory at video rates. A double-buffering scheme lets user programs run asynchronously. When the user program wants a stereo pair, it requests it with the sequence svsVideoWaitImage - svsVideoGetImage. This will give the program a pointer to the most recent acquired stereo pair. The buffer is managed by SVS, so the application should not free it up after processing.
Small Vision System User Manual 5.4 52 Single Capture In single capture mode, the program requests a single stereo pair, and then waits until it is delivered. The sequence of function calls is svsVideoGrabSingle - svsVideoWaitImage – svsVideoGetImage. As in continuous capture, the frame size and digitization parameters are downloaded to the camera hardware before the acquisition. int svsVideoGrabSingle(svsSP *sp) Requests a single stereo pair be captured.
Small Vision System User Manual 5.5 53 Stereo Engine functions These functions compute a stereo disparity or confidence image from an input stereo pair. Stereo processing is affected by many parameters, such as the correlation window size, confidence filter threshold, etc. A parameter structure (svsSP) contains most of these; a few others are global variables. 5.5.1 Parameter Structure svsSP This structure is used by most Stereo Engine functions. It holds image size and stereo processing parameters.
Small Vision System User Manual 54 svsRIGHT svsLEFTCOLOR 5.5.4 Stereo Calculation These functions perform the stereo calculations. The svsCalcStereo function performs the full stereo calculation from intensity images, including pre-filtering, correlation, and post-filtering. It does not perform a warping step on the initial images to undistort them; to do this, first call the svsWarpImage function.
Small Vision System User Manual width RESTOP(sp) Left intensity image height RESLEN(sp) Disparity image height RESLEFT(sp) Dest image RESWIDTH(sp) width Figure 5-1 Spatial placement of the disparity image relative to the left input image. void svsCalcConfidence(unsigned char *dest, unsigned char *il, svsSP *sp) Compute the confidence and return it in dest. See svsCalcStereo for a description of the various arguments. 5.5.
Small Vision System User Manual 5.6 56 Host display functions These functions display images from the Stereo Engine in their proper format. As of SVS Version 2.0, display functions are derived from the FLTK cross-platform windowing system. All of the FLTK functionality is available in SVS, upon loading the FLTK shared library (libfltk.so or fltkdll.dll). Documentation on FLTK is available at www.fltk.org. class svsWindow : public Fl_Window Window class for display of input images or stereo results.
Small Vision System User Manual 5.7 57 Host file functions unsigned char *svsReadFile(char *name, unsigned char *buffer, int *height, int *width, int *num) This function reads from the SSI file given by the string name into the unsigned character buffer specified in buffer. If buffer is NULL, then a new one will be created, and is the return value of the function.
