Range Imaging
and
Visual Servoing
Range Imaging
and
Visual Servoing
for industrial applications
Proefschrift
ter verkrijging van de graad van doctor aan de Technische Universiteit Delft,
op gezag van de Rector Magnificus prof. dr. ir. J. T. Fokkema, voorzitter van het College voor Promoties,
in het openbaar te verdedigen op maandag 7 januari 2008 om 15:00 uur door
Frank DE JONG
natuurkundig ingenieur, geboren te Gorinchem
Prof. dr. ir. L. J. van Vliet Prof. dr. ir. P.P. Jonker
Samenstelling promotiecommissie:
Rector Magnificus, voorzitter
Prof. dr. ir. L. J. van Vliet, Technische Universiteit Delft, promotor Prof. dr. ir. P. P. Jonker, Technische Universiteit Delft, promotor Prof. dr. ir. J. van Eijk, Technische Universiteit Delft
Prof. dr. ir. J. J. M. Braat, Technische Universiteit Delft Prof. dr. ir. M. Steinbuch, Technische Universiteit Eindhoven Dr. ir. M. Brok, Assembl´eon
Dr. ir. M. de Bakker, European Patent Office
Dit werk is uitgevoerd in het kader van het Innovatiegerichte Onderzoekspro-gramma Beeldverwerking onder auspici¨en van SenterNovem.
De foto op de kaft is gebruikt met permissie van Assembl´eon.
Advanced School for Computing and Imaging Dit werk is uitgevoerd in onderzoeksschool ASCI. ASCI dissertatie serienummer 159.
de Jong, Frank
Proefschrift Technische Universiteit Delft ISBN 978-90-79365-01-2
Keywords: Range Imaging, Visual Servoing, Printed Circuit Board c
Contents
List of symbols ix
List of abbreviations xi
Summary xiii
Samenvatting xv
1 Computer Vision in Printed Circuit Board Manufacturing 1
1.1 Introduction . . . 1
1.2 PCB Manufacturing . . . 2
1.2.1 Bare board manufacturing . . . 2
1.2.2 Solder paste application . . . 3
1.2.3 Component placement . . . 4
1.2.4 Soldering . . . 5
1.3 Future requirements . . . 5
1.4 Project Goals . . . 5
1.5 Overview of this thesis . . . 6
I
PSD Range Imaging
7
2 Range Imaging Systems 9 2.1 Introduction . . . 92.2 Range imaging basics . . . 9
2.3 Industrial Range Imaging Techniques . . . 11
2.3.1 Requirements . . . 11
2.3.2 Passive Versus Active Techniques . . . 11
2.3.3 Triangulation . . . 12
2.3.4 Depth from Focus . . . 13
2.3.5 Scanning Techniques . . . 14
2.4 PSD Range imaging . . . 16
3.1 Introduction . . . 19
3.2 Previous Work . . . 19
3.3 Calibration of Scheimpflug Cameras . . . 20
3.3.1 The Scheimpflug condition . . . 20
3.3.2 Initial Considerations . . . 23
3.3.3 Modeling lens aberrations . . . 24
3.3.4 Calibrating cameras . . . 25
3.3.5 Vanishing points . . . 28
3.3.6 Vanishing Point Lens Calibration . . . 29
3.3.7 Scheimpflug Parameter Calibration . . . 33
3.4 Experiments . . . 34
3.4.1 Lens calibration on normal cameras . . . 34
3.4.2 Scheimpflug camera calibration . . . 44
3.5 Summary . . . 45
4 PSD Range Imaging Artifacts 47 4.1 Introduction . . . 47 4.2 PSD range imaging . . . 47 4.2.1 Basics . . . 47 4.2.2 From 1D to 2D . . . 49 4.2.3 From 2D to 3D . . . 49 4.2.4 Image formation . . . 50 4.2.5 Out-of-range signal . . . 52 4.3 Artifact Types . . . 53 4.3.1 Surface reflection . . . 53 4.3.2 Occlusion . . . 56 4.3.3 Shape discontinuities . . . 57 4.3.4 Reflectivity discontinuities . . . 60 4.3.5 Secondary reflections . . . 61
4.3.6 Other systematic errors . . . 66
4.4 PSD Simulator . . . 67
4.4.1 Overview . . . 67
4.4.2 Direct diffuse reflection . . . 67
4.4.3 Specular reflections . . . 70
4.4.4 Diffuse-specular secondary reflections . . . 70
4.4.5 Diffuse-diffuse secondary reflections . . . 73
4.4.6 Models . . . 75 4.5 Experiments . . . 75 4.5.1 Calibration . . . 76 4.5.2 Measurements . . . 76 4.6 Recommendations . . . 82 4.6.1 Reflection characteristics . . . 82
4.6.2 Dual sensor systems . . . 83
4.6.3 Sensor masking . . . 83
II
Visual Servoing for PCB Manufacturing
85
5 Component Placement Machine Design 87
5.1 Introduction . . . 87
5.2 Pick-and-place machines . . . 87
5.3 Attaining high accuracy . . . 89
5.4 Look-While-Place Design . . . 89 5.4.1 Mechanical design . . . 90 5.4.2 Frames of reference . . . 91 5.4.3 PCB artwork selection . . . 92 5.4.4 Dynamic behavior . . . 93 5.4.5 Image processing . . . 94
5.4.6 Camera system requirements . . . 94
5.5 Summary . . . 95
6 Fast Visual Matching to CAD Data 97 6.1 Introduction . . . 97
6.2 Visual matching to CAD data . . . 97
6.2.1 Global self-localization . . . 98
6.2.2 Local self-localization . . . 100
6.3 Visual Matching for Look-While-Place . . . 102
6.3.1 PCB features . . . 102
6.3.2 Finding Vias . . . 103
6.3.3 Finding Fiducials . . . 104
6.3.4 Finding Traces . . . 104
6.3.5 Exploratory Experiments . . . 106
6.3.6 Delay versus frame time . . . 109
6.3.7 CAD selection . . . 111
6.4 Visual Matching for RoboCup . . . 113
6.4.1 Soccer field features . . . 114
6.4.2 Two-Tiered Approach . . . 114
6.4.3 Global Search Algorithm . . . 115
6.4.4 Local Search . . . 117
6.5 Summary . . . 118
7 CMOS and Smart Sensors 119 7.1 Introduction . . . 119
7.2 System Setup . . . 119
7.2.1 Line versus Area sensors . . . 119
7.2.2 Feature detection stages . . . 120
7.2.3 Minimum pixel bandwidth . . . 120
7.2.4 Smart sensors . . . 120
7.3.2 Flashing . . . 121 7.3.3 Shuttering . . . 121 7.3.4 Number of photons . . . 121 7.4 Image grabbing . . . 122 7.4.1 Direct-read . . . 122 7.4.2 Integration . . . 122 7.5 Data transport . . . 122 7.5.1 Taps . . . 122 7.5.2 Windowing / ROI . . . 122
7.5.3 Coarse sensing / Binning . . . 123
7.5.4 Buffering . . . 123
7.6 Overview Camera Requirements . . . 123
7.7 Camera Overview . . . 123 7.8 Recommendations . . . 124 7.9 PhotoBit PB1024 . . . 126 7.10 Summary . . . 126 8 Look-While-Place Implementation 129 8.1 Introduction . . . 129 8.2 System Overview . . . 129 8.2.1 Hardware Design . . . 129
8.2.2 Vision Hardware Design . . . 132
8.2.3 Vision Software Design . . . 132
8.3 Coordinate systems . . . 134
8.3.1 Camera coordinate system . . . 135
8.3.2 Nozzle marker coordinate system . . . 135
8.3.3 Motion coordinate system . . . 135
8.3.4 PCB coordinate system . . . 135
8.4 Calibration . . . 136
8.4.1 Camera coordinate system calibration . . . 136
8.4.2 Nozzle marker calibration . . . 137
8.4.3 Motion controller coordinate system calibration . . . 139
8.5 Measurements . . . 140
8.5.1 PCB Measurements . . . 140
8.5.2 Nozzle measurements . . . 140
8.6 Details on methods . . . 141
8.6.1 Pose estimation . . . 141
8.6.2 Finding PCB Pose with Least-Squares Optimization . . . 142
8.6.3 Finding PCB track shifts . . . 142
8.6.4 Robust line fit . . . 143
8.6.5 Rulers . . . 144
8.7 Experiments . . . 146
8.7.1 Vision Software Results . . . 146
8.7.2 Motion-Vision Results . . . 147
Appendix A 155 A.1 Analysis of delay for a simple system . . . 155 A.2 Summary . . . 159
Bibliography 165
Curriculum Vitae 167
List of Symbols
Lens distortions (c, r) pixel coordinates
(C0, R0) distorted pixel coordinates
(C, R) undistorted pixel coordinates
(Cc, Rc) pixel coordinates of optical axis pixel
(Cv, Rv) pixel coordinates of vanishing point pixel
(Ckij, Rkij) undistorted pixel coordinates of point j on
line i through vanishing point k pixel
Pu undistorted image coordinates m
Pd distorted image coordinates m
r2 squared distance to optical axis pixel2
W aberration function
(X, Y, Z) world coordinates m
²kij distance of (Ckij, Rkijk) to line pixel
²2 sum of squared errors rad
κ1 first radial distortion parameters m−2
κ2 second radial distortion parameters m−4
θi line equation parameter rad
(θki, ρki) line parameters
PSD geometry
C1− C4 De Bakker’s triangulation parameters
-D1− D6 extended triangulation parameters
-(G, H, L) sheet-of-light coordinates m
H0 H coordinate corresponding to u = 0 m
Ilef t, Iright PSD currents A
Lpsd PSD strip length m
(u, v) PSD sensor coordinates m
Reflectance
Ei incoming power flux density (irradiance) Wm−2
Er reflected power flux density Wm−2
E(y) reflected power flux density (1D) Wm−1
Li incoming radiance Wm sr
Lr reflected radiance Wm−2sr−1
R(τ ) reflectivity function
δI sensor surface patch m2
δO scene surface patch m2
Ω solid angle sr
Φ incoming radiant power W
ρ albedo (in Lambertian reflectivity)
(θi, φi, θr, φr) spherical coordinates (BRDF) rad
LWP system design
Fa PCB artwork frame-of-reference
Fc camera frame-of-reference
Fe expected artwork frame-of-reference Fp pipette frame-of-reference
List of Abbreviations
ADC Analog to Digital Converter API Application Programming Interface APS Active Pixel Sensor
BGA Ball Grid Array
BRDF Bidirection Reflectance Distribution Function
CA Component Alignment
CAD Computer Aided Design CCD Charge Coupled Device
CFT Centrum voor Fabricage Technologie (Philips) CMOS Complementary Metal Oxide Semiconductor COG Center Of Gravity
DAC Digital to Analog Converter FA Fiducial Alignment
FPGA Field Programmable Gate Array FR Fibreglass Resin
JTAG Joint Test Action Group LED Light Emitting Diode
LS Long stroke
LWP Look-While-Place
LZ Landing zone
MRI Magnetic Resonance Imaging NMS Nelder-Mead Simplex PCB Printed Circuit Board
PCI Peripheral Component Interconnect PD Proportional Derivative (controller) PGA Pin Grid Array
PI Proportional Integral
PID Proportional Integral Derivative PIV Philips Industrial Vision PSD Position Sensitive Device RMS Root Mean Square ROI Region of Interest SMD Surface Mounted Device SRAM Static Random Access Memory
VGA Video Graphics Array VP Vanishing Point ZOH Zero Order Hold
Summary
High-tech systems play a key role in Printed Circuit Board (PCB) manufactur-ing. Two of these systems are presented in this thesis. The first system is range imaging, which is a measuring technique for determining three-dimensional shape. The second system is visual servoing, which is a method of adaptively controlling a mechanical system by acting upon measurements done with a com-puter vision system. Range imaging is used for quality inspection of the PCB during and after the manufacturing process. Visual servoing is used to quickly and accurately guide a component placement robot to the correct position on a PCB.
Part I (Chapters 2-4) of this thesis discusses range imaging. In Chapter 2, a general introduction to range imaging is given, while in Chapters 3 and 4, a special type of range imaging sensors is examined in detail: the Position Sensitive Device (PSD) range imaging sensor. Chapter 3 discusses ways of calibrating such sensors. Chapter 4 describes methods of predicting artifacts, i.e. systematic errors that occur under specific circumstances. Using this knowledge, recommendations for designing and using PSD range imaging sensors are given. In Part II (Chapters 5-8) of this thesis, the design and implementation of a vision-guided component placement robot is discussed. In Chapter 5, a general design for the entire component placement system is given, in which speed and accuracy are the two main design requirements. In Chapter 6, a design for the vision software is given. The software includes an offline (preparatory) part and an online (real-time) part. The offline part uses a PCB design description to compose a set of fast image processing operations. The online (real-time) part executes this set of operations to guide the component placement robot to the correct position within a very small time. The online vision part uses a pipelined architecture which, despite its high speed, introduces a non-negligible delay in the feedback loop. In Chapter 7, requirements for a vision sensor for the application are drawn up, and a particular sensor (the PB1024) is proposed. In Chapter 8, implementation details and experimental results of a working prototype component placement robot are presented.
Samenvatting
High-tech systemen spelen een belangrijke rol in printplatenfabricage. Twee van deze systemen worden in dit proefschrift gepresenteerd.
Het eerste systeem is dieptebeeldvorming, een meettechniek voor het bepalen van drie-dimensionale vorm. Het tweede systeem is visuele sturing, een methode voor het adaptief besturen van een mechanisch systeem door gebruik te maken van metingen van een computer-visie-systeem. Dieptebeeldvorming wordt ge-bruikt voor kwaliteitscontrole van de printplaat tijdens of na het fabricagepro-ces. Visuele sturing wordt gebruikt om een componentplaatsingsrobot snel en nauwkeurig naar de juiste positie op de printplaat te loodsen.
Deel I (Hoofdstukken 2-4) van dit proefschrift bespreekt dieptebeeldvorming. In Hoofdstuk 2 wordt een algemene inleiding tot dieptebeeldvorming gegeven, terwijl in Hoofdstuk 3 en Hoofdstuk 4 een bepaald type sensor voor dieptebeeld-vorming in detail wordt bekeken: de PSD dieptebeelddieptebeeld-vormingssensor. Hoofd-stuk 3 bespreekt de calibratie van deze sensoren. HoofdHoofd-stuk 4 beschrijft meth-odes voor het voorspellen van artefacten (systematische fouten die optreden onder specifieke omstandigheden). Gebruikmakend van deze kennis worden aanbevelingen gedaan voor het ontwerp en gebruik van PSD dieptebeeldvor-mingssensor.
In Deel II (Hoofdstukken 5-8) van dit proefschrift worden ontwerp en imple-mentatie besproken van een componentplaatsingsrobot welke door computer-visie wordt gestuurd. In Hoofdstuk 5 wordt een algemeen ontwerp gegeven voor het gehele componentplaatsingssysteem, waarin snelheid en nauwkeurigheid de drijvende ontwerpfactoren zijn. In Hoofdstuk 6 wordt een ontwerp gepresen-teerd voor de visie-software. De software bevat een voorbereidend deel en een operationeel deel. Het voorbereidend deel gebruikt een beschrijving van een printplaat om een set van snelle beeldverwerkingsopdrachten te formuleren. Het operationele deel voert deze set uit om een componentplaatsingsrobot binnen zeer korte tijd naar de juiste positie te leiden. Het operationele deel gebruikt een lopendebandarchitectuur waarin meerdere subtaken tegelijkertijd worden uitgevoerd maar welke ondanks haar snelheid toch een vertraging in de terugkop-pelingslus teweeg brengt. In Hoofdstuk 7 worden voorwaarden opgesteld voor een beeldsensor voor de toepassing en wordt een bepaalde sensor (de PB1024) aanbevolen. In Hoofdstuk 8 worden details van de uitvoering en experimentele resultaten gepresenteerd van een werkend prototype van de componentplaat-singsrobot.
Chapter 1
Computer Vision in Printed
Circuit Board
Manufacturing
1.1
Introduction
In today’s high-tech world, electronics have become an indispensable commod-ity. Especially in portable devices such as mobile phones, an increasing demand for higher levels of miniaturization and complexity with a simultaneous (but conflicting!) demand for decreasing production costs forces the electronics man-ufacturers to search for improved manufacturing processes.
In modern electronics manufacturing, the use of robots and computer vision has become the solution to these conflicting demands; the robots perform the work of assembling the electronics quickly and accurately, while the computer vision systems perform quality inspection at various stages of the production process and do measurements to guide the robots in their task.
For quality inspection, various types of vision sensors can be used, depending on the task at hand. For one type of inspection tasks, it is enough to verify the presence or location of a certain object on an intermediate product; for another type of inspection tasks, 3D measurements of (parts of) the intermediate product are needed. For the first type of tasks, usually a standard (2D) sensor system can be used, but for the second type of tasks, 3D vision sensors are required. These 3D vision sensors are the subject of Chapters 2 to 4.
The task of guiding the robots can be divided roughly into two types: the type in which measurements are made before the robot performs its task and which define the robot’s task beforehand, and the type in which measurements are made while the robot performs its task and which update the robot’s task while it performs it (so called real-time control). This real-time control is the subject of Chapters 5 to 8.
The remainder of this chapter gives an introduction to electronics manufac-turing, and the challenges and pitfalls involved with this complicated high-tech industry.
1.2
PCB Manufacturing
Most electronics consist of one or more processor chips and several other com-ponents, attached to a carrier surface. The carrier surface contains connectors for the components and electric wiring to interconnect the components.
Connecting the components to the carrier surface’s wiring is done mostly through soldering, i.e. melting a quantity of electrically conductive material between the component and the carrier surface and subsequently cooling the melted material, forming a stiff, electrically conducting bond between the carrier and the component.
In most cases, the carrier surface is a Printed Circuit Board (PCB), but other carriers also exist. In general, PCB manufacturing can be roughly divided into four stages:
1. Bare board manufacturing
2. Solder paste application (for some types of components) 3. Component placement
4. Soldering
To reduce manufacturing overhead costs due to errors in the early stages of the PCB manufacturing, each stage is ideally followed by an inspection step, where the result of the phase is checked for errors [1] [2] [3] [4] [5] [6].
Since the third phase is the most costly in terms of added value (the com-ponents being by far the most expensive parts of the PCB), inspection of the result up to just before the third phase is very important.
Instead of having an inspection step after component placement, it is also possible to have inspection during the placement itself, by adding a 2D or 3D vision camera to the component placement tool, which not only checks the placement result, but also helps in accurately placing the component relative to the PCB.
To allow for recycling of the more expensive components, a visual inspection step after the soldering phase can be useful before any electronic testing is carried out which may damage the expensive components.
1.2.1
Bare board manufacturing
In the bare board manufacturing phase, contact points for components and the wiring between those contact points (both mostly in copper) are created on a substrate. The classic PCB substrates are FR-4/5 (Fiberglass Resin laminate) and BT-epoxy. In applications with awkwardly shaped PCBs such as cameras
and mobile phones, the polymide flexible PCB (’Flex’) is increasing in market-share.
The steps of depositing copper wiring and contact points can be repeated several times, interleaving insulator layers and wiring layers. The resulting product is referred to as multi-layer PCB.
In the silk-screening step, one adds text labels (usually in white) to the board, which helps identifying components during and after manufacturing. As this layer is usually added in a separate production step from the copper wiring, the actual alignment between the silk-screen layer and the contact layers can have a relatively high imprecision.
To prevent melted solder from running off the component contact points and creating bridges between different wires during the soldering phase, top surface layers are often finished with a solder mask layer. The solder mask layer forms a barrier between adjacent contact points (pads), and completely shields off the wiring.
The creation of a bare board is finished with the drilling of holes for so-called through-hole components. Through-hole technology is the classic way of soldering components to the PCB board where the components signal leads stick through the PCB itself and are soldered on the backside of the PCB.
The actual design of the layers mentioned above is usually done with Com-puter Aided Design (CAD) tools, and the information about the layers is usually formatted in the industry-standard named Gerber. We will make use of these Gerber files in chapter 6.
1.2.2
Solder paste application
After the bare board is finished, the soldering material is applied to the bare board at the locations where components will be placed later on in the process. For the high-tech industry, most components will be Surface Mounted Devices (SMDs), which are components that are soldered on top of one of the surfaces of the PCB, as opposed to the through-hole devices mentioned in the previous section. In the case of SMDs, the solder is usually applied to the PCB with screening techniques, where the solder is pushed through a plate with tiny holes corresponding to contact point locations.
Since solder paste application is usually done in a separate manufacturing step, the alignment of the solder paste stencil with the PCB board itself requires extra measures to prevent misplaced solder dots. The solder mask mentioned in the previous section helps correct for slight misalignments (slightly misaligned solder flows back into the cup formed by the solder mask during the soldering phase).
For SMD technology, the amount of solder deposited is important; too large quantities can lead to short circuits after the melting phase, too small quantities will make for unstable contacts that may break easily under thermal or tensile stress. The measurement of volume and position of the solder paste dots is an interesting target for inspection with a 3D sensor.
Figure 1.1: Several popular integrated circuit packages.
1.2.3
Component placement
After the solder paste is applied, the components are placed. Depending on the component’s packaging, different techniques are used to do the actual component placement. Figure 1.1 gives an overview of the more common packages used in today’s industry.
The most common way of automatically placing components on a PCB is done with air-pressure pipetting. In this technique, components are picked up with a small vacuum tube; the low pressure in the tube causes the components to stick to the tube until it is placed on the board, and the pressure is changed to high pressure.
For current automatic placement machines, rates of several component place-ments per second are not uncommon.
With all of these packaging methods, it is important for the component’s contact points to connect precisely to the contact points on the board. So for any machine placing a component it is imperative to know the leads’ position with respect to the board. For so-called leaded components (named for the contact leads sticking out of the package), aligning the leads with the board can be done directly using vision systems that observe both the board and the leads simultaneously. For contact grid components such as Ball Grid Arrays (BGA, see Figure 1.1) where the leads are underneath the package, one must first determine the location of the leads with respect to the package, and then place the package precisely with respect to the board. This indirect method is
also often used for leaded packages, since the optimal illumination conditions for observing leads and PCB are different.
1.2.4
Soldering
For SMD technology, in this phase, the solder paste is melted, forming bonds between the PCB’s contact points and the components.
For through-hole technology, one employs wave or flow soldering, where the board is placed just above a surface of liquid solder, and a wave is passed through the solder in such a way that the top of the wave touches the board, leaving behind solder on the uncovered metal surfaces.
The heat applied during the soldering phase can cause a number of problems. First, the PCB can deform due to thermal stress between the layers. Second, if the solidifying time is not uniform across the PCB, simple two-terminal com-ponents such as resistors can exhibit tomb stoning. This effect can occur when the not yet solidified end of the resistor is pulled upwards (away from the PCB) due to solidification stress on the other, solidified end.
1.3
Future requirements
Due to the simultaneous demand for increased miniaturization and increasing number of I/Os per chip, chip packages that have leads along the edges, such as TSOP (see Figure 1.1) will have to be replaced more and more by packages that have leads on the bottom such as Ball Grid Array (BGA) and Pin Grid Array (PGA). Even with grid array packages, the distance between the center of the leads, the lead pitch, will have to go down from current values of 300 µm to 150 µm or even 100 µm. Obviously, this presents a formidable challenge for the machines that have to place the components with this accuracy.
The challenge is increased by the speed requirements. Component placement speeds of up to 20,000 components per hour are not uncommon, which roughly translates to 5 components per second.
1.4
Project Goals
To meet the requirements, a cooperation project between the Delft University of Technology, the Dutch governmental organization IOP Beeldverwerking and several Dutch industrial partners (notably Philips CFT1 and Philips EMT2)
was started in early 1997, to develop innovative techniques for use in the PCB industry. In this project, the PCB manufacturer Philips EMT was the problem owner, the Delft University of Technology and Philips CFT were the research partners, and IOP Beeldverwerking[7] was the project sponsor.
1Now operating under the name Philips Applied Technologies. 2Now operating under the name Assembl´eon.
At the start of the project, the aim of the project was to develop a 3D vision system that could be used for several stages of PCB manufacturing:
• Inspection of component glue dots
• Component placement under guidance of 3D sensor information • Post-solder inspection of placed components
By the time the project actually started, Philips had already designed and created a machine that could do glue dot inspection with 3D sensors, which elim-inated the scientific and engineering challenges for that subgoal of the project.
Furthermore, it became clear early in the project’s lifetime that doing com-ponent placement under guidance of 3D sensor information as well as doing post-solder inspection of placed components would be too complex a task for the project’s limited lifetime, and so the goals were redefined as follows:
• Research into accuracy and applicability of 3D sensors
• Component placement under guidance of 2D sensor information
1.5
Overview of this thesis
This work deals with the most important issues inherent to the challenges men-tioned in the previous sections. The remainder of this work is split up into two major parts corresponding to the redefined project goals, which can be read independently.
In Part I (Chapters 2-4), we will look at 3D sensors, and in particular PSD range sensors. In Part II (Chapters 5-8), we will look at systems that are capable of high precision, high speed placement of components under guidance of 2D vision.
Part I
Chapter 2
Range Imaging Systems
2.1
Introduction
Many people have experienced the disappointment of looking at panoramic holiday photographs that fail to command the sense of awe that the original panorama evoked. Most of this disappointment can be ascribed to the fact that a photograph makes a projection from 3D to 2D, causing the loss of information about distances to points (and therefore, the size of objects in the scene). Range imaging is a general term for a class of imaging techniques that do not exhibit this loss of distance information.
This chapter describes the properties of range imaging, and gives an overview of existing systems. A particular range imaging solution (PSD triangulation) is explained in more detail, hinting at the challenges that come with PSD trian-gulation (the subjects of Chapter 3 and Chapter 4).
2.2
Range imaging basics
A range image is a set of distance measurements from a sensor to points on one or more objects in a scene.
The distance measurements can be done in many different ways. It can be as simple as laying a yard stick from an object to a sensor, and having a sensor read out the value on the stick. A more sophisticated variant of this technique carefully lowers a mechanical probe until it meets any resistance.
It is also possible to count the time elapsed between the sending of a signal and the receiving of an echo (sonar, radar). These methods are referred to a
time-of-flight methods.
Another possibility is to use the difference between the measurements of two optical sensors that are separated by some distance - this is one of the mechanisms used by humans to perceive depth. These methods are commonly grouped under the name stereo vision methods.
Figure 2.1: Two common range image representations. On the left, the 3D represen-tation; on the right, the corresponding 2D range image.
Yet another possibility is using active triangulation (explained below). For an extensive overview of range imaging techniques, the reader is referred to Jarvis [8], Besl [9], and De Bakker [10].
There are also different ways in which the set of distance measurements is structured, geometrically. In industrial inspection, range measurements are often done along a line perpendicular to a conveyor belt that moves objects through the field-of-view of the sensor. In navigation, range measurements are mostly done in continuous circular sweeps (e.g. sonar and radar). In traffic surveillance, range measurements are done by police officers by simply aiming a ”laser gun” at a car for a while - measuring the speed of that car by the rate of change in distance.
Although range imaging is often referred to as a 3D imaging, the term is not really appropriate for most range imaging systems, where the measurements give only the distance to the object surface closest to the sensor; any objects behind that first surface are not measured at all. In contrast, techniques like mag-netic resonance imaging (MRI) actually do perform 3D measurements, thereby acquiring a 3D measurement volume. Since range imaging provides higher di-mensional information than strict 2D techniques such as photography, but less than the aforementioned 3D techniques, it is often referred to as 2.5D imaging. Figure 2.1 shows two representations of a range image. On the left, the 3D representation is shown which gives insight in the shape of the object; on the right, the range measurements are displayed as a grey level image with lighter shades indicating a shorter distance to the sensor. The latter representation is more appropriate when standard 2D image processing techniques are used for tasks such as noise suppression, object recognition, etc.
Figure 2.2: Active illumination. The illuminated object contour (thick line) is mea-sured.
2.3
Industrial Range Imaging Techniques
2.3.1
Requirements
As already noted, there are many ways to do range imaging. This work focuses on those range imaging techniques that are suitable for industrial inspection applications. In these applications, range imaging usually has to be done while satisfying the following demands: a) contactless (to remove risk of damaging), b) high speed c) high accuracy, and d) low cost.
Although it is not possible to give a single figure for the resolution ranges that industrial inspection applications require (due to the diversity of the field), the resolution range for inspection of PCBs is roughly between 10 µm and 100 µm. This range prohibits the use of time-of-flight methods which are typically used for resolutions in the order of centimeters or more.
2.3.2
Passive Versus Active Techniques
The demand of contactless measurements is satisfied by using optical methods. We divide the optical methods into two main classes by distinguishing between the methods that do not use special illumination on the objects of interest (passive techniques), and those that do (active techniques).
The previous section mentioned one example of a passive technique, stereo vision. The main problem with this technique is that corresponding points in two sensor images have to be matched before the distance to the source point can be computed (the correspondence problem). This matching step can be computationally expensive, limiting the speed that can be attained. To alleviate the computational load of the matching process, it is possible to track points of interest in both sensor images once the correspondence has been found. Unfortunately, even with this tracking technique, the industrial applicability is still quite limited.
C A B α β Figure 2.3: Triangulation.
Active methods apply (structured) illumination to the scene to select sub-spaces of the 3D space, eliminating correspondance problems. An example of an active method is given in Figure 2.2; by illuminating an object with a sheet of light and observing the illuminated object contour from outside the sheet’s plane, the 3D location of the contour points can be found unambiguously: all points lie in the plane of the light sheet, and the 2D projection of the observed contour defines the position within the plane. For instance, in Figure 2.2, the more a contour point lies to the left, the closer it is to the light source.
2.3.3
Triangulation
Both stereo vision and the active illumination method of Figure 2.2 are actually forms of triangulation range imaging. In triangulation, the distances AC and
BC can be found if the distance AB between two other points A and B and
the angles α and β are known (Figure 2.3).
In stereo vision (Figure 2.4, left), A and B correspond to the sensors, and
C is the focus position. As C moves to the point C0, its projections move to A0 and B0 on the sensors. However, the distance A0B0 and the angles α0 and β0 can be calculated from the stereo system geometry, so that the distance to
point C0 is still unambiguously determined.
In laser triangulation (Figure 2.4, right), a beam of light is sent along line
AC, and the point at which the beam hits the object (C0) is projected on a
sensor at point B0. Again, α and β and the distance AB0 are known, so that
the position of C0 is unambiguously determined. The line AC0 in this figure
corresponds to the dotted line in Figure 2.2.
The actual geometry of laser triangulation systems can be tuned to the reflection properties of the object. For diffuse objects, it customary to use a laser beam direction perpendicular to the transport direction of the objects to be measured, and a sensor at a medium angle to the laser beam (6 (AC, BC) =
30-60o).
For highly reflective surfaces, it is also possible to have the laser beam and sensor at equal but opposite angles θ relative to the expected surface normals. As long as smooth and mostly planar surfaces are used, the main reflection will
C A B α β C' B' β' A' α' C A B α β C' B' β'
Figure 2.4: Two triangulation range imaging techniques: left, stereo vision; right, laser triangulation.
sample grid
Figure 2.5: Irregular object sampling grid on 45o degree beam direction.
be caught in the sensor [11]. An important disadvantage of this method is the fact that a range scan of an object on a regular grid in X and Y direction does not lead to a regularly sampled object because the measurement direction partly coincides with the X and Y directions (Figure 2.5). This irregular sampling complicates the use of standard image processing techniques (which assume regular sampling grids).
2.3.4
Depth from Focus
Another method to perform range imaging is called depth from focus. In depth from focus techniques, one makes use of the Gaussian lens law for thin lenses:
1/f = 1/LI+ 1/LO; (2.1)
which specifies that for a lens with focal distance f , an object at distance LO
from the lens will have a sharp image at distance LI from the lens. By finding
the sensor position LI along the optical axis that produces a sharp image, one
knows the distance LO to the object.
In Figure 2.6, a depth from focus system is shown that was used in industrial inspection of PCBs [2]. This particular system uses a telecentric optical system
pupil detector intensity light beam
Figure 2.6: Depth from focus with parallel sensors.
with partly transparent mirrors to sample the image ’sharpness’ at several image distances (b) in parallel.
An illuminated point casts images with different degrees of defocus on a pinhole in the exit pupils of the individual lenses. The sharper the image of the point, the more radiation energy passes through the pinhole and reaches the detector. If the transparency factors of the mirrors are chosen correctly, matching a Gaussian profile through the measured intensities yields a potentially high degree of range accuracy.
2.3.5
Scanning Techniques
Implementation of the discussed depth from focus and laser triangulation tech-niques is relatively easy when the range measurements are made one at a time
R L
H
x
Figure 2.7: Polygon mirror scanning.
(single point measurements). When dealing with industrial applications, how-ever, usually a rectangular grid of point measurements is required, so that entire objects can be inspected.
In industrial applications that make use of transport systems such as con-veyor belts, one dimension (say, Y) of the rectangular grid is formed by time multiplied by the transport speed. This requires a one-dimensional range sensor for the X dimension; the point measurement needs to be extended along a line perpendicular to the transport (Y) and range (Z) directions.
One technique that is used often in industrial applications makes use of a rotating polygonal mirror (Figure 2.7). As the polygonal mirror rotates from the position indicated by the dotted lines to the position indicated by the solid lines, the indicated light beam intersects with the current mirror segment at increasing angles. Consequently, the reflection angle also increases, causing the beam to intersect the object surface at decreasing x position.
When the beam goes from one mirror segment to the next, the reflection angle changes abruptly, and the object intersection point jumps back to the right. The sensor observes the object intersection through the same polygonal mirror system, so the actual range sensor can be the simple point measurement sensor.
Note that because of the non-perpendicular irradiant beam, this system suf-fers from irregular surface sampling (see also Figure 2.5). Depending on the number of mirrors, additional high-order terms are introduced in the depen-dency between x and the rotation φ (which usually increases at a constant rate). Figure 2.8 shows the functional behavior of x(φ), scaled by H, for a number of ratios between H and R, for the setup shown in Figure 2.7.
By increasing the number of mirror segments, the system operates in a smaller φ domain, and the non-linearity between x and φ can be reduced to within specifications. Alternatively, a non-regular time sampling can be used to
-0.4 -0.2 0.2 0.4 φ -0.75 -0.5 -0.25 0.25 0.5 0.75 x / H 10.0 5.0 1.5 1.1
Figure 2.8: Position x on the surface, scaled by H, as a function of rotation angle φ (in rads) for different ratios of stand-off height H and polygon radius R.
remove any higher-order terms.
Instead of ”scanning” the laser beam with polygon mirrors (or otherwise), it is also possible to increase the dimensionality of the entire range imaging system: instead of using a single system such as in Figure 2.4 (right), an array of N of these systems (in the direction perpendicular to the paper) is used to make N parallel measurements.
To save complexity and cost, one lens and one laser beam can be used in conjunction with an array of sensors. To increase the beam’s dimensionality, it is possible to shape the laser beam into a sheet of light (as in Figure 2.2).
One simple way to do this is with cylindrical lenses, but because the resulting beam intensity profiles are Gaussian along the sheet, there is a loss of sensitivity away from the center, which is not very desirable for industrial applications.
Fortunately, commercial optical systems exist [12] that can generate more or less constant intensity profiles.
2.4
PSD Range imaging
One special form of laser triangulation sensors are so-called position-sensitive
devices (PSDs). These sensors are composed of one or more strips of
photo-sensitive material that output two photo currents on their longitudinal ends. The magnitude of these photo currents is determined by the incoming light distribution: the closer the incoming light is to a particular longitudinal end, the higher the photo current on that end with respect to the photo current on the other longitudinal end.
Assuming that the incoming light from a single scene point is sufficiently focused onto a single sensor position, the ratio between the photo currents can be used to calculate the incoming light intensity distribution’s center-of-gravity
position on the PSD. Because it is the ratio between photo currents that deter-mines this center-of-gravity position, PSDs perform well over a wide range of light intensities.
Since PSD sensors can be implemented entirely in hardware, they are very fast (typically in the order of MHz) and therefore very interesting for industrial inspection.
Apart from being fast, PSDs also score well on the other requirements for industrial inspection; since they can be implemented in simple hardware pro-cesses [10] they are relatively cheap, and dynamic ranges of 1:1000 (for the range measurement) are not uncommon for PSD sensor systems.
The above-mentioned assumption of ”sufficient focusing” is, however, quite an important one as PSDs cannot distinguish between a single well-focused spot, several spots, or one or more out-of-focus light distributions. As we will see in Chapter 4, PSD range sensors exhibit several errors (so-called artifacts) under special conditions, most of which can be led back to violations of the focusing assumption.
2.5
Summary
In this chapter, we have presented a number of range imaging techniques. We have shown the industrial relevance of the laser triangulation technique, and discussed methods to increase the dimensionality of this technique so that they can perform two-dimensional scans on industrial parts.
A case was made for PSD triangulation range imaging, because of its good characteristics for industrial inspection applications, although it was noted that the assumption of ”sufficient focusing” of the laser spot or sheet-of-light may not always be valid, introducing errors in the range measurement.
In Chapter 3, we will see how to calibrate a special setup of a PSD system that ensures proper focusing under normal conditions. In Chapter 4, we will see how geometric and reflection properties of the objects to be measured can introduce errors in the range measurement result.
Chapter 3
Triangulation System
Calibration
3.1
Introduction
Calibration of 3D laser triangulation sensors is of utmost importance if accu-rate measurements are required. Factors such as lens aberrations and camera misalignments can influence absolute range measurements (location) and even relative range measurements (size) of objects considerably. Especially in the case of cheap equipment, pixel shifts due to these effects may be in the order of 1% (or more) of the sensor dimension, while accuracy requirements are usually in the order of 0.1% (or less) of the sensor dimension.
By modeling the aberrations and calibrating the model parameters, we may reduce the errors to acceptable levels. Obviously, higher demands on the accu-racy usually imply increasing model and calibration complexity.
This chapter discusses the relevant parameters in camera models, and comes to a proposed calibration routine that can calibrate a triangulation range imag-ing camera with high accuracy, even if consumer-grade optic components are used.
3.2
Previous Work
Most of the previous work in the field of camera calibration for computer vision starts with a model of a pinhole camera and then adds complexity between the pinhole and the sensor plane to model the lens aberrations.
A distinction is made between intrinsic and extrinsic parameters. The in-trinsic parameters define the camera’s optical system (lens focal distance, lens aberrations, sensor size, etc); the extrinsic (or external) parameters describe the relative position and orientation of the calibration object(s) with respect to the camera.
A definite milestone is set by Tsai’s method [13]. In Tsai’s method, the sensor plane is assumed to be perpendicular to the optical axis, and lens aberrations are assumed to be limited to radial distortions only. Skews in the sensor plane (which arise when the pixel row and pixel column axes not perpendicular) are modeled, as well as scaling in the horizontal direction due to frame grabber re-sampling of the analog video signal. The calibration procedure is done using a single planar dot pattern assumed to be on Z=0, or using a set of non-coplanar yet well-defined world points. Tsai’s method computes the parameters by alternatingly solving the external and intrinsic camera parameters in least square error sense.
The disadvantages of Tsai’s early method are that certain intrinsic parame-ters of the sensor have to be set in advance by the user, and obtaining accurate data points in the non-coplanar form is relatively hard.
Zhang [14] extended Tsai’s work, and discovered a way to combine a number of non-coplanar views of a single flat calibration object, and find the external and intrinsic camera parameters in closed form. The method has a significant practical advantage over Tsai’s method, because the observed orientations of the grid pattern do not have to be known or set beforehand, and can be anything as long as the set of grid patterns are sufficiently non-coplanar.
More recently, Devernay & Faugeras [15] and Perˇs & Kovaˇciˇc [16] have noted that to ease the task of calibration, we can use the fact that straight lines remain straight under perspective transformation. We will use this information later in this chapter.
3.3
Calibration of Scheimpflug Cameras
3.3.1
The Scheimpflug condition
A factor not incorporated in the previously mentioned calibration methods is the fact that triangulation setups usually obey the Scheimpflug condition [17]. This condition, named after the Austrian Theodor Scheimpflug, states that an object plane O0 that is not perpendicular to the optical axis, is focused sharply
in an image plane I0 if and only if the lens plane, image plane and object plane
intersect on a single line (the so-called Scheimpflug line, Figure 3.1).
The relevance of this condition to range imaging becomes clear when we realize that in ideal laser triangulation systems only points in the sheet-of-light are imaged onto the sensor. If we want every point in the sheet-of-light to be in focus simultaneously, the sensor plane must be tilted with respect to the lens plane.
For range imaging with a single laser beam (instead of a sheet-of-light), the Scheimpflug condition translates to a setup in which the laser beam line, lens line and sensor line intersect in a single point.
If the sensor plane is positioned in accordance with the Scheimpflug condi-tion, and undergoes a significant tilt relative to the lens plane (angle α), we will observe a distortion as illustrated in Figure 3.2.
image plane object plane of sharp focus lens plane lens Scheimpflug line optical axis
Figure 3.1: The Scheimpflug condition; image plane, object plane and lens plane intersect in a single line.
u v H Y sensor sheet of light * *
Figure 3.2: Adhering to the Scheimpflug condition distorts the square in the laser plane to the truncated pyramid in the sensor plane.
If we model the Scheimpflug camera as a standard pinhole camera without radial lens distortions, then the same distortion effect can be ascribed to a rotation of the camera over an angle α around the optical center, as is shown in Figure 3.3.
f f ' α β ζ side view top view S S'
Figure 3.3: The Scheimpflug camera as a rotated normal camera.
projected through the pinhole onto the sensor plane, which has an angle α with respect to the lens plane. If we introduce a virtual optical system S0, rotated
around the pinhole by angle α, and indicated by the dashed lines, its sensor plane is again perpendicular to the optical axis.
Compared to the original Scheimpflug system, the new virtual system be-haves like a system with shortened effective focal length (f < f0), and rotated
relative to the sheet-of-light by an angle ζ = α + β. The focal length foreshortening is simply:
where f is the effective focal length of the Scheimpflug lens.
Note however that the virtual optical system S0 has two differences with
a normal camera system: the sensitive sensor area is not centered around the optical axis, and the focal plane is not perpendicular to the optical axis. Neither of these differences influence the image formation model in the case of pinhole cameras.
Although this observation seems to imply that we can calibrate a Scheimpflug camera with existing techniques, the radial distortion actually breaks the simple correspondence between S and S0 which makes the existing techniques
inaccu-rate. So what we need is a method to calibrate the lens radial distortions first, and then find the camera rotation and translation. The following sections will describe such a method.
3.3.2
Initial Considerations
To better understand the errors involved in triangulation measurements, we start with the basic triangulation setup as depicted in Figure 3.4. To calibrate the 3D region of interest (ROI), we have to put a certain well-defined calibration object in the ROI (see Van den Ouden [18]), or alternatively, move it through the ROI in a very precise way. During the movement, we measure the sensor values for the reflected laser, and then fit a model to the measured sensor values and the coordinates of the calibration object.
The actually measured values are dependent on many factors, the most ob-vious of which are:
• triangulation parameters, for instance a, α, and β • reflected light intensity
• sensor sensitivity
but also on some less obvious factors, which come into play when high accuracies are required:
• degree of planarity of the laser sheet
• laser intensity distribution (usually assumed Gaussian) • directional light reflection behavior (BRDF) of the object • amount of background illumination
• lens distortions and out-of-focus effects • sensitivity range of the sensor
laser
beam
β α a Z X optical axis θ H G sens or u lens Z0 bFigure 3.4: Triangulation parameters and coordinates. The Y axis is pointing out toward the reader, and the v axis is pointing away from the reader.
Note that, although light travels on a straight path, for both single and multi laser stripe measurements, there is no guarantee that the projected stripe is straight (not even in object space!) since it is optically formed from a single beam, and the beam splitting optics introduces errors of its own.
The other factors that we have mentioned above can influence the measure-ments during calibration significantly, but since calibration usually takes place in a controlled environment, we will assume that we can reduce their influence to negligible levels.
3.3.3
Modeling lens aberrations
Lens aberrations arise from inaccuracies in the fabrication of the lens system, such as the deviation of the shape of a lens from a perfectly spherical shape to some higher order shape, misalignment of lens components, or number of lenses. The result of these imperfections is that a wave front that emanates from a certain point in space is not precisely focused in one point in the image plane, but is moved or spread out over a three-dimensional volume. By studying these wave front aberrations in the exit pupil of the lens, one can discriminate various types of errors.
In the middle of the 19th century, Seidel and Petzval investigated these effects by expanding Snell’s refraction law to a third order polynomial and
dis-carding higher order terms. Using this approximation, an aberration function
W can be defined for a quasi-spherical wavefront emanating from the exit pupil
towards a paraxial image point. The function quantifies the deviation from a perfectly spherical wave front, and has the form of a Taylor series:
W (x, y; x0, y0) =
X
αijklxiyjxk0y0l (3.2)
where (x, y) are exit pupil coordinates and (x0, y0) are (paraxial) image coor-dinates. After limiting the scope to optical systems with rotational symmetry and rewriting to polar coordinates (ρ, θ) and (ρ0, θ0), only specific terms remain (see Braat [19] or Born & Wolf [20]):
W (ρ2, ρ2
0, ρρ0cos(θ − θ0)) =
X
βijk(ρ20)i(ρ2)j(ρρ0cos(θ − θ0))k (3.3)
Choosing coordinate systems such that θ0 = 0 and discarding higher order
terms, 3 paraxial terms and 5 types of aberrations remain, which are listed in Table 3.1.
i, j, k Aberration Effect
1 0 0 Piston Phase shift
0 0 1 Tilt Magnification changed
0 1 0 Focus Image plane shifted
1 1 0 Field curvature Image plane curved
1 0 1 Distortion Barrel and pincushion distortion 0 1 1 Coma Defocus and asymmetric distortion 0 2 0 Spherical Defocus along optical axis
0 0 2 Astigmatism Different focus for x and y Table 3.1: Paraxial and Seidel Aberrations.
For today’s low-end market lenses, all the effects except radial distortion are sufficiently small (Tsai [13]) or can be neutralized by reducing the diaphragm opening (the errors get worse with increasing distance from the optical axis). Therefore, we may limit ourselves to modeling the radial lens distortions only.
3.3.4
Calibrating cameras
We will now have a detailed look at some existing calibration algorithms. Polynomial approximation
If the camera is assumed to be stationary with respect to a certain measurement plane that is roughly perpendicular to the camera’s optical axis, a simple, yet effective correction for lens distortions can be obtained by expressing each mea-surement plane point (Xp, Yp) as a polynomial function of the measured pixel
coordinates (c, r):
Xp = fx(c, r, ~p) (3.4)
Yp = fy(c, r, ~q) (3.5)
where ~p and ~q are vectors of polynomial coefficients.
By measuring an image of a high-accuracy calibration grid (for instance a checker board pattern), finding the grid points’ pixel coordinates (c, r), and linking them to (Xp, Yp) coordinates, we can optimize the polynomial coefficients
with a standard numerical optimization package.
For approximate alignment of the measurement plane’s axes and the cam-era’s pixel axes, and moderate lens aberrations, the effects of magnification, rotation, radial distortion and perspective distortion can be collected into one polynomial [21] of the form:
fx(c, r, ~p) = p0+ p1c + p2r + p3c2+ p4rc + p5r2+ p6r2c + p7c3 (3.6) fy(c, r, ~q) = q0+ q1r + q2c + q3r2+ q4cr + q5c2+ q6c2r + q7r3 (3.7)
Note that the polynomials are not ’complete’; some cross-terms are skipped because of the approximate alignment of axes. Less terms can also be used, depending on the required accuracy.
A disadvantage of this model is that the coefficients ~p and ~q do not directly correspond to the physical entities of the imaging system.
Tsai
According to Tsai, a camera system has 5 intrinsic camera parameters: effective focal length, radial distortion parameter of the lens, frame grabber scaling, and point of intersection of the optical axis and the sensor grid (row, column).
For a detailed description of the parameters, the reader is referred to the original paper, but a short review is given here1.
In Figure 3.5, we see how point P = (Xp, Yp, Zp) is projected onto the image
plane, which is at distance f (the effective focal length) from the pinhole origin. This yields the undistorted image coordinates Pu= (Xu, Yu):
Xu = f Xp/Zp
Yu = f Yp/Zp (3.8)
Starting from pixel coordinates (c, r), if we convert the pixel coordinates to image plane coordinates relative to the optical center, then correct for frame grabber scaling in the X direction and finally correct for radial distortion, this yields for Xu and Yu:
1Note that some variables have been renamed to clarify the distinction between pixel coordinates and world coordinates
P Pu X Y Z c r Pd f
Figure 3.5: Pinhole model projection of point P to undistorted image coordinate Pu,
and subsequent radial distortion to image coordinate Pd.
Xd = s−1x d0x(c − Cc) Yd = d0y(r − Rc) r =pX2 d+ Yd2 Xu = (Xd)(1 + κ1r2) Yu = (Yd)(1 + κ1r2) (3.9)
where (c, r) are the distorted pixel coordinates, (Cc, Rc) are the pixel coordinates
of the optical axis intersection, d0
x and d0y are the pixel sizes, sx is the frame
grabber scaling, and κ1 the radial distortion parameter.
Having two sets of expressions for Xuand Yu(Eqs 3.8 and 3.9), Tsai obtained
a way to go from an approximation of the external parameters to an approxima-tion of the intrinsic parameters (and vice versa), iteratively converging toward a solution.
Due to the Scheimpflug condition, however, we have an additional parameter
α for the sensor tilt, which can not be easily inserted into the Tsai solving
method, because it acts both as a rotation (as we concluded from Figure 3.3 in the previous section) and as an influence to the pixel coordinates after the radial distortion is applied.
De Bakker
De Bakker [10] modified Tsai’s calibration routine by putting the correction for the sensor plane tilt inside the intrinsic parameter step, but this required a priori knowledge of the sensor tilt. Although a general indication of the sensor tilt can usually be given, we would like to have a calibration routine that actually calibrates the sensor tilt as well.
vanishing points
Figure 3.6: Vanishing points for perspective-transformed, parallel lines.
To see how we can find the intrinsic parameters including the sensor tilt, we need to introduce the concept of vanishing points.
3.3.5
Vanishing points
Anyone who has ever seen a field that has just been plowed knows that the lines of the plow marks, which are parallel, nevertheless seem to go to a single point on the horizon. The effect is referred to as perspective, and the point is referred to as the vanishing point (VP), or point at infinity. Although the effect is most readily observed for a single pattern of long lines, in fact all features on parallel lines deform in this way towards a single VP under perspective transformation, as is illustrated in Figure 3.6: not only the lines coinciding with the grid lines meet in a single point (left and right top), but also the lines which are diagonal to the grid lines meet in one point (center top).
A vanishing point can end up at infinity when the object plane is exactly parallel (in one or two directions) to the sensor plane. For instance, in Figure 3.6, horizontal lines crossing the rectangular segments diagonally remain (almost) parallel under perspective transformation.
The interesting thing to note here is that under the Scheimpflug condition, the focal plane is never parallel to the sensor plane. Therefore, we are guaranteed to find vanishing points at finite distance for line patterns that are not parallel to the v axis.
So if we use a calibration object which has points on parallel lines, these lines must intersect at one point, and all non-linearity in the sensor image of these lines is a direct consequence of radial distortion.
Therefore, we can solve the intrinsic parameters of the camera indepen-dently of the exact value of the 6 external parameters and indepenindepen-dently of the
Scheimpflug angle α as well. Instead, we only determine, for each set of parallel
lines, the image position of the vanishing point (just 2 parameters). Once the intrinsic parameters are known with sufficient precision, the external parameters can be found with the first stage of Tsai’s method.
3.3.6
Vanishing Point Lens Calibration
We will now give a detailed description of the algorithm for the calibration using vanishing points.
If we denote the undistorted image coordinates of the jth point on the ith line through vanishing point (Cv, Rv) by (Cij, Rij), then the lines can be described
by the equations:
²ij = (Cij− Cv) cos θi+ (Rij− Rv) sin θi = 0 (3.10)
where θiis the line equation parameter for the ith line.
If we assume only radial distortions that are centered around image coordi-nate (Cc, Rc), and drop the subscripts and superscripts for a moment, we can
express the undistorted parameters (C, R) in terms of the distorted (measured) coordinates (C0, R0) as follows:
C = Cc+ (C0− Cc)(1 + κ1r2+ κ2r4) (3.11) R = Rc+ (R0− Rc)(1 + κ1r2+ κ2r4) (3.12) r2= (R0− R
c)2+ (C0− Cc)2 (3.13)
where κ1,2 denote the first and second radial distortion parameters.
The error we wish to minimize to find the optimal parameters is simply the sum of squared errors from the line equation given above:
²2=X ij
(²ij)2 (3.14)
We can improve our results in a straightforward way by combining measure-ments from N line patterns from one or more images. Each line pattern will have its own vanishing point, but the same intrinsic parameters. Hence, adding a subscript k for the kth line pattern to all the equations above, we get:
²2=X kij
(²kij)2 (3.15)
Note that we are not necessarily limited to a single vanishing point per image. If we use a CCD or CMOS sensor we can image a regular dot pattern or checkerboard pattern in which we can find many parallel lines, as shown in Figure 3.7. To find out which points can form a particular parallel line pattern, it is necessary to identify each measured point (thick dot) with its corresponding original grid point. This identification process is simplified with checkerboard
Figure 3.7: Parallel lines in checkerboard and dot patterns.
patterns, where, after finding a grid point, we can simply follow the black-white edges in the image to find the four neighboring points on the grid.
For PSD cameras, we can only measure linear features, so we perform mea-surements on a flat object that is moved along the H direction in the sheet-of-light (see Figure 3.4). The shift does not need to be exactly in the H direction, nor does the shift need to be exactly known. The measured line, however, should be slanted with respect to the YZ plane to prevent numerical errors for vanish-ing points at infinity. The followvanish-ing discussion of the minimization algorithm will however deal with the general case of a grid.
Initialization
As with most minimization strategies of complex functions with possibly several local minima, an important first step is to find initial parameter values which come as close as possible to the actual parameters.
Initialization of Vanishing Points
For convenience, we start with the assumption that the calibration grid was imaged in such a way that (after removal of the radial distortions) the grid points can be divided into a set of approximately horizontal and/or vertical lines intersecting in a vanishing point (Cv, Rv), as in Figure 3.6.
0 50 100 150 200 250 300 0
100 200
Figure 3.8: Distorted line (top), its curvature (bottom), the radial distortion center (+) and the point of minimum curvature closest to distortion center (top of dotted line).
each line i of vanishing point k the line parameters (ρki, θki) of the line equation Ckijcos θki+ Ckijsin θki= ρki. We then calculate the intersection point of each
pair of lines (i, i0) and take the average of those intersection points as a first
estimate of the kth vanishing point.
Initialization of Radial Distortion Parameters
As a starting point for the the radial distortion center (Cc, Rc), we can use the
center of the sensor. However, it is also possible to obtain a better estimate of the radial distortion center by observing the curvature of a distorted line. The closer to the radial distortion center, the smaller (in absolute sense) the curvature of the line.
In Figure 3.8, we show the curvature of a line with radial distortion relative to a preset radial distortion center (Cc, Rc) = (150, 100). We see that the point of
minimum curvature is the closest to the distortion center. Hence, the distortion center is on a line perpendicular to the distorted line and intersecting in the point of maximum curvature.
perpen-dicular lines that ideally intersect in (Cc, Rc). In practise, the perpendicular
lines will not intersect in a single point, but intersect each other within a cer-tain area around (Cc, Rc). We can find a good approximation of the distortion
center by finding the coordinates (Cc, Rc) that have the smallest distance (in
least-squares sense) to the perpendicular lines.
As a starting point for the radial distortion parameters, we take κ1,2 = 0.
Minimizing the error
Minimizing the squared distance error of Eq 3.15 to all of the parameters at the same time is a rather complex task. For a grid of 5x9 grid points on 5 lines intersecting in a single vanishing point, we already need to estimate the two vanishing point coordinates, 5 line parameters and 4 lens parameters – a total of 11 parameters. Using a multidimensional minimization procedure such as available in commercial mathematics packages, the result of minimization often ends up in a local minimum because of the high dimensionality of the parameter space in combination with the complexity of the functional.
Instead, we split the minimization into two steps. In the first step, we find the k optimal vanishing points (and associated line parameters θki) keeping the
lens distortion parameters fixed. In the second step, we find the optimal lens distortion parameters, keeping the vanishing points fixed. These two steps need to be repeated several times to obtain accurate results.
Optimal Vanishing Points
To determine the optimal vanishing points (and the accompanying line param-eters), we perform a minimum search using the Nelder-Mead Simplex (NMS) method in Matlab (fminsearch). This function performs a multidimensional unconstrained nonlinear minimization. As function to be minimized we use the sum of squared distances of the undistorted grid points to the lines through the vanishing point. The line parameters are recalculated each time the minimiza-tion procedure iteratively updates a vanishing point. These line parameters are taken as the least-squares optimum line through that vanishing point and the grid points associated with the line.
Optimal Lens Parameters
To determine the optimal lens parameters (κ1, κ2, Cc, Rc), we can use a number
of different strategies. The first strategy uses the NMS method, this time with the four optical parameters as variables, and keeping the vanishing points fixed. The function to minimize is again the squared distance of the undistorted points to the lines.
A second strategy uses a steepest-descent method. We calculate the first derivative of the undistorted row and column coordinates ~u = (C, R) to each of the lens parameters:
