Preface |
|
xv | |
|
1 Image Super-Resolution: Historical Overview and Future Challenges |
|
|
1 | (34) |
|
|
|
1.1 Introduction to Super-Resolution |
|
|
1 | (4) |
|
|
5 | (1) |
|
1.3 Techniques for Super-Resolution |
|
|
5 | (15) |
|
1.3.1 Image Observation Model |
|
|
5 | (2) |
|
1.3.2 Super-Resolution in the Frequency Domain |
|
|
7 | (1) |
|
1.3.3 Interpolation-Restoration: Non-Iterative Approaches |
|
|
8 | (1) |
|
1.3.4 Statistical Approaches |
|
|
9 | (2) |
|
1.3.4.1 Maximum Likelihood |
|
|
11 | (1) |
|
1.3.4.2 Maximum a Posteriori |
|
|
12 | (1) |
|
1.3.4.3 Joint MAP Restoration |
|
|
13 | (1) |
|
1.3.4.4 Bayesian Treatments |
|
|
14 | (1) |
|
1.3.5 Example-Based Approaches |
|
|
15 | (3) |
|
1.3.6 Set Theoretic Restoration |
|
|
18 | (2) |
|
1.4 Challenge Issues for Super-Resolution |
|
|
20 | (4) |
|
|
20 | (1) |
|
1.4.2 Computation Efficiency |
|
|
21 | (1) |
|
|
22 | (1) |
|
|
23 | (1) |
|
|
24 | (11) |
|
2 Super-Resolution Using Adaptive Wiener Filters |
|
|
35 | (28) |
|
|
|
36 | (2) |
|
|
38 | (9) |
|
2.2.1 Image Formation Model |
|
|
38 | (3) |
|
|
41 | (1) |
|
|
42 | (2) |
|
2.2.4 System Point-Spread Function |
|
|
44 | (3) |
|
|
47 | (4) |
|
|
51 | (6) |
|
2.4.1 SR Results for Simulated Data |
|
|
51 | (2) |
|
2.4.2 SR Results for Infrared Video Data |
|
|
53 | (4) |
|
|
57 | (1) |
|
|
58 | (1) |
|
|
58 | (5) |
|
3 Locally Adaptive Kernel Regression for Space-Time Super-Resolution |
|
|
63 | (34) |
|
|
|
|
64 | (3) |
|
3.2 Adaptive Kernel Regression |
|
|
67 | (16) |
|
3.2.1 Classic Kernel Regression in 2-D |
|
|
67 | (3) |
|
3.2.2 Steering Kernel Regression in 2-D |
|
|
70 | (2) |
|
3.2.3 Space-Time (3-D) Steering Kernel Regression |
|
|
72 | (6) |
|
3.2.4 Kernel Regression with Rough Motion Compensation |
|
|
78 | (2) |
|
3.2.5 Implementation and Iterative Refinement |
|
|
80 | (3) |
|
|
83 | (7) |
|
3.3.1 Spatial Upscaling Examples |
|
|
83 | (3) |
|
3.3.2 Spatiotemporal Upscaling Examples |
|
|
86 | (4) |
|
|
90 | (1) |
|
|
91 | (1) |
|
3.5.1 Steering Kernel Parameters |
|
|
91 | (1) |
|
3.5.2 The Choice of the Regression Parameters |
|
|
91 | (1) |
|
|
92 | (1) |
|
|
92 | (5) |
|
4 Super-Resolution with Probabilistic Motion Estimation |
|
|
97 | (26) |
|
|
|
|
98 | (1) |
|
4.2 Classic Super-Resolution: Background |
|
|
99 | (2) |
|
4.3 The Proposed Algorithm |
|
|
101 | (7) |
|
4.3.1 The New Formulation |
|
|
101 | (2) |
|
4.3.2 Separating the Blur Treatment |
|
|
103 | (1) |
|
4.3.3 The Algorithm: A Matrix-Vector Version |
|
|
104 | (1) |
|
4.3.4 The Algorithm: A Pixel-Wise Version |
|
|
104 | (2) |
|
4.3.5 Computing the Weights |
|
|
106 | (1) |
|
4.3.6 Other Resampling Tasks |
|
|
107 | (1) |
|
4.4 Experimental Validation |
|
|
108 | (10) |
|
4.4.1 Experimental Results |
|
|
108 | (8) |
|
4.4.2 Computational Complexity |
|
|
116 | (2) |
|
|
118 | (1) |
|
|
119 | (4) |
|
5 Spatially Adaptive Filtering as Regularization in Inverse Imaging: Compressive Sensing, Super-Resolution, and Upsampling |
|
|
123 | (32) |
|
|
|
|
|
|
124 | (1) |
|
5.2 Iterative Filtering as Regularization |
|
|
125 | (4) |
|
5.2.1 Spectral Decomposition of the Operator |
|
|
126 | (1) |
|
5.2.2 Nonlocal Transform Domain Filtering |
|
|
126 | (3) |
|
|
129 | (8) |
|
5.3.1 Observation Model and Notation |
|
|
130 | (1) |
|
5.3.2 Iterative Algorithm with Stochastic Approximation |
|
|
130 | (2) |
|
5.3.2.1 Comments on the Algorithm |
|
|
132 | (1) |
|
|
133 | (1) |
|
5.3.3.1 Radon Inversion from Sparse Projections |
|
|
134 | (1) |
|
5.3.3.2 Limited-Angle Tomography |
|
|
134 | (1) |
|
5.3.3.3 Reconstruction from Low-Frequency Data |
|
|
134 | (3) |
|
|
137 | (13) |
|
5.4.1 Spectral Decomposition for the Super-Resolution Problem |
|
|
139 | (1) |
|
|
140 | (1) |
|
5.4.3 Scaling Family of Transforms |
|
|
140 | (2) |
|
5.4.4 Multistage Iterative Reconstruction |
|
|
142 | (1) |
|
|
143 | (1) |
|
5.4.5.1 Implementation Details |
|
|
144 | (1) |
|
|
144 | (1) |
|
|
145 | (5) |
|
|
150 | (1) |
|
|
150 | (5) |
|
6 Registration for Super-Resolution: Theory, Algorithms, and Applications in Image and Mobile Video Enhancement |
|
|
155 | (32) |
|
|
|
|
|
157 | (4) |
|
|
161 | (1) |
|
6.3 Super-Resolution as a Multichannel Sampling Problem |
|
|
162 | (4) |
|
|
163 | (3) |
|
6.4 Registration of Totally Aliased Signals |
|
|
166 | (2) |
|
6.4.1 Variable Projection Method |
|
|
166 | (1) |
|
6.4.2 Frequency Analysis Method |
|
|
167 | (1) |
|
|
168 | (1) |
|
6.5 Registration of Partially Aliased Signals |
|
|
168 | (15) |
|
6.5.1 Super-Resolution Using Frequency Domain Registration |
|
|
168 | (1) |
|
6.5.1.1 Image Registration |
|
|
168 | (2) |
|
6.5.1.2 Image Reconstruction |
|
|
170 | (1) |
|
|
170 | (6) |
|
6.5.2 Super-Resolution from Low-Quality Videos |
|
|
176 | (1) |
|
|
176 | (1) |
|
6.5.2.2 Image Registration |
|
|
177 | (2) |
|
6.5.2.3 Image Reconstruction |
|
|
179 | (2) |
|
6.5.2.4 Results on Video Sequences |
|
|
181 | (2) |
|
|
183 | (1) |
|
|
184 | (3) |
|
7 Towards Super-Resolution in the Presence of Spatially Varying Blur |
|
|
187 | (32) |
|
|
|
|
|
188 | (6) |
|
7.1.1 Representation of Spatially Varying PSF |
|
|
189 | (1) |
|
7.1.2 General Model of Resolution Loss |
|
|
189 | (2) |
|
7.1.3 Bayesian View of Solution |
|
|
191 | (3) |
|
7.2 Defocus and Optical Aberrations |
|
|
194 | (8) |
|
|
195 | (2) |
|
7.2.2 Approximation of PSF by 2D Gaussian Function |
|
|
197 | (1) |
|
7.2.3 General Form of PSF for Axially-Symmetric Optical Systems |
|
|
197 | (1) |
|
|
198 | (3) |
|
|
201 | (1) |
|
|
202 | (2) |
|
|
202 | (1) |
|
|
203 | (1) |
|
|
204 | (1) |
|
|
204 | (10) |
|
7.5.1 Super-Resolution of a Scene with Local Motion |
|
|
205 | (2) |
|
7.5.2 Smoothly Changing Blur |
|
|
207 | (3) |
|
7.5.3 Depth-Dependent Blur |
|
|
210 | (4) |
|
|
214 | (1) |
|
|
215 | (1) |
|
|
215 | (4) |
|
8 Toward Robust Reconstruction-Based Super-Resolution |
|
|
219 | (28) |
|
|
|
|
220 | (1) |
|
|
221 | (4) |
|
8.2.1 Super-Resolution Reconstruction |
|
|
221 | (1) |
|
8.2.2 Robust SR Reconstruction |
|
|
222 | (2) |
|
8.2.3 Robust Registration |
|
|
224 | (1) |
|
8.3 Robust SR Reconstruction with Pixel Selection |
|
|
225 | (7) |
|
8.3.1 Displacement and Similarity Measure |
|
|
225 | (2) |
|
8.3.2 Proposed Pixel Selection Algorithm |
|
|
227 | (1) |
|
8.3.2.1 Pixel Selection Based on Similarity Measure and Displacement Estimation |
|
|
227 | (2) |
|
8.3.3 Luminance Correction |
|
|
229 | (1) |
|
|
230 | (2) |
|
8.4 Robust Super-Resolution Using MPEG Motion Vectors |
|
|
232 | (5) |
|
8.4.1 Registration Using MPEG Motion Vectors |
|
|
232 | (1) |
|
8.4.2 Experiments of Robust SR Reconstruction |
|
|
233 | (4) |
|
8.5 Robust Registration for Super-Resolution |
|
|
237 | (7) |
|
8.5.1 Proposed Multiple Motion Estimation |
|
|
238 | (1) |
|
8.5.1.1 Motion Estimation and Region Extraction for a Single Object |
|
|
239 | (1) |
|
8.5.1.2 Multiple Motion Estimation |
|
|
240 | (1) |
|
8.5.2 Super-Resolution for Multiple Motions |
|
|
241 | (1) |
|
|
241 | (3) |
|
|
244 | (1) |
|
|
244 | (3) |
|
9 Multiframe Super-Resolution from a Bayesian Perspective |
|
|
247 | (38) |
|
|
|
|
|
|
248 | (10) |
|
9.1.1 Considerations in the Forward Model |
|
|
249 | (2) |
|
9.1.2 A Probabilistic Setting |
|
|
251 | (1) |
|
9.1.2.1 The Maximum Likelihood Solution |
|
|
251 | (1) |
|
9.1.2.2 The ML Solution in Practice |
|
|
252 | (2) |
|
9.1.2.3 The Maximum a Posteriori Solution |
|
|
254 | (1) |
|
9.1.3 Selected Priors Used in MAP Super-Resolution |
|
|
255 | (3) |
|
9.2 Where Super-Resolution Algorithms Go Wrong |
|
|
258 | (5) |
|
9.2.1 Point-Spread Function Example |
|
|
259 | (2) |
|
9.2.2 Photometric Registration Example |
|
|
261 | (1) |
|
9.2.3 Geometric Registration Example |
|
|
262 | (1) |
|
9.3 Simultaneous Super-Resolution |
|
|
263 | (10) |
|
9.3.1 Super-Resolution with Registration |
|
|
264 | (1) |
|
9.3.2 Learning Prior Strength Parameters from Data |
|
|
265 | (1) |
|
9.3.3 Scaling and Convergence |
|
|
266 | (1) |
|
|
267 | (2) |
|
9.3.5 Evaluation on Synthetic Data |
|
|
269 | (2) |
|
9.3.6 Experiments on Real Data |
|
|
271 | (2) |
|
9.4 Bayesian Marginalization |
|
|
273 | (9) |
|
9.4.1 Marginalizing over Registration Parameters |
|
|
274 | (3) |
|
9.4.2 Marginalizing over the High-Resolution Image |
|
|
277 | (1) |
|
9.4.3 Implementation Notes |
|
|
278 | (1) |
|
9.4.4 Experimental Evaluation |
|
|
279 | (3) |
|
|
282 | (1) |
|
|
282 | (1) |
|
|
283 | (2) |
|
10 Variational Bayesian Super-Resolution Reconstruction |
|
|
285 | (30) |
|
|
|
|
|
285 | (3) |
|
|
288 | (1) |
|
10.3 Bayesian Framework for Super-Resolution |
|
|
288 | (5) |
|
10.3.1 Observation Models |
|
|
289 | (1) |
|
|
290 | (1) |
|
|
291 | (1) |
|
10.3.4 Motion (Registration) Models |
|
|
292 | (1) |
|
10.3.5 Hyperpriors on the Hyperparameters |
|
|
292 | (1) |
|
|
293 | (3) |
|
10.5 Variational Bayesian Inference Using TV Image Priors |
|
|
296 | (5) |
|
10.5.1 Estimation of the HR Image Distribution |
|
|
297 | (1) |
|
10.5.2 Estimation of the Hyperparameter Distributions |
|
|
298 | (3) |
|
|
301 | (4) |
|
10.7 Estimation of Motion and Blur |
|
|
305 | (3) |
|
|
308 | (1) |
|
|
309 | (1) |
|
|
309 | (6) |
|
11 Pattern Recognition Techniques for Image Super-Resolution |
|
|
315 | (40) |
|
|
|
|
316 | (2) |
|
11.2 Nearest Neighbor Super-Resolution |
|
|
318 | (8) |
|
11.2.1 k-Nearest Neighbor |
|
|
320 | (1) |
|
11.2.2 k-Nearest Neighbor Regression |
|
|
321 | (2) |
|
11.2.3 Adaptive k-NN for Super-Resolution |
|
|
323 | (2) |
|
11.2.4 Heuristics for Insufficient Training in Adaptive k-NN Regression |
|
|
325 | (1) |
|
11.3 Markov Random Fields and Approximations |
|
|
326 | (3) |
|
11.4 Kernel Machines for Image Super-Resolution |
|
|
329 | (9) |
|
11.4.1 Support Vector Regression |
|
|
330 | (2) |
|
11.4.2 Inductively Learning the Kernel Matrix for Regression |
|
|
332 | (3) |
|
11.4.3 The Quadratically Constrained Quadratic Programming Problem |
|
|
335 | (2) |
|
11.4.4 Applications to Super-Resolution |
|
|
337 | (1) |
|
11.5 Multiple Learners and Multiple Regressions |
|
|
338 | (8) |
|
11.5.1 Neural Networks and Super-Resolution |
|
|
339 | (1) |
|
11.5.2 Unsupervised Clustering |
|
|
340 | (2) |
|
11.5.3 Supervised Clustering |
|
|
342 | (1) |
|
11.5.4 Integrating Regression |
|
|
343 | (3) |
|
11.6 Design Considerations and Examples |
|
|
346 | (2) |
|
|
348 | (1) |
|
|
349 | (1) |
|
|
349 | (6) |
|
12 Super-Resolution Reconstruction of Multichannel Images |
|
|
355 | (28) |
|
|
|
|
356 | (2) |
|
|
358 | (2) |
|
12.3 Image Acquisition Model |
|
|
360 | (6) |
|
12.3.1 Motion Compensation |
|
|
361 | (1) |
|
|
362 | (1) |
|
12.3.3 Spectral Filtering |
|
|
363 | (2) |
|
12.3.4 Multichannel Observation Model |
|
|
365 | (1) |
|
12.4 Subspace Representation |
|
|
366 | (3) |
|
12.4.1 Blind Source Separation |
|
|
367 | (1) |
|
12.4.2 Observation Model with BSS |
|
|
368 | (1) |
|
12.5 Reconstruction Algorithm |
|
|
369 | (4) |
|
12.5.1 The Subspace Observation Model |
|
|
370 | (1) |
|
12.5.2 POCS with Outliers of Residual |
|
|
371 | (1) |
|
12.5.3 POCS with Variance of Residual |
|
|
372 | (1) |
|
12.6 Experiments and Discussions |
|
|
373 | (4) |
|
|
374 | (1) |
|
12.6.2 Robustness against Noise |
|
|
375 | (1) |
|
12.6.3 Simultaneous Spatial and Spectral Super-Resolution |
|
|
376 | (1) |
|
|
377 | (3) |
|
|
380 | (3) |
|
13 New Applications of Super-Resolution in Medical Imaging |
|
|
383 | (30) |
|
|
|
|
|
|
|
|
384 | (1) |
|
13.2 The Super-Resolution Framework |
|
|
385 | (3) |
|
13.2.1 Image Capture Model |
|
|
385 | (2) |
|
13.2.2 Super-Resolution Estimation Framework |
|
|
387 | (1) |
|
13.3 New Medical Imaging Applications |
|
|
388 | (17) |
|
13.3.1 Super-Resolution in Low Radiation Digital X-Ray Mammography |
|
|
389 | (2) |
|
13.3.1.1 Multiframe Shift Estimation |
|
|
391 | (2) |
|
13.3.1.2 Multiframe forWaRD Deconvolution and Denoising |
|
|
393 | (1) |
|
13.3.1.3 Experimental X-Ray Results |
|
|
394 | (3) |
|
13.3.2 Super-Resolution in Optical Coherence Tomography |
|
|
397 | (2) |
|
13.3.2.1 Proposed Method: Sparse Repeated Imaging |
|
|
399 | (2) |
|
13.3.2.2 Multiframe Joint Registration |
|
|
401 | (2) |
|
13.3.2.3 Experimental Results |
|
|
403 | (2) |
|
|
405 | (1) |
|
|
406 | (1) |
|
|
407 | (6) |
|
14 Practicing Super-Resolution: What Have We Learned? |
|
|
413 | (36) |
|
|
|
414 | (1) |
|
|
414 | (6) |
|
14.2.1 Video Quality Trends |
|
|
415 | (1) |
|
14.2.2 The Need for Postprocessing |
|
|
415 | (2) |
|
14.2.3 Why is Super-Resolution not Used More? |
|
|
417 | (1) |
|
14.2.4 Automation versus User Interaction |
|
|
418 | (1) |
|
14.2.5 Modeling Motion for Super-Resolution |
|
|
418 | (1) |
|
14.2.6 Performance Issues |
|
|
419 | (1) |
|
14.2.7 Relationship to Existing Standards |
|
|
419 | (1) |
|
14.3 MotionDSP: History and Concepts |
|
|
420 | (2) |
|
|
421 | (1) |
|
14.4 Markets and Applications |
|
|
422 | (5) |
|
14.4.1 Forensic and Real-Time Markets: MotionDSP's Ikena |
|
|
423 | (2) |
|
14.4.2 Consumers: MotionDSP's vReveal |
|
|
425 | (2) |
|
|
427 | (4) |
|
14.5.1 Robust Parametric Motion Estimation |
|
|
428 | (3) |
|
|
431 | (6) |
|
14.6.1 Mobile and Digital Still Camera Video |
|
|
432 | (1) |
|
|
433 | (1) |
|
14.6.3 Handling Complex Motion |
|
|
433 | (2) |
|
14.6.4 Practical Limits of Super-Resolution |
|
|
435 | (2) |
|
|
437 | (7) |
|
|
444 | (1) |
|
|
444 | (5) |
Index |
|
449 | |