Computational techniques for image reconstruction problems enable imaging technologies including high-resolution microscopy, astronomy and seismology, computed tomography, and magnetic resonance imaging. Until recently, methods for solving such inverse problems were derived by experts without any learning. Now, the best performing image reconstruction methods are based on deep learning.
This textbook gives the first comprehensive introduction to deep learning based image reconstruction methods. This book first introduces important inverse problems in imaging, including denoising and reconstructing an image from few and noisy measurements, and explains what makes those problems hard and interesting. Then, the book briefly discusses traditional optimization and sparsity based reconstruction methods, as well as optimization techniques as a basis for training and deriving deep neural networks for image reconstruction.
The main part of the book is about how to solve image reconstruction problems with deep learning techniques: The book first disuses supervised deep learning approaches that map a measurement to an image as well as network architectures for imaging including convolutional neural networks and transformers. Then, reconstruction approaches based on generative models such as variational autoencoders and diffusion models are discussed, and how un-trained neural networks and implicit neural representations enable signal and image reconstruction. The book ends with a discussion on the robustness of deep learning based reconstruction as well as a discussion on the important topic of evaluating models and datasets, which are a critical ingredient of deep learning based imaging.
This textbook offers an introduction to deep learning for solving inverse problems. It introduces deep neural networks and deep neural network based signal and image reconstruction techniques. It discusses robustness aspects, how to evaluate and test different methods, and data-centric aspects.
1: Introduction
2: Solving inverse problems with optimization tasks
3: Solving optimization problems
4: Sparse modelling
5: Plug-and-play methods
6: Learning to solve inverse problems end-to-end
7: Unrolled neural networks
8: Self-supervised learning
9: Signal reconstruction via imposing generative priors
10: Diffusion models
11: Signal reconstruction with un-trained neural networks
12: Coordinate-based multi-layer perceptrons
13: Robustness to perturbations
14: Datasets and evaluation of image reconstruction methods
15: Advanced reconstruction problems
16: Mathematical background
Reinhard Heckel is a Professor of Machine Learning (Tenured Associate Professor) at the Department of Computer Engineering at the Technical University of Munich (TUM), and adjunct faculty at Rice University, where he was an assistant professor of Electrical and Computer Engineering from 2017-2019. Before that, he was a postdoctoral researcher in the Berkeley Artificial Intelligence Research Lab at UC Berkeley, and before that a researcher at IBM Research Zurich. He completed his PhD in 2014 at ETH Zurich and was a visiting PhD student at Stanfords University's Statistics Department. Reinhard's work is centered on machine learning, artificial intelligence, and information processing, with a focus on developing algorithms and foundations for deep learning, particularly for medical imaging, on establishing mathematical and empirical underpinnings for machine learning, and on the utilization of DNA as a digital information technology.
