|
|
xiii | |
Preface |
|
xix | |
|
Chapter 1 Tensor decompositions: computations, applications, and challenges |
|
|
1 | (30) |
|
|
|
|
|
|
|
1 | (2) |
|
|
1 | (1) |
|
1.1.2 Why do we need tensors? |
|
|
2 | (1) |
|
|
3 | (10) |
|
|
3 | (1) |
|
|
4 | (2) |
|
1.2.3 Tensor transformations |
|
|
6 | (1) |
|
|
7 | (4) |
|
|
11 | (2) |
|
|
13 | (1) |
|
1.3 Tensor decompositions |
|
|
13 | (11) |
|
1.3.1 Tucker decomposition |
|
|
13 | (1) |
|
1.3.2 Canonical polyadic decomposition |
|
|
14 | (2) |
|
1.3.3 Block term decomposition |
|
|
16 | (2) |
|
1.3.4 Tensor singular value decomposition |
|
|
18 | (1) |
|
|
19 | (5) |
|
1.4 Tensor processing techniques |
|
|
24 | (1) |
|
|
25 | (6) |
|
|
26 | (5) |
|
Chapter 2 Transform-based tensor singular value decomposition in multidimensional image recovery |
|
|
31 | (30) |
|
|
|
|
|
32 | (2) |
|
2.2 Recent advances of the tensor singular value decomposition |
|
|
34 | (10) |
|
2.2.1 Preliminaries and basic tensor notations |
|
|
34 | (1) |
|
2.2.2 The t-SVD framework |
|
|
35 | (3) |
|
2.2.3 Tensor nuclear norm and tensor recovery |
|
|
38 | (3) |
|
|
41 | (3) |
|
|
44 | (1) |
|
2.3 Transform-based t-SVD |
|
|
44 | (5) |
|
2.3.1 Linear invertible transform-based t-SVD |
|
|
45 | (2) |
|
2.3.2 Beyond invertibility and data adaptivity |
|
|
47 | (2) |
|
2.4 Numerical experiments |
|
|
49 | (4) |
|
2.4.1 Examples within the t-SVD framework |
|
|
49 | (2) |
|
2.4.2 Examples of the transform-based t-SVD |
|
|
51 | (2) |
|
2.5 Conclusions and new guidelines |
|
|
53 | (8) |
|
|
55 | (6) |
|
|
61 | (30) |
|
|
|
|
|
Ioannis Marios Papagiannakos |
|
|
|
|
|
|
62 | (2) |
|
|
62 | (1) |
|
|
63 | (1) |
|
|
64 | (6) |
|
3.2.1 Matrix least-squares problems |
|
|
65 | (4) |
|
3.2.2 Alternating optimization for tensor decomposition |
|
|
69 | (1) |
|
3.3 Tensor decomposition with missing elements |
|
|
70 | (5) |
|
3.3.1 Matrix least-squares with missing elements |
|
|
71 | (3) |
|
3.3.2 Tensor decomposition with missing elements: the unconstrained case |
|
|
74 | (1) |
|
3.3.3 Tensor decomposition with missing elements: the nonnegative case |
|
|
75 | (1) |
|
3.3.4 Alternating optimization for tensor decomposition with missing elements |
|
|
75 | (1) |
|
3.4 Distributed memory implementations |
|
|
75 | (8) |
|
3.4.1 Some MPI preliminaries |
|
|
75 | (2) |
|
3.4.2 Variable partitioning and data allocation |
|
|
77 | (2) |
|
3.4.3 Tensor decomposition |
|
|
79 | (2) |
|
3.4.4 Tensor decomposition with missing elements |
|
|
81 | (1) |
|
3.4.5 Some implementation details |
|
|
82 | (1) |
|
3.5 Numerical experiments |
|
|
83 | (4) |
|
3.5.1 Tensor decomposition |
|
|
83 | (1) |
|
3.5.2 Tensor decomposition with missing elements |
|
|
84 | (3) |
|
|
87 | (4) |
|
|
88 | (1) |
|
|
88 | (3) |
|
Chapter 4 A Riemannian approach to low-rank tensor learning |
|
|
91 | (30) |
|
|
|
|
|
91 | (2) |
|
4.2 A brief introduction to Riemannian optimization |
|
|
93 | (4) |
|
4.2.1 Riemannian manifolds |
|
|
94 | (1) |
|
4.2.2 Riemannian quotient manifolds |
|
|
95 | (2) |
|
4.3 Riemannian Tucker manifold geometry |
|
|
97 | (7) |
|
4.3.1 Riemannian metric and quotient manifold structure |
|
|
97 | (3) |
|
4.3.2 Characterization of the induced spaces |
|
|
100 | (2) |
|
|
102 | (1) |
|
|
103 | (1) |
|
|
104 | (1) |
|
|
104 | (1) |
|
4.4 Algorithms for tensor learning problems |
|
|
104 | (3) |
|
|
105 | (1) |
|
4.4.2 General tensor learning |
|
|
106 | (1) |
|
|
107 | (9) |
|
|
108 | (1) |
|
4.5.2 Low-rank tensor completion |
|
|
109 | (4) |
|
4.5.3 Low-rank tensor regression |
|
|
113 | (2) |
|
4.5.4 Multilinear multitask learning |
|
|
115 | (1) |
|
|
116 | (5) |
|
|
117 | (4) |
|
Chapter 5 Generalized thresholding for low-rank tensor recovery: approaches based on model and learning |
|
|
121 | (32) |
|
|
|
|
|
121 | (2) |
|
5.2 Tensor singular value thresholding |
|
|
123 | (8) |
|
5.2.1 Proximity operator and generalized thresholding |
|
|
123 | (3) |
|
5.2.2 Tensor singular value decomposition |
|
|
126 | (2) |
|
5.2.3 Generalized matrix singular value thresholding |
|
|
128 | (1) |
|
5.2.4 Generalized tensor singular value thresholding |
|
|
129 | (2) |
|
5.3 Thresholding based low-rank tensor recovery |
|
|
131 | (5) |
|
5.3.1 Thresholding algorithms for low-rank tensor recovery |
|
|
132 | (2) |
|
5.3.2 Generalized thresholding algorithms for low-rank tensor recovery |
|
|
134 | (2) |
|
5.4 Generalized thresholding algorithms with learning |
|
|
136 | (5) |
|
|
137 | (3) |
|
|
140 | (1) |
|
|
141 | (4) |
|
|
145 | (8) |
|
|
147 | (6) |
|
Chapter 6 Tensor principal component analysis |
|
|
153 | (62) |
|
|
|
|
|
153 | (2) |
|
6.2 Notations and preliminaries |
|
|
155 | (6) |
|
|
156 | (1) |
|
6.2.2 Discrete Fourier transform |
|
|
157 | (2) |
|
|
159 | (1) |
|
|
160 | (1) |
|
6.3 Tensor PCA for Gaussian-noisy data |
|
|
161 | (5) |
|
6.3.1 Tensor rank and tensor nuclear norm |
|
|
161 | (4) |
|
6.3.2 Analysis of tensor PCA on Gaussian-noisy data |
|
|
165 | (1) |
|
|
166 | (1) |
|
6.4 Tensor PCA for sparsely corrupted data |
|
|
166 | (25) |
|
|
167 | (5) |
|
6.4.2 Tensor low-rank representation |
|
|
172 | (14) |
|
|
186 | (5) |
|
|
191 | (1) |
|
6.5 Tensor PCA for outlier-corrupted data |
|
|
191 | (16) |
|
6.5.1 Outlier robust tensor PCA |
|
|
192 | (4) |
|
6.5.2 The fast OR-TPCA algorithm |
|
|
196 | (2) |
|
|
198 | (8) |
|
|
206 | (1) |
|
6.6 Other tensor PCA methods |
|
|
207 | (1) |
|
|
208 | (1) |
|
|
208 | (7) |
|
|
209 | (6) |
|
Chapter 7 Tensors for deep learning theory |
|
|
215 | (34) |
|
|
|
|
|
|
|
215 | (2) |
|
7.2 Bounding a function's expressivity via tensorization |
|
|
217 | (6) |
|
7.2.1 A measure of capacity for modeling input dependencies |
|
|
218 | (2) |
|
7.2.2 Bounding correlations with tensor matricization ranks |
|
|
220 | (3) |
|
7.3 A case study: self-attention networks |
|
|
223 | (19) |
|
7.3.1 The self-attention mechanism |
|
|
223 | (4) |
|
7.3.2 Self-attention architecture expressivity questions |
|
|
227 | (3) |
|
7.3.3 Results on the operation of self-attention |
|
|
230 | (5) |
|
7.3.4 Bounding the separation rank of self-attention |
|
|
235 | (7) |
|
7.4 Convolutional and recurrent networks |
|
|
242 | (3) |
|
7.4.1 The operation of convolutional and recurrent networks |
|
|
243 | (1) |
|
7.4.2 Addressed architecture expressivity questions |
|
|
243 | (2) |
|
|
245 | (4) |
|
|
245 | (4) |
|
Chapter 8 Tensor network algorithms for image classification |
|
|
249 | (44) |
|
|
|
|
|
249 | (2) |
|
|
251 | (7) |
|
|
251 | (2) |
|
8.2.2 Tensor decompositions |
|
|
253 | (3) |
|
8.2.3 Support vector machines |
|
|
256 | (1) |
|
8.2.4 Logistic regression |
|
|
257 | (1) |
|
8.3 Tensorial extensions of support vector machine |
|
|
258 | (26) |
|
8.3.1 Supervised tensor learning |
|
|
258 | (2) |
|
8.3.2 Support tensor machines |
|
|
260 | (3) |
|
8.3.3 Higher-rank support tensor machines |
|
|
263 | (2) |
|
8.3.4 Support Tucker machines |
|
|
265 | (4) |
|
8.3.5 Support tensor train machines |
|
|
269 | (6) |
|
8.3.6 Kernelized support tensor train machines |
|
|
275 | (9) |
|
8.4 Tensorial extension of logistic regression |
|
|
284 | (4) |
|
8.4.1 Rank-1 logistic regression |
|
|
285 | (1) |
|
8.4.2 Logistic tensor regression |
|
|
286 | (2) |
|
|
288 | (5) |
|
|
289 | (4) |
|
Chapter 9 High-performance tensor decompositions for compressing and accelerating deep neural networks |
|
|
293 | (48) |
|
|
|
|
|
|
9.1 Introduction and motivation |
|
|
294 | (1) |
|
|
295 | (10) |
|
|
295 | (1) |
|
|
295 | (3) |
|
9.2.3 Fully connected neural networks |
|
|
298 | (2) |
|
9.2.4 Convolutional neural networks |
|
|
300 | (3) |
|
|
303 | (2) |
|
9.3 Tensor networks and their decompositions |
|
|
305 | (16) |
|
|
305 | (3) |
|
9.3.2 CP tensor decomposition |
|
|
308 | (2) |
|
9.3.3 Tucker decomposition |
|
|
310 | (3) |
|
9.3.4 Hierarchical Tucker decomposition |
|
|
313 | (2) |
|
9.3.5 Tensor train and tensor ring decomposition |
|
|
315 | (3) |
|
9.3.6 Transform-based tensor decomposition |
|
|
318 | (3) |
|
9.4 Compressing deep neural networks |
|
|
321 | (12) |
|
9.4.1 Compressing fully connected layers |
|
|
321 | (1) |
|
9.4.2 Compressing the convolutional layer via CP decomposition |
|
|
322 | (3) |
|
9.4.3 Compressing the convolutional layer via Tucker decomposition |
|
|
325 | (2) |
|
9.4.4 Compressing the convolutional layer via TT/TR decompositions |
|
|
327 | (3) |
|
9.4.5 Compressing neural networks via transform-based decomposition |
|
|
330 | (3) |
|
9.5 Experiments and future directions |
|
|
333 | (8) |
|
9.5.1 Performance evaluations using the MNIST dataset |
|
|
333 | (3) |
|
9.5.2 Performance evaluations using the CIFAR10 dataset |
|
|
336 | (1) |
|
9.5.3 Future research directions |
|
|
337 | (1) |
|
|
338 | (3) |
|
Chapter 10 Coupled tensor decompositions for data fusion |
|
|
341 | (30) |
|
|
|
|
|
|
341 | (1) |
|
10.2 What is data fusion? |
|
|
342 | (6) |
|
10.2.1 Context and definition |
|
|
342 | (1) |
|
10.2.2 Challenges of data fusion |
|
|
343 | (4) |
|
10.2.3 Types of fusion and data fusion strategies |
|
|
347 | (1) |
|
10.3 Decompositions in data fusion |
|
|
348 | (7) |
|
10.3.1 Matrix decompositions and statistical models |
|
|
350 | (1) |
|
10.3.2 Tensor decompositions |
|
|
351 | (1) |
|
10.3.3 Coupled tensor decompositions |
|
|
352 | (3) |
|
10.4 Applications of tensor-based data fusion |
|
|
355 | (3) |
|
10.4.1 Biomedical applications |
|
|
355 | (2) |
|
|
357 | (1) |
|
10.5 Fusion of EEG and fMRI: a case study |
|
|
358 | (3) |
|
|
361 | (2) |
|
10.6.1 SDF demo - approximate coupling |
|
|
361 | (2) |
|
10.7 Conclusion and prospects |
|
|
363 | (8) |
|
|
364 | (1) |
|
|
364 | (7) |
|
Chapter 11 Tensor methods for low-level vision |
|
|
371 | (56) |
|
|
|
|
11.1 Low-level vision and signal reconstruction |
|
|
371 | (7) |
|
11.1.1 Observation models |
|
|
372 | (2) |
|
|
374 | (4) |
|
11.2 Methods using raw tensor structure |
|
|
378 | (31) |
|
11.2.1 Penalty-based tensor reconstruction |
|
|
379 | (14) |
|
11.2.2 Tensor decomposition and reconstruction |
|
|
393 | (16) |
|
11.3 Methods using tensorization |
|
|
409 | (6) |
|
11.3.1 Higher-order tensorization |
|
|
411 | (2) |
|
11.3.2 Delay embedding/Hankelization |
|
|
413 | (2) |
|
11.4 Examples of low-level vision applications |
|
|
415 | (4) |
|
11.4.1 Image inpainting with raw tensor structure |
|
|
415 | (1) |
|
11.4.2 Image inpainting using tensorization |
|
|
416 | (1) |
|
11.4.3 Denoising, deblurring, and superresolution |
|
|
417 | (2) |
|
|
419 | (8) |
|
|
420 | (1) |
|
|
420 | (7) |
|
Chapter 12 Tensors for neuroimaging |
|
|
427 | (56) |
|
|
|
|
427 | (2) |
|
12.2 Neuroimaging modalities |
|
|
429 | (2) |
|
12.3 Multidimensionality of the brain |
|
|
431 | (2) |
|
12.4 Tensor decomposition structures |
|
|
433 | (4) |
|
12.4.1 Product operations for tensors |
|
|
434 | (1) |
|
12.4.2 Canonical polyadic decomposition |
|
|
435 | (1) |
|
12.4.3 Tucker decomposition |
|
|
435 | (2) |
|
12.4.4 Block term decomposition |
|
|
437 | (1) |
|
12.5 Applications of tensors in neuroimaging |
|
|
437 | (34) |
|
12.5.1 Filling in missing data |
|
|
438 | (3) |
|
12.5.2 Denoising, artifact removal, and dimensionality reduction |
|
|
441 | (3) |
|
|
444 | (1) |
|
12.5.4 Registration and longitudinal analysis |
|
|
445 | (2) |
|
|
447 | (4) |
|
12.5.6 Activity recognition and source localization |
|
|
451 | (5) |
|
12.5.7 Connectivity analysis |
|
|
456 | (6) |
|
|
462 | (1) |
|
12.5.9 Feature extraction and classification |
|
|
463 | (5) |
|
12.5.10 Summary and practical considerations |
|
|
468 | (3) |
|
|
471 | (1) |
|
|
472 | (11) |
|
|
473 | (10) |
|
Chapter 13 Tensor representation for remote sensing images |
|
|
483 | (54) |
|
|
|
|
|
|
|
|
|
483 | (5) |
|
13.2 Optical remote sensing: HSI and MSI fusion |
|
|
488 | (29) |
|
13.2.1 Tensor notations and preliminaries |
|
|
488 | (1) |
|
13.2.2 Nonlocal patch tensor sparse representation for HSI-MSI fusion |
|
|
488 | (8) |
|
13.2.3 High-order coupled tensor ring representation for HSI-MSI fusion |
|
|
496 | (8) |
|
13.2.4 Joint tensor factorization for HSI-MSI fusion |
|
|
504 | (13) |
|
13.3 Polarimetric synthetic aperture radar: feature extraction |
|
|
517 | (20) |
|
13.3.1 Brief description of PolSAR data |
|
|
518 | (1) |
|
13.3.2 The tensorial embedding framework |
|
|
519 | (3) |
|
13.3.3 Experiment and analysis |
|
|
522 | (10) |
|
|
532 | (5) |
|
Chapter 14 Structured tensor train decomposition for speeding up kernel-based learning |
|
|
537 | (28) |
|
|
|
|
|
|
|
|
538 | (2) |
|
14.2 Notations and algebraic background |
|
|
540 | (1) |
|
14.3 Standard tensor decompositions |
|
|
541 | (4) |
|
14.3.1 Tucker decomposition |
|
|
542 | (1) |
|
|
542 | (1) |
|
14.3.3 Tensor networks and TT decomposition |
|
|
543 | (2) |
|
14.4 Dimensionality reduction based on a train of low-order tensors |
|
|
545 | (3) |
|
14.4.1 TD-train model: equivalence between a high-order TD and a train of low-order TDs |
|
|
546 | (2) |
|
14.5 Tensor train algorithm |
|
|
548 | (3) |
|
14.5.1 Description of the TT-HSVD algorithm |
|
|
548 | (1) |
|
14.5.2 Comparison of the sequential and the hierarchical schemes |
|
|
549 | (2) |
|
14.6 Kernel-based classification of high-order tensors |
|
|
551 | (4) |
|
14.6.1 Formulation of SVMs |
|
|
552 | (1) |
|
14.6.2 Polynomial and Euclidean tensor-based kernel |
|
|
553 | (1) |
|
14.6.3 Kernel on a Grassmann manifold |
|
|
553 | (1) |
|
14.6.4 The fast kernel subspace estimation based on tensor train decomposition (FAKSETT) method |
|
|
554 | (1) |
|
|
555 | (3) |
|
|
555 | (2) |
|
14.7.2 Classification performance |
|
|
557 | (1) |
|
|
558 | (7) |
|
|
560 | (5) |
Index |
|
565 | |