Preface |
|
xiii | |
PART 1 Understanding and Simplifying Networks |
|
1 | (100) |
|
1 Analyzing the Internal Representations of Trained Neural Networks |
|
|
3 | (24) |
|
|
3 | (2) |
|
|
5 | (19) |
|
1.2.1 Hierarchical cluster analysis |
|
|
6 | (4) |
|
1.2.2 Principal component analysis |
|
|
10 | (1) |
|
1.2.3 Multi-dimensional scaling |
|
|
11 | (2) |
|
1.2.4 Discriminant analysis |
|
|
13 | (5) |
|
1.2.5 Output weight projections |
|
|
18 | (5) |
|
1.2.6 Contribution analysis |
|
|
23 | (1) |
|
|
24 | (3) |
|
2 Information Maximization to Simplify Internal Representation |
|
|
27 | (34) |
|
|
27 | (1) |
|
2.2 Information maximization method |
|
|
28 | (6) |
|
2.2.1 Outline of information maximization |
|
|
28 | (1) |
|
2.2.2 The concept of information |
|
|
28 | (2) |
|
2.2.3 An information theoretic formulation |
|
|
30 | (1) |
|
2.2.4 Information maximization |
|
|
31 | (2) |
|
2.2.5 Cross entropy minimization |
|
|
33 | (1) |
|
2.3 Modified information maximization methods |
|
|
34 | (4) |
|
2.3.1 Outline of modified methods |
|
|
34 | (1) |
|
2.3.2 Information maximizer |
|
|
35 | (2) |
|
2.3.3 Information maximizer with weigth elimination |
|
|
37 | (1) |
|
2.4 Application to the XOR problem |
|
|
38 | (3) |
|
2.5 Application to the symmetry problem |
|
|
41 | (3) |
|
2.6 Application to well-formedness estimation |
|
|
44 | (13) |
|
2.6.1 Well-formedness by sonority |
|
|
44 | (1) |
|
2.6.2 Information and generalization |
|
|
45 | (1) |
|
2.6.3 Interpretation of internal representation |
|
|
46 | (5) |
|
2.6.4 Improved generalization with weight elimination |
|
|
51 | (6) |
|
2.7 Competitive learning and information maximization |
|
|
57 | (2) |
|
|
59 | (2) |
|
3 Rule Extraction From Trained Artificial Neural Networks |
|
|
61 | (40) |
|
|
61 | (1) |
|
3.2 The importance of rule extraction |
|
|
62 | (4) |
|
3.2.1 Provision of a `user explanation' capability |
|
|
63 | (1) |
|
3.2.2 Extension of ANN systems to `safety-critical' problem domains |
|
|
64 | (1) |
|
3.2.3 Software verification and debugging of ANN components in software systems |
|
|
64 | (1) |
|
3.2.4 Improving the generalization of ANN solutions |
|
|
65 | (1) |
|
3.2.5 Data exploration and the induction of scientific theories |
|
|
65 | (1) |
|
3.2.6 Knowledge acquisition for symbolic AI systems |
|
|
65 | (1) |
|
|
66 | (1) |
|
3.4 A classification scheme for rule-extraction algorithms |
|
|
67 | (3) |
|
3.4.1 The expressive power of the extracted rules |
|
|
67 | (1) |
|
3.4.2 The translucency of the underlying ANN units |
|
|
67 | (2) |
|
3.4.3 The portability of the rule-extraction algorithm |
|
|
69 | (1) |
|
3.4.4 Quality of the extracted rules |
|
|
69 | (1) |
|
3.4.5 Complexity of the rule-extraction algorithm |
|
|
70 | (1) |
|
3.5 Rule-extraction techniques |
|
|
70 | (12) |
|
3.5.1 Decompositional approaches to rule extraction |
|
|
70 | (2) |
|
3.5.2 Decompositional algorithms that directly decompile weights to rules |
|
|
72 | (5) |
|
3.5.3 Pedagogical approaches to rule extraction |
|
|
77 | (1) |
|
|
78 | (3) |
|
3.5.5 Eclectic rule-extraction techniques |
|
|
81 | (1) |
|
3.6 Extraction of fuzzy rules |
|
|
82 | (2) |
|
3.7 Techniques for performing rule refinement |
|
|
84 | (4) |
|
3.8 Rule refinement and recurrent networks |
|
|
88 | (4) |
|
3.9 Current issues in rule extraction and refinement |
|
|
92 | (6) |
|
3.9.1 Limitations imposed by inherent algorithmic complexity |
|
|
92 | (1) |
|
3.9.2 Limitations on achieving simultaneously high accuracy, high fidelity, and high comprehensibility |
|
|
93 | (1) |
|
3.9.3 Rule extraction and the quality of ANN solutions |
|
|
94 | (1) |
|
3.9.4 Functional dependences, causal factors and rule extraction |
|
|
95 | (3) |
|
3.9.5 Extension to connectionist knowledge representation techniques |
|
|
98 | (1) |
|
|
98 | (3) |
PART 2 Novel Architectures and Algorithms |
|
101 | (82) |
|
4 Pulse-Stream Techniques and Circuits for Implementing Neural Networks |
|
|
103 | (7) |
|
|
103 | (1) |
|
4.2 Pulse-stream encoding |
|
|
103 | (1) |
|
4.3 Basic neural computation on a chip |
|
|
104 | (2) |
|
4.4 Computation using an analogue, two-quadrant multiplier |
|
|
106 | (2) |
|
4.5 Designing a pulsed multiplier-a case study |
|
|
108 | (3) |
|
|
111 | (1) |
|
4.7 Results from the multiplier circuit |
|
|
112 | (2) |
|
4.8 Converting analogue outputs and inputs into pulses |
|
|
114 | (1) |
|
|
115 | (1) |
|
|
115 | (2) |
|
4.11 Process dependence of signals as VLSI technologies are scaled down |
|
|
117 | (1) |
|
4.12 ANN, and other, applications |
|
|
118 | (1) |
|
4.13 Further reading on applications of the technique |
|
|
118 | (2) |
|
5 Cellular Neural Networks |
|
|
120 | (16) |
|
|
120 | (1) |
|
|
121 | (6) |
|
5.3 CNN dynamical systems |
|
|
127 | (7) |
|
5.4 Conclusions: the CNN as supercomputer or neural network? |
|
|
134 | (2) |
|
6 Efficient Training of Feed-Forward Neural Networks |
|
|
136 | (38) |
|
|
136 | (1) |
|
6.2 Notation and basic definitions |
|
|
137 | (1) |
|
6.3 Optimization strategy |
|
|
138 | (1) |
|
|
139 | (13) |
|
|
140 | (1) |
|
|
141 | (2) |
|
6.4.3 Gradient descent with momentum |
|
|
143 | (1) |
|
6.4.4 Adaptive learning rate and momentum |
|
|
144 | (3) |
|
6.4.5 Learning rate schedules for on-line gradient descent |
|
|
147 | (1) |
|
6.4.6 The quickprop method |
|
|
148 | (1) |
|
6.4.7 Estimation of optimal learning rate and reduction of large curvature components |
|
|
149 | (3) |
|
|
152 | (13) |
|
6.5.1 Non-interfering directions of search |
|
|
152 | (3) |
|
|
155 | (2) |
|
6.5.3 Scaled conjugate gradient |
|
|
157 | (6) |
|
6.5.4 Stochastic conjugate gradient |
|
|
163 | (2) |
|
6.6 Newton related methods |
|
|
165 | (1) |
|
6.7 On-line versus off-line discussion |
|
|
166 | (6) |
|
|
172 | (2) |
|
7 Exploiting Local Optima in Multiversion Neural Computing |
|
|
174 | (9) |
|
7.1 Local optima as a neural computing problem |
|
|
174 | (1) |
|
7.2 The group properties of local optima |
|
|
175 | (2) |
|
7.3 Engineering productive sets of local optima |
|
|
177 | (1) |
|
7.4 Optimizing performance over local optima sets |
|
|
177 | (2) |
|
7.5 Applications of the multiversion approach |
|
|
179 | (1) |
|
7.6 Discussion and conclusions |
|
|
180 | (3) |
PART 3 Applications |
|
183 | (68) |
|
8 Neural and Neuro-Fuzzy Control Systems |
|
|
185 | (19) |
|
|
185 | (1) |
|
|
185 | (2) |
|
|
187 | (1) |
|
|
188 | (5) |
|
8.4.1 Approximating the inverse transfer function |
|
|
188 | (1) |
|
8.4.2 Emulating a controller |
|
|
189 | (1) |
|
8.4.3 Reinforcement learning |
|
|
190 | (3) |
|
|
193 | (4) |
|
|
193 | (1) |
|
|
193 | (1) |
|
8.5.3 Fuzzy sets and membership functions |
|
|
193 | (1) |
|
8.5.4 Fuzzy logic control (FLC) |
|
|
194 | (3) |
|
|
197 | (6) |
|
8.6.1 Adaptive fuzzy associative memory (AFAM) |
|
|
197 | (3) |
|
8.6.2 Takagi-Sugeno-Kang (TSK) |
|
|
200 | (1) |
|
|
201 | (2) |
|
|
203 | (1) |
|
9 Image Compression using Neural Networks |
|
|
204 | (20) |
|
|
204 | (1) |
|
|
205 | (1) |
|
9.3 Basic image compression |
|
|
206 | (8) |
|
9.3.1 Basic compression implementations |
|
|
208 | (2) |
|
9.3.2 CODECs with multiple compression networks |
|
|
210 | (1) |
|
|
210 | (2) |
|
9.3.4 Different neural network models |
|
|
212 | (2) |
|
9.4 The random neural network model |
|
|
214 | (4) |
|
9.4.1 Image compression with the RNN |
|
|
216 | (2) |
|
|
218 | (4) |
|
9.5.1 Adaptive vector quantization |
|
|
219 | (1) |
|
9.5.2 Adaptive Kohonen networks |
|
|
220 | (1) |
|
9.5.3 VLSI Kohonen networks |
|
|
220 | (1) |
|
9.5.4 Predictive vector quantization |
|
|
221 | (1) |
|
|
222 | (2) |
|
10 Oil Spill Detection: a Case Study using Recurrent Artificial Neural Networks |
|
|
224 | (27) |
|
|
224 | (1) |
|
|
225 | (7) |
|
|
225 | (1) |
|
10.2.2 The SLAR simulation model |
|
|
225 | (2) |
|
|
227 | (3) |
|
10.2.4 Design of network architecture |
|
|
230 | (2) |
|
|
232 | (9) |
|
|
232 | (1) |
|
10.3.2 Reliability and sensitivity |
|
|
233 | (4) |
|
10.3.3 Robustness to varying sea states |
|
|
237 | (2) |
|
10.3.4 Further experiments |
|
|
239 | (2) |
|
10.4 Analysis and discussion |
|
|
241 | (7) |
|
10.4.1 Functional analysis |
|
|
242 | (2) |
|
10.4.2 Analysis of internal states |
|
|
244 | (4) |
|
10.5 Summary and conclusion |
|
|
248 | (3) |
Bibliography |
|
251 | (12) |
Index |
|
263 | |