Big Data in Radio Astronomy: Scientific Data Processing for Advanced Radio Telescopes provides the latest research developments in big data methods and techniques for radio astronomy. Providing examples from such projects as the Square Kilometer Array (SKA), the world’s largest radio telescope that generates over an Exabyte of data every day, the book offers solutions for coping with the challenges and opportunities presented by the exponential growth of astronomical data. Presenting state-of-the-art results and research, this book is a timely reference for both practitioners and researchers working in radio astronomy, as well as students looking for a basic understanding of big data in astronomy.
- Bridges the gap between radio astronomy and computer science
- Includes coverage of the observation lifecycle as well as data collection, processing and analysis
- Presents state-of-the-art research and techniques in big data related to radio astronomy
- Utilizes real-world examples, such as Square Kilometer Array (SKA) and Five-hundred-meter Aperture Spherical radio Telescope (FAST)
Contributors |
|
xi | |
Preface |
|
xiii | |
Acknowledgments |
|
xv | |
|
|
|
Chapter 1 Introduction to radio astronomy |
|
|
3 | (26) |
|
|
|
1 The history of astronomy |
|
|
3 | (2) |
|
2 What is radio astronomy |
|
|
5 | (10) |
|
3 Advanced radio telescope |
|
|
15 | (5) |
|
4 The challenge of radio astronomy |
|
|
20 | (5) |
|
5 The development tendency of radio astronomy |
|
|
25 | (2) |
|
|
27 | (2) |
|
Chapter 2 Fundamentals of big data in radio astronomy |
|
|
29 | (32) |
|
|
|
|
29 | (8) |
|
2 Increasing data volumes of telescopes |
|
|
37 | (3) |
|
3 Existing methods for the value chain of big data |
|
|
40 | (10) |
|
4 Current statistical methods for astronomical data analysis |
|
|
50 | (2) |
|
5 Platforms for big data processing |
|
|
52 | (6) |
|
|
58 | (3) |
|
Part B Big data processing |
|
|
|
Chapter 3 Preprocessing pipeline on FPGA |
|
|
61 | (22) |
|
|
|
|
|
61 | (4) |
|
|
65 | (4) |
|
3 Time-frequency domain transposing |
|
|
69 | (4) |
|
4 Correlators based on FPGA |
|
|
73 | (2) |
|
5 General architectures for data reduction design and implementation |
|
|
75 | (5) |
|
|
80 | (1) |
|
|
80 | (3) |
|
Chapter 4 Real-time stream processing in radio astronomy |
|
|
83 | (30) |
|
|
|
83 | (1) |
|
|
84 | (2) |
|
3 Heterogeneous signal processing |
|
|
86 | (4) |
|
|
90 | (5) |
|
5 First-stage data processing |
|
|
95 | (2) |
|
|
97 | (3) |
|
7 Second-stage processing |
|
|
100 | (7) |
|
|
107 | (2) |
|
|
109 | (1) |
|
|
109 | (4) |
|
Chapter 5 Digitization, channelization, and packeting |
|
|
113 | (26) |
|
|
|
|
113 | (6) |
|
|
119 | (7) |
|
|
126 | (7) |
|
|
133 | (6) |
|
Chapter 6 Processing data of correlation on GPU |
|
|
139 | (26) |
|
|
|
|
|
|
|
|
139 | (3) |
|
2 GPU-based cross-correlator engines |
|
|
142 | (3) |
|
3 Applying and implementing gridding algorithm after cross-correlator |
|
|
145 | (10) |
|
4 Applying and implementing deconvolution algorithm and parallel implementation after cross-correlator |
|
|
155 | (6) |
|
|
161 | (1) |
|
|
161 | (4) |
|
Chapter 7 Flux calibration for single-dish radio telescopes |
|
|
165 | (20) |
|
|
|
|
165 | (2) |
|
|
167 | (1) |
|
3 Processing spectral line data |
|
|
168 | (9) |
|
4 Observations of a brown dwarf by Arecibo single dish |
|
|
177 | (6) |
|
|
183 | (2) |
|
Chapter 8 Imaging algorithm optimization for scale-out processing |
|
|
185 | (30) |
|
|
|
|
|
|
|
185 | (6) |
|
2 Gridding and degridding |
|
|
191 | (6) |
|
3 The choice of the gridding function in the era of big data |
|
|
197 | (10) |
|
4 Bayesian source discrimination |
|
|
207 | (3) |
|
|
210 | (5) |
|
Part C Computing technologies |
|
|
|
Chapter 9 Execution framework technology |
|
|
215 | (30) |
|
|
|
|
|
|
|
|
215 | (2) |
|
|
217 | (7) |
|
|
224 | (18) |
|
|
242 | (1) |
|
|
242 | (3) |
|
Chapter 10 Application design for execution framework |
|
|
245 | (26) |
|
|
|
|
|
|
|
1 OpenCluster applications design |
|
|
245 | (3) |
|
2 MUSER pipeline using OpenCluster |
|
|
248 | (4) |
|
3 Design CHILES on AWS using DALiuGE |
|
|
252 | (2) |
|
4 The migration of SAGECal/MPI to DALiuGe |
|
|
254 | (14) |
|
|
268 | (1) |
|
|
268 | (1) |
|
|
269 | (2) |
|
Chapter 11 Heterogeneous computing platform for backend computing tasks |
|
|
271 | (34) |
|
|
|
|
|
|
|
271 | (1) |
|
2 Computing architecture and platform |
|
|
272 | (9) |
|
|
281 | (13) |
|
4 Telescopes and applications |
|
|
294 | (4) |
|
|
298 | (1) |
|
|
298 | (4) |
|
|
302 | (3) |
|
Chapter 12 High-performance computing for astronomical big data |
|
|
305 | (20) |
|
|
|
|
|
|
|
|
305 | (2) |
|
2 Execution framework and prototype test |
|
|
307 | (10) |
|
3 Improving SKA algorithm reference library on high-performance computing platform |
|
|
317 | (5) |
|
|
322 | (1) |
|
|
322 | (3) |
|
Chapter 13 Spark and dask performance analysis based on ARL image library |
|
|
325 | (22) |
|
|
|
|
|
|
|
|
325 | (2) |
|
2 Preliminaries and notations |
|
|
327 | (4) |
|
|
331 | (7) |
|
4 Task scheduling based on data processing capacity |
|
|
338 | (4) |
|
5 Network connection model and routing topology model |
|
|
342 | (2) |
|
|
344 | (1) |
|
|
345 | (2) |
|
Chapter 14 Applications of artificial intelligence in astronomical big data |
|
|
347 | (32) |
|
|
|
|
|
347 | (1) |
|
2 Machine learning for astronomical data calibration and repair |
|
|
348 | (1) |
|
3 Artificial intelligence algorithms in astronomy data classification and preprocessing |
|
|
349 | (14) |
|
4 Artificial intelligence application in astronomy data analysis |
|
|
363 | (10) |
|
|
373 | (1) |
|
|
373 | (6) |
|
Part D Future developments |
|
|
|
Chapter 15 Mapping the universe with 21cm observations |
|
|
379 | (28) |
|
|
1 The neutral hydrogen and 21 cm line |
|
|
379 | (6) |
|
|
385 | (10) |
|
|
395 | (8) |
|
|
403 | (2) |
|
|
405 | (1) |
|
|
406 | (1) |
Index |
|
407 | |
Linghe Kong is currently a Research Professor in Department of Computer Science and Engineering at Shanghai Jiao Tong University and an engineer in the scientific data processing group in SKA China. Before that, he was a postdoctoral researcher at Columbia University and McGill University. He received his Ph.D. degree from Shanghai Jiao Tong University, China, his Masters degree from TELECOM SudParis, France, and his B. E. degree from Xidian University, China. His research interests include big data, Internet of things, and mobile computing systems. He has published more than 60 papers in refereed journals and conferences, such as ACM MobiCom, IEEE INFOCOM, IEEE RTSS, IEEE ICDCS, IEEE TMC, and IEEE TPDS. He serves on the editorial boards of several journals including Springer Telecommunication Systems and KSII Transactions on Internet and Information Systems. He organized several special issues such as in IEEE Communications Magazine and in the Computer Journal. He is a senior member of IEEE. Tian Huang is Research Associate of the Astrophysics Group, Cavendish Lab, University of Cambridge. He takes part in multiple radio telescope array projects and mainly focuses on data preprocessing and quality metrics. In March 2016, he graduated from the School of Microelectronics at Shanghai Jiao Tong University, where he completed his PhD thesis. His main research interest is Data Mining for time series, including time series big data indexing, anomaly detecting, and computer architecture for time series data mining and statistical models for time series data. He has published 9 SCI journal and 18 EI conference papers. He has rich experience on software and hardware co-designing. Yongxin Zhu is a full Professor at Shanghai Advanced Research Institute, Chinese Academy of Sciences (CAS). He is also an Adjunct Professor with the School of Microelectronics at the Shanghai Jiao Tong University (SJTU). He is currently the technical leader of Chinese Consortium of Science Data Processor (SDP) for Square Kilometre Array Telescope. He has published over 130 English journal and conference papers, 40 Chinese journal papers and 20 China patent approvals in the areas of computer architecture, embedded systems, and big data processing. With around 1,000 citations of these works in recent years, he has received recognition in China and Asia with IEEE best paper award, Shanghai innovation award, SJTU Annual Outstanding Teacher Award and Bilingual Teaching Award. To date, he has received around 20 million RMB in grants from various funding agencies and industrial partners in China. Prior to his tenure with CAS and SJTU, he worked as a research fellow with the National University of Singapore in 2002-2005, a senior consultant with S1 Incorporation (inventor of the 1st Internet banking in the world) in 1999-2002 and a teaching assistant with the Department of Computer Science and Engineering, SJTU in 1994-1995. He is a guest editor of Journal of Systems Architecture, senior member of IEEE and China Computer Federation (CCF). He has been also a Visiting Professor with National University of Singapore since 2013. Shenghua Yu is an Associate Professor at National Astronomical Observatories, Chinese Academy of Sciences (CAS). He received his PhD degree in astrophysics from Queens University of Belfast in 2012, and worked as a post-doc fellow at University of Western Australia in 2014-2015. His main research interests include gravitational wave astrophysics and astronomy, double compact objects, radio emission from ultracool dwarfs and radiation mechanisms. He has published ~13 SCI journal papers, 3 Chinese journal and conference papers, 2 China patent approvals, and 3 China computer software copyrights in the research areas.