Use Case
Transient Detection in Radio Astronomy Search Data
J. Cordes (Cornell) with M. McLaughlin (West Virginia University), V. Kaspi (McGill University), B. Allen (Albert Einstein Institute), A. Brazier, P. Vaillancourt (Cornell CAC), and PALFA collaborators located at the National Radio Observatory, Max Planck Institute for Radio Astronomy, University of British Columbia-Vancouver, University of Texas at Brownville, Naval Research Laboratory, Franklin and Marshall College, University of Wisconsin-Milwaukee, and Columbia University
What?
- Low-latency/on-demand searching of time-varying, including one-time signals in radio astronomy data
- Builds on existing Cornell/CAC collaboration
Why Aristotle?
- Bursting on-demand to analyze new data
- High-speed transport of very large data sets into and around the Federation
Accomplishments
- Developed a new containerized solution that features a full suite of all relevant pulsar-processing software, Python Conda distributions, the core PRESTO pulsar search package to perform transient search using the Spitler modulation index method, and the decimation code.
- Implemented an improved reproduction of the PALFA2 pipeline for detecting single pulse candidates that may be a Fast Radio Burst (FRB) source; this pipeline includes PRESO functionality, modulation index calculation, parameter customization, and the production of graphic data output.
- Built a new flexible framework for running radio astronomy searches; data are read from their native format into NumPy arrays, and the pipeline’s routines are selected from a configuration file and includes a friends-of-friends search, and also allows the running of the pipeline of Laura Spitler who discovered FRB 1211102.
- Improved the Fast Radio Burst (FRB) Pipeline, a customizable scientific software pipeline for detecting single pulse candidates that may be FRB sources in radio astronomy data, by preparing to parallelize certain methods, updating the Python 3 version of PRESTO, and adding functionality for general radio transient detection beyond just FRBs.
- Added checkpoint/restart functionality to the pipeline to support tests on AWS spot instances as part of the RightScale investigation.
- Updated container and pipeline documentation.
- Implemented the workflow for the "Friends-Of-Friends" algorithm in parallel and prepared to deploy it in containerized form to a cluster of Spot instances in an automated cloud deployment.
- Demonstrated Friends of Friends workflows work well for smaller data sizes, but are prohibitively expensive if attempting to run large amounts of data for a long time. Exploration of how to do more processing in parallel is needed.
- Completed and tested the single container of radio astronomy software combining the pipeline components developed for Pulsar and other transient detections that can be deployed either on the cloud (Docker) or on an XSEDE HPC resource (Singularity), and released the container build files and images publicly via GitHub and DockerHub.
- Created a Singularity container packaging our pulsar and single pulse search pipeline, and secured trial XSEDE allocations for deploying the container on Bridges and for running pulsar/FRB searches on large data volumes.
Plans
- Continue to make improvements to the FOF algorithm in the FRB_pipeline and complete the comparison with Single Pulse Search method in PRESTO with the modulation index calculation. A new dataset that has not been processed yet will then be explored with the FOF algorithm in search of FRB detections.
- Continue to use Aristotle for the prototyping of FRB search codes and the development of deployable Docker and Singularity containers for deployment on large-scale XSEDE resources and public clouds.
Products
