Agile methodologies are current best practice in software development. They are favored for, among other reasons, preventing premature optimization by taking a somewhat short-term focus, and allowing frequent replans/reprioritizations of upcoming development work based on recent results and current backlog. At the same time, funding agencies prescribe earned value management accounting for large projects which, these days, inevitably include substantial software components. Earned Value approaches emphasize a more comprehensive and typically longer-range plan, and tend to characterize frequent replans and reprioritizations as indicative of problems. Here we describe the planning, execution and reporting framework used by the LSST Data Management team, that navigates these opposite tensions.
KEYWORDS: Large Synoptic Survey Telescope, Databases, Data storage, Data modeling, Data archive systems, Data centers, Image processing, Lanthanum, Cameras, Telescopes
The Large Synoptic Survey Telescope (LSST) project is a proposed large-aperture, wide-field, ground-based telescope
that will survey half the sky every few nights in six optical bands. LSST will produce a data set suitable for answering a
wide range of pressing questions in astrophysics, cosmology, and fundamental physics. The 8.4-meter telescope will be
located in the Andes mountains near La Serena, Chile. The 3.2 Gpixel camera will take 6.4 GB images every 15
seconds, resulting in 15 TB of new raw image data per night. An estimated 2 million transient alerts per night will be
generated within 60 seconds of when the camera’s shutter closes. Processing such a large volume of data, converting the
raw images into a faithful representation of the universe, automated data quality assessment, automated discovery of
moving or transient sources, and archiving the results in useful form for a broad community of users is a major
challenge. We present an overview of the planned computing infrastructure for LSST. The cyberinfrastructure required
to support the movement, storing, processing, and serving of hundreds of petabytes of image and database data is
described. We also review the sizing model that was developed to estimate the hardware requirements to support this
environment beginning during project construction and continuing throughout the 10 years of operations.
KEYWORDS: Large Synoptic Survey Telescope, Data centers, Data storage, Prototyping, Data archive systems, Space telescopes, Telescopes, Failure analysis, Lanthanum, Image processing
Large ground-based and space-based telescopes are expected to make exciting discoveries in the upcoming decade.
These large projects start their construction phase many years before first-light and continue to operate for many years
after first-light and usually span multiple countries. The file-storage cyberinfrastructure ("file-storage CI") of these largescale
projects has to evolve over several years from a conceptual prototype to a highly flexible data distribution network.
During this long period the file-storage CI has to transition into multiple stages, starting with a conceptual prototype
before first-light, to a large-scale distributed network in production, and finally into a persistent archive once the project
is decommissioned. While the project makes these transitions, the file-storage CI has to incorporate several requirements
including but not limited to: Technology Evolution, due to changes in Cyberinfrastructure (CI) software or hardware
during the lifetime of the project; International Partnerships that are updated during the various phases of the project;
and Data Lifecycle that exists in the project. The file-storage and management software's architecture has to be designed
with significant consideration of these requirements for these large projects. In this paper, we provide the generic
requirements, for file-storage and management cyberinfrastructure in a large project similar to LSST before first-light.
KEYWORDS: Databases, Large Synoptic Survey Telescope, Data centers, Data archive systems, Image processing, Data storage, Prototyping, Observatories, Data modeling, Cameras
The 3.2 giga-pixel LSST camera will produce approximately half a petabyte of archive images every month. These data need to be reduced in under a minute to produce real-time transient alerts, and then added to the cumulative catalog for further analysis. The catalog is expected to grow about three hundred terabytes per year. The data volume, the real-time transient alerting requirements of the LSST, and its spatio-temporal aspects require innovative techniques to build an efficient data access system at reasonable cost. As currently envisioned, the system will rely on a database for catalogs and metadata. Several database systems are being evaluated to understand how they perform at these data rates, data volumes, and access patterns. This paper describes the LSST requirements, the challenges they impose, the data access philosophy, results to date from evaluating available database technologies against LSST requirements, and the proposed database architecture to meet the data challenges.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.