NASA’s artificial intelligence (AI) capabilities through joint efforts with NASA’s FDL

As the volume of data in the EOSDIS archive continues to grow, the EOSDIS archive’s data ingest rate is expected to increase dramatically along with it. By 2030, the volume of data in the EOSDIS archive is expected to surpass 320 PB.

For the Earthdata Cloud to meet users’ needs, NASA's Earth Science Data and Information System (ESDIS) Project is working to ensure it provides services in several key areas, including:

  • Data acquisition from data providers (such as NASA science teams)

  • Data ingest: The system must support multi-mission and multi-discipline data ingest

  • Data validation and processing

  • Data archive: The system must preserve and protect NASA Earth observation data

  • Data distribution, including disaster recovery: The system must support distribution of data, subsetting, and visualization, and must be adaptable to future technologies

  • Metadata: The harvest, creation, and publication of dataset metadata to the CMR

  • Data management: The system must meet the development and execution of information lifecycle needs of NASA mission-based Earth science datasets

  • Metrics: Publication of metrics to the ESDIS Metrics System (EMS), which collects and organizes various metrics from the DAACs and other data providers

  • NASA’s agreement with AWS has resulted in collaborations to improve the discovery, access, and use of NASA science datasets; the creation of data storage and staging areas to facilitate the community evaluation of data products; and workshops to expand the use of cloud-computing resources

  • NASA’s collaboration with Google has led to investigations into the transfer, storage, and value of making large volumes of NASA science datasets available on the Google Cloud and Google Earth Engine; making NASA Earth Science data accessible to users via the Google Cloud Public Dataset search engine and Earth Engine Catalog; and growing NASA’s artificial intelligence (AI) capabilities through joint efforts with NASA’s Frontier Development Lab (FDL) Challenges and SpaceML projects

  • NASA’s partnership with Microsoft has launched investigations into the value of making high-value NASA science datasets available on Azure; cost and performance evaluations of data storage methods and technologies and support analytics; the exploration of strategies to enable cloud-based analytics to promote science in the cloud; and analysis of the approaches to build and share training datasets for AI at a scale

For example, NASA implemented Cumulus, which provides a range of functionality in the cloud, including data acquisition from providers (such as NASA science teams); data ingest, including validation and processing; the harvest, creation, and publication of dataset metadata to the CMR; the storage and distribution of data, including disaster recovery; and publication of metrics to the EMS, which collects and organizes various metrics from the DAACs and other data providers.

Further, Cumulus is integrated with the NASA-Compliant General Application Platform (NGAP), a custom-built cloud optimized platform, which provides highly flexible cloud native infrastructure, NASA-compliant IT Security controls, networking services, and business cost control in Amazon Web Services (AWS).

Moving the collective data archive from the DAACs into the cloud puts NASA Earth observation data “close to compute,” giving users improved access to data, the ability to use large datasets more efficiently, and the ability to conduct a broader range of research. This move will not change existing methods of user interaction with EOSDIS data, but it does require new methods of accessing NASA data that differs from on-premises platforms. Further, as more datasets migrate to the cloud, the DAACs will continue to serve as the gateways to EOSDIS data holdings and provide a wide range of support services for users.

Previous
Previous

The Next Big Solar Storm Could Fry the Grid

Next
Next

Frontier Development Lab Launches FDL-X to Advance AI Capabilities Ahead of Increased Solar Activity