Publications of Torsten Hoefler

Langwen Huang, Luigi Fusco, Florian Scheidl, Jan Zibell, Michael Armand Sprenger, Sebastian Schemm, Torsten Hoefler:

Error bounded compression for weather and climate applications

(Presentation - EGU General Assembly 2025. presented in Vienna, Austria, Apr. 2025)

Abstract

As the resolution of weather and climate simulations increases, the amount of data produced is growing rapidly from hundreds of terabytes to tens of petabytes. The huge size becomes a limiting factor for broader adoption, and its fast growth rate will soon exhaust all the available storage devices. To address these issues, we present EBCC (Error Bounded Climate Compressor). It follows a two-layer compression approach: a base compression layer using JPEG2000 to capture the bulk of the data with a high compression ratio, and a residual compression layer using wavelet transform and SPIHT (Set Partitioning In Hierarchical Trees) encoding to efficiently eliminate long-tail extreme errors introduced by the base compression layer. It incorporates a feedback rate-control mechanism for both layers that adjusts compression ratios to achieve the specified maximum error target. We evaluate EBCC alongside other established error-bounded compression methods on several benchmarks related to weather and climate science. The benchmarks include energy spectrum, case study on primitive and derived variables near a hurricane, evaluation of the closure of the energy budget, and a Lagrangian air parcel trajectory simulation. This is the first time that trajectories are used to verify compressing methods. According to the histogram of the compression errors, our method concentrates most errors near zero, while other error bounded methods tend to distribute errors uniformly within the error bound. It outperforms other methods in most benchmarks at relative error targets ranging from 0.1% to 10% and achieves compression ratios from 16x to 542x, respectively.

Documents

download slides:

BibTeX

@misc{langwen-egu25-ebcc,
  author={Langwen Huang and Luigi Fusco and Florian Scheidl and Jan Zibell and Michael Armand Sprenger and Sebastian Schemm and Torsten Hoefler},
  title={{Error bounded compression for weather and climate applications }},
  journal={EGU General Assembly 2025},
  year={2025},
  month={Apr.},
  location={Vienna, Austria},
  source={http://www.unixer.de/~htor/publications/},
}


serving: 216.73.216.217:15155	© Torsten Hoefler