Skip to content

Latest commit

 

History

History
86 lines (76 loc) · 17.1 KB

README.md

File metadata and controls

86 lines (76 loc) · 17.1 KB
Raw

Compiler-benchmark-suites

A list of benchmark suites (and some loose kernels) used in the research related to compilers, program performance, scientific computations etc.

Benchmark suites

Benchmark Lang Platform Published in Content Additional info
AnghaBench C CPU [DaSilva2021] ~1 million benchmarks Obtained from open-source repositories; for analysis of predictive compilers
BEEBS C Embedded [Pallister2013] 10 benchmarks For analysis of energy consumption on embedded platforms
Berkeley Multimedia Workload C CPU (MMX/SSE/...) [Slingerland2002] 14 applications Architectural support for multimedia applications
Embench™ C Embedded [Patterson2019] 19 benchmarks Modern benchmarks for performance analysis built on top of selected BEEBS benchmarks
GridNBP 3.1 Fortran, Java (v3.0) CPU [VanDerWijngaart2002] 4 benchmarks Evaluation of the computational grids, their communications and distributed tasks.
LCALS v1.0.2 C CPU [Hornung2013] 32 loops For analysis of compiler optimisations; modern extension to Livermore loops; more information
Livermore loops (LFK) FORTRAN CPU [McMahon1986] 24 loops For performance analysis
Livermore loops C CPU 1992 24 loops Port to C
Mälardalen WCET Benchmarks C CPU [Gustafsson2010] 35 benchmarks For Worst-Case Execution Time (WCET) analysis
MediaBench C CPU [Lee1997] 19 applications For analysis of Instruction-Level Parallelism (ILP) in compilers; source code
MediaBench II C CPU [Fritts2005, Fritts2009] 11 applications General analysis of compilers on video/media related application
MiBench C Embedded [Guthaus2001] 6 benchmarks For performance analysis
NAS Parallel Benchmarks 3.4.2 C, Fortran, MPI, OpenMP, Java (v3.0) CPU [Bailey1991] 5 kernels, 3 programs, 4 other benchmarks Evaluation of the performance of parallel supercomputers; more information
PARKBENCH 2.1.1 Fortran, MPI, PVM CPU [Hockney1994] 10 measurement codes, 7 kernels, 3 application For performance analysis of parallel architectures; more information
PARSEC 3.0 C, OpenMP, pthreads CPU [Bienia2008] 13 programs For research on parallelization; other versions
PolyBench/ACC C, CUDA, OpenCL, OpenACC, HMPP, OpenMP GPU, accelerators [Grauer-Gray2012] 29 kernels Updated PolyBench/GPU; GitHub
PolyBench/C 4.2 C CPU 2010 30 kernels For the analysis of performance and compiler optimisations (especially related to polyhedra compilation); more information; SourceForge
PolyBench/Fortran 1.0 FORTRAN CPU 2011 30 kernels Port to FORTRAN; more information
PolyBench/GPU 1.0 C, CUDA, OpenCL, OpenHMPP CPU, GPU [Grauer-Gray2012] 15 kernels Port to heterogeneous architectures; more information
Rodinia v3.1 C, OpenMP, OpenCL, CUDA CPU, GPU [Che2009] 23 applications (2 under evaluations) Performance analysis of heteregenous architectures
SPLASH C CPU [Singh1992] 4 kernels, 8 programs For research on parallelization; source code unavailable
SPLASH-2 C CPU [Woo1995] 4 kernels, 8 programs For research on parallelization; original source code is unavailable, several modifications: staceyson/splash2, liuyix/splash2_benchmark, andysan/splash2
SPLASH-3 C CPU [Sakalis2016] 4 kernels, 8 programs For research on parallelization
TSVC FORTRAN CPU [Callahan1988] 135 loops For testing automatic vectorizing compilers; Single precision version
TSVC C CPU [Maleki2011] 151 loops Extended port to C
TSVC 2 C CPU 2015 151 loops Update to TSVC in C
UTDSP C CPU 1992 6 loops, 12 programs For testing compilers on Digitial Signal Processing (DSP) applications; more information
VersaBench C Custom CPUs [Rabbah2004] 15 programs For the performance analysis of new flexible CPU architectures

Full names

  • BEEBS — Bristol/Embecosm Embedded Benchmark Suite
  • LCALS — Livermore Compiler Analysis Loop Suite
  • LFK — Livermore FORTRAN Kernels
  • NPB — NAS Parallel Benchmarks
  • PARKBENCH — PARallel Kernels and BENCHmarks
  • PARSEC — The Princeton Application Repository for Shared-Memory Computers
  • PolyBench — The Polyhedral Benchmark Suite
  • SPLASH — Stanford Parallel Applications for Shared-Memory
  • TSVC — Test Suite for Vectorizing Compilers
  • VersaBench — A Benchmark Suite for Versatile Architectures

Collections of benchmarks

Collections of loose kernels

References

  • [Bailey1991] D. H. Bailey et al., "The NAS parallel benchmarks summary and preliminary results," Supercomputing '91: Proceedings of the 1991 ACM/IEEE Conference on Supercomputing, 1991, pp. 158-165, doi: https://doi.org/10.1145/125826.125925.
  • [Bienia2008] Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. 2008. "The PARSEC benchmark suite: characterization and architectural implications". In Proceedings of the 17th international conference on Parallel architectures and compilation techniques (PACT '08). Association for Computing Machinery, New York, NY, USA, 72–81. doi:https://doi.org/10.1145/1454115.1454128
  • [Callahan1988] Callahan, D., Dongarra, J., & Levine, D. (1988). "Vectorizing compilers: a test suite and results". In Proceedings. SUPERCOMPUTING ’88 (pp. 98–105). IEEE Comput. Soc. Press. https://doi.org/10.1109/SUPERC.1988.44642
  • [Che2009] S. Che et al., "Rodinia: A benchmark suite for heterogeneous computing," 2009 IEEE International Symposium on Workload Characterization (IISWC), 2009, pp. 44-54, doi: https://doi.org/10.1109/IISWC.2009.5306797
  • [DaSilva2021] A. F. da Silva, B. C. Kind, J. W. de Souza Magalhães, J. N. Rocha, B. C. Ferreira Guimarães and F. M. Quinão Pereira, "ANGHABENCH: A Suite with One Million Compilable C Benchmarks for Code-Size Reduction," 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), 2021, pp. 378-390, doi: https://doi.org/10.1109/CGO51591.2021.9370322
  • [Fritts2005] Jason E. Fritts, Frederick W. Steiling, Joseph A. Tucek, "MediaBench II video: expediting the next generation of video systems research," Proc. SPIE 5683, Embedded Processors for Multimedia and Communications II, (8 March 2005); https://doi.org/10.1117/12.587955
  • [Fritts2009] Jason E. Fritts, Frederick W. Steiling, Joseph A. Tucek, and Wayne Wolf. 2009. MediaBench II video: Expediting the next generation of video systems research. Microprocess. Microsyst. 33, 4 (June, 2009), 301–318. https://doi.org/10.1016/j.micpro.2009.02.010
  • [Guthaus2001] M. R. Guthaus, J. S. Ringenberg, D. Ernst, T. M. Austin, T. Mudge and R. B. Brown, "MiBench: A free, commercially representative embedded benchmark suite," Proceedings of the Fourth Annual IEEE International Workshop on Workload Characterization. WWC-4 (Cat. No.01EX538), Austin, TX, USA, 2001, pp. 3-14, doi:https://doi.org/10.1109/WWC.2001.990739
  • [Grauer-Gray2012] Scott Grauer-Gray, Lifan Xu, Robert Searles, Sudhee Ayalasomayajula, and John Cavazos. "Auto-tuning a High-Level Language Targeted to GPU Codes". Proceedings of Innovative Parallel Computing (InPar '12), 2012. https://doi.org/10.1109/InPar.2012.6339595
  • [Gustafsson2010] Jan Gustafsson, Adam Betts, Andreas Ermedahl, and Björn Lisper. "The Mälardalen WCET benchmarks: Past, present and future". 10th International Workshop on Worst-Case Execution Time Analysis, WCET 2010, July 6, 2010, Brussels, Belgium. https://doi.org/10.4230/OASIcs.WCET.2010.136
  • [Hockney1994] Roger Hockney, and Michael Berry. "Public International Benchmarks for Parallel Computers". Technical report. February 7, 1994. https://netlib.org/parkbench/parkbench.ps
  • [Hornung2013] Richard D. Hornung and Jeffrey A. Keasler, “A Case for Improved C++ Compiler Support to Enable Performance Portability in Large Physics Simulation Codes”, LLNL-TR-635681 (2013). https://computing.llnl.gov/projects/co-design/compilersupportwhitepaper_tr-635681_cover.pdf
  • [Lee1997] Chunho Lee, M. Potkonjak and W. H. Mangione-Smith, "MediaBench: a tool for evaluating and synthesizing multimedia and communications systems," Proceedings of 30th Annual International Symposium on Microarchitecture, 1997, pp. 330-335, doi:https://doi.org/10.1109/MICRO.1997.645830
  • [Maleki2011] Maleki, S., Gao, Y., Garzarán, M. J., Wong, T., & Padua, D. A. (2011). "An evaluation of vectorizing compilers". Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT, 7, 372–382. https://doi.org/10.1109/PACT.2011.68
  • [McMahon1986] F. H. McMahon. "Livermore fortran kernels: A computer test of numerical performance range". Technical Report UCRL-53745, Lawrence Livermore National Laboratory, Livermore, CA, December 1986; COMMENT: no PDF on the web.
  • [Patterson2019] David Patterson, Jeremy Bennett, Palmer Dabbelt, Cesare Garlati, G. S. Madhusudan and Trevor Mudge. "Embench™: An Evolving Benchmark Suite for Embedded IoT Computers from an Academic-Industrial Cooperative: Towards the Long Overdue and Deserved Demise of Dhrystone" (2019). https://riscv.org/wp-content/uploads/2019/06/9.25-Embench-RISC-V-Workshop-Patterson-v3.pdf
  • [Pallister2013] James Pallister, Simon Hollis, Jeremy Bennett (2013). "BEEBS: Open Benchmarks for Energy Measurements on Embedded Platforms" http://arxiv.org/abs/1308.5174.
  • [Rabbah2004] Rodric M. Rabbah, Ian Bratt, Krste Asanovic, and Anant Agarwal, "Versatility and VersaBench: A New Metric and a Benchmark Suite for Flexible Architectures", MIT CSAIL Technical Memo, MIT-LCS-TM-646, June 2004, http://catfish.csail.mit.edu/~rabbah/versabench/MIT-LCS-TM-646.pdf
  • [Sakalis2016] C. Sakalis, C. Leonardsson, S. Kaxiras and A. Ros, "Splash-3: A properly synchronized benchmark suite for contemporary research", 2016 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2016, pp. 101-111, doi: https://www.doi.org/10.1109/ISPASS.2016.7482078.
  • [Singh1992] Jaswinder Pal Singh, Wolf-Dietrich Weber, and Anoop Gupta. 1992. "SPLASH: Stanford parallel applications for shared-memory". SIGARCH Comput. Archit. News 20, 1 (March 1992), 5–44. DOI:https://doi.org/10.1145/130823.130824
  • [Slingerland2002] Slingerland, N., Smith, A. Design and characterization of the Berkeley multimedia workload. Multimedia Systems 8, 315–327 (2002). https://doi.org/10.1007/s005300200052
  • [VanDerWijngaart2002] Rob F. Van der Wijngaart et al. "NAS Grid Benchmarks Version 1.0". NASA Technical Report NAS-02-005. July, 2002. https://www.nas.nasa.gov/assets/pdf/techreports/2002/nas-02-005.pdf
  • [Woo1995] S. C. Woo, M. Ohara, E. Torrie, J. P. Singh and A. Gupta, "The SPLASH-2 programs: characterization and methodological considerations," Proceedings 22nd Annual International Symposium on Computer Architecture, 1995, pp. 24-36, doi: https://www.doi.org/10.1109/ISCA.1995.524546.