SPEC Benchmarks

Can't locate strict.pm

When this happens, you have probably forgotten to run shrc.bat. This batch file sets the SPECPERLLIB environment variable, among other things.

Older info: SPEC '95

When the Spec '95 benchmark software is installed, there are no binary versions of the benchmarks installed. To make these, you are supposed to make a configuration file first, and then you run runspec. For our purposes, the standard configuration file is fine, so the benchmarks can be made as follows. As an example, the integer benchmark 129.compress will be used:

cd ~/extra/spec95
. ./shrc
runspec -a build -c sun.cfg -T base -i ref 129.compress

The first line just changes to the appropriate directory. You should op to binstaff before this. The shrc is a shell script that sets up the environment for spec95. The runspec does the building (it can also run the benchmarks a specified number of times, and so on). The -T base builds the base (-T peak builds peak version, -Tall build both) version of the benchmark; base is with optimisation suitable for all benchmarks, and peak is with agressive optimisation specific to one benchmark. Most times we would use the former. Note that in order to get all the special optimisations for Suns (in sun.cfg), you need to use base. Peak will give you a generic optimisation (e.g. "-O") unless there is a special entry in the .cfg file for that benchmark! The -i argument specifies the input set; we use ref since this is required when publishing results. The benchmarks take 5 minutes or so each to build.

Note: When changing flags or the like, be sure to change to the src directory (see below) and do a make clean. It may be worth running the make without runspec, to really see what is happening. Or just examine the make.out file in the src directory.

What flags were used?

To find out exactly what flags were used, you can go to the src directory and examine the make.out file. This is the output from the compiler, etc. If it's not as you expect, you can also examine Makefile.spec, which is apparently generated for each run of runspec. It is included from the main Makefile, and provides values for special variables such as OPTIMIZE, EXTRA_CFLAGS etc.

Example (starting at the spec95 directory):

cd benchspec/CINT95/129.compress/src
cat make.out
cat Makefile.spec

Making without runspec

It is possible to make a benchmark without using the runspec script. Note: gnu make is required, not standard make. Change to the src directory (see above). Use make clean and make build after editing the Makefile.spec file if needed. This is handy when adding printf style debugging information to the benchmark. The binary is put into the exe directory, e.g. ../exe/base.

Using the benchmarks

The resulting benchmark binaries are in these directories:

~/extra/spec95/benchspec/CINT95/129.compress/exe/base
~/extra/spec95/benchspec/CINT95/129.compress/exe/peak

They can of course be disassembled or whatever right in place, but to actually run them may need some more effort. To run a benchmark and nothing else, you would use something like

runspec -c sun.cfg -T base -i ref 129.compress

This will copy the appropriate executable and some data files to the directory

./benchspec/CINT95/129.compress/run/work_compress.000

By running ps -fu binstaff after it has settled down and has displayed running '029.compress', you will find that it is running

sh -c ./compress95.v8 < bigtest.in >bigtest.out 2>bigtest.err

Other programs may have completely different ways of running them, e.g. for 126.gcc:

sh -c ./cc1.v8 -quiet -funroll-loops -fforce-mem -fcse-follow-jumps -fcse-skip-

Note that this is not the full set of arguments for this run; it has been truncated by ps. You can figure out the full argument list from the .pm file in the Spec directory.

It therefore seems reasonable, if you want to do something other than just run the Compress benchmark (e.g. profile or trace it), then you could run it like this:

indjump -- compress95.v8 < bigtest.in > bigtest.out 2>bigtest.err

where here indjump is a program that counts the number of indirect jumps in the program. You can get suitable data files from

./benchspec/CINT95/129.compress/data/ref/input

(In the case of compress, these are not the data itself, just the parameters; the benchmark appears to generate its own random data).