Thoughts on SPEC CPU2000 results

Thoughts on SPEC CPU2000 results

Post by Tony Bourk » Tue, 14 Oct 2003 03:50:58


Hi All,

I wanted to see what people's opinions on SPEC's ( http://www.yqcomputer.com/ )
CPU2000 benchmarks as a basis for system performance evaluation between
the same architecture (such as SPARC) and other architectures (SPARC
versus Power4, for instance).

Comparing one platform to another is of course apples to oranges and
often draws emotional and religious remarks, but on a basic and
non-partisan level what are people's thoughts?

Does anyone know other comparable bechmarking results/suites?

Cheers,

Tony
 
 
 

Thoughts on SPEC CPU2000 results

Post by Douglas Si » Tue, 14 Oct 2003 05:15:33

Tony Bourke < XXXX@XXXXX.COM > writes:





They aren't perfect, but they are probably about as good as it gets for
realistic comparisons between different systems. A couple of the SPEC2K
benchmarks have been "broken" by Sun (compiler getting a bit too smart
making a particular benchmark component no longer relevant) but then
the compiler maturity and resources are different between different
vendors anyway (it isn't like HP is investing anything to further improve
its PA-RISC and Alpha compilers, instead investing in their IA-64
compilers, etc.)

For server type loads I'd ignore SPECint (and certainly SPECfp) entirely,
just look at SPECintrate. There are other benchmarks like TPC and the
SAP SD stuff that are more realistic for commercial loads, but how good
you perform depends a lot on how much hardware your benchmark team has
available to put together and how much time they have to experiment and
properly tune for the best possible results.

--
Douglas Siebert XXXX@XXXXX.COM

"Suppose you were an idiot. And suppose you were a member of Congress.
But I repeat myself." -- Mark Twain

 
 
 

Thoughts on SPEC CPU2000 results

Post by David B. C » Thu, 16 Oct 2003 02:03:42

In <mohib.64911$ XXXX@XXXXX.COM > Tony Bourke < XXXX@XXXXX.COM > writes:



CPU2000 stresses a variety of compute intensive tasks, many of them
numerical. If that's going to be the primary use of your computer,
look at the specific items which match your workload. It does not
test interactive features, operating system performance, such as
file system or network performance, or other aspects which are
covered by some of the other SPEC benchmarks.

I have chosen platforms based on specific CPU2000 benchmarks, because
I knew precisely what the workload for the platform is. But even
then there are compiler differences, interaction with os issues,
etc., which prevent the SPEC benchmarks from being more than
suggestive.





--
David B. Chorlian
Neurodynamics Lab SUNY/HSCB
XXXX@XXXXX.COM
XXXX@XXXXX.COM
 
 
 

Thoughts on SPEC CPU2000 results

Post by Tony Bourk » Thu, 16 Oct 2003 03:50:01

Thanks for everyone's reply.

The reason I asked was to do an evaluation of price/performance between
x86 systems and SPARC systems. I used the SPEC CPU2000 integer and
floating point results, as they had the most comprehensive results.


While not a perfect evaluation, it does highlight a problem Sun has to
face, which is that the price/performance story is lacking on Sun
systems. I made special note of the weaknesses in the evaluation.

I'll probably have it published in a few days.

If you'd like to see a PDF copy of it, email me offline and I'll send
you a copy.

Tonyb (at) vegan (.) net

Cheers,

Tony
 
 
 

Thoughts on SPEC CPU2000 results

Post by Douglas Si » Thu, 16 Oct 2003 06:32:06

Tony Bourke < XXXX@XXXXX.COM > writes:






Well, the big problem with this comparison is that you are comparing x86
systems using Intel's compilers when you would actually be using x86
systems using Sun's compilers. It is only $500 to buy a SPEC license
for commercial entities ($125 if you are educational) so it might be
worth that plus a day or two for someone to install Solaris x86 on a
representative PC you are considering and run SPEC on it and a Sparc
for a better comparison.

I'm not claiming Sparcs aren't slower, but Intel is widely regarded as
having the best compilers around, so you'd probably lose a fair bit
using Sun's. I've seen some tests on Linux x86 boxes comparing Intel's
compilers and the latest (3.3) gcc and gcc is 30-40% behind Intel's
results for SPECint. For FP it is even worse (for those tests g77 can
run, a few of the tests require Fortran 90 which g77 can't do)

--
Douglas Siebert XXXX@XXXXX.COM

"I feel sorry for people who don't drink. When they wake up in the morning,
that's as good as they're going to feel all day" -- Frank Sinatra
 
 
 

Thoughts on SPEC CPU2000 results

Post by Tony Bourk » Thu, 16 Oct 2003 07:53:47

Well, my evaluation went through a wide variety of systems, none of
which I could reasonably get access to run SPEC rests myself, unless
Sun, Intel, and AMD want to give me gobs of hardware. Which they can if
they want :)

The SPEC listings compares Intel's best compiler to Sun's best compiler,
which I think is fair. In that sense, each leverages the best they can
from the architecture. What the processors might be able to pull off
isn't really relevant to what actual compilers can leverage, and thereby
compete.

Tony
 
 
 

Thoughts on SPEC CPU2000 results

Post by Rich Tee » Thu, 16 Oct 2003 08:16:29


Please don't top post.


The trouble with using SPEC2000 as a benchmark (or anything
else for that matter) is that the results are only valid if
your application has a similar profile.

If your application doesn't have a similar profile, the results
are academic at best, and completely useless at worst.

--
Rich Teer, SCNA, SCSA

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.yqcomputer.com/
 
 
 

Thoughts on SPEC CPU2000 results

Post by Tony Bourk » Thu, 16 Oct 2003 08:41:16


My apologies


What would you suggest then.
 
 
 

Thoughts on SPEC CPU2000 results

Post by Rich Tee » Thu, 16 Oct 2003 09:44:28


No problem.


The application that you intend to run is the application
that should be used for benchmarking. To use a car analogy,
the vehicle with the fastest 0-60 probably isn't the best
choice for ploughing a field. In the same way, the machine
with the best SPEC2000 score isn't necessarily the best
database server (for exmaple).

Even if the point of the exercise is to compare price/performance,
other things need to be taken into consideration, like I/O
performance, ease of adminstration, etc. In other words,
price/performance needs to consider TCO, not just TCA.

Also bear in mind that unless your machine is very busy,
it'll spend most of it's time idly twiddling its thumbs,
no matter what MHz it runs at. While I pause to think of
the next sentence in this posting, this 200 MHz Ultra 1 is
idling just as quickly as a 3 GHz PC would. :-)

--
Rich Teer, SCNA, SCSA

President,
Rite Online Inc.

Voice: +1 (250) 979-1638
URL: http://www.yqcomputer.com/