[Info-vax] VUPS.COM relevance for modern CPUs

[Arne Vajhøj]

On 12/21/2022 7:42 AM, abrsvc wrote:
> On Tuesday, December 20, 2022 at 7:07:50 PM UTC-5, Arne Vajhøj wrote:
>> On 12/20/2022 11:43 AM, abrsvc wrote:
>>> I will disagree. How many standard benchmarks bear any relevance to
>>> an actual application? I suppose you can use them for relative
>>> machine performance information, but without knowing how your own
>>> application performs relative to those, they are useless. SPEC
>>> benchmarks mean little to I/O bound applications. Great, my new
>>> machine can perform calculations 10 times as fast. But... the
>>> application is bound by disk performance limits, so I see little to
>>> nothing for the speed improvement. just one extreme example.

>> Testing with the actual application instead of an
>> artificial benchmark is obviously better.
>>
>> But given how much effort has gone into developing
>> the modern benchmarks, then they should be better
>> than a simple homegrown benchmark unless one has a rather
>> unique context.
>>
>> Obviously one need to pick the right benchmark. Like:
>>
>> CPU integer => SPEC CPU SPECint
>> CPU floating point => SPEC CPU SPECfp
>> CPU floating point linear algebra => LINPACK
>> Database OLTP => TPC-C
>> Database DWH => TPC-H
>> Java app servers => SPECjEnterprise
>>
>> If we talk old 1980's benchmarks like Dhrystone/Whetstone, then
>> it is probably not too much work to come up a homegrown benchmark
>> as good or better.

> Standard benchmarks can provide raw throughput numbers for certain
> classes of functions (CPU, raw I/O , database functions, etc.). 
> But...  How these relate to a real application environment is
> required in order to use these to predict performance of a system.  A
> home grown benchmark is less of a raw performance indicator than a
> more accurate predictor of the specific application environment for
> any new hardware. If you know the relationship, then I would guess
> that industry standard benchmarks are useful.  In many cases where I
> have been involved, no simple correlation could be made.  You mileage
> will vary...

If ones application does not match one of the standard benchmarks then
one has to create a custom benchmark.

But creating good benchmarks is not easy. Most quickly put
together custom benchmarks are pretty bad. There is a long
history of bad custom benchmarks producing misleading results.

Arne