[Info-vax] VUPS.COM relevance for modern CPUs

[Mark Daniel]

On 20/12/2022 5:40 am, Simon Clubley wrote:
> On 2022-12-19, abrsvc <dansabrservices at yahoo.com> wrote:
>> On Monday, December 19, 2022 at 8:23:23 AM UTC-5, Simon Clubley wrote:
>>> On 2022-12-19, Mark Daniel <mark.... at wasd.vsm.com.au> wrote:
>>>>
>>>> With this in mind I knocked-up a small program to repeatedly call a
>>>> function using $CMEXEC which calls a function using $CMKRNL and that is
>>>> that. It measures how much effort is required compared to the simple
>>>> USER mode loop and reports it as b[ogo]VUPs.
>>>>
>>>> https://wasd.vsm.com.au/wasd_tmp/bogovups.c
>>>>
>>>> The real disappointment is my X86 VM. The rest of the results seem in
>>>> line with expectations.
>>>>
>>> What numbers did you see ?
>>>> PS. Looking for ideas, suggestions, criticism(s), etc. here...
>>>>
>>> The tests still "feel" rather artificial.

Certainly they are artificial.  I really didn't want to write a test 
suite, just a simple measuring stick for a quick comparison of 
platforms.  For performance testing the only real metric is "the 
application" or a representative code base.

>>> My suggested alternative would be to write actual files away to disk
>>> using RMS sequential files and also indexed files. Maybe read them
>>> back as well.
>>>
>>> Repeat the sequential files test using direct QIO access and see what
>>> performance difference that gives when you bypass the transition to
>>> executive mode.
>>>
>>> (Yes, I know, RMS will add its own fixed overheads, but you will still
>>> be able to see a percentage difference across the various machines you
>>> are testing on and whether x86-64 VMS imposes a much higher performance
>>> overhead.)
>>>
>>> One obvious problem is that you will have to address issues round
>>> caching in your tests.
>>>
>> Why would you want I/O involved in a measurement of relative CPU power?  That makes no sense.  The VUPs rating has always been a relative CPU performance test.  You can argue whether User mode vs other modes makes sense.  I suppose that using Mark's updated program may make sense given that the newer versions of OpenVMS use software for some functions (modes really).  I don't believe that this is accurate as it will compare hardware vs software between a few models.

Agreed.  Not for a "simple measuring stick" (to quote myself).

Disagree.  It compares *VMS* performance.  Not necessarily just CPU 
power (which is the thrust of BogoMips) but a combination of 
hardware/bus/software.

> Because Mark was reporting bad performance on x86-64 VMS compared to his
> other machines, but are the results artifically bad ?

X86 actually performs quite responsively, so this is a fair question.

My EXEC mode function calling a KERNEL mode function is pretty intense. 
Perhaps it would be better calling KERNEL mode every tenth time.  Or 
some combination such as that.  I have no information on the respective 
call rates apart from @VUPS.COM which on my Alpha shows

| Executive Mode               18 |▒▒▒▒▒▒▒
| Supervisor Mode              81 |▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒

and on the X86 VM

Combined for  2 CPUs
| Kernel Mode                  22 |▒▒▒▒
| Executive Mode               16 |▒▒▒
| Supervisor Mode              62 |▒▒▒▒▒▒▒▒▒▒▒▒
| Idle Time                   100 |▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒

Of course there is no SUPER mode in my code.

The numbers certainly support the idea that X86 inner-modes are 
"emulated" in some way, mediated by KERNEL mode processing.

> We already know there is going to be an overhead because of the need to
> emulate executive and supervisor modes on x86-64 VMS, but Mark's results
> don't cover the elapsed time taken by doing "real" work while in executive
> mode.

Quite deliberately.  It was intended to measure only the overhead of 
inner-mode calls.  Is this fair and equitable?  Who knows?

> Now that I have seen the numbers (thanks Jan-Erik :-)), they would appear
> to be not only bad in general, but bad even when compared to Alpha (provided
> Mark's VirtualBox instance isn't somehow constraining performance and
> assuming he is running it on modern x86-64 hardware).

A pre-loved AU$300.00 Dell SFF bought via eBay -- Optiplex 9020, 
Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz RAM 16.0 GB, Windows 10 Pro 
Version 22H2 Installed 17/03/2021, VirtualBox 6.1.40.  A nice little 
system for the money.

> As such, doing real work in executive mode, and then comparing that
> performance to direct QIO calls, might give better insights into more
> real-world performance results.
> 
> Also, to help eliminate differences in disk hardware performance, might
> it be a good idea to run those tests on a RAM disk instead of a physical
> disk ?

Not interested in I/O.

Doesn't meet the project criterion of quick (and dirty) platform 
comparison (a la VUPS.COM).  Perhaps this approach is fraught with all 
sorts of limitations and attention should be redirected to VUPS.COM.

> Regarding your comment about testing in an emulated environment, don't
> forget that VirtualBox is NOT an emulator, but a virtualisation mechanism.

Certainly.

> That means operating systems running under it should be running at close
> to native hardware performance speeds and it means you should be testing
> the x86-64 performance against physical Alpha machines, not emulated ones.

As a performance comparison tool, why not?

> Simon. 

Thanks for your (plural) interest.

-- 
Anyone, who using social-media, forms an opinion regarding anything 
other than the relative cuteness of this or that puppy-dog, needs 
seriously to examine their critical thinking.