[Info-vax] VUPS.COM relevance for modern CPUs

[Simon Clubley]

On 2022-12-19, abrsvc <dansabrservices at yahoo.com> wrote:
> On Monday, December 19, 2022 at 8:23:23 AM UTC-5, Simon Clubley wrote:
>> On 2022-12-19, Mark Daniel <mark.... at wasd.vsm.com.au> wrote: 
>> > 
>> > With this in mind I knocked-up a small program to repeatedly call a 
>> > function using $CMEXEC which calls a function using $CMKRNL and that is 
>> > that. It measures how much effort is required compared to the simple 
>> > USER mode loop and reports it as b[ogo]VUPs. 
>> > 
>> > https://wasd.vsm.com.au/wasd_tmp/bogovups.c 
>> > 
>> > The real disappointment is my X86 VM. The rest of the results seem in 
>> > line with expectations. 
>> >
>> What numbers did you see ?
>> > PS. Looking for ideas, suggestions, criticism(s), etc. here... 
>> >
>> The tests still "feel" rather artificial. 
>> 
>> My suggested alternative would be to write actual files away to disk 
>> using RMS sequential files and also indexed files. Maybe read them 
>> back as well. 
>> 
>> Repeat the sequential files test using direct QIO access and see what 
>> performance difference that gives when you bypass the transition to 
>> executive mode. 
>> 
>> (Yes, I know, RMS will add its own fixed overheads, but you will still 
>> be able to see a percentage difference across the various machines you 
>> are testing on and whether x86-64 VMS imposes a much higher performance 
>> overhead.) 
>> 
>> One obvious problem is that you will have to address issues round 
>> caching in your tests.
>> 
> Why would you want I/O involved in a measurement of relative CPU power?  That makes no sense.  The VUPs rating has always been a relative CPU performance test.  You can argue whether User mode vs other modes makes sense.  I suppose that using Mark's updated program may make sense given that the newer versions of OpenVMS use software for some functions (modes really).  I don't believe that this is accurate as it will compare hardware vs software between a few models.
>

Because Mark was reporting bad performance on x86-64 VMS compared to his
other machines, but are the results artifically bad ?

We already know there is going to be an overhead because of the need to
emulate executive and supervisor modes on x86-64 VMS, but Mark's results
don't cover the elapsed time taken by doing "real" work while in executive
mode.

Now that I have seen the numbers (thanks Jan-Erik :-)), they would appear
to be not only bad in general, but bad even when compared to Alpha (provided
Mark's VirtualBox instance isn't somehow constraining performance and
assuming he is running it on modern x86-64 hardware).

As such, doing real work in executive mode, and then comparing that
performance to direct QIO calls, might give better insights into more
real-world performance results.

Also, to help eliminate differences in disk hardware performance, might
it be a good idea to run those tests on a RAM disk instead of a physical
disk ?

Regarding your comment about testing in an emulated environment, don't
forget that VirtualBox is NOT an emulator, but a virtualisation mechanism.

That means operating systems running under it should be running at close
to native hardware performance speeds and it means you should be testing
the x86-64 performance against physical Alpha machines, not emulated ones.

Simon.

-- 
Simon Clubley, clubley at remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.