[Info-vax] OpenVMS async I/O, fast vs. slow

[Dan Cross]

[sorry for the delayed response; I was out of pocket for a few
days doing some volunteer outdoors conservation work....]

In article <uiovnc$td7$1 at news.misty.com>,
Johnny Billquist  <bqt at softjar.se> wrote:
>On 2023-11-11 16:06, Dan Cross wrote:
>> In article <uij9oc$st3$1 at news.misty.com>,
>> Johnny Billquist  <bqt at softjar.se> wrote:
>>> On 2023-11-09 17:50, Dan Cross wrote:
>>>> In article <uig3nn$2ke$2 at news.misty.com>,
>>>> Johnny Billquist  <bqt at softjar.se> wrote:
>>>>> On 2023-11-08 03:00, Dan Cross wrote:
>>>>>> [snip]
>>>>>> Yes.  See below.
>>>>>
>>>>> :-)
>>>>>
>>>>> And yes, I know how the cache coherency protocols work. Another thing
>>>>> that was covered already when I was studying at University.
>>>>
>>>> Cool.  MESI wasn't presented until 1984, so you must have seen
>>>> it pretty early on.
>>>
>>> I think I was looking at it in 1995 (what made you think I would have
>>> looked at it close to 1984? How old do you think I am??? :) ).
>> 
>> *cough* Uh, ahem...sorry.... *cough*
>
>Well, it's kindof funny. :-)

Glad you took it in the spirit it was offered. :-D

>>>>> The owning CPU then tries to
>>>>> release the lock, at which time it also access the data, writes it, at
>>>>> which point the other CPUs will still have it in shared, and the owning
>>>>> CPU gets it as owned.
>>>>
>>>> Hey now?  Seems like most spinlock releases on x86 are simply
>>>> going to be a memory barrier followed by a store.  That'll push
>>>> the line in the local into MODIFIED and invalidate all other
>>>> caches.
>>>
>>> Why would it invalidate other caches?
>> 
>> Because that's what the protocol says it must do?  :-)
>
>If you want to move it to MODIFIED, then yes, you need to invalidate all 
>other CPU caches. But there is no good reason to do that in general.

Perhaps not, but that's how the protocol works.

>To quote the wikipedia article:
>
>"Owned
>This cache is one of several with a valid copy of the cache line, but 
>has the exclusive right to make changes to it—other caches may read but 
>not write the cache line. When this cache changes data on the cache 
>line, it must broadcast those changes to all other caches sharing the 
>line. The introduction of the Owned state allows dirty sharing of data, 
>i.e., a modified cache block can be moved around various caches without 
>updating main memory. The cache line may be changed to the Modified 
>state after invalidating all shared copies, or changed to the Shared 
>state by writing the modifications back to main memory. Owned cache 
>lines must respond to a snoop request with data."
>
>Note the "when the cache changes the data on the cache line, it must 
>broadcast those changes to all other caches sharing the line".

Hmm, in this case, I think the wikipedia article may be wrong,
or at least misleading.  How would a CPU know what other CPUs
are sharing the cache line?  A write to an OWNED line (by the
owner) will generate a WrReq and move the line into OWNED; any
other CPU holding the line will get a probe WrReq and transition
the line to INVALID.  But the transition to OWNED only happens
from MODIFIED, and won't happen unless the cache snoops a RdReq,
in which case it can supply the value on the cache bus.


>So basically - no. If you have a cache in owned state, and you change 
>the content, you already have the data modified, but you own it, and any 
>changes you broadcast to anyone else who also have the data in their 
>cache. All others have the cache line in SHARED state, so they cannot 
>modify it.

No, sorry, that's just not how the protocol works.  Here's a
decent reference:
https://inst.eecs.berkeley.edu/~cs61c/su18/disc/11/Disc11Sol.pdf

>But eventually you need to flush it, at which point it goes back to main 
>memory, and your copy is invalidated.

Writing to main memory doesn't invalidate _your_ cache copy, it
just puts it into SHARED state.

>Others still have it shared. Of 
>course, someone else might want to try and write, at which point the 
>ownership needs to move over there. More handshaking...

The difference with MESI here is that a probe read hit on a
MODIFIED cache line transitions the line on the owning core to
OWNED and allows the other cache to receive data from the owning
CPU's cache.  A subsequent write to an OWNED line will
transition the line to MODIFIED and invalidate anyone who has it
in SHARED state until a subsequent read request is probed.

>Basically, OWNED is to allow sharing of dirty cache without it being in 
>main memory. You're the one who even reminded me about it. :D

Yes, but you only get to OWNED after you probe a read request.
Preemptively pushing to other caches seems unsporting; how would
you know that the other cache line would actually be _used_?
Putting additional contention on another CPUs cache is mean. :-)

>>> It could just push the change into
>>> their caches as well. At which point the local CPU would have "owned"
>>> and others "shared".
>> 
>> Well, no....  Even in MOESI, a write hit puts an OWNED cache
>> line into MODIFIED state, and a probe write hit puts the line
>> into INVALID state from any state.  A probe read hit can move a
>> line from MODIFIED into OWNED (presumably that's when it pushes
>> its modified contents to other caches).
>
>No. OWNED means you already modified it. You can modify it additional 
>times if you want to. It don't change anything in cache state.

I'm afraid it does.  See the above linked document.

>But anyone else who have that same line in SHARED needs to either be 
>invalidated or updated. But as noted above, the normal thing is to update.
>
>You don't get to OWNED state until you actually do modify the cache. 
>It't not that modifying the cache moves out out of OWNED. It moves you 
>INTO OWNED.

No.  You only get into OWNED from MODIFIED if you probe a read
request.

>[snip]
>The PDP-11 never had write-back caching. Only write-through. So no need 
>to flush as such.
>But you want to bypass (as well as invalidate) what's in there, in case 
>physical memory holds something else.

That makes sense.

	- Dan C.