[Info-vax] debugging stack corruption on Macro64?

[John E. Malmberg]

On 3/9/2015 9:43 AM, John Reagan wrote:
>
>>           $ROUTINE ffi_call_osf,-
>>                    SAVED_REGS=<R2,R3,R4,FP>,-
>>                    KIND=STACK,-
>>                    STANDARD_PROLOGUE=FALSE
>>           $CODE_SECTION
>>
>>           lda     sp,-24(sp)
>>           stq     r27,(sp)
>>           stq     r26,8(sp)
>>           stq     fp,16(sp)
>> ; step/inst to here ok.
>>           mov     sp,fp
>> ; step/inst
>> ; DEBUG-I-INCMPSTACK, WARNING: stack corrupted; session integrity not
>> guaranteed.
>> ; fp and sp now contain 07ad8b9a8
>
> In STACK routines with BASE_REG_IS_FP (the default is TRUE and you
> didn't change that), the contents of FP must always point to the
> stack base *AND* the sop of the stack must point to the procedure
> descriptor.  You stored the current value in 16(FP) but when you did
> the MOV to overwrite the FP, but the top of the stack at 24(FP) does
> not point to a procedure descriptor.  The debugger now gets confused
> since it cannot determine the active routine and how large the stack
> should be.

I tried to model this after example 3.1 in the OpenVMS Calling standard, 
but discovered after I posted this that I had missed editing in actual 
register save area.

This is what the code was intended to be, but it still corrupts the stack.

         lda     sp,-$SIZE(sp)
         stq     r27,(sp)
         stq     r26,16(sp)
         stq     r2,24(sp)
         stq     r3,32(sp)
         stq     r4,40(sp)
         stq     fp,48(sp)
         mov     sp,fp

> I'm not even sure why you need a STACK routine (in either dynamic or
> fixed form).

The original code from Philippe Vouters used a STACK routine.

One of the things I am confused on is that Philippe's code stores R26 at 
8(sp) and the Example 3-1 stores R26 at 16(SP).

> And your STANDARD_PROLOGUE=FALSE covers over your bogus SAVED_REGS
> mask since STANDARD_PROLOGUE=FALSE prevents $ROUTINE from generating
> any code.

I realize that it does not save the registers.  I forgot to put the 
register save routine in what I posted previously.

I am assuming that the SAVED_REGS mask is needed for the $RETURN macro 
to restore them.

> If you are trying to dead-reckon a Macro64 routine by looking at the
> generated code from a C compilation, you'll probably have problems.
> Besides looking at the generated code, you need to look at the
> procedure descriptor that GEM formatted into the procedure descriptor
> also.  The problem is that it isn't to map the GEM formatted PDSC
> fields to the arguments of $ROUTINE.  And I'm not even sure that GEM
> formats all of them...

I have been trying to learn the differences between the Tru64 calling 
standard and assembler and VMS.

Regards,
-John
wb8tyw at qsl.network