[Info-vax] debugging stack corruption on Macro64?

[John Reagan]

On Monday, March 9, 2015 at 8:31:55 PM UTC-4, John E. Malmberg wrote:
> On 3/9/2015 8:56 AM, "VAXman- "@SendSpamHere.ORG wrote:
> > In article <UeOdnWOeg9Uo2WDJnZ2dnUU7-S-dnZ2d at mchsi.com>, "John E. Malmberg" <wb8tyw at qsl.network> writes:
> >> On 3/8/2015 11:26 PM, John E. Malmberg wrote:
> >>>
> >>> The R17 value appears to be quadword aligned when the corruption occurs.
> >>
> >> I am now reproducing the stack corruption with this program once I made
> >> a change to save some registers.  The test macro code does not use the
> >> registers, the actual program does.
> >>
> >>
> >> EAGLE> type [.vms]test.c,test2.m64
> >>
> >> #define __NEW_STARLET 1
> >> #include <psigdef.h>
> >> #include <stdio.h>
> >> #include <builtins.h>
> >> #pragma message disable badptrarith
> >>
> >> extern void ffi_call_osf(void *stack, void *frame, unsigned flags,
> >>                           void *raddr, void (*fn)(void), void *closure);
> >>
> >>
> >> int main(int argc, char **argv) {
> >>
> >> unsigned long * argp;
> >> void * frame;
> >> unsigned flags;
> >> int ret_value;
> >> long long * stack;
> >> AIDEF *vms_ai;
> >> unsigned long long *vms_ai_llong;
> >>
> >>
> >>      argp = frame = __ALLOCA(8 * (6 + 4));
> >>      frame += (8 * 6);
> >>      vms_ai = (AIDEF *) frame;
> >>      vms_ai_llong = (unsigned long long *) frame;
> >>      vms_ai_llong[0] = 0;
> >>      vms_ai->ai$b_arg_count = 1;
> >>
> >>      flags = 1;
> >>      stack = (long long *) argp;
> >>      stack[0] = (long long)"test string";
> >>
> >>      ffi_call_osf(argp, frame, flags, &ret_value, (void (*)(void))puts,
> >> NULL);
> >>
> >> }
> >>
> >> VMS_ROOT:[libffi.vms]test2.m64
> >>
> >>          .TITLE "VMS"
> >>
> >>          .PSECT MY_CODE,EXE,NOMIX
> >>          .align long
> >>
> >>          $ROUTINE ffi_call_osf,-
> >>                   SAVED_REGS=<R2,R3,R4,FP>,-
> > -------------------^^^^^^^^^^^^^^^^^^^^^^^^^
> >
> > Why???  Save them yourself in your prologue is they're needed.  As far as
> > I can tell, you're not touching R2,R3 or R4 and you save frame pointer at
> > 16 off the new stack space you've reserved.
> 
> The code that touches r2, r3, and r4 was removed from the example that I 
> am getting the stack corruption on.
> 
> How does the $return macro know to restore them if I do not specify it 
> as part of the $ROUTINE macro invocation.

You don't have to use $RETURN (especially if you think you must use STANDARD_PROLOGUE=FALSE, which is rather rare).

> 
> I also have not found any documentation as to what order that I should 
> be saving the registers on the stack.
> 

In the Calling Standard for both OpenVMS has a section named "Register Save Area for All Stack Frames"

The algorithm for packing saved registers in the quadword-aligned register save
area is:

1. The return address is saved at the lowest address of the register save area
(offset 0).

2. All saved integer registers (as indicated by the corresponding bit in PDSC$L_
IREG_MASK being set to 1) are stored, in register-number order, in
consecutive quadwords, beginning at offset 8 of the register save area.

3. All saved floating-point registers (as indicated by the corresponding bit in
PDSC$L_FREG_MASK being set to 1) are stored, in register-number order, in
consecutive quadwords, following the saved integer registers.

You can find the Tru64 manual online.  I found a copy at:

http://h30097.www3.hp.com/docs/base_doc/DOCUMENTATION/V51B_ACRO_DUX/ARH9MCTE.PDF

It has a section "Register Save Area" which is essentially identical.