[Info-vax] DCL crashing bug update

[Simon Clubley]

I finally found some time to look in some detail at the DCL bug I found
and have managed to get some firm answers as well as a surprisingly
simple reproducer now that I understand the problem better.

There are also some questions for VSI at the end of this post.

On the plus side, I can't think of a way to turn this into a security
vulnerability because it isn't what I originally thought it was. On the
minus side, it turns out that Unix isn't the only operating system with
an in-band null character (ie: 0x00) problem. :-)

Basically, if you craft a specific sequence of command lines with lots
of embedded null characters in one of the command lines and inject
those command lines directly into DCL, then you will get a crash on
VMS Alpha v8.4. I don't know if VAX or IA64 is also affected.

Before going any further, here's a sample command procedure which
demonstrates the problem. READ THE NOTES FULLY BEFORE USING.

=======================================================================
$ set verify
$ !
$ ! WARNING: You run this command procedure and use the output from it at
$ ! your own risk! Do NOT run this command procedure or the bug triggering
$ ! sequence on production systems or on any system which is important to you.
$ !
$ ! While exploring this bug on VMS Alpha running under the FreeAXP emulator,
$ ! I experienced multiple crashes during a reboot. I do not know for sure
$ ! if this has anything to do with the triggering of this bug or not so
$ ! you have been warned.
$ !
$ ! ==> To be safe, I recommend you perform a reboot immediately after you
$ ! have finished examining this bug even though I am having a hard time
$ ! seeing how supervisor mode code could mess up the system to this level
$ ! so for now I am assuming this is probably just an emulator artifact.
$ !
$ ! Run this command procedure to generate simon_test.tmp and then do the
$ ! following:
$ !
$ !	$ recall/erase
$ !	$ recall/input=simon_test.tmp
$ !	$ recall/all
$ !
$ ! At this point, DCL crashes on me when using the VMS Alpha v8.4 hobbyist
$ ! distribution.
$ !
$ ! Simon.
$ !
$ set noverify
$	write sys$output "Generating test file"
$
$	open/write outch simon_test.tmp
$	call generate_line "2ab" 1197 0
$	call generate_line "" 1500 66
$	call generate_line "" 1400 67
$
$	close outch
$	write sys$output "Test file generated"
$	exit
$
$ generate_line:
$ subroutine
$	chr[0,7] = f$integer(p3)
$	oline = p1
$	i = 1
$
$ next_char:
$	if i .gt. p2 then $ goto line_complete
$	oline = oline + chr
$	i = i + 1
$	goto next_char
$
$ line_complete:
$	write/symbol outch oline
$	exit
$
$ endsubroutine
=======================================================================

and here is the crash I get with the above sequence:

[snip]

$ set noverify
Generating test file
Test file generated
$ recall/erase
$ recall/input=simon_test.tmp
$ recall/all

[snip first two command lines which are ok]

  3 

  Improperly handled condition, bad stack or no handler specified.
    Signal arguments:   Number = 0000000000000005
                        Name   = 000000000000000C
                                 0000000000000000
                                 000000007FFC9CD0
                                 000000007AF96F98
                                 0000000000000012

    Register dump:
    R0  = 000000007FF9CC30  R1  = 000000007FF9FEA3  R2  = 000000000000000A
    R3  = 000000007FF9CC30  R4  = 000000007AEF85F0  R5  = 0000203320200000
    R6  = 0000000000000001  R7  = 000000007FFA4F28  R8  = 000000007FF9CDE8
    R9  = 000000007FFC9CD1  R10 = 000000007FFA4F28  R11 = 000000007FFCDC18
    R12 = 000000007FFCDA98  R13 = 00000000000000FB  R14 = 0000000000000000
    R15 = 000000000000002F  R16 = 000000007FFCEDE2  R17 = 00000000000011CA
    R18 = 000000007FFC8CAF  R19 = 0000000000000001  R20 = 000000007FFC8CB1
    R21 = 000000007AEF84B0  R22 = 000000000000612F  R23 = 000000007FF9FEA4
    R24 = 000000007FF9FEA6  R25 = 0000000000000001  R26 = 000000007AF97268
    R27 = 000000007AEF84B0  R28 = 000000007AF97404  R29 = 000000007FF9CC00
    SP  = 000000007FF9CC00  PC  = 000000007AF96F98  PS  = 0000000000000012

My test payloads were generated using HP Pascal programs because it's
a lot easier to work with binary records with nulls embedded in them
in Pascal than in C. The above command procedure is just a Q&D
command procedure so you can easily explore this bug if you wish.

Note that it's only 0x00 which causes the crash. If you generate a
file with the same structure but with 0x00 replaced with 0x01 then
DCL does not crash (although it doesn't show the command line with
the 0x01 characters embedded in it).

The DCL command history buffer is a fixed size wrap-around buffer.
Originally, I thought this was a problem with my fuzzing data causing
DCL to get confused about where the start of the buffer is[*], but my
attempts to embed a valid internal representation of a DCL command
within another larger command always resulted in DCL displaying the
larger command correctly (and ignoring the embedded command) so it
became clear that wasn't the problem.

[*] I quickly found the longword which identifies the next free byte
to write to within the buffer (it's immediately prior to the buffer
data itself) but I couldn't find anything which identified the offset
to the first command within the history buffer area. This is why
I originally thought DCL might be scanning for all the 0x00 characters
which are immediately prior to the first command when the first command
is not at the start of the buffer.

If you want to examine the command history buffer from another process
then running SDA, setting the target PID you want to examine and then
using the following command will get you started:

SDA> examine CTL$AG_CLIDATA;1400

Also note that if you increase the first record from 1200 to 1300 bytes,
I get buffer wrap-around and if the first record is all nulls then the
trailing line length field is not removed from the command history when
the first record is purged from the history. Also note that this does
not occur if 0x01 is used instead of 0x00 (for example).

My questions for VSI are these:

1) I've been concerned for some time that as new functionality was
added to VMS then maybe this new functionality allowed the existing
functionality to be manipulated in ways that could cause problems
because the new functionality was never considered as part of the
original design and security audit process.

DCL clearly has code which assumes it is never going to see a stream
of 0x00 characters because you could probably never get those characters
into the history buffer via the terminal driver. However, $ recall/input
now gives you the ability to directly inject any characters you choose
directly into the DCL command history buffer.

When adding new functionality to VMS, how much work is done to evaluate
the original design assumptions in light of any new functionality ?

2) While this specific problem turned out not to be an actual vector
for a security exploit, does this same style of buffer management
exist elsewhere in VMS and in a form which could cause problems if
the utility/command allows direct injection of command history ?

Simon.

-- 
Simon Clubley, clubley at remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world