[Info-vax] NetBackup Performance Woes

[Geek Nerdly]

On Wednesday, May 20, 2015 at 5:31:29 PM UTC-4, Stephen Hoffman wrote:
> On 2015-05-20 21:15:40 +0000, Geek Nerdly said:
> 
> > It's very consistent:
> > 
> > Queue Length spikes to as high as 50 and does not dip below 3 on node B 
> > (where backup runs) while backup is running, where at the same time on 
> > node A Queue Length hovers between 0 & 1.
> 
> That's a disk volume that's saturated, or potentially an HBA or storage 
> controller that's saturated.
> 
> Any disk I/O queue length past 0.5 means that half of all I/O requests 
> are waiting.
> 
> Your backup tool has sufficient quotas and an I/O design that allows 
> this load to be generated, unfortunately.
> 
> For an example of a throttling mechanism within the OpenVMS BACKUP 
> tool, see HELP BACKUP /IO_LOAD on OpenVMS V8.3 and later.
> 
> Given the disk is apparently active at the time of the backup, the 
> resulting archived may not be entirely consistent nor entirely 
> reliable, either.
> 
> 
> -- 
> Pure Personal Opinion | HoffmanLabs LLC

I would love to be able to try the /IO_LOAD qual and may with some disk-disk backups to see if it has any effect.

This NetBackup agent may use BACKUP under the covers but I doubt we can affect what it does or how it does it, except for LIMIT_BANDWIDTH and perhaps tweaking DIOLM on our NETBACKUP account (the service uses that account) down to below 100 to see if that has the effect I've read about...

Anyway, we will be going the trial-and-error route because it most likely will get us somewhere faster than trying to escalate with Symantec support.

I am no expert with FC storage or HBAs, but at the highest level, I am inclined to consider engage HP support to review firmware revs.  However, I think that with this EMC SAN platform, we had to take a certain (not current at the time) fw revision to make it work.  It may still be that way and we might have to have a support party with HP and EMC pointing fingers at each other for a while before we get that answer.

So I guess what I would like to find is how to tell if the HBA is getting saturated.  I know that'd be a lot to ask for, and I don't want to dive down a rabbit hole.  It might just be trivia, because I think the application is what's misbehaving.

If we can see no change by taking some actions to tune the account or the application, then I will consider whether the hardware/firmware is the problem.

Thanks again!