[Info-vax] Sudden problems with slow sftp transfers and slow disk accesses

[gwilliams at cfa.harvard.edu]

On Tuesday, February 18, 2014 6:23:22 AM UTC-5, Jim wrote:
> On Monday, February 17, 2014 11:42:27 PM UTC-5, gwil... at cfa.harvard.edu wrote:
> 
> > Cluster of 8 Alpha boxes running V8.3 + patches, recently moved
> 
> > 
> 
> > from one building to another.  Since the move, we've been experiencing
> 
> > 
> 
> > odd behaviors: very slow network access (via sftp) and slow disk IO.
> 
> > 
> 
> > 
> 
> > 
> 
> > Disk storage is mostly on three eternal disk boxes (five three-member
> 
> > 
> 
> > shadow sets).  No errors are reported via SHOW DEV DSA.
> 
> > 
> 
> > 
> 
> > 
> 
> > Network cards on all machines are set to 100 MB, full duplex,
> 
> > 
> 
> > non-autonegotiate, connected via an 8-port GB switch.
> 
> > 
> 
> > 
> 
> > 
> 
> > An sftp from one of our machine to a local Linux system transferred 288 KB
> 
> > 
> 
> > of a ~ 6MB file in the first second, the current rate is now down to 5KB/s.
> 
> > 
> 
> > MONITOR PROCESS/TOPCPU doesn't show the process getting even 1% of the
> 
> > 
> 
> > CPU and there is nothing else running on the system.  Another attempt on
> 
> > 
> 
> > the same file transferred 1.5 MB in the first second, then dropped to
> 
> > 
> 
> > < 50 KB/s.
> 
> > 
> 
> > 
> 
> > 
> 
> > SHOW MEMORY doesn't show any problems.
> 
> > 
> 
> > 
> 
> > 
> 
> > A filing operation merging two large files took a matter of seconds
> 
> > 
> 
> > when both files were on a locally-attached disk.  When both files were
> 
> > 
> 
> > on a shadow set, the merge took 6+ minutes.  MONITOR LOCK while
> 
> > 
> 
> > running the latter test showed ENQ/DEQ rates < 1.
> 
> > 
> 
> > 
> 
> > 
> 
> > Image activation is slow.  It can take several seconds to begin
> 
> > 
> 
> > running an .exe stored on the shadow set.  
> 
> > 
> 
> > 
> 
> > 
> 
> > MCR SCACP SHOW LAN_DEV/ALL showed numerous errors occurring over the
> 
> > 
> 
> > past 24 hours, so this evening we replaced the switch connecting these
> 
> > 
> 
> > 8 machines.  Errors are continuing to appear.
> 
> > 
> 
> > 
> 
> > 
> 
> > What am I missing?
> 
> > 
> 
> > 
> 
> > 
> 
> > Gareth
> 
> 
> 
> The output of the following command might be interesting
> 
> 
> 
> $ MCR LANCP SHOW DEVICE/INTERNAL Exxx ! where Exxx is the suspect NIC

Contrary to what DO MCR LANCP SHOW DEV EW/CHAR shows in SYSMAN
(all interface cards set to Full duplex enable YES, Full duplex
operational YES, 100 Mb/s), most of the interfaces carrying IP
traffic display "possible duplex mismatch" when running the SHOW
DEV/INT command.  The driver messages all show "Link State: UP"
and "Full Duplex 100base TX connection selected". I have reconfigured
all interfaces, but I'm still seeing sftp issues (the 6 MB file
mentioned earlier transferred 2.8 MB in the first second, then
trailed off).  The next driver message isn't due for another 30 minutes
or so.

Gareth