[Info-vax] ABS *TLE.DAT catalog files and duplicates on CONVERT (catalogue)

[Baldrick]

On Thursday, 24 October 2019 04:37:24 UTC+1, Hein RMS van den Heuvel  wrote:
> On Monday, October 21, 2019 at 12:02:07 PM UTC-4, Baldrick wrote:
> > I think this affects ABS V various... On Alpha VMS.
> > 
> > Doing catalog converts for the *TLE.DAT, the convert fails saying there is a 
> > duplicate key but duplicates not allowed. 

> Is that a 'fresh' FDL of the file to be converted, or a target FDL which might not match the source.
> Did you try a convert WITHOUT FDL and did that also fail?

Thank you Rob and Hein...

A CONVERT with no FDL completes without error - but the duplicate record appears in the resultant file, as trying to CONVERT WITH the FDL results in the error.

> If convert for a file with the same structure/FDL results in a duplicate, then that source file is corrupted and other seemingly random behaviour my happen. 
> Possibly, but unlikely, a process started to insert a record, but failed before updating the alternate key not allowing dups, or possible it ran into the duplicate, but failed to unravel the insert all the way leaving the base record.
> Note - this is why you really want the keys which do not allow duplicate to be the 'first keys' - possibly primary keys.
> You do not want RMS to start updating structures only to find out it cannot be done well down the path.
> 
>  
> Beside the troubleshoot/correction list you could consider
> - Allowing duplicates! Then use tools (sort, datatrieve, ...) to determine which record is undesirable.
> - Make a test convert with a file just using that key which does not allow duplicates, and convert with /exception to find out the attributes (primary key) of the record(s) which issues.
> 
> Sorting first is only useful if you use CONVERT/NOFAST, inserting records one-at-a-time.... very slowly.
> 
> The primary key order will determine which of the duplicates comes first.

Primary key is date, so I'm guessing because the backups are renewed older 
overwritten, that the new dates are more important than the old dates.

I did an ANAL/RMS/FDL, and looking at that FDL it says DUPLICATES: NO against 
that same key, so its not that the source FDL is faulty, it is correct. 
Presumably that file would have been created with the FDL in the first place 
too.

It may be possible that updating that file has been interrupted and/or the 
catalog cleanup process has been interrupted too but I'm baffled as to what 
would allow that duplicate key in that file, surely whatever was trying to 
insert that key would error and fail.

I don't have datatrieve on that cluster, I may have it on another system with 
the appropriate security rating I can copy it onto to examine.

The question has to be why is ABS generating a duplicate key in the first place, regardless of RMS's ability to reject that key so that the file integrity is not compromised. 

I'm not a great fan of ABS. It fails a tape (volume) if a disk isn't mounted. Journal files are more efficient at storing content than this strange database. And tape rotation can be achieved using manual or other programmed methods (some I have implemented in the past to ensure even tape rotation). MDMS can waste free tapes if an allocation stats but a tape is never used for operational reasons, a newly INITed but unused tape is out of the cycle for the backup lifetime. And /TAPE_EXP is not used... The only thing going for it is the scheduler, but again a bit of forethought can emulate the functionality.

If this is a comment on what could be improved in ABS, while perhaps it is a tool to help you make backups of your system correctly, some of the shortcomings are a liability and should be addressed in future versions.

Just yesterday I was reminiscing about walking from the tape library to the tape units with both arms full of magtape spools (arms through the holes) and a pocket full of write permit rings.