[Info-vax] OpenVMS - DCL - Data entry filtering

[Stephen Hoffman]

On 2015-03-29 21:24:24 +0000, mcleanjoh at gmail.com said:

> On Sunday, March 29, 2015 at 6:41:29 AM UTC+11, Stephen Hoffman wrote:
>> On 2015-03-28 17:53:45 +0000, David Froble said:
>> 
>>> What type of application will be creating 10,000 or more files each 
>>> day, and, after created, what is done with the files?
>> 
>> I've encountered OpenVMS sites with that many log files from ftp> 
>> operations, some with transaction-related files, sometimes outbound> 
>> mail messages, all sorts of things.  (More than a few of these folks 
>> should probably switch tactics here, but... excuses are made.)  Ten 
>> thousand files is a little better than one a second, though most have 
>> traffic rates that aren't that smooth.
>> 
>> Having five and ten thousand active network connections in parallel, 
>> and having connection rates of five thousand network connections a 
>> second aren't particularly unusual in larger (usually non-OpenVMS) 
>> configurations.
> 
> I think you forgot to add the fact that many sites seem to want to use 
> a prefix on filenames so that they can easily identify what they are.   
> Adding a prefix that consists of the same characters each time, even as 
> few as less than 4, further impacts directory performance because of 
> the internal indexing system that's used.

MAIL is a (good?) (bad?) example of what happens when you don't have a 
modern database, when you do have to work within the limitations on how 
fast you can scan directories (and this also from within a 
memory-constrained era of computing), and where you have to deal with 
different filename naming schemes depending on versions, and where you 
have to maintain compatibility for mixed-version clusters, when your 
designs can end up used at scales vastly beyond what was likely 
originally intended, and, yes, when you optimize for something other 
than directory performance by using a fixed prefix on the filenames.

Related:
  <https://groups.google.com/d/msg/comp.os.vms/xfo2gbKePUk/QFNDYrc3pBsJ>
  <https://groups.google.com/d/msg/comp.os.vms/UUe0YZhInW0/QG6Uo6oNpBEJ>
Among others.

Big directories have lower performance on OpenVMS.  Some filename 
designs will further reduce that performance.  Unfortunately, there are 
folks with such designs.  What they're doing with all those files, 
depends.


-- 
Pure Personal Opinion | HoffmanLabs LLC