[Info-vax] Still no DIR/SORT_BY_TIME

[AEF]

On Saturday, August 15, 2015 at 11:13:32 AM UTC-4, Stephen Hoffman wrote:
> On 2015-08-15 14:56:40 +0000, AEF said:
> 
> > On Wednesday, August 12, 2015 at 6:55:22 PM UTC-4, Stephen Hoffman wrote:
> >> On 2015-08-12 22:29:26 +0000, xxxxx said:
> >> 
> >>> ... I'm just surprised that after ~38 years someone at DEC/... would 
> >>> have gotten around to it.
> >> 
> >> It's certainly reasonable -- maybe as part of that newer file system.
> > 
> > Newer file system?
> 
> See previous VSI discussions -- they're planning on providing a path for 
> folks on ODS-2 and ODS-5 that might want to migrate to something 
> slightly less limited, less encrufted and better performing.

OK. Thanks.

> 
> >> Or you can put the timestamp in the filename.
> > 
> > Too much work.
> > We are running a file-transfer system with files coming and going on a 
> > daily basis. We add our own time stamps. In some cases the customer is 
> > nice enough to put in _their own_ time stamps. And we have to add our 
> > own anyway!
> 
> Are files even the appropriate container for that?

So you're saying files could be indexed by what would be their time stamp?

> 
> >> But then I'm increasingly working a whole lot less with writing or 
> >> storing zillions of files -- and trying to create or store even fewer 
> >> -- and a whole lot more with data stored in databases.
> > 
> > I'm migrating a file-transfer system. And there's a real crap-load of files!
> 
> If it's typical path, what you're porting was a simple design that was 
> well suited to its intended environment, and has since been scaled way 
> past sanity and efficacy, and it's undoubtedly become entwined 
> throughout.

We are not porting anything. On the VMS side everything is done with numerous batch jobs, command procedures, and a flat database. The new system was built on the Unix side from scratch at least 2 years ago. New jobs are always set up on this new system. It's like buying a new car. You have something new that is functionally the same, but nothing has been ported.

The VMS side is command procedure driven. The new system on the Unix side is table driven.

There's a whole lot to both systems that you don't know. It's taken me weeks to learn. I can't put that all down here and I shouldn't. You don't know enough about the system to be so specific. Trust me. What we need to do is more complicated than you think.

The crapload of files is what is being sent about. Customers, including internal departments, send us files we need. We send them files they need. It is these that add up to a huge number. And different customers insist on different protocols. Some files are encrypted. Some need special processing for a variety of reasons. Each filename is in the database with a lot of metadata that tells the scripts what to do with the files and where to send them and where to archive them and where to retain them and what not. 

But right now we have a hard deadline. We can redesign or improve it otherhow it later. There may well be a better way to do this. But we don't have time for that now. 

> 
> 
> >>> And then write a nice front end for the database.
> >> 
> >> It seems maybe this is more about the Oracle database than the 
> >> operating system platform?
> > 
> > Most of the work is setting up jobs on the Unix/Sybase side. And most 
> > of that is entering data into the database that tells the 
> > well-established perl scripts what to do. Each and every file must be 
> > carefully processed and tracked, with numerous things happening 
> > differently for different jobs. Some are pgp-encrypted. Protocols vary 
> > (ftp, sftp, NDM, etc.).
> > 
> > And AutoSys runs the show; there's that too. Once the job is set up on 
> > the Unix side, we simply stop the corresponding job on the VMS side. 
> > Then something breaks. We go back to the VMS side, fix the Unix side, 
> > and then do take 2.
> 
> 
> So you're also rolling your own a distributed scheduler, too?

What? AutoSys does that.

> 
> 
> -- 
> Pure Personal Opinion | HoffmanLabs LLC