Writing data to native file

General APL language issues

Writing data to native file

Postby gil on Thu Jul 09, 2015 6:00 am

I'm processing a large csv file into separate, typed native files that are then used as mapped vars from the ws. When writing the data to file, I also do a simple health check to test the content by mapping the file and comparing with the data in the ws. This has never reported a problem before.
Today it did. One out of 2 billion 4-byte integers had an incorrect value on file. I understand this can happen for all sorts of reasons, and I expect it doesn't happen very frequently. Does anybody have any experience of inconsistencies like this when reading/writing to native files?
gil
 
Posts: 68
Joined: Mon Feb 15, 2010 12:42 am

Re: Writing data to native file

Postby Morten|Dyalog on Thu Jul 09, 2015 7:26 am

I don't think we have other recorded, documented cases of anything like this, but then again most people wouldn't think to check for it. Well done for writing the "paranoia check" for something that should really never fail!

I presume this is not repeatable if you re-run the process, so we're looking at some kind of hardware failure (memory or disk)?
User avatar
Morten|Dyalog
 
Posts: 409
Joined: Tue Sep 09, 2008 3:52 pm

Re: Writing data to native file

Postby gil on Fri Jul 10, 2015 5:10 am

I just re-ran it once and it was fine, so I don't think it was related to the actual data written. This was done on a hosted system, but I don't know if the equipment is new. The storage used is a RAID 10 array, not that I know if that makes any difference.
gil
 
Posts: 68
Joined: Mon Feb 15, 2010 12:42 am

Re: Writing data to native file

Postby Morten|Dyalog on Fri Jul 10, 2015 7:35 am

If anything, a RAID array is supposed to decrease the chances of data loss or corruption. My personal experience, however, is that the fancier / more expensive your storage solution is, the MORE likely it is that it will fail in some spectacular fashion :-)
User avatar
Morten|Dyalog
 
Posts: 409
Joined: Tue Sep 09, 2008 3:52 pm

Re: Writing data to native file

Postby PGilbert on Fri Jul 10, 2015 12:38 pm

In our case we are writing the data of the WS every 5 minutes to a native file and also to a backup file. If reading back the first file does not work (usually hard disk problem like bad sector) than we read the backup file (hopefully this one is good). The problem we found is releasing the handle of the file that is corrupted, coming from another APL vendor it did not work all the time. Now we will try reading/writing the file with .NET because Microsoft guaranties that the handle will be released in case of error while reading/writing the file.

If the handle is not released than the following writing will not work (file already tied error) and the customer has to close and restart the program which is a problem in our case.
User avatar
PGilbert
 
Posts: 419
Joined: Sun Dec 13, 2009 8:46 pm
Location: Montréal, Québec, Canada

Re: Writing data to native file

Postby kai on Fri Jul 10, 2015 2:04 pm

My personal experience, however, is that the fancier / more expensive your storage solution is, the MORE likely it is that it will fail in some spectacular fashion :-)


Schrödinger's Backup: The condition of any backup is unknown until a restore is attempted.
User avatar
kai
 
Posts: 123
Joined: Thu Jun 18, 2009 5:10 pm
Location: Hillesheim / Germany


Return to Language

Who is online

Users browsing this forum: No registered users and 1 guest