Writing data to native file

[gil]

I'm processing a large csv file into separate, typed native files that are then used as mapped vars from the ws. When writing the data to file, I also do a simple health check to test the content by mapping the file and comparing with the data in the ws. This has never reported a problem before.
Today it did. One out of 2 billion 4-byte integers had an incorrect value on file. I understand this can happen for all sorts of reasons, and I expect it doesn't happen very frequently. Does anybody have any experience of inconsistencies like this when reading/writing to native files?

[Morten|Dyalog]

I don't think we have other recorded, documented cases of anything like this, but then again most people wouldn't think to check for it. Well done for writing the "paranoia check" for something that should really never fail!

I presume this is not repeatable if you re-run the process, so we're looking at some kind of hardware failure (memory or disk)?

[gil]

I just re-ran it once and it was fine, so I don't think it was related to the actual data written. This was done on a hosted system, but I don't know if the equipment is new. The storage used is a RAID 10 array, not that I know if that makes any difference.

[Morten|Dyalog]

If anything, a RAID array is supposed to decrease the chances of data loss or corruption. My personal experience, however, is that the fancier / more expensive your storage solution is, the MORE likely it is that it will fail in some spectacular fashion :-)

[PGilbert]

In our case we are writing the data of the WS every 5 minutes to a native file and also to a backup file. If reading back the first file does not work (usually hard disk problem like bad sector) than we read the backup file (hopefully this one is good). The problem we found is releasing the handle of the file that is corrupted, coming from another APL vendor it did not work all the time. Now we will try reading/writing the file with .NET because Microsoft guaranties that the handle will be released in case of error while reading/writing the file.

If the handle is not released than the following writing will not work (file already tied error) and the customer has to close and restart the program which is a problem in our case.

[kai]

My personal experience, however, is that the fancier / more expensive your storage solution is, the MORE likely it is that it will fail in some spectacular fashion :-)


Schrödinger's Backup: The condition of any backup is unknown until a restore is attempted.