We have been working today to find the cause of the file system error
we experienced this morning with no real result yet. Due to the nature
of the error we have no logs of the moment the error happened which
makes it much harder than normal to troubleshoot. We have verified the
SMART status of the drives, and have checked the array, all with no
errors returned. Due to this we are thinking this issue may be a
software bug, or possibly load related (due to backups being performed
at the time).

We shall be updating some of the server software tonight, this will be
after midnight so as to least affect service. The updates will
require a reboot at which point we shall run a full disk check, taking
around 30 minutes to an hour to complete.

We have also setup a remote logging server to monitor all server
messages and should the
file system error occur again we should have logs on the remote server
allowing us to identify what error occurred.

Once the updates are finished this evening this issue should be
resolved. Thanks for your patience.