We will be performing work on our primary DNS server tomorrow night,
this should have no noticeable effect on service as our secondary DNS
server will take over for the period of the maintenance. Preparation
work will begin at around 10pm and we will take the server offline at
around midnight with an aim to complete the work by 6am, at which
point the primary will be brought back online again.

During this time we will also be performing various security and
software updates on the server which will require a reboot. The reboot
will be performed after midnight so as to least affect business.

We have been working today to find the cause of the file system error
we experienced this morning with no real result yet. Due to the nature
of the error we have no logs of the moment the error happened which
makes it much harder than normal to troubleshoot. We have verified the
SMART status of the drives, and have checked the array, all with no
errors returned. Due to this we are thinking this issue may be a
software bug, or possibly load related (due to backups being performed
at the time).

We shall be updating some of the server software tonight, this will be
after midnight so as to least affect service. The updates will
require a reboot at which point we shall run a full disk check, taking
around 30 minutes to an hour to complete.

We have also setup a remote logging server to monitor all server
messages and should the
file system error occur again we should have logs on the remote server
allowing us to identify what error occurred.

Once the updates are finished this evening this issue should be
resolved. Thanks for your patience.

We have had some server issues this morning. We're currently looking
into the issue to see what exactly went wrong.

It may be hard drive related but at this stage there are no errors
being reported by the drives themselves.

If we find a solution to the issue today then we'll post a second
update with maintenance information. This will be scheduled for
tonight.