We will be performing work on our primary DNS server tomorrow night,
this should have no noticeable effect on service as our secondary DNS
server will take over for the period of the maintenance. Preparation
work will begin at around 10pm and we will take the server offline at
around midnight with an aim to complete the work by 6am, at which
point the primary will be brought back online again.

During this time we will also be performing various security and
software updates on the server which will require a reboot. The reboot
will be performed after midnight so as to least affect business.

We have been working today to find the cause of the file system error
we experienced this morning with no real result yet. Due to the nature
of the error we have no logs of the moment the error happened which
makes it much harder than normal to troubleshoot. We have verified the
SMART status of the drives, and have checked the array, all with no
errors returned. Due to this we are thinking this issue may be a
software bug, or possibly load related (due to backups being performed
at the time).

We shall be updating some of the server software tonight, this will be
after midnight so as to least affect service. The updates will
require a reboot at which point we shall run a full disk check, taking
around 30 minutes to an hour to complete.

We have also setup a remote logging server to monitor all server
messages and should the
file system error occur again we should have logs on the remote server
allowing us to identify what error occurred.

Once the updates are finished this evening this issue should be
resolved. Thanks for your patience.

We have had some server issues this morning. We're currently looking
into the issue to see what exactly went wrong.

It may be hard drive related but at this stage there are no errors
being reported by the drives themselves.

If we find a solution to the issue today then we'll post a second
update with maintenance information. This will be scheduled for
tonight.

We are performing some updates across all servers on our network over
the coming weekend. This will involve a reboot of each server, and
also there may be a short period where each service (Web, Mail, FTP)
is offline/unresponsive.

These updates are to bring all software up to date and also add
additional security. Updates will take place between 10pm and 4am
Friday and Saturday night. Any service affecting updates will take
place after midnight.

If you have any questions please don't hesitate to get in touch.

Our network providers have made us aware of some routine software
updates scheduled to be carried out on various routers on Friday night
(16/10/09).

As we have redundant connections this should not affect our
connectivity, however should a fault arise during the maintenance then
an outage will occur.

We will be monitoring our network throughout this period as usual.
Please don't hesitate to contact us should you have any problems.

Since the hard drive errors a month ago we have been starting to
experience more frequent SATA errors in the server logs. We believe
the motherboard is most likely responsible for this problem (we have
already replaced PSU and SATA cables).
 
We are therefore taking the decision to replace the motherboard this
coming Friday (10th July). We shall be working in the data center from
around 11pm, and will start the motherboard replacement after midnight
to minimise disruption.
 
We expect a downtime of around 20 minutes.

Having spent the last two days monitoring the server we are now
confident that the early morning sickness has been resolved. We
updated the Linux kernel/libraries last night and ran the evening
backups manually. The backups completed without issue and the server
did not re-mount the hard drive as "read-only". This issue is now
closed.
 
Please get in touch if you have any questions.

We replaced the PSU and SATA cables on our main server last night and
as a result web services were down for a very brief period between
10:00 and 10:20. Upon further investigation we now believe these not
to be the cause of the problem.
 
We have narrowed the problem down to either load related issue, or
something that occurs during early morning backups. The backups that
we are running tonight are in debug mode, so we should have a large
amount of info should the problem happen again before the backups
complete.
 
If you have any troubles with your sites today then please get in
touch. It is likely that this problem will be solved during the course
of today and while we locate the cause of the error, the server
continues to operate as expected.

We will be taking our main server off-line within the next hour to
replace both the power supply and the hard drive cables. This
emergency maintenance is being carried out in response to an issue
that occurred on Monday morning and again today. We expect the
downtime this evening to be brief. If you have any questions please
contact us via email.