toggle

toggle http://support.toggle.uk.com support blog posterous.com Fri, 04 Mar 2011 01:25:00 -0800 Server Monitoring & Statistics http://support.toggle.uk.com/server-monitoring-statistics http://support.toggle.uk.com/server-monitoring-statistics

Today we are launching a new service that allows you to track our server performance and outages. The tool provides up to the minute statistics on downtime, response time and a complete archive of hosting performance since we set this running last August. We have placed a link to this service on the right hand side of this support blog (Server Stats) so you can keep an eye on how we are doing.

Stats1

Stats2

Stats3

Stats4

Stats5

See the full gallery on Posterous

Permalink

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Mon, 14 Feb 2011 05:34:00 -0800 Planned Maintenance: 16th Feb 01:00 - 02:00 http://support.toggle.uk.com/planned-maintenance-16th-feb-0100-0200 http://support.toggle.uk.com/planned-maintenance-16th-feb-0100-0200

To reduce the risk of extended downtime in the event of a major power issue at the data centre, we have decided to upgrade our power distribution units to enable us to better control the load and also control the sequence that servers and network equipment boots up.

This upgrade will be taking place on Wednesday 16th Feb at around 01:00.

Due to the nature of this work, we will need to shutdown all equipment and reconnect it to the new setup. We do not anticipate downtime to be longer than 30 minutes, however we are allowing a longer maintenance window to allow for any complications with individual equipment while booting up.

Permalink

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Thu, 30 Dec 2010 02:50:29 -0800 Issue Resolved: 30th December 08:00 http://support.toggle.uk.com/issue-resolved-30th-december-0800 http://support.toggle.uk.com/issue-resolved-30th-december-0800 Rebooting the server yesterday has fixed this database issues. We are keeping a close eye on the software to see if we can find out what caused the issue.

Yesterday the WordPress development team also released a security update for WordPress 3.0. We have been busy updating your sites this morning and that maintenance has been completed successfully. Those of you still on the WordPress 2.8 & 2.9 branches will be contacted separately next week to discuss ongoing support.

Permalink

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Wed, 29 Dec 2010 05:00:00 -0800 Database errors: 29th December 12:50 http://support.toggle.uk.com/database-errors-29th-december-1250 http://support.toggle.uk.com/database-errors-29th-december-1250

We have determined that all data is in tact within the database and our backups. The current issue resides with the database software. We have requested that this part of the server is rebooted and will let you know once it is back online. From what we have established there has not been a security breach.

We would like to recommend that no data on your site is modified (via WordPress or any other CMS) until the issue is resolved.

Permalink

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Wed, 29 Dec 2010 02:16:00 -0800 Database errors: 29th December 11:00 http://support.toggle.uk.com/database-errors-29th-december-2010 http://support.toggle.uk.com/database-errors-29th-december-2010

We are currently investigating issues with WordPress databases on our main web server. We are looking to restore the sites and their databases very shortly and will post back with updates.

At this time we are unable to tell if this is a security breach or server crash.

Permalink

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Sun, 07 Nov 2010 11:10:39 -0800 Updated: 7th November 19:00 http://support.toggle.uk.com/updated-7th-november-1900 http://support.toggle.uk.com/updated-7th-november-1900 Our main server has been experiencing high loads this afternoon which
in turn has affected database connections and MySQL performance. The
server has been rebooted and we are currently running a disk check as
a precaution. Load times are still much higher than expected and until
we find the root cause of this the server will continue to operate
intermittently. We will be back soon with more updates.

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Sun, 07 Nov 2010 10:39:46 -0800 Server Down: 7th November 18:00 http://support.toggle.uk.com/server-down-7th-november-1800 http://support.toggle.uk.com/server-down-7th-november-1800 Our main server is currently offline. We are investigating why this
has happened and will have more news from the onsite engineers
shortly.

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Thu, 22 Jul 2010 10:56:56 -0700 Planned Maintenance: 23rd June 22:00 - 06:00 http://support.toggle.uk.com/planned-maintenance-23rd-june-2200-0600 http://support.toggle.uk.com/planned-maintenance-23rd-june-2200-0600 We will be performing work on our primary DNS server tomorrow night,
this should have no noticeable effect on service as our secondary DNS
server will take over for the period of the maintenance. Preparation
work will begin at around 10pm and we will take the server offline at
around midnight with an aim to complete the work by 6am, at which
point the primary will be brought back online again.

During this time we will also be performing various security and
software updates on the server which will require a reboot. The reboot
will be performed after midnight so as to least affect business.

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Wed, 07 Jul 2010 08:20:37 -0700 Maintenance Update: 8th June 00:00 http://support.toggle.uk.com/maintenance-update-8th-june-0000 http://support.toggle.uk.com/maintenance-update-8th-june-0000 We have been working today to find the cause of the file system error
we experienced this morning with no real result yet. Due to the nature
of the error we have no logs of the moment the error happened which
makes it much harder than normal to troubleshoot. We have verified the
SMART status of the drives, and have checked the array, all with no
errors returned. Due to this we are thinking this issue may be a
software bug, or possibly load related (due to backups being performed
at the time).

We shall be updating some of the server software tonight, this will be
after midnight so as to least affect service. The updates will
require a reboot at which point we shall run a full disk check, taking
around 30 minutes to an hour to complete.

We have also setup a remote logging server to monitor all server
messages and should the
file system error occur again we should have logs on the remote server
allowing us to identify what error occurred.

Once the updates are finished this evening this issue should be
resolved. Thanks for your patience.

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Wed, 07 Jul 2010 03:05:43 -0700 Server Problems http://support.toggle.uk.com/server-problems-1 http://support.toggle.uk.com/server-problems-1 We have had some server issues this morning. We're currently looking
into the issue to see what exactly went wrong.

It may be hard drive related but at this stage there are no errors
being reported by the drives themselves.

If we find a solution to the issue today then we'll post a second
update with maintenance information. This will be scheduled for
tonight.

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Fri, 26 Feb 2010 08:02:00 -0800 Planned Maintenance: 26th & 27th Feb 22:00 - 04:00 http://support.toggle.uk.com/planned-maintenance-26th-and-27th-february-22 http://support.toggle.uk.com/planned-maintenance-26th-and-27th-february-22

We are performing some updates across all servers on our network over
the coming weekend. This will involve a reboot of each server, and
also there may be a short period where each service (Web, Mail, FTP)
is offline/unresponsive.

These updates are to bring all software up to date and also add
additional security. Updates will take place between 10pm and 4am
Friday and Saturday night. Any service affecting updates will take
place after midnight.

If you have any questions please don't hesitate to get in touch.

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Wed, 14 Oct 2009 06:15:00 -0700 Network Maintenance: Friday 23rd October http://support.toggle.uk.com/network-maintenance-friday-23rd-october http://support.toggle.uk.com/network-maintenance-friday-23rd-october

Our network providers have made us aware of some routine software
updates scheduled to be carried out on various routers on Friday night
(16/10/09).

As we have redundant connections this should not affect our
connectivity, however should a fault arise during the maintenance then
an outage will occur.

We will be monitoring our network throughout this period as usual.
Please don't hesitate to contact us should you have any problems.

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Wed, 08 Jul 2009 10:57:44 -0700 Planned Maintenance: 10th July 23:00 http://support.toggle.uk.com/planned-maintenance-10th-july-2300 http://support.toggle.uk.com/planned-maintenance-10th-july-2300 Since the hard drive errors a month ago we have been starting to
experience more frequent SATA errors in the server logs. We believe
the motherboard is most likely responsible for this problem (we have
already replaced PSU and SATA cables).

We are therefore taking the decision to replace the motherboard this
coming Friday (10th July). We shall be working in the data center from
around 11pm, and will start the motherboard replacement after midnight
to minimise disruption.

We expect a downtime of around 20 minutes.

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Fri, 12 Jun 2009 03:28:13 -0700 Issue Resolved: 12th June 08:00 http://support.toggle.uk.com/issue-resolved-12th-june-0800 http://support.toggle.uk.com/issue-resolved-12th-june-0800 Having spent the last two days monitoring the server we are now
confident that the early morning sickness has been resolved. We
updated the Linux kernel/libraries last night and ran the evening
backups manually. The backups completed without issue and the server
did not re-mount the hard drive as "read-only". This issue is now
closed.

Please get in touch if you have any questions.

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Wed, 10 Jun 2009 03:32:00 -0700 Maintenance Update: 10th June 11:00 http://support.toggle.uk.com/maintenance-update-10th-june-1100 http://support.toggle.uk.com/maintenance-update-10th-june-1100

We replaced the PSU and SATA cables on our main server last night and
as a result web services were down for a very brief period between
10:00 and 10:20. Upon further investigation we now believe these not
to be the cause of the problem.

We have narrowed the problem down to either load related issue, or
something that occurs during early morning backups. The backups that
we are running tonight are in debug mode, so we should have a large
amount of info should the problem happen again before the backups
complete.

If you have any troubles with your sites today then please get in
touch. It is likely that this problem will be solved during the course
of today and while we locate the cause of the error, the server
continues to operate as expected.

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle Tue, 09 Jun 2009 13:06:51 -0700 Emergency Maintenance: 9th June 21:00 http://support.toggle.uk.com/emergency-maintenance-9th-june-2100 http://support.toggle.uk.com/emergency-maintenance-9th-june-2100 We will be taking our main server off-line within the next hour to
replace both the power supply and the hard drive cables. This
emergency maintenance is being carried out in response to an issue
that occurred on Monday morning and again today. We expect the
downtime this evening to be brief. If you have any questions please
contact us via email.

]]> http://files.posterous.com/user_profile_pics/70105/toggle_logo_green.png http://posterous.com/users/4Ttzgrm7o toggle toggle toggle