We are currently experiencing an internal fault on our VoIP Centrex Server lon-pbx-12.

Our server team are urgently investigating the issue and we are aiming to resolve the problem as soon as possible. Customers may experience issues with calls terminating at present and we apologise for this.

Update 14:43

Our server team have created an emergency fix for two unrelated problems which were causing an intermittent fault on lon-pbx-12 and the server has now been stable for 75 minutes.  Once the changes have been reviewed, they will be rolled out to customers on other servers.

We apologise for the inconvenience caused by this unavoidable problem.

We are currently experiencing an issue where customer facing control panels are unavailable.

Customers will receive an error message when trying to access either the Hosting or VOIP Control Panels, this outage is also affecting our partners.

We have engineers currently investigating this problem and will post an update once further information has been received.

This issue has now been resolved by our oncall engineering team, if you still experience problems accessing your Control Panel, please raise a support incident via the Gradwell Portal.

Starts: 2008/09/04 18:30 Ends: 2008/09/04 19:00

This evening we will be releasing a series of improvements to our partner VoIP CSR’s. Partners may have difficulty using the CSR admin area for a short period of time while the upgrade takes place.

This upgrade will only affect the CSR admin area, and will not affect the functioning of end users VoIP service.

Starts: 2008/09/03 22:00 Ends: 2008/09/03 23:00

We will be running a full file system check on all mailboxes hosted on our v-mail-file-8 server this evening. If you are unsure whether your mailbox will be affected or not, please check under the mailboxes menu within the hosting control panel, where the server which hosts your email will be displayed, on a per box basis.

We expect this maintenance to last less than an hour. However, we have noticed some minor anomalies in the file system and it is possible that file system checking may take longer than expected.

Update:

The maintenance finished successfully at around 22:25.

09:45

We are currently aware of a problem with our FreeBSD shell server ‘Ochre’ which also handles customer FTP. FTP is currently working correctly, however ssh access is not. We are currently awaiting assistance from on-site datacentre engineers in connecting to the server and will update customers via GradwellStatus once more information is available.

10:10 update

Shell access to Ochre has been restored from 10:03, no other services appear to have been affected or existing sessions interrupted.

17:35 We are currently experiencing a very high load on one of our back-end mail storage server (v-mail-file-7).
We are currently rebooting the server and expect it to back back online very shortly.

17:44: The v-mail-file-7 server is now up and responding to mail requests normally, we apologise for any inconvenience caused.

13:20

We have now brought our v-mail-file-3 server back online. If you are experiencing issues with mail access on this server, please retry and if problems persist, please let our support team know and we will be happy to help.

As an update to customers on our v-mail-file-3 mail server, we are currently run-running our file system integrity scans due to an internal error in the software which performs the checking and repairing. Further updates will be released as soon as possible and we will follow up any mailboxes which have issues. We will not be able to give an exact time on resolution at present and will update gradwellstatus regularly to keep all customers informed.

We are currently experiencing an issue with lon-pbx-12, customers on this server may experience loss of audio. We have found an intermittent software fault on the server which are server team are urgently investigating. We must emphasise that the majority of calls to this server are progessing properly and the server has been functioning normally for the past 30 minutes, however we are currently examining traces to see if there is an underlying issue to resolve on the server.

11:51

We have experienced an issue with one of our our back-end mail storage servers which has caused one of the mailbox servers to be marked read-only for security. We have taken this server our of service as an emergency response. The file system is now currently being thoroughly checked. Mailboxes not on v-mail-file-3 as shown within the hosting control panel will not be affected. We apologise for the inconvenience which we expect to last less than thirty minutes. We will update customers again via Gradwell Status as soon as possible.