Network Status & Advisories

Summary

MySQL Server Failure

Status

Resolved

Start

Fri, 9th May - 8:54 am

End

Fri, 9th May - 9:28 am

ID

A33

Our primary MySQL server hosting customer’s databases failed this morning at 8.54 am (UTC+10) and did not automatically failover to our secondary/backup server. Engineers resolved the problem at 9.28 am, and confirmed that all services are operational again.

We apologise for the downtime and we are investigating why the server did not automatically failover to the backup server like designed.

Affects customers with the following services:

• Shared Hosting Services

Updates:

Summary

Important: Contacting Support and the separation of The Mac Experts operations

Status

Start

Thu, 1st May - 12:01 am

ID

A32

Beginning May 1st the business operations for our two businesses; TotalServe and The Mac Experts will be operating completely separately, albeit with the same backend staff.

We’re internally restructuring the two businesses to function as separate entities to allow us to take on board new employees who will work for one of our businesses only. Eventually we plan for The Mac Experts to be run and managed by new staff (we’ll still own/control the business) allowing us to concentrate on our TotalServe products and services.

As a TotalServe customer it’s critical you contact our TotalServe support channels, and discontinue contacting The Mac Experts for support - beginning May 1st, you will not be able to obtain support via The Mac Experts for TotalServe products and services.

Additionally, we’ll be enabling telephone support for all new and existing customers, making contacting us a lot easier. In the next few days we’ll be issuing new Welcome Packs for existing customers that will include updated contact information and our new phone numbers

If you have any questions, please contact support@totalserve.net.au and we’ll be happy to answer any questions you may have.

Affects customers with the following services:

• I.T. Support & Maintenance Services

Updates:

Wed, 30th Apr - 4:35 pm

​Our new 1300 number is now set-up and operational. All customers should update their address books with this new number. You can begin using this new number immediately for any support, on-site booking and general enquiries.

Currently, if you call please leave your name, number and return telephone number which will internally open a new support ticket/callback request which we’ll then action and call you back within 15-minutes. We’ll be making refinements over the next few weeks with how we operate telephone calls.

The new number will be posted to our website in the next few days once our new site design is ready.

Summary

Data Connectivity Issues

Status

Resolved

Start

Tue, 29th Apr - 9:22 am

ID

A29

Canberra based customers may be experiencing a disruption to their service with high packet loss, dropouts and data connectivity issues. This is under investigation and we will endeavour to resolve this matter as soon as possible.

Affects customers with the following services:

• All Services (All Customers)

Updates:

Tue, 29th Apr - 10:41 am

The data connectivity issues are now resolved. We apologise for the downtime and if customers continue to observe issues should open a support ticket.

Summary

Extremely Critical Defect Found in Cryptographic Software

Status

Resolved

Start

Tue, 8th Apr - 9:14 pm

End

Wed, 9th Apr - 5:00 am

ID

A28

​Security researches have discovered an extremely critical defect in the cryptographic software libraries that our load balancers, secured web-servers and mail servers utilise that prevents the eavesdropping of passwords, banking credentials, and other sensitive data.

This is an incredibly major bug and possibly one of the biggest threats we’ve ever seen, affecting approximately 66% of the entire Internet worldwide. We’ve rushed and completed updates to all our front-end load balancers, web-servers and mail servers, so they are now safe and protected from possible leakage of confidential information.

This vulnerability can potentially expose customers confidential information to man-in-the-middle type attacks. Like many hundreds of thousands of system administrators tonight, we will be preventing any potential or theoretical attacks to our customers by revoking all our SSL security certificates and deploying new secure certificates. We’ll also be taking the extraordinary pre-cautious step of changing all our passwords for all critical and non-critical systems, which is going to be a lengthly process.

In short we’ve taken steps to address this security vulnerability, and we’re as of now considered “secure”. However it is theoretically possible that your passwords used for any web-services whether with us, or another company could be compromised by someone directly targeting either our services, or your connection. While we have no evidence that any of our customers or our services have been targeted, as this defect affects a very large percentage of the entire Internet such as your hosting with us, your Gmail account, Facebook account, eBay account and even your Internet banking we are recommending customers also take extraordinary steps to change their passwords for any and all web-services.

Affects customers with the following services:

Updates:

Wed, 9th Apr - 10:19 am

​All customer facing SSL certificates have been revoked, re-issued and re-deployed. As an extraordinary precaution we’ve also completed changing all root, system administrator and staff passwords, plus private/public keys.

Summary

Canberra Co-Location Transit Failure

Status

Resolved

Start

Wed, 12th Mar - 12:42 pm

ID

A26

It appears we’ve lost IP data connectivity within our Canberra Co-location facilities. We are currently investigating the issue.

Affects customers with the following services:

Updates:

Wed, 12th Mar - 12:59 pm

​Data connectivity has been fully restored and is operation again. We apologise for the outage and are continuing investigation of the root cause.

Thu, 20th Mar - 11:28 am

After further investigation it was found a core router had been rebooted by our upstream provider which resulted in a complete loss of data connectivity in the A.C.T. As we understand it, due to the integration work between TransACT, iiNet and their carrier services (Agile, Chime) greater redundancy is being added to their core networks in the A.C.T. which will further reduce this problem reoccurring in the future.

We are currently investigating with our provider about upgrading our data connectivity to include multi-homed connectivity which will protect us from a core router failure in the future. An upgrade to this connectivity will incur a downtime window to be scheduled. We will announce the changes and schedule a downtime window once the details have been finalised.

Summary

Anti-spam/anti-virus Processing Failure

Status

Resolved

Start

Mon, 3rd Mar - 8:24 am

ID

A25

We’ve just received notification from our monitoring system that our anti-spam/anti-virus server is offline and efforts to bring it back online are not successful.

This failure does not affect customers sending or receiving of e-mails, however you may see an increase of the amount of spam received in your inbox.

We’ve temporary placed an anti-virus solution in place and expect to have full anti-spam capabilities restored in 2-4 hours time.

An update will be provided no later than 12.55 PM.

Affects customers with the following services:

Updates:

Mon, 3rd Mar - 11:05 am

​A replacement anti-spam/anti-virus appliance is currently being brought online. We expect normal operations to resume within 45-60 minutes.

Mon, 3rd Mar - 12:10 pm

​The replacement appliance is now online and fully operation. All anti-spam and anti-virus processing for e-mail is restored.

Summary

Support tickets sometimes not being submitted

Status

Resolved

Start

Tue, 26th Feb - 11:19 pm

End

Wed, 27th Feb - 5:41 pm

ID

A3

​We’ve been notified by some customers that they are having difficulties opening tickets via the Control Panel.

It appears that some newly created tickets are not successfully submitted to our help desk system, with no errors returned and no other indication that the ticket wasn’t successfully lodged.

We utilise Webgroup Media’s Cerb6 helpdesk product and it appears that our control panel is making a successful connection to our helpdesk interface, but the ticket is vanishing. We’ve opened a support call with Webgroup Media for assistance in tracking down why tickets are not always being created and will update this ticket once we hear from them (likely in the next 12-18 hours).

In the meantime we advice all customers to e-mail support@totalserve.net.au to obtain support and a notice will be posted in the Control Panel to advice of these current problems.

Affects customers with the following services:

• All Services (All Customers)

Updates:

Wed, 27th Feb - 8:21 am

​We’ve been provided an emergency software update that we will be applying at 10.00 am. Please be aware any tickets created between 10.00 am - 10.15 am customers might not receive an auto-reply confirmation with their ticket id.

Wed, 27th Feb - 11:27 am

​The software update has been applied, and we’re now in the process of resolving issues with the control panel.

Wed, 27th Feb - 5:37 pm

​We’ve resolved these issues and creating tickets via the control panel will now be reliable. Improvements made: customers will now receive an instant auto-reply confirming ticket creation as-well as visual feedback within the control panel that the ticket was successful created.

Additionally a handful of corrupted tickets have been removed and we’ve put in place better logging capabilities to catch any future errors to ensure these issues don’t reoccur.

Summary

Telephone/PBX System Failure

Status

Resolved

Start

Mon, 1st Apr - 1:00 am

End

Mon, 9th Dec - 8:39 am

ID

A2

​A hard drive in our telephone/PBX system has failed and we’ve been unable to restore the server, or recover it’s data (we also didn’t have a backup, whoops!).

We’ve setup a temporary phone system for Service Level Agreement customers only, such customers will notice that the interactive voice response menus have been simplified until we re-build the system.

Affects customers with the following services:

• All Services (All Customers)

Updates:

Mon, 9th Dec - 3:16 pm

​This issue is now completely resolved. For customers with phone support, please login to our website to view our contactable phone numbers. We won’t be making general enquiry numbers available to the general public.

Mon, 6th May - 3:22 pm

​New hardware has arrived to begin rebuilding our phone system. We’re just waiting on some new line cards and expect to have the PBX operational again in the next several days.