As part of our continued commitment to customer security we've rolled out a significant windows update to all our Windows servers today. This was to resolve an issue with MS14-066 applied last week. We started this roll out at 3.15am this morning (November 19th) to minimize downtime for our customers. Unfortunately this update has effected some hosted customers and our main web site today and I just wanted to provide some further information.UPDATE:
After further investigation and working with the Azure support team it appears the outage yesterday was not caused due to the security patch we applied to our servers but a general outage with Azure Storage Services within Europe, Asia & America. Jason Zander, Corporate VP of Microsoft Azure, has posted a blog to provide an Update on Azure Storage Service Interruption. This blog provides a high level description of the issue and what has been done to date. Please accept our apologies for this interruption.https://azure.microsoft.com/blog/2014/11/19/update-on-azure-storage-service-interruption
We've also posted a full background to the issues here...https://blog.instantasp.co.uk/General/Apology-for-the-Azure-outage-yesterday-November-19th
You can also learn more on twitter using the hash tag #AzureDownhttps://twitter.com/hashtag/AzureDown
This security update we applied resolves a critical security vulnerability within the TLS / SSL layer within Windows Server 2008 & 2012 that could allow a remote attacker to execute code on the server. Due to the severity of this issue and ease at which it can be executed we didn't provide any notice to our hosted customers and went ahead to immediately patch our servers. We have contacted all of our hosted customers this morning to inform of this update.
If you’re interested you can learn about this vulnerability within the following TechNet article…https://technet.microsoft.com/en-us/library/security/ms14-066.aspx
If you perform a quick Google search for MS14-066
you’ll see this update has caused a few major issues for many different vendors. Both Amazon AWs and IBM application services have been effected.
Unfortunately it’s not hard to find how to execute this vulnerability and I expect there will be pre-built modules for tools like Metasploit very soon. .
You can also learn more about this problem within the SecurityNow episode below…
After applying the various Windows updates 3 of our virtual machines didn't come fully back online. We've worked with a Microsoft Azure support engineer remotely today to resolve these issues with the affected VMs and all services should now be fully operational once again.
In addition we have also applied workarounds for the MS14-066 vulnerability to ensure we are not effected.
We are continuing to closely monitor our servers after the update. We sincerely apologize for any inconvenience this unexpected downtime may have caused. We've invested heavily to ensure a reliable service for our hosted customers and it’s frustrating for us when you are effected by issues like this.
I’ll continue to keep this topic updated with developments if we have any further issues.
If your affected or we can assist with any questions please don’t hesitate to respond here, email us on email@example.com
or call us directly on +44 ( 0 ) 1565 832 827. UPDATE: 19th November 6.30pm
We received the following advisory notice from Microsoft...
Virtual Machines - North Europe and West Europe - Advisory
We are investigating issues impacting a limited subset of Virtual Machines customers in North Europe and West Europe. Further information is available in the Management Portal of potentially impacted customers.
UPDATE: 19th November 7.45pm
We received a response from the Azure support team which indicates there may be "residual impact due to the initial outage". All services are currently online however please know if you do encounter any further connectivity issues we will be aware and working to resolve.
From Azure Support...
Please accept our apologies for the problems this outage has cause you and your customers. We have full engagement from the engineering team and senior management and will be working around the clock until this is resolved. While this issue is largely mitigated, we understand there are customers who are still being affected by residual impact due to the initial outage. To assist us in our continuing efforts if you are still being impacted by this issue would you please reply to this message with the following...UPDATE 19th November 10pm
We received the following update from Microsoft...."This issue is now mitigated for North Europe. We continue to investigate and address issues impacting a limited subset of Virtual Machines customers in West Europe. A subset of customers may see their VMs in continual "Start state”, and limited subset of customers may have difficulty in connecting to their VMs. Potentially impacted customers are advised to continue to visit the Management Portal"
Our primary VMs for hosted customers are in the North Europe data center however for geo-redundancy we do sister with the Western Europe data center so will continue to closely monitor our customer installations. . UPDATE: 20th November 1am
Update from Azure support...
Thank you for your patience in this matter. Jason Zander, Corporate VP of Microsoft Azure, has posted a blog to provide an Update on Azure Storage Service Interruption. This blog provides a high level description of the issue and what has been done to date. Please accept our apologies for this interruption.https://azure.microsoft.com/blog/2014/11/19/update-on-azure-storage-service-interruption
UPDATE: 20th November 11am
Just a quick update to let you know we've not seen any further problems after the outage yesterday. We are continuing to closely monitor our services and will post here should we encounter any further issues.
firstname.lastname@example.orgBlog | Community