Power Outage at Lon1 Monday 10th January 2022 19:01:00

IPI Cloud at Lon1 is currently down, due to power loss at the data center. Power was lost at 18:00 (appox) Engineer sent to site at 18:40

Interxion has now provided some initial communication of the incident RFO. This confirms the timeline of events that we had seen but does not provide a full RFO or detailed follow up actions, it does, however, infer these will be provided, once investigations into the incident have been completed.

Once we have the final RFO we will provide a further update.

Update - After work being completed by our carrier, lon1 transit has now returned and is now fully resilient.

Update from the Datacentre: "Further to our updates yesterday we have continued to run on uninterrupted utility power overnight and our monitoring has shown a stable supply throughout the building. Temperatures within the datacentre returned to normal parameters and has also remained stable. Our facilities systems have also been closely monitored for the last 8 hours and full system redundancy has been confirmed for the site returning us to a fully operational status. A root cause investigation is currently being carried out and an update on our findings will be issued today."

Update - Issue in lon1 is resolved.

IPI SIP is now full functional.

Update - Due to customers seeing issues with Voice Audio at Lon1 (reported at about 04:00) - we have taken the SBCs for IPI SIP out of service at Lon1.

All customer services have failed over to Reading and are working successfully.

Update - All MPLS servers are functional. Transit is still effected at Lon1.

Update - Whilst power is back, we are still seeing some carrier equipment down.

  1. Wave from Lon1-> Link33
  2. Transit from at Lon1

No impact to VMs or servers expected just FYI

We are now seeing power as up in the DC (time 22:03) workings to restore services

Update from INX: "The site has experienced a failure on multiple power feeds in to the building and despite the generators powering up a failed electronic switchgear has prevented us from switching between mains and the generator. The third party supplier of our electrical switchgear has been engaged and is working with our own engineers and other third party specialists in the datacentre to restore normal operations"

Engineer has arrived at site. However due to active power working going on is not allowed access to the DC floor until work is completed. Confirmed as a major outage by the DC and there are over 20 engineers waiting to get access to the DC (other datacentre customers) No estimated time to fix from the DC yet

We have confirmed that power has been lost to the ground floor of the Lon1 building @ Interxion. Engineer still on route. No estimated fix time from the DC