Quality Location GmbH - Storage and SIP trunk failure – Incident details

All systems operational

Storage and SIP trunk failure

Resolved
Operational
Started over 2 years agoLasted about 6 hours

Affected

Managed Services

Partial outage from 9:13 AM to 10:18 AM, Operational from 9:13 AM to 10:10 AM, Partial outage from 10:10 AM to 12:00 PM, Degraded performance from 10:18 AM to 3:03 PM, Operational from 12:00 PM to 3:03 PM

Storage

Partial outage from 9:13 AM to 10:18 AM, Degraded performance from 10:18 AM to 3:03 PM

Webhosting

Partial outage from 9:13 AM to 10:18 AM, Degraded performance from 10:18 AM to 3:03 PM

Nextcloud

Partial outage from 9:13 AM to 10:18 AM, Degraded performance from 10:18 AM to 3:03 PM

vServer Cloud

Partial outage from 9:13 AM to 10:18 AM, Degraded performance from 10:18 AM to 3:03 PM

Telephone

Operational from 9:13 AM to 10:10 AM, Partial outage from 10:10 AM to 12:00 PM, Operational from 12:00 PM to 3:03 PM

Updates
  • Resolved
    UTC
    Resolved

    After testing and observation by us, we can no longer detect any further impairments.

    If you still notice any problems, please let us know.

  • Update
    UTC
    Update

    The cause was 2 faulty NVMES SSDs with greatly increased latency; these were removed from the storage cluster. The faulty disks will be replaced during the course of the day.

    We do not expect any further disruptions and the systems are now running normally again. However, we will continue to monitor this for some time.

  • Monitoring
    UTC
    Monitoring

    As far as we can currently see, telephony is working completely again.

    If you still have problems, please contact us.

    The storage malfunction is currently being resolved by technicians on site in the data center.

    We will keep you updated on this.

  • Investigating
    UTC
    Investigating

    Dear Customers,

    We have been experiencing problems with our storage and its performance for several minutes.

    Furthermore, there is currently a problem with the SIP trunk provider that provides telephony for our customers using the telephone system hosted by us. We have received the following message:

    Due to a network error in our data center, there are currently disruptions to telephony, cloud telephone system registrations and the Teams gateway. We are working hard to resolve the problem.

    Unfortunately, it is currently not possible for customers with telephone systems to make external calls until the problem has been resolved.

    Internal calls are not affected.

    As soon as we have further information, we will inform you immediately.

    We regret the current inconvenience and would like to apologize. We are dependent on the SIP Trunk provider and their cooperation.