Instability caused by an infrastructure failure at the US-EAST (Newark) data center

Resolved · Partial outage · Started: July 27, 2025 at 8:30 AM · Resolved: July 29, 2025 at 9:49 PM

Updates

  1. Resolved · July 29, 2025 at 9:49 PM

    Post-mortem

    What happened:

    The access problems were caused by an outage in part of our hosting provider's infrastructure at the US-EAST (Newark-NY) data center, due to complications in the cooling system after a power outage.

    You can follow the technical details and all of our provider's public communication about the incident here: https://status.linode.com/incidents/6yw88b0ft94g

    Relevant fact:

    This problem with this specific provider affected not only Cademí's services but those of thousands of other companies around the world that also rely on this solution for part of the infrastructure behind their technology services.

    The failure overloaded the replication and failover services in cascade, which in turn also compromised our redundancy servers, affecting not only Newark but also the Fremont, Dallas, Washington, Tokyo (Japan) and Sydney (Australia) data centers.

    Actions taken:

    Alongside this incident, after 4 updates from the provider and once it was clear the situation could escalate, our team acted promptly to restructure and activate the application's backup service on another hosting provider to ensure service continuity.

    Because of this, over the next 24 hours, some platform features may show small performance adjustments while the new network addresses fully stabilize.

    Data and information:

    No relevant information or data from your project was affected or lost during this period.

    Transactions:

    All purchases made during this window were saved in a dedicated database and will be reprocessed, guaranteeing your students' access.

    Thank you again for your understanding. We are available to support you with anything you need.

    Our team is actively monitoring all support channels for any questions.

    We reaffirm our commitment to the security, transparency and continuous improvement of our systems.

    Best regards, Cademí Support Team

  2. Resolved · July 29, 2025 at 9:49 PM

    All services have been restored. We keep monitoring closely to ensure the platform's continued stability. If you are still having any trouble, please open a ticket with our Support team. We are ready to help.

  3. Identified · July 28, 2025 at 1:45 PM

    The images, attachments, certificate issuing and upload service is 100% functional again.

  4. Identified · July 28, 2025 at 9:30 AM

    We have identified an instability in the platform's image service. Our team is working to bring it back to normal right away.

  5. Identified · July 28, 2025 at 6:45 AM

    We have started processing the purchases stored in contingency during the incident. Access is being granted gradually, with full validation of each transaction.

  6. Identified · July 28, 2025 at 2:45 AM

    We have fully reactivated the platform's main services: webhook queues, email sending, the API and system access. Image and attachment loading is also back to normal.

  7. Identified · July 27, 2025 at 2:30 PM

    We are finishing bringing up the essential services in an alternative environment. The platform is already showing signs of partial recovery, and new updates will be posted soon.

  8. Identified · July 27, 2025 at 11:15 AM

    It has been confirmed that the impact extends to other regions because of service interdependence, also affecting the Dallas, Fremont, Washington, Sydney, Tokyo and Toronto data centers. We are continuing with the contingency and recovery plans.

  9. Identified · July 27, 2025 at 10:45 AM

    The infrastructure provider confirmed that the failure was caused by a problem in the data center's cooling system after a power outage. Power has been restored, but the impact continues. Given the severity of the situation, our team is starting to migrate the critical infrastructure to another environment.

  10. Identified · July 27, 2025 at 8:30 AM

    Our team activated a contingency to safely store every sale notification sent by the payment gateways. This ensures no transaction is lost: all of them will be reprocessed later, guaranteeing students' access.

  11. Investigating · July 27, 2025 at 8:30 AM

    We are investigating a widespread instability affecting our services. The incident originates from a structural failure at the US-EAST (Newark) data center, run by our hosting provider. We keep monitoring and will update as soon as we have new information.