Post-incident report: 20 October 2026 service outage

On the morning of Tuesday 20 October 2026, Payaca experienced a service outage between 07:00 BST and 09:38 BST affecting the Payaca web app, Field App, customer portal and API. We know how much your teams rely on Payaca to see their work schedules and travel to customers every morning, and we are sorry for the disruption this caused to your workday. We posted updates throughout the morning on our status page at status.payaca.com, where you can subscribe to be notified of any future incidents.

Here is a straightforward breakdown of what happened, what we did to restore service and recover data, and the changes we are making to ensure our infrastructure is resilient against regional cloud outages in the future.

What caused the outage

Shortly after 07:00 BST on 20 October, Google Cloud experienced a major infrastructure incident in its London region following a reported power event at its data centre complex. As reported by the BBC, the incident affected a wide range of UK businesses and public services, including major banks, delivery networks and the courts.

Payaca's primary database and its high-availability replica are hosted in Google Cloud's London region across separate availability zones. Under normal circumstances, if one zone in a data centre experiences a hardware or network fault, our database automatically fails over to another zone in the same region within moments.

During this incident, however, the underlying disruption affected database services across the entire London region simultaneously. When Google Cloud's automated failover triggered at 07:06 BST, it was unable to complete because the wider region was degraded. Standard recovery actions within the London region - including restarting the primary database and restoring backups to other London zones - also could not complete while the regional incident was ongoing.

How we mitigated the issue and recovered data

Failing over to a different region

Once it became clear that database operations within the London region could not complete, we moved our recovery efforts out of the UK region entirely. At 08:32 BST, we initiated a restore of our production database into Google Cloud's Paris region, and re-routed our backend to the new database.

Full service was restored across the Payaca web app, Field App, customer portal and API at 09:38 BST.

Recovering data from the 3-hour backup window

To bring the platform back online in a new region as quickly as possible, our initial failover restored from our most recent scheduled backup, which had been taken roughly three hours before the outage began (at 04:29 BST). This meant that when Payaca came back online at 09:38 BST, activity recorded between 04:29 BST and 07:03 BST was not yet present in the active database.

Later that morning, at 10:09 BST, our original London database recovered enough to accept read-only connections. We extracted all records created or updated between 04:29 BST and 07:03 BST, and merged them into the active database at 12:14 BST.

Handling data conflicts

Because Payaca was live and taking new updates between 09:38 BST and 12:14 BST while we prepared the data recovery, there were a small number of cases where newly created records overlapped with records from earlier that morning.

Where a record had already been edited more recently than 09:38 BST, we kept the newer version so that current work was never overwritten. Where sequential numbers clashed - such as a new project or proposal being assigned the same reference number as one created between 04:29 BST and 07:03 BST - we preserved both records and ensured they could be distinguished.

Only a small proportion of accounts were affected by a data conflict. We have already dealt with these cases and will contact you directly if your account was one of the few affected.

What we are changing for the future

While full-region cloud outages are rare, an incident like this is a valuable test of where our architecture relies on assumptions that can fail under extreme conditions. Two limitations stood out during our response: our automated failover was confined to a single geographic region, and our cross-region recovery relied on a scheduled backup rather than continuous replication.

We are making two structural changes to address both:

1. Reducing the window of potential data loss

Relying on a periodic backup meant our initial cross-region restore carried a three-hour gap that had to be reconciled afterwards. We have now enabled continuous point-in-time recovery on our production database and are pairing this with continuous cross-region replication. In a future disaster recovery scenario, this reduces the potential window of data loss from hours down to seconds.

2. Automatic cross-region failover

Until today, our high-availability setup was designed to survive the loss of an availability zone within London, while failing over to another region required manual engineering intervention to restore a backup and update service configuration.

We are adding a dedicated disaster recovery replica in a secondary region and updating how our services connect to the database. Should a regional outage happen again, our infrastructure will be able to fail over to a healthy region automatically.

Once again, we apologise for the disruption this morning's outage caused to you and your team.

Matt Bessey,
CTO