Duration: 2 minutes

Affected Components: https://news.kagi.com

Mar 12, 00:48:21 GMT+0
Investigating - https://news.kagi.com cannot be accessed at the moment. This incident was created by an automated monitoring service..

Mar 12, 00:50:39 GMT+0
Resolved - https://news.kagi.com is now operational! This update was created by an automated monitoring service..

Duration: 1 minute

Affected Components: https://news.kagi.com

Mar 12, 00:39:19 GMT+0
Investigating - https://news.kagi.com cannot be accessed at the moment. This incident was created by an automated monitoring service..

Mar 12, 00:40:33 GMT+0
Resolved - https://news.kagi.com is now operational! This update was created by an automated monitoring service..

Duration: 43 minutes

Affected Components: , [Assistant] asia-southeast1, [Assistant] europe-west2, [Assistant] us-west2, [Assistant] southamerica-east1, [Search] asia-east2, [Search] europe-west4, [Search] us-west2, [Search] asia-southeast1, [Assistant] australia-southeast1, [Search] us-east4, [Search] australia-southeast1, [Assistant] europe-west4, [Assistant] us-east4, [Assistant] us-central1, [Search] us-central1, [Assistant] asia-east2, , [Search] southamerica-east1, [Search] europe-west2, Kagi Search → Kagi Assistant →

Mar 8, 06:00:00 GMT+0
Identified - We plan to perform a minor upgrade to the [**Kagi.com**](http://Kagi.com) database backend on this time. During the maintenance window, write operations such as updating your profile or changing settings may be temporarily slower. However, we expect there won't be much disruption when accessing [**Search**](https://kagi.com/search) or [**Assistant**](https://kagi.com/assistant)..

Mar 8, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 8, 06:42:48 GMT+0
Completed - Maintenance has completed successfully..

Duration: 12 minutes

Affected Components: https://orionfeedback.org

Mar 6, 09:30:14 GMT+0
Investigating - https://orionfeedback.org cannot be accessed at the moment. This incident was created by an automated monitoring service..

Mar 6, 09:42:41 GMT+0
Resolved - https://orionfeedback.org is now operational! This update was created by an automated monitoring service..

Duration: 17 minutes

Affected Components: https://kagifeedback.org

Mar 6, 09:24:53 GMT+0
Investigating - https://kagifeedback.org cannot be accessed at the moment. This incident was created by an automated monitoring service..

Mar 6, 09:42:13 GMT+0
Resolved - https://kagifeedback.org is now operational! This update was created by an automated monitoring service..

Duration: 2 minutes

Affected Components: https://news.kagi.com

Mar 4, 13:37:27 GMT+0
Investigating - https://news.kagi.com cannot be accessed at the moment. This incident was created by an automated monitoring service..

Mar 4, 13:39:52 GMT+0
Resolved - https://news.kagi.com is now operational! This update was created by an automated monitoring service..

Duration: 1 minute

Affected Components: https://news.kagi.com

Mar 4, 12:19:41 GMT+0
Investigating - https://news.kagi.com cannot be accessed at the moment. This incident was created by an automated monitoring service..

Mar 4, 12:21:01 GMT+0
Resolved - https://news.kagi.com is now operational! This update was created by an automated monitoring service..

Duration: 3 minutes

Affected Components: https://news.kagi.com

Mar 4, 12:15:05 GMT+0
Investigating - https://news.kagi.com cannot be accessed at the moment. This incident was created by an automated monitoring service..

Mar 4, 12:18:29 GMT+0
Resolved - https://news.kagi.com is now operational! This update was created by an automated monitoring service..

Duration: 1 hour

Affected Components: https://news.kagi.com

Mar 4, 12:11:39 GMT+0
Investigating - We noticed that the news generation has not finished in time for today, it should be available soon. This incident was created by an automated monitoring service..

Mar 4, 13:11:48 GMT+0
Resolved - https://news.kagi.com is now operational! This update was created by an automated monitoring service..

Duration: 2 minutes

Affected Components: https://news.kagi.com

Mar 4, 12:11:28 GMT+0
Investigating - https://news.kagi.com cannot be accessed at the moment. This incident was created by an automated monitoring service..

Mar 4, 12:13:47 GMT+0
Resolved - https://news.kagi.com is now operational! This update was created by an automated monitoring service..

Duration: 2 minutes

Affected Components: https://news.kagi.com

Mar 4, 12:07:31 GMT+0
Investigating - https://news.kagi.com cannot be accessed at the moment. This incident was created by an automated monitoring service..

Mar 4, 12:09:45 GMT+0
Resolved - https://news.kagi.com is now operational! This update was created by an automated monitoring service..

Duration: 6 minutes

Affected Components: https://news.kagi.com

Mar 4, 11:59:52 GMT+0
Investigating - https://news.kagi.com cannot be accessed at the moment. This incident was created by an automated monitoring service..

Mar 4, 12:06:20 GMT+0
Resolved - https://news.kagi.com is now operational! This update was created by an automated monitoring service..

Duration: 10 minutes

Affected Components: https://news.kagi.com

Mar 3, 03:21:58 GMT+0
Investigating - https://news.kagi.com cannot be accessed at the moment. This incident was created by an automated monitoring service..

Mar 3, 03:31:46 GMT+0
Resolved - https://news.kagi.com is now operational! This update was created by an automated monitoring service..

Duration: 1 minute

Affected Components: https://news.kagi.com

Mar 2, 07:01:11 GMT+0
Investigating - https://news.kagi.com cannot be accessed at the moment. This incident was created by an automated monitoring service..

Mar 2, 07:02:22 GMT+0
Resolved - https://news.kagi.com is now operational! This update was created by an automated monitoring service..

Duration: 26 minutes

Affected Components: [Assistant] asia-southeast1, [Assistant] europe-west2, [Assistant] us-west2, [Assistant] southamerica-east1, [Assistant] australia-southeast1, [Assistant] europe-west4, [Assistant] us-east4, [Assistant] us-central1, [Assistant] asia-east2, , Kagi Assistant →

Feb 23, 21:36:19 GMT+0
Investigating - We are currently investigating this incident..

Feb 23, 22:01:57 GMT+0
Resolved - Our read replicas were lagging behind the main DB, which was causing queries that depend on the replication latency to be low, to fail. The source of the replication lag has been identified and stopped..

Feb 24, 17:37:35 GMT+0
Postmortem - # Summary On February 23, 2025 at approximately 9:12 PM UTC we started receiving reports of Assistant threads disappearing, and were quickly able to replicate the problem ourselves. After a quick investigation, we found that the read replicas in multiple regions had gotten out of sync with the main DB, and were moderately loaded. This was causing issues where writes are acknowledged by the main DB, but then due to replication lag the data is not present in a Read Replica. At this point several Read Replicas had already recovered from what they were working on, and were back in sync with the main DB. Within 7 additional minutes all Read Replicas had recovered and were back in sync with the main DB. Recovery did not require any additional work to be done, and happened without intervention. # Timeline | 7:00 PM UTC | An engineer starts debugging a slow query that had been previously identified in Postgresql. | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 7:10 PM UTC | The nightly process that cleans up the table the slow query is referencing is discovered to have been failing for at least the last month, and the table has data from several years ago still stored in it. | | 7:28 PM UTC | The engineer decides to manually clean up the table to avoid a situation where the nightly cleanup process (once fixed) causes an outage due to deleting too many rows in a single statement. Deletes are run in separate statements only doing 50 rows at a time. This way the process can be aborted if it takes too many DB resources. | | 8:07 PM UTC | An index is added to a secondary table that holds a foreign key referencing the original table. This is done to speed up deletes and avoid scanning the whole secondary table on every query. | | 8:20 PM UTC | Read replicas start to fall behind by a few seconds. Replication lag slowly increases. | | 9:12 PM UTC | First ticket comes in about Assistant Threads disappearing. | | 9:17 PM UTC | Second ticket comes in about Assistant Threads disappearing. | | 9:22 PM UTC | Issue was replicated by a Kagi team member | | 9:23 PM UTC | The Kagi team member starts pulling engineers to a combined channel to investigate the issue with the Assistant threads. | | 9:24 PM UTC | The manual cleanup has finished. | | 9:34 PM UTC | The [status.kagi.com](http://status.kagi.com) page is updated. | | 9:37 PM UTC | The issue is identified as the Read Replicas being out of sync, and the source of the lag being the manual cleanup. | | 9:44 PM UTC | Read Replicas are back in sync with the Primary. | # Root Cause ## Why were Assistant threads disappearing and other queries failing? We run a replicated PostgreSQL infrastructure for low latency access to user account data across multiple regions. We have one Primary that accepts writes, and Read Replicas in each region that handle (most) reads. This allows us to scale reads by adding more replicas, and more regions, as needed. The problem that can happen with this style of scaling is that the read replicas MUST be up to date with the Primary. If a Read Replica is more than a few seconds behind, then common database access patterns start to break down as writes will go to the Primary, and not be reflected in the Read Replica in time to fulfill additional queries. ## Why were the Read Replicas lagging behind on replication? 1. They were processing a lot of deletes that had already been run on the Primary Database. These deletes require more processing than normal due to foreign key references needing to be updated. ## Why were so many deletes happening in the Primary? 1. A table was being cleaned up manually because the automated clean up had been failing for a long time. This was split up into small batches of 50 rows at a time to avoid a single delete statement causing issues. ## Why didn’t the cleanup get paused at the first sign of trouble? 1. The Read Replicas did not have any replication issues for almost an hour during the cleanup. This changed near the end of the cleanup, when all the Read Replicas started falling behind. ## Why didn’t the engineer doing the cleanup get notified? 1. There isn’t any monitoring setup specifically for replication lag, and the other alerts that we use to monitor the PostgreSQL Primary instance didn’t get near their configured alerting levels. # Resolution and Recovery As this was due to replication lag of a manual process that had already finished, the only thing that was done was to wait. The cluster recovered by itself. # Corrective and Preventative Measures The main thing that we are doing to correct this problem in the future is to add an alert for Replication Lag. If there was an existing alert for this, the engineer would have been notified and then would have spaced out the queries cleaning up the table and given the read replicas time to catch up. The second thing that we will be doing is watching both the Read Replicas and the Primary instance when doing this kind of manual work to ensure that actions taken on the Primary do not adversely affect the Read Replicas..

Duration: 18 minutes

Affected Components: https://kagifeedback.org, https://orionfeedback.org

Feb 23, 09:00:00 GMT+0
Identified - We will be deploying new features to our feedback sites (kagifeedback.org and orionfeedback.org) at this time. During maintenance, you won't be able to access them to ensure data consistency. The ETA could take up to 1 hour, but we expect it to be faster. We're sorry for the inconvenience and will keep you posted..

Feb 23, 09:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 23, 09:18:18 GMT+0
Completed - Maintenance has completed successfully..

Duration: 3 minutes

Affected Components: https://kagi.com/summarizer

Feb 18, 14:00:10 GMT+0
Investigating - https://kagi.com/summarizer cannot be accessed at the moment. This incident was created by an automated monitoring service..

Feb 18, 14:03:06 GMT+0
Resolved - https://kagi.com/summarizer is now operational! This update was created by an automated monitoring service..

Duration: 13 minutes

Affected Components: [Assistant] europe-west2

Feb 12, 18:41:53 GMT+0
Investigating - The Assistant in the europe-west2 region is taking longer to respond then normal. This incident was created by an automated monitoring service..

Feb 12, 18:55:13 GMT+0
Resolved - \\\[Assistant\\\] europe-west2 is now operational! This update was created by an automated monitoring service..

Duration: 7 minutes

Affected Components: [Assistant] europe-west2

Feb 12, 18:29:10 GMT+0
Investigating - The Assistant in the europe-west2 region is taking longer to respond then normal. This incident was created by an automated monitoring service..

Feb 12, 18:36:29 GMT+0
Resolved - \\\[Assistant\\\] europe-west2 is now operational! This update was created by an automated monitoring service..

Duration: 11 minutes

Affected Components: [Assistant] europe-west2

Feb 12, 18:12:30 GMT+0
Investigating - The Assistant in the europe-west2 region is taking longer to respond then normal. This incident was created by an automated monitoring service..

Feb 12, 18:23:51 GMT+0
Resolved - \\\[Assistant\\\] europe-west2 is now operational! This update was created by an automated monitoring service..

Duration: 6 minutes

Affected Components: [Assistant] europe-west2

Feb 12, 17:52:21 GMT+0
Investigating - The Assistant in the europe-west2 region is taking longer to respond then normal. This incident was created by an automated monitoring service..

Feb 12, 17:58:25 GMT+0
Resolved - \\\[Assistant\\\] europe-west2 is now operational! This update was created by an automated monitoring service..

Duration: 1 hour and 8 minutes

Affected Components: [Assistant] us-west2

Feb 12, 17:48:41 GMT+0
Investigating - The Assistant in the us-west2 region is taking longer to respond then normal. This incident was created by an automated monitoring service..

Feb 12, 18:56:11 GMT+0
Resolved - \\\[Assistant\\\] us-west2 is now operational! This update was created by an automated monitoring service..

Duration: 1 hour and 7 minutes

Affected Components: [Assistant] us-east4

Feb 12, 17:48:39 GMT+0
Investigating - The Assistant in the us-east4 region is taking longer to respond then normal. This incident was created by an automated monitoring service..

Feb 12, 18:55:59 GMT+0
Resolved - \\\\\\\[Assistant\\\\\\\] us-east4 is now operational! This update was created by an automated monitoring service..

Duration: 9 minutes

Affected Components: [Assistant] europe-west2

Feb 12, 17:26:59 GMT+0
Investigating - The Assistant in the europe-west2 region is taking longer to respond then normal. This incident was created by an automated monitoring service..

Feb 12, 17:35:46 GMT+0
Resolved - \\\[Assistant\\\] europe-west2 is now operational! This update was created by an automated monitoring service..

Duration: 5 hours and 39 minutes

Affected Components: [Assistant] asia-southeast1, [Assistant] europe-west2, [Assistant] us-west2, [Assistant] southamerica-east1, [Assistant] australia-southeast1, [Assistant] europe-west4, [Assistant] us-east4, [Assistant] us-central1, [Assistant] asia-east2, , Kagi Assistant →

Feb 13, 19:01:41 GMT+0
Postmortem - Assistant Slowdowns - February 11-12, 2025 # Summary This issue was only affecting our account used to monitor the response times of the Assistant prompt endpoint. Due to it not affecting any users at all, this will be a very short explanation of the last two days. Our automated alerting service started alerting us that the Assistant prompt endpoint was taking longer than it should and timing out. After a lot of different investigations, it was discovered that a preview feature that had been rolled out to Kagi team members had been applied to the user account we use to monitor [kagi.com](http://kagi.com). After disabling the feature, response time recovered. As this only affected the monitoring account, no users were affected at all. # Timeline | November | The feature was turned on for Kagi team members. | | ----------------------- | -------------------------------------------------------------------------------------------- | | Nov - Feb | The prompt for the monitoring account grew slowly | | February 11 1:48 PM UTC | First alert fired | | February 12 6:55 PM UTC | The feature was disabled for the monitoring account, and response times instantly recovered. | # Root Cause A feature being tested by team members at Kagi was inadvertently enabled for our monitoring account. It was rolled out to Kagi team members to get feedback and to understand how it would work with real people and their interactions. The monitoring account sends 8 requests/minute to different servers around the world. Due to the high number of requests from the monitoring account and the feature not being ready for production as it was still being tested, the prompt slowly grew larger without anyone knowing. This ended up with the prompt being large enough that it would cause our LLM providers to take too long to parse the prompt being sent, and resulted with us ultimately timing out the request, and reporting an error back to the monitoring service. This was difficult to debug as it manifested itself as the provider slowing down. # Resolution and Recovery During the outage we had paused some of our monitoring probes due to the fact that they are very noisy, and can interrupt us while we are investigating the issues. We noticed a pattern that when they were paused, errors from specific providers would drop to lower rates, but when they were resumed, errors would increase to previous levels. Once we realized that this was an issue with just the account doing the monitoring we were able to quickly determine that the feature was enabled on the monitoring account. Once we had disabled it, error rates dropped to normal levels and the monitoring account requests ran in the normal amount of time. # Corrective and Preventative Measures We are making some changes to ensure that the monitoring account does not have features enabled on it that are not what our customers would see. We will also be adding in additional metrics so that we can monitor the monitoring account more thoroughly than we currently do..

Feb 12, 14:41:54 GMT+0
Investigating - The Assistant has been responding slower then normal. We are looking into it..

Feb 12, 19:07:31 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Feb 12, 20:21:20 GMT+0
Resolved - This incident has been resolved..

Duration: 3 hours and 11 minutes

Affected Components: [Assistant] europe-west2

Feb 12, 14:09:06 GMT+0
Investigating - The Assistant in the europe-west2 region is taking longer to respond then normal. This incident was created by an automated monitoring service..

Feb 12, 17:20:15 GMT+0
Resolved - \\\[Assistant\\\] europe-west2 is now operational! This update was created by an automated monitoring service..

Duration: 9 minutes

Affected Components: [Assistant] europe-west2

Feb 12, 13:44:44 GMT+0
Investigating - The Assistant in the europe-west2 region is taking longer to respond then normal. This incident was created by an automated monitoring service..

Feb 12, 13:53:28 GMT+0
Resolved - \\\[Assistant\\\] europe-west2 is now operational! This update was created by an automated monitoring service..

Duration: 19 minutes

Affected Components: https://kagifeedback.org, https://orionfeedback.org

Jan 22, 08:00:00 GMT+0
Identified - We will be deploying new features to our feedback sites (kagifeedback.org and orionfeedback.org) at this time. During maintenance, you won't be able to access them to ensure data consistency. The ETA could take up to 1 hour, but we expect it to be faster. We're sorry for the inconvenience and will keep you posted..

Jan 22, 08:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 22, 08:18:56 GMT+0
Completed - Maintenance has completed successfully..

Duration: 27 minutes

Affected Components: [Assistant] europe-west4, , [Assistant] asia-southeast1, [Assistant] europe-west2, [Assistant] us-west2, [Assistant] southamerica-east1, [Search] asia-east2, [Search] europe-west4, [Search] us-west2, [Search] asia-southeast1, [Assistant] australia-southeast1, [Search] us-east4, [Search] australia-southeast1, [Assistant] us-east4, Search API, [Assistant] us-central1, Summarizer API, [Search] us-central1, [Assistant] asia-east2, , [Search] southamerica-east1, [Search] europe-west2, Kagi Search → Kagi Assistant →

Dec 7, 06:00:00 GMT+0
Identified - We plan to perform a minor upgrade to the [**Kagi.com**](http://Kagi.com) database backend on December 7, 2025\. During the maintenance window, write operations such as updating your profile or changing settings may be temporarily affected. However, we do not expect any interruptions to [**Search**](https://kagi.com/search) or [**Assistant**](https://kagi.com/assistant)..

Dec 7, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 7, 06:27:29 GMT+0
Completed - Maintenance has completed successfully..

Duration: 21 minutes

Affected Components: https://kagifeedback.org, https://orionfeedback.org

Nov 27, 08:00:00 GMT+0
Identified - We will be deploying new features to our feedback sites (kagifeedback.org and orionfeedback.org) at this time. During maintenance, you won't be able to access them to ensure data consistency. The ETA could take up to 1 hour, but we expect it to be faster. We're sorry for the inconvenience and will keep you posted..

Nov 27, 08:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 27, 08:20:55 GMT+0
Completed - Maintenance has completed successfully..

Duration: 1 hour

Affected Components: https://orionfeedback.org

Nov 21, 13:45:00 GMT+0
Completed - We are planning for a scheduled maintenance during that time..

Duration: 35 minutes

Affected Components: , [Assistant] europe-west4, [Assistant] asia-southeast1, [Assistant] europe-west2, [Assistant] us-west2, [Assistant] southamerica-east1, [Search] asia-east2, [Search] europe-west4, [Search] us-west2, [Search] asia-southeast1, [Assistant] australia-southeast1, [Search] us-east4, [Search] australia-southeast1, [Assistant] us-east4, [Assistant] us-central1, [Search] us-central1, [Assistant] asia-east2, , [Search] southamerica-east1, [Search] europe-west2, Kagi Search → Kagi Assistant →

Oct 12, 06:00:00 GMT+0
Identified - We plan to perform a minor upgrade to the [Kagi.com](http://Kagi.com) database backend on October 12, 2025\. During the maintenance window, write operations such as updating your profile or changing settings may be temporarily affected. However, we do not expect any interruptions to [Search](https://kagi.com/search) or [Assistant](https://kagi.com/assistant)..

Oct 12, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 12, 06:34:31 GMT+0
Completed - Maintenance has completed successfully..

Duration: 1 hour and 1 minute

Affected Components: https://kagifeedback.org, https://orionfeedback.org

Sep 24, 08:00:00 GMT+0
Identified - We're planning to release new features for [**Kagifeedback.org**](http://Kagifeedback.org) and will be performing scheduled maintenance. During this maintenance window, <https://kagifeedback.org> and <https://orionfeedback.org> will be temporarily unavailable, with expected downtime of up to 1 hour..

Sep 24, 09:01:07 GMT+0
Completed - Maintenance has completed successfully..

Sep 24, 08:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 1 hour

Affected Components: https://kagifeedback.org, https://orionfeedback.org

Sep 10, 09:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 10, 08:00:00 GMT+0
Identified - We're planning to release new features for [Kagifeedback.org](http://Kagifeedback.org) and will be performing scheduled maintenance. During this maintenance window, <https://kagifeedback.org> and <https://orionfeedback.org> will be temporarily unavailable, with expected downtime of up to 1 hour..

Sep 10, 08:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 1 hour

Affected Components: https://kagi.com/summarizer, [Assistant] asia-east2, [Search] asia-east2, [Search] asia-southeast1, [Search] us-east4, [Assistant] australia-southeast1, FastGPT, [Assistant] europe-west4, [Search] us-west2, [Search] australia-southeast1, [Search] southamerica-east1, [Assistant] us-east4, Summarizer API, https://kagi.com/smallweb, Smallweb, FastGPT API, [Search] us-central1, [Search] europe-west4, [Assistant] southamerica-east1, [Search] europe-west2, [Assistant] asia-southeast1, [Assistant] us-central1, https://kagi.com/fastgpt, Search API, [Assistant] us-west2, [Assistant] europe-west2

Jun 30, 07:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jun 30, 06:00:00 GMT+0
Identified - Continuing the upgrade that started last week. A few additional instances need to be addressed. A whole hour is scheduled, but downtime should be limited to under 10 minutes..

Jun 30, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 10 minutes

Affected Components: https://kagifeedback.org, https://orionfeedback.org

Jun 29, 07:09:46 GMT+0
Completed - Maintenance has completed successfully..

Jun 29, 07:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 29, 07:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Duration: 1 hour

Affected Components: , , [Search] asia-east2, https://kagi.com/summarizer, [Search] asia-southeast1, [Search] us-east4, [Assistant] asia-east2, [Assistant] australia-southeast1, FastGPT, [Assistant] europe-west4, [Search] us-west2, , [Search] australia-southeast1, [Search] southamerica-east1, [Assistant] us-east4, Summarizer API, https://kagi.com/smallweb, , Smallweb, FastGPT API, [Search] us-central1, [Search] europe-west4, [Assistant] southamerica-east1, [Search] europe-west2, [Assistant] asia-southeast1, [Assistant] us-central1, https://kagi.com/fastgpt, Search API, [Assistant] us-west2, [Assistant] europe-west2, Other Services → Kagi Assistant → Kagi Search → API →

Jun 22, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 22, 07:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jun 22, 06:00:00 GMT+0
Identified - Our production database is running an older version that will not have good support in the next few months. We are upgrading it to a newer version to be able to continue to provide excellent performance and reliability to you our customers. This version has been tested locally and in staging environments for around 2 months without any issues. We expect no issues with using it in production. There will be minimal downtime during this upgrade (estimated to be less then 10 minutes), and access will be restored automatically when it is finished..

Duration: 25 minutes

Affected Components: [Search] us-east4, [Search] asia-east2, [Search] europe-west4, [Search] us-west2, [Search] australia-southeast1, [Search] southamerica-east1, [Search] europe-west2, [Search] asia-southeast1, [Search] us-central1

Jun 1, 06:00:00 GMT+0
Identified - We will maintain all our database servers to upgrade the network architecture, improving connectivity, reliability, and security. During the maintenance window, there may be minor disruptions, but there will not be significant downtime. We apologize for this inconvenience. .

Jun 1, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 1, 06:25:11 GMT+0
Completed - Maintenance has completed successfully..

Duration: 38 minutes

Affected Components: [Search] us-east4, [Search] us-west2, [Search] europe-west2, [Search] asia-east2, [Search] us-central1, [Search] southamerica-east1, [Search] europe-west4, [Search] asia-southeast1, [Search] australia-southeast1, Search API, Summarizer API, FastGPT API

Mar 23, 06:38:14 GMT+0
Completed - All maintenance has been completed, and all services are now operational. .

Mar 23, 06:18:23 GMT+0
Identified - The maintenance takes more time than expected. We're working on it and will update soon. .

Mar 23, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 23, 06:15:00 GMT+0
Completed - Maintenance has completed successfully.

Mar 23, 06:00:00 GMT+0
Identified - We are planning to maintain the database in all regions during that time. During the maintenance, all supported regions are expected to have less than 1 second of disruption, except for region asia-east2, which is expected to have up to 60 seconds of disruption. .

Duration: 30 minutes

Affected Components: [Search] us-east4, [Search] us-west2, [Search] europe-west2, [Search] asia-east2, [Search] us-central1, [Search] southamerica-east1, [Search] europe-west4, [Search] asia-southeast1, [Search] australia-southeast1

Oct 27, 06:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 27, 06:00:00 GMT+0
Identified - Kagi will be temporaily unavailable in all regions while we migrate the infrastructure for our primary database instance. This is in compliance with required maintenance from our hosting provider (GCP), but we are taking the opportunity to increase the resources allocated to the database for improved performance for our customers. We will update here as the maintenance progresses - thank you for your patience..

Oct 27, 06:30:00 GMT+0
Completed - Maintenance has completed successfully.