Incident History
History
September 15 9:18 AM UTCCreating new dbs and schema migrations fail in N. Virginia region
The root cause was identified as a bug that surfaced during routine maintenance operations. The issue has been resolved permanently with a code deployment.
The issue has been mitigated. Creation of new databases and schema migrations are restored in the affected region N. Virginia (us-east-1). We continue monitoring and working on root cause analysis.
We are currently investigating an issue which disrupted the creation of new databases and schema migrations in N. Virginia region (us-east-1)
Search store in degraded state
Replication to the Search store has been restored. The issue was caused by an error in the deployment pipeline. We are taking all relevant measures to ensure such occurrences will be prevented in the future.
We have identified the root cause and we are rolling out a fix. The Search store issue will begin to recover in all regions soon.
Search and aggregation functionality is degraded because replication from the primary store is currently stopped.
Error while loading app.xata.io
The issue has been resolved, all Xata services are now fully operational.
The Xata Web UI at app.xata.io is available again. We are monitoring closely the recovery and stability of all service components in Xata.
We are investigating impact of AWS service degradation in us-east-1 on aspects of the Xata platform (Web UI, API) and taking all possible actions to minimize impact.
We are investigating errors "500: INTERNAL_SERVER_ERROR" while loading the Xata Web UI.
API Response serialization issues
The rollback completed in all regions and we have verified that the API response serialization issue has been resolved.
The rollback that we anticipate to resolve the issue is ongoing and we are monitoring progress across all regions.
We have identified the root cause of the serialization issues and we are currently working on reverting the change that introduced it.
We are currently investigating reports of incorrect serialization in the API response content which may affect various API operations.
Error when creating new databases from the Web UI
The issue has been resolved. Xata users are now able to create new databases from the Web UI, in all regions.
We are deploying a fix that we are confident will mitigate the issue.
The problem has been attributed to a new deployment.
We have identified that users are not able to create new databases in any region from the Xata Web UI.
API errors in the presence of ascii control characters
We have confirmed that the fix which was deployed resolved the issue successfully.
We have identified the root cause of the issue, deployed a fix and monitoring the results closely
We have identified an issue causing HTTP 400 error responses to be returned by the Xata API, when records with text columns containing ascii control characters are inserted or updated.