Upstream AWS incident in US-East-1 in impacting ClickHouse Cloud services

Incident Report for ClickHouse Cloud

Resolved

ClickHouse has completed its testing and the issue looks resolved - no further issues or errors are observed for the US-East-1 region

Posted Jun 13, 2023 - 21:36 UTC

Monitoring

AWS is reporting the fix has been implemented and they are seeing reduced errors. AWS is monitoring recovery. ClickHouse is testing and monitoring recovery.

Posted Jun 13, 2023 - 20:53 UTC

Identified

The issues in US-East-1 have been identified as related to the AWS upstream degradation at AWS. AWS is reporting that they have identified the root cause as an issue with AWS Lambda and are working on resolution https://health.aws.amazon.com/health/status

Posted Jun 13, 2023 - 20:10 UTC

Investigating

ClickHouse is currently investigating an issue with service management in AWS region US-East-1. We believe this is due to an issue with upstream provider AWS as reported here:
https://health.aws.amazon.com/health/status
https://status.aws.amazon.com/rss/multipleservices-us-east-1.rss

Operational issue - Multiple services (N. Virginia)
Affected AWS services
The following AWS services have been affected by this issue. Degradation (3 services)
* AWS CloudFormation
* AWS Lambda
* Amazon Connect

ClickHouse Services impacted in US-East-1 include Control Plane functions such as the Cloud console, service creation or deletion, user management, API, usage metrics and billing.

Posted Jun 13, 2023 - 19:42 UTC

This incident affected: Amazon Web Services (AWS) (us-east-1) and ClickHouse Cloud Console (SQLConsole).