ClickHouse Cloud permissions errors
Incident Report for ClickHouse Cloud
Resolved
After investigation and identification, this incident has been resolved. As noted previously, this new permissions checking in ClickHouse 23.9 is a better and more proper method of ensuring grants on these uses of asynchronous inserts and dictionaries. In order to properly apply these grants, please refer to the following ClickHouse Knowledge Base (KB) article:
https://clickhouse.com/docs/knowledgebase/ingest-failures-23-9-release

We apologize for any interruption from this change in behavior, as it should have been better communicated prior to the release. If you require any additional assistance with updating your user grants in ClickHouse Cloud, please reach out to ClickHouse Support via a new case or chat:
https://clickhouse.cloud/support
Posted Nov 17, 2023 - 20:06 UTC
Monitoring
As noted, the workaround for this new behavior in ClickHouse 23.9 is described in more detail in the following KB article:
https://clickhouse.com/docs/knowledgebase/ingest-failures-23-9-release

If you require any additional assistance in updating your user grants in ClickHouse Cloud, please reach out to ClickHouse Support via a new case or chat:
https://clickhouse.cloud/support
Posted Nov 17, 2023 - 19:57 UTC
Identified
The root cause of these permissions errors has been identified and is due to more strict checking of grants for asynchronous inserts and dictionaries in the 23.9 release. If you are using a non-default user with limited permissions to perform asynchronous inserts or dictionary creation, you must apply additional grants to this user.

The workaround for this issue is described in more detail in the following KB article: https://clickhouse.com/docs/knowledgebase/ingest-failures-23-9-release
Posted Nov 17, 2023 - 19:53 UTC
Investigating
ClickHouse has been alerted to user-level database 4xx permissions errors in ClickHouse Cloud and is actively investigating the issue
Posted Nov 17, 2023 - 18:18 UTC
This incident affected: Amazon Web Services (AWS) (us-east-1, us-east-2, us-west-2, eu-central-1, eu-west-1, ap-south-1, ap-southeast-1, ap-southeast-2) and Google Cloud Provider (GCP) (us-central1, europe-west4, asia-southeast1).