ClickHouse as a main operational databaseFirst of all, this post is not a recommendation but more like a “what if” story. What if we use ClickHouse (which is a columnar analytical database) as our main datastore? Well, typically, an analytical database is not a replacement for a transactional or key/value datastore. However, ClickHouse is super efficient for timeseries and provides “sharding” out of the box (scalability beyond one node).  So can we use it as our main datastore?

Let’s imagine we are running a webservice and provide a public API. Public API as -a-service has become a good business model: examples include social networks like Facebook/Twitter, messaging as a service like Twilio, and even credit card authorization platforms like Marqeta. Let’s also imagine we need to store all messages (SMS messages, email messages, etc) we are sending and allow our customers to get various information about the message. This information can be a mix of analytical (OLAP) queries (i.e. how many messages was send for some time period and how much it cost) and a typical key/value queries like: “return 1 message by the message id”.

Using a columnar analytical database can be a big challenge here. Although such databases can be very efficient with counts and averages, some queries will be slow or simply non existent. Analytical databases are optimized for a low number of slow queries. The most important limitations of the analytical databases are:

  1. Deletes and updates are non-existent or slow
  2. Inserts are efficient for bulk inserts only
  3. No secondary indexes means that point selects (select by ID) tend to be very slow

This is all true for ClickHouse, however, we may be able to live with it for our task.

To simulate text messages I have used ~3 billion of reddit comments (10 years from 2007 to 2017), downloaded from pushshift.io . Vadim published a blog post about analyzing reddit comments with ClickHouse. In my case, I’m using this data as a simulation of text messages, and will show how we can use ClickHouse as a backend for an API.

Loading the JSON data to Clickhouse

I used the following table in Clickhouse to load all data:

Then I used the following command to load the JSON data (downloaded from pushshift.io) to ClickHouse:

The data on disk in ClickHouse is not significantly larger than compressed files, which is great:

We have ~4 billion rows:

The data is partitioned and sorted by created_utc so queries which include created_utc will be able to using partition pruning: therefore skip the not-needed partitions. However, let’s say our API needs to support the following features, which are not common for analytical databases:

  1. Selecting a single comment/message by ID
  2. Retrieving the last 10 or 100 of the messages/comments
  3. Updating a single message in the past (e.g. in the case of messages, we may need to update the final price; in the case of comments, we may need to upvote or downvote a comment)
  4. Deleting messages
  5. Text search

With the latest ClickHouse version, all of these features are available, but some of them may not perform fast enough.

Retrieving a single row in ClickHouse

Again, this is not a typical operation in any analytical database, those databases are simply not optimized for it. ClickHouse does not have secondary indexes, and we are using created_utc as a primary key (sort by). So, selecting a message by just ID will require a full table scan:

Only if we know the timestamp (created_utc)… Then it will be lighting fast: ClickHouse will use the primary key:

Actually, we can simulate an additional index set by creating a materialized view in ClickHouse:

Here I’m creating a materialized view and populating it initially from the main (rc) table. The view will be updated automatically when there are any inserts into table reddit.rc. The view is actually another MergeTree table sorted by id. Now we can use this query:

This is a single query which will join our materialized view to pass the created_utc (timestamp) to the original table. It is a little bit slower but still less than 100ms response time.

Using this trick (materialized views) we can potentially simulate other indexes.

Retrieving the last 10 messages

This is where ClickHouse is not very efficient. Let’s say we want to retrieve the last 10 comments:

In a conventional relational database (like MySQL) this can be done by reading a btree index sequentially from the end, as the index is sorted (like “tail” command on linux). In a partitioned massively parallel database system, the storage format and sorting algorithm may not be optimized for that operation as we are reading multiple partitions in parallel. Currently, an issue has been opened to make the “tailing” based on the primary key much faster: slow order by primary key with small limit on big data. As a temporary workaround we can do something like this:

It is still a five seconds query. Hopefully, this type of query will become faster in ClickHouse.

Updating / deleting data in ClickHouse

The latest ClickHouse version allows running update/delete in the form of “ALTER TABLE .. UPDATE / DELETE” (it is called mutations in ClickHouse terms). For example, we may want to upvote a specific comment.

“Mutation” queries will return immediately and will be executed asynchronously. We can see the progress by reading from the system.mutations table:

Now we can try deleting comments that have been marked for deletion (body showing “[deleted]”):

After a while, we can do the count again:

As we can see our “mutation” is done.

Text analysis

ClickHouse does not offer full text search, however we can use some text functions. In my previous blog post about ClickHouse I used it to find the most popular wikipedia page of the month. This time I’m trying to find the news keywords of the year using all reddit comments: basically I’m calculating the most frequently used new words for the specific year (algorithm based on an article about finding trending topics using Google Books n-grams data). To do that I’m using the ClickHouse function alphaTokens(body) which will split the “body” field into words. From there, I can count the words or use arrayJoin to create a list (similar to MySQL’s group_concat function). Here is the example:

First I created a table word_by_year_news:

This will store all frequent words (I’m filtering by subreddits; the examples are: “news, politics and worldnews” or “programming”) as well as its occurrence this year; actually I want to store “relative” occurrence which is called “ratio” above: for each word I divide its occurrence by the number of total words this year (this is needed as the number of comments grows significantly year by year).

Now we can actually calculate the words of the year:

And the results are (here I’m grouping words for each year):

For “programming” subreddit:

For news subreddit:

Conclusion

ClickHouse is a great massively parallel analytical system. It is extremely efficient and can potentially (with some hacks) be used as a main backend database powering a public API gateway serving both realtime and analytical queries. At the same time, it was not originally designed that way. Let me know in the comments if you are using ClickHouse for this or similar projects.


Photo by John Baker on Unsplash

 

 

10 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Denis Golius

Alexander Rubin showed the worst things… So maybe you are not realy good in CH?

Kevin English

GROUP BY is not OLAP. Please stop associating single-index column stores with OLAP. They are miles apart. If you can not analyze and pivot in N dimensions, it is not OLAP.

David

Additional index simulation is just what I was looking for. Thanks!

sourabh S

we run adservers which process almost a several hunderd million to close to a billion requests per day.
we are using clickhouse for our analytical storage engine for over 1.5 yeras now.

few months ago when updated/deletes came out for clickhouse we tried to do exactly what is mentioned above .i.e convert everything to clickhouse from mysql , including user,product table etc.

we used clickhouse as our primary storage (replicated engines with kafka) in the development mode everything was running smoothly even the updates and deletes , so we were happy and pushed the code to production.. however this turned out to be a major blunder and almost ruined our production server.

What happened was there were to many updates queries piling up and choking up the server ,clickhouse was not able to process them in time. as a result , updates were taking more then 30 minutes .

we knew from he start that updates would be slow but we dint expected this. in short clickhouse must not be used for the table where updates are happening frequently.

Merge engine is very unreliable about when would it merge the duplicate entries. usage of ‘final’ keyword query is very bad ideas once you table starts growing.also not to mention frequent data corruption/loss due to replication errors.

Adnan

Did you try ReplacingMergeTree or CollapsingMergeTree for update/delete ? Is your system working good now with update/delete ? which engine are you using now then ?

sourabh S

Regarding Update and Deletes : any table that requires constant updates/delete we migrated it back to MYSQL Galera Cluster, If we need to update any row in clickhouse we just replace the row with another row keeping the Primary key/touple the same ,for this reason we are mostly using Replacing MergeTree. we tried to use collapsingTree earlier but we changed it to ReplicatingMergeTree later , i dont really remember the reason why we favored ReplicatingMergeTree instead of collapisngTree right now.
We completely quit the idea of using Alter Update and Alter Delee queries as they get extremely slow as the table size increases ,it doesnt matter if its replacaing or collapsing tree ,the permormace is unpredicatable about when will the changes actually happen and it gets worse as the table size grows.

Adnan

Exactly bro! i was thinking same, thats why i asked you why didn’t you use ReplacingMergeTree .. So now its all stable for you ? But on merge new record, does it do it instantly ? or you have to wait ? Me too working on some telecom company stuff, so what i thought is to keep mysql and clickhouse on same application .. what i am trying is to select record from clickhouse, then update it and insert it back in the clickhouse table ReplacingMergeTree, i dont know if it will success but i am trying for it .. Share your opinion too it will be appreciated 😉

Adnan

BTW bro can you tell me why MergeTree is unreliable ? and did you try ReplacingMergeTree ?

sourabh S

I guess i failed to explain my point. MergeTree is not unreliable,its the best and the fastest engine which we should use. The only problem is the updates and deletes queries, which is not only the problem of MergreTree but the whole clickhouse. ,it doesnt matter if its MergeTree, ReplacingMergeTree or collapsingtree ,the performance is unpredictable about when will the changes actually happen and it gets worse as the table size grows.

Adnan

Well if we plan better before diving into CH database, i think we can avoid many issues, CH is very good for analytics, and also not bad for update/delete, we just need to follow their docs and limits. So we can achieve our goal with less hassles .. well your article is nice and i like it though 😉