Case Study: A Keybase Live Chat Feed for Chia Network

TL;DR: In 2 weeks, we built an open-source live feed of Chia Network’s Keybase community. Check it out here!

Overview

Chia Network is a “decentralized open source global blockchain which is less wasteful, more decentralized, and more secure”. It was founded by Bram Cohen, the creator of BitTorrent.

Building a strong developer community is key to the success of a cryptocurrency like Chia. Their community was on Keybase Teams, an end-to-end encrypted instant-messaging application similar to Slack.

But without a Keybase account, visitors to their site could not view the latest updates about Chia. We worked with Chia to swiftly develop and launch a live chat feed for their website.

The Challenge

Chia wanted visitors to keep coming back to their website, and it was important that any visitor could stay up-to-date on news about Chia.

However, visitors without a Keybase account would need to:

1. Sign up for Keybase.
2. Download & install the application.
3. Join the Keybase group using its chat interface.

Those visitors also had no visibility into how active Chia’s community was, nor its size. This resulted in friction joining the community and getting the latest news.

Results

Reception from Chia’s community was overwhelmingly positive! Visitors could now see a real-time preview of the discussion right in their browser, and without requiring a Keybase account.

It’s the first of its kind and has been open-sourced here. We hope that it enables other developers to easily integrate a Keybase feed with their own website.

The Project

To reduce this friction, we built a custom integration that previews Chia’s #general Keybase channel. It’s robust, mobile-friendly, and the message feed updates in real-time (including emoji reacts!).

Design

• Frontend: React, React-Markdown, Bootstrap, Socket.IO
• Backend: Node, Express, Socket.IO, and Keybase.

Architecture diagram of the Keybase Live Feed.

Frontend

We wanted the code to be simple to use, update, and maintain. We went with a time-tested, minimal React stack — only 8 dependencies in total!

The chat UI is implemented with React for view layer, and Bootstrap to help with layout.

Chat messages stream in via Socket.IO, and are immediately shown on screen — either appended to the messages list, or rewriting existing messages. There’s a separate message type for metadata, such as the number of people in the conversation.

Because Keybase Teams supports a small subset of Markdown in chat messages, we implemented this using the React-Markdown library. It’s important to use escapeHtml: false here to prevent XSS vulnerabilities.

Emoji rendering in text messages was a bit of a challenge. To match the Keybase behaviour as much as possible, we wrote our own spritesheet display component. This uses regular expressions to find emoji.

Some of the interesting behaviours it handles are:

1. The syntax for an emoticon is :some_identifier: — emoticons like :) and :O are left alone.
2. Emoji identifiers are based on names from the emoji-datasource NPM package.
3. Modifiers are also supported — you can write :smile::skin-tone-2: and see the emoji with a different skin tone.

Backend

As of this writing, Keybase doesn’t offer a complete API for their Teams product. It does have endpoints for basic functionality such as identity management.

However, the keybase command-line tool offers a semi-documented way to directly interact with their Teams product, via various subcommands such as keybase chat api and keybase chat api-listen.

We looked into duplicating what the command-line tool does, but in the end, the command-line tool’s API surface looked more stable than their undocumented HTTP/Websockets API surface. To use it, we decided to simply run the tool in the background and capture its standard output.

There were only a few things we needed from the API:

1. List of all conversations/channels in the team.
2. List of all messages in a conversation/channel.
3. List of all members of a conversation/channel.
4. Profile picture of each member of each conversation/channel (for the avatar images).

After automating the command-line Keybase tool, we had a reliable, consistent way of obtaining all of these. We then abstracted it into the keybaseChatApi library.

On top of this, we built keybaseChatHistory, a scrollback-management library that keeps track of created/edited/deleted messages and reactions, as well as providing an up-to-date view of the current chat history. This allows us to easily retrieve messages with their related content, such as reactions and edit markers, and synchronize that state across all clients.

The Live Feed itself now becomes a simple Express/Socket.IO server. There are three listeners running in parallel in each Node process:

1. Chat Listener: streams in messages arriving from Keybase via keybaseChatApi, reifies them with the in-memory chat history using keybaseChatHistory, and broadcasts the changes to all connected Socket.IO clients. This broadcasted data includes user lists and profile pictures from the Metadata listener.
2. Metadata Listener: periodically retrieves a lists all members of the team, as well as their profile pictures, maintaining an in-memory team members cache and profile-pictures cache used by the Chat Listener. This only runs a couple times a minute since retrieving all this data is quite expensive.
3. Socket.IO Listener: listens to incoming connection requests and hands them off to the Socket.IO handler.

Finally, the entire application was written to be stateless — all data is stored using in-memory data structures and can be restarted at any time without side effects. Upon start, the chat history and the various caches are rebuilt automatically, and the application is immediately ready to serve users.

Deployment

The live feed was deployed to AWS EC2 instances with Debian images. We set them up with the usual best practices, such as:

1. Running under a dedicated user with restricted permissions.
2. Running as a SystemD service (including auto-restart on crash).
3. Nginx reverse-proxying with Let’s Encrypt for HTTPS (including certificate auto-renewal).
4. Runbook with step-by-step instructions for common tasks, such as applying updates and viewing logs.

For larger use cases, Keybase Live Feed also supports running within Google Kubernetes Engine, StackDriver for logging, and its own K8S-specific runbook.

Hypotenuse Labs is an elite team of software consultants. Hailing from Facebook, Amazon, Uber, and Snap, we specialize in delivering web and AI software products for startups and SMBs.

If custom integration struggles are keeping you up at night, contact us at hello@hypotenuselabs.com.