TL;DR : Code is on Github. In this post I explain how you can build a chat room using Django Channels.

A standard Django application handles http requests using a request-response lifecycle. A request is sent from the user’s browser, Django calls the relevant view which then returns a response to the user. The request-response lifecycle has certain limitations though : it’s not great for realtime applications which usually require communicating with the backend server frequently. New standards such as websockets and HTTP2 address some of these shortcomings. WebSockets is a recent communications protocol which provides full-duplex communication channels over a single TCP connection and is well suited for realtime applications. Opening and maintaining a websocket connection with a server is very cheap in terms of memory and cpu resources required. To give you some real world numbers, Chris McCord was able to hold 2 million open websocket connections on a single server with 40 cores and 128 GB of RAM. Though he used the Phoenix framework as the backend of choice instead of Django channels, the important takeaway is that websockets are extremely lightweight full-duplex communication channels.

Channels extends Django and allows us to handle websocket connections in a way that’s very similar to normal views. So, what is a channel? A channel is an ordered, first-in first-out queue with message expiry and at-most-once delivery to only one listener at a time. In simpler terms, a channel is a task queue which accepts messages from producers and delivers them to consumers.

So how does Channels extend Django? It seperates Django into two processes :

  • an interface server which handles the incoming HTTP and Websocket connection, eg- Daphne

  • worker process that runs the views to process the websocket and http requests.

They communicate over a protocol called ASGI which is routed over Redis. What’s interesting is that since the interface server and worker process are decoupled in Channels, it’s possible to add and remove worker processes without closing websocket connections.

The following representation gives you an idea of how Django serves requests the traditional way vs using Channels:

Django handling only HTTP

Browser <——–> Web server <——–> Django View function

Django with Channels

Browser <——–> Interface server <——–> Channel layer <——–> Django View function + Websocket Consumer

Web server - handles http connections.

Interface server - handles http and websocket connections.

Channel layer - transports http and websocket messages and delivers them to worker processes that run normal Django views as well as Channels specific consumer code.

Note that this is a simplified representation. In production deployments it’s likely to be different. Eg- Daphne is usually not exposed to the outside world directly and instead sits behind an ngnix server. As another example, it’s possible to run a WSGI server alongside websocket specific workers to serve normal Django http requests.

I will be explaining the code that is specific to running a chat room. I won’t be covering topics as setting up a Django project from scratch, setting up an auth system etc.

Channels installation instructions:

  1. pip install channels

  2. Add channels to INSTALLED_APPS setting

  3. Install redis using this guide which is for Ubuntu 16.04

  4. Use redis as the channel by following this guide

The chat room will be minimal :

  • a single chat room where users can…chat

  • a column to show the logged in users that are currently online as well as the number of onlookers i.e people who are watching the chat but are not logged in

An often overlooked aspect of chat room demos is that the count of how many users are online is incorrect. To give you an example, check HN Chat which shows that 68 users are online (as of this writing) even though that’s not true. This happens because there are a varienty of events that cause clients to disconnect websocket connections without notifying the server of the same. The server has no easy way of updating the number of users connected to the chat room without a periodic pruning process. A Django package that helps out in this situation is django-channels-presence. It allows us to prune stale websocket connections and keep an accurate count of the number of users currently connected to the chat room. How does this pruning work? In order to keep track of which websockets are actually still connected, we must regularly update the last_seen timestamp for all present connections, and periodically remove connections if they haven’t been seen in a while.

So let’s jump in to the code. We are going to have a single hardcoded room called all so there’s no need to create a separate Room model. So how do we represent the actual chat message that users send? Take a look at the following model :

class ChatMessage(models.Model):
    """
    Model to represent a chat message
    """

    #Fields
    user = models.ForeignKey(User, on_delete=models.CASCADE)
    message = models.TextField(max_length=3000)
    message_html = models.TextField()
    created = models.DateTimeField(auto_now_add=True)
    updated = models.DateTimeField(auto_now=True)

    def __str__(self):
        """
        String to represent the message
        """

        return self.message

The fields are straightforward:

  • user represents the user object

  • message is the raw text that the user types in to the chat box

  • message_html is the html rendered version of message. The html version will be escaped and only link tags will be allowed in the chat room. All other tags such as <script></script> won’t work. The main reason to prerender the html version of each chat message is to save on processing resources by avoiding rendering the same html everytime a user requests it.

Let’s move on to designing the homepage view. This view is the chat room, so we need to retrieve the chat messages from the database and display them using a simple html template.

class IndexView(generic.View):

    def get(self, request):
        # We want to show the last 10 messages, ordered most-recent-last
        chat_queryset = ChatMessage.objects.order_by("-created")[:10]
        chat_message_count = len(chat_queryset)
        if chat_message_count > 0:
            first_message_id = chat_queryset[len(chat_queryset)-1].id
        else:
            first_message_id = -1
        previous_id = -1
        if first_message_id != -1:
            try:
                previous_id = ChatMessage.objects.filter(pk__lt=first_message_id).order_by("-pk")[:1][0].id
            except IndexError:
                previous_id = -1
        chat_messages = reversed(chat_queryset)
        
        return render(request, "chatdemo/chatroom.html", {
            'chat_messages': chat_messages,
            'first_message_id' : previous_id,
        })

The IndexView retrieves the last 10 messages, ordered most-recent-last i.e normal chat oder and sets the following variables :

  • first_message - The id of the first message in the chat scrollback. Eg- let’s say our database contains the messages with id 4,3,2,1. We send 4,3,2 to be rendered in the chat window. When I hit “Load previous messages” from the chat room, I want all messages starting from the id that’s just before what’s currently in the chat room. In this case it will be 1.

The chat room is rendered using the chatroom.html template.

Now that we have a chat room page that displays the ten most recent messages, we can start designing the websocket endpoints that allow users to send a chat message as well as load previous chat messages. The file structure to enable this is simple : we define the routes in routing.py which is the Channels version of urls.py, and we define the consumers (views) in consumers.py which is the Channels version of views.py.

Let’s define 2 endpoints in routing.py:

  • /ws/ will be the endpoint to which chat messages will be sent

  • /loadhistory/ will be the endpoint which handles the requests for previous chat messages

Each endpoint has 3 events :

  • websocket.connect is called when a new websocket connection is opened

  • websocket.receive handles the actual message which is either a chat message from the user or a request for chat history in this project.

  • websocket.disconnect is called when the client disconnects from the websocket connection

Here’s the code for routing.py where we define the mentioned websocket endpoints:

from channels.routing import route
from channels import include
from chatdemo.consumers import chat_connect, chat_disconnect, chat_receive, loadhistory_connect, loadhistory_disconnect, loadhistory_receive

chat_routing = [
    route("websocket.connect", chat_connect),
    route("websocket.receive", chat_receive),
    route("websocket.disconnect", chat_disconnect)
]

loadhistory_routing = [
    route("websocket.connect", loadhistory_connect),
    route("websocket.receive", loadhistory_receive),
    route("websocket.disconnect", loadhistory_disconnect)
]

channel_routing = [
    include(chat_routing, path=r"^/ws/$"),
    include(loadhistory_routing, path=r"^/loadhistory/$"),
]

Moving on to defining the views that process the websocket requests in the consumers.py file, we start by understanding how chat messages are handled by Django channels.

When a user establishes a websocket connection to the /ws/ endpoint, here’s what happens :

  • The reply_channel of the connection is added to the all group. Why is this required? All users will be part of the same group which allows us to send chat messages in a one-to-many fashion. Eg- when User A sends a message, all users connected in the all group will receive the message.

  • The connection is also added to the all room. The Room here referes to a model in the channels_presence package. This is used to keep track of the number of users connected to the chat room.

  • Finally the following response is sent to the user to confirm that the connection has been accepted : {"accept": True}.

Here’s the code that we just discussed :

@channel_session_user_from_http
def chat_connect(message):
    Group("all").add(message.reply_channel)
    Room.objects.add("all", message.reply_channel.name, message.user)
    message.reply_channel.send({"accept": True})

Let’s move on to the code that handles the chat messages. Here’s the outline of what happens inside the function:

  • Receive and decode the json message

  • Confirm that the json contains a message key which holds the content of the message

  • Confirm that the user is authenticated since it does not make sense to accept chat messages from unathenticated users

  • Escape the message using the escape function from django.utils.html

  • Check if the message contains any valid urls and convert the urls to links. Eg- https://google.com becomes <a href="https://google.com">https://google.com</a>

  • Finally return a json containing the username and html message.

Here’s the implementation of the above flow

@touch_presence
@channel_session_user
def chat_receive(message):
    data = json.loads(message['text'])
    if not data['message']:
        return
    if not message.user.is_authenticated:
        return
    current_message = escape(data['message'])
    urlRegex = re.compile(
            u'(?isu)(\\b(?:https?://|www\\d{0,3}[.]|[a-z0-9.\\-]+[.][a-z]{2,4}/)[^\\s()<'
            u'>\\[\\]]+[^\\s`!()\\[\\]{};:\'".,<>?\xab\xbb\u201c\u201d\u2018\u2019])'
        )
    
    processed_urls = list()
    for obj in urlRegex.finditer(current_message):
        old_url = obj.group(0)
        if old_url in processed_urls:
            continue
        processed_urls.append(old_url)
        new_url = old_url
        if not old_url.startswith(('http://', 'https://')):
            new_url = 'http://' + new_url
        new_url = '<a href="' + new_url + '">' + new_url + "</a>"
        current_message = current_message.replace(old_url, new_url)
    m = ChatMessage(user=message.user, message=data['message'], message_html=current_message)
    m.save()

    my_dict = {'user' : m.user.username, 'message' : current_message}
    Group("all").send({'text': json.dumps(my_dict)})

The @touch_presence decorator is used to note the periodic beat sent from the users’ browser which ensures that users are not removed from the list of active users connected to the chat room.

Finally, the chat_disconnect function just removes the user from the all group and room.

@channel_session_user
def chat_disconnect(message):
    Group("all").discard(message.reply_channel)
    Room.objects.remove("all", message.reply_channel.name)

Let’s look at the function that sends the latest list of users and the count of anonymous users. This function is called everytime a user connects or disconnects from the chat room.

@receiver(presence_changed)
def broadcast_presence(sender, room, **kwargs):
    # Broadcast the new list of present users to the room.
    Group(room.channel_name).send({
        'text': json.dumps({
            'type': 'presence',
            'payload': {
                'channel_name': room.channel_name,
                'members': [user.username for user in room.get_users()],
                'lurkers': int(room.get_anonymous_count()),
            }
        })
    })

The json data returned is :

  • members : the list of logged in users connected to the chat room

  • lurkers : an integer count representing how many anonymous users are connected to the chat

Moving on to the second websocket endpoint, /loadhistory/, the loadhistory_connect(message) function just accepts the websocket connect by sends the standard response to indicate that the connection has been accepted : {"accept": True}. On disonnecting the connection, we don’t need to do anything.

@channel_session_user_from_http
def loadhistory_connect(message):
    message.reply_channel.send({"accept": True})

@channel_session_user
def loadhistory_disconnect(message):
    pass

Finally, here’s how the loadhistory_receive returns previous messages:

  • It decodes the json and extracts the last_message_id which represents the chat message id before the last message that has been rendered in the chat room for the current user.

  • It makes a query to the database to retrieve 10 messages (if they exist) having an id less than or equal to last_message_id and returns the chat messages in json format.

Here’s the code:

@channel_session_user
def loadhistory_receive(message):
    data = json.loads(message['text'])
    chat_queryset = ChatMessage.objects.filter(id__lte=data['last_message_id']).order_by("-created")[:10]
    chat_message_count = len(chat_queryset)
    if chat_message_count > 0:
        first_message_id = chat_queryset[len(chat_queryset)-1].id
    else:
        first_message_id = -1
    previous_id = -1
    if first_message_id != -1:
        try:
            previous_id = ChatMessage.objects.filter(pk__lt=first_message_id).order_by("-pk")[:1][0].id
        except IndexError:
            previous_id = -1

    chat_messages = reversed(chat_queryset)
    cleaned_chat_messages = list()
    for item in chat_messages:
        current_message = item.message_html
        cleaned_item = {'user' : item.user.username, 'message' : current_message }
        cleaned_chat_messages.append(cleaned_item)

    my_dict = { 'messages' : cleaned_chat_messages, 'previous_id' : previous_id }
    message.reply_channel.send({'text': json.dumps(my_dict)})

We now have the websocket endpoints, so let’s move on to the the client side javascript which calls these endpoints.

Before I explain the client side javascript that connects to the websocket endpoints, a quick note about the usage of Reconnecting websockets : it’s a javascript library that automatically reconnects if the connection is dropped and is used because by default, websockets do not automatically reconnect if the connection is closed.

There are 2 clientside javascript files :

I’ll give you a high level overview of what happens inside the realtime.js file:

  • Establish a ReconnectingWebSocket connection to the /ws/ endpoint. Like http and https, websocket connections can be either ws or wss i.e unencrypted or encrypted. The code uses wss if the site is loaded over https, otherwise it uses the unencrypted ws mode.

  • Send a periodic heartbeat to the /ws endpoint every 10 seconds to let the server know that the connection is active

  • Convert the chat message text that the user submits into json format and send it to the server

  • There are two types of server responses that this file handles:

    1. When the server sends a presence payload containing an updated list of active users along with a lurkers count. The javascript uses this data to update the users list and lurkers count.

    2. When a new message is posted, the server sends the username of the user who sent the message along with the rendered html. The javascript uses this data to update the user’s chat room.

Here’s the javascript implementation of what we just discussed:

$(function() {
    // When we're using HTTPS, use WSS too.
    $('#all_messages').scrollTop($('#all_messages')[0].scrollHeight);
    var to_focus = $("#message");
    var ws_scheme = window.location.protocol == "https:" ? "wss" : "ws";
    var chatsock = new ReconnectingWebSocket(ws_scheme + '://' + window.location.host + "/ws/");

    chatsock.onmessage = function(message) {

        if($("#no_messages").length){
            $("#no_messages").remove();
        }

        var data = JSON.parse(message.data);
        if(data.type == "presence"){
            //update lurkers count
            lurkers = data.payload.lurkers;
            lurkers_ele = document.getElementById("lurkers-count");
            lurkers_ele.innerText = lurkers;

            //update logged in users list
            user_list = data.payload.members;
            document.getElementById("loggedin-users-count").innerText = user_list.length;
            user_list_obj = document.getElementById("user-list");
            user_list_obj.innerText = "";
            
            //alert(user_list);
            for(var i = 0; i < user_list.length; i++ ){
                var user_ele = document.createElement('li');
                user_ele.setAttribute('class', 'list-group-item');
                user_ele.innerText = user_list[i];
                user_list_obj.append(user_ele);
            }

            return;
        }
        var chat = $("#chat")
        var ele = $('<li class="list-group-item"></li>')
        
        ele.append(
            '<strong>'+data.user+'</strong> : ')
        
        ele.append(
            data.message)
        
        chat.append(ele)
        $('#all_messages').scrollTop($('#all_messages')[0].scrollHeight);
    };

    $("#chatform").on("submit", function(event) {
        var message = {
            message: $('#message').val()
        }
        chatsock.send(JSON.stringify(message));
        $("#message").val('').focus();
        return false;
    });

    setInterval(function() {
    chatsock.send(JSON.stringify("heartbeat"));
    }, 10000);
});

The second javascript file handles the loading of previous chat messages. It works as follows:

  • When the user clicks on the “Load old messages” button, the javascript sends the last_message_id variable to the /loadhistory endpoint.

  • When the server responds with the previous messages, the javascript updates the client side chat room with the historical chat messages. If the server indicates that there are no more messages in the scrollback, the javascript removes the “Load old messages” button.

Here is the loadhistory.js code:

$(function() {
    // When we're using HTTPS, use WSS too.
    var ws_scheme = window.location.protocol == "https:" ? "wss" : "ws";

    var loadhistorysock = new ReconnectingWebSocket(ws_scheme + '://' + window.location.host + "/loadhistory/");

    loadhistorysock.onmessage = function(message) {

        var data = JSON.parse(message.data);

        new_messages = data.messages

        last_id = data.previous_id
        
        if(last_id == -1){
            $("#load_old_messages").remove();
            $("#last_message_id").text(last_id)
            if(new_messages.length == 0){
                return;
            }
        }
        else{
            $("#last_message_id").text(last_id)
        }

        var chat = $("#chat")

        for(var i=new_messages.length - 1; i>=0; i--){
            var ele = $('<li class="list-group-item"></li>')
            
            ele.append(
                '<strong>'+new_messages[i]['user']+'</strong> : '
                )
            
            ele.append(
                new_messages[i]['message'])
            
            chat.prepend(ele)
        }

    };

    $("#load_old_messages").on("click", function(event) {
        var message = {
            last_message_id: $('#last_message_id').text()
        }
        loadhistorysock.send(JSON.stringify(message));
        return false;
    });
});

We talked about pruning stale websocket connections earlier, but how do we implement the code? We use a celery task that runs every 10 seconds and prunes the stale websocket connections.

The code to prune the websocket connections is very simple:

def prune():
    from channels_presence.models import Room
    Room.objects.prune_presences()

The django-channels-presence package makes this task easy. If you want to view the full code in the celery.py file, link is here. If you are not familiar with celery, refer to the official documentation.

That’s it! We went over the process of building a chat room using Django Channels. You can see the code for this project on Github which has instructions to setup and run this project on your local machine.

If you have any questions, comments or suggestions to improve this guide let me know in the comments section below (which I built as a Disqus alternative). You can contact me by email or Twitter and the links to them are in the footer.

Here is a screenshot of the chat room for future reference. The demo page will be taken down a few days after this post is published.

Chat room demo

References: