How to create a streamlined communication channel between the server and client sides
Recently, I came across a programming task required to display the last ‘n’ lines of a log file on a web interface, echoing the functionality ‘tail -[n] -f.’ The tail -f -30 /var/log/nginx/error.log command that we’ve probably done millions of times on the terminal. At first, I thought this was a well-charted territory. I mean, it’s a common task, right? After doing some research, there aren’t many solutions lying around. As a problem-solver by nature, I decided to tackle it head-on.
To continuously feed data from the backend to the frontend, we must venture beyond traditional REST APIs. I explored the possibilities of Server-Side Events and File Streaming APIs, but their server-side implementations appeared rather complex. Therefore, I opted for WebSockets, providing a more streamlined communication channel between the server and client sides.
As I already had an existing web API server written in GoLang that serves up a couple of REST endpoints as a starting point. Please refer to this if you have questions about this is done.
Let’s add the required packages for hosting a WebSocket service.
go get github.com/gorilla/websocket
We need to upgrade an HTTP endpoint for WebSocket. We will need this method to upgrade a regular HTTP endpoint to a WebSocket endpoint later.
var upgrader = websocket.Upgrader{
ReadBufferSize: 1024,
WriteBufferSize: 1024,
}
Next, we would prepare two types of structs: one for the client object and another for a broadcaster object. The client object represents a single client. It holds a socket connection reference and the sending channel. The broadcaster object is responsible for managing multiple client connections.
type Client struct {
socket *websocket.Conn
send chan []byte
}
type Broadcaster struct {
clients map[*Client]bool
broadcast chan string
register chan *Client
unregister chan *Client
}
Let’s create a broadcaster producer function to create Broadcaster instance objects. A broadcaster keeps track of the WebSocket connections. It also keeps a channel to receive messages to broadcast to all connections.
ster() *Broadcaster {
return &Broadcaster{
broadcast: make(chan string),
register: make(chan *Client),
unregister: make(chan *Client),
clients: make(map[*Client]bool),
}
}
func (b *Broadcaster) run() {
for {
select {
case client := <-b.register:
b.clients[client] = true
case client := <-b.unregister:
if _, ok := b.clients[client]; ok {
delete(b.clients, client)
close(client.send)
}
case message := <-b.broadcast:
for client := range b.clients {
client.send <- []byte(message)
}
}
}
}
We can use this Go package to do the ‘tailing’ read of the file. So, we can include this package in our code. It will send the lines of text to the broadcaster, which sends them to the send channel — a Go routine that we will define.
func (b *Broadcaster) tailFile(filepath string) {
t, err := tail.TailFile(
filepath,
tail.Config{Follow: true, Location: &tail.SeekInfo{Offset: 0, Whence: 2}},
)
if err != nil {
log.Fatalf("tail file err: %v", err)
}
for line := range t.Lines {
if line.Text != "" {
b.broadcast <- line.Text
}
}
}
After setting up the broadcaster logic, we can hook it up in our main() routine. We add a static file route to test this feature with JavaScript easily.
import (
"encoding/json"
"log"
"net/http"
"github.com/gorilla/mux"
"github.com/gorilla/websocket"
"github.com/hpcloud/tail"
)
// ...
func main() {
broadcaster := newBroadcaster()
go broadcaster.run()
go broadcaster.tailFile("./sample_data/sample_log.log") // file path, last n number of lines
staticServer := http.FileServer(http.Dir("./public_html"))
router := mux.NewRouter()
// or just http.HandleFunc(...) for not using mux.
router.HandleFunc("/ws", handleWebSocketConnection(broadcaster))
router.Handle("/", staticServer) // we can also add a static file server to test it using JavaScript
// ...
log.Fatal(http.ListenAndServe(":8000", router))
// Original map
}
Put some index.html content under /public-html.
<html>
<head>
<title>WebSocket Tester</title>
</head>
<body>
<script>
var ws = new WebSocket('ws://localhost:8000/ws');
ws.onopen = function(event) {
console.log('Connection is open ...');
};
ws.onerror = function(err) {
console.log('err: ', err, err.toString());
};
// Event handler for receiving text from the server
ws.onmessage = function(event) {
console.log('Received: ' + event.data);
};
ws.onclose = function() {
console.log('Connection is closed...');
};
</script>
</body>
</html>
And we need to create the connection handler for WebSocket. Notice we are using two Go Routines here. They are under a for loop, which keeps them running until they break. The first Go routine is used to clean the WebSocket connection when we receive errors from the socket. The other is used to write messages we receive from the send channel, which was passed from the broadcaster we defined earlier.
func handleWebSocketConnection(b *Broadcaster, filePath string, n int) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
ws, err := upgrader.Upgrade(w, r, nil)
if err != nil {
log.Println(err)
return
}
client := &Client{socket: ws, send: make(chan []byte)}
b.register <- client
go func() {
defer func() {
b.unregister <- client
ws.Close()
}()
for {
_, _, err := ws.ReadMessage()
if err != nil {
b.unregister <- client
ws.Close()
break
}
}
}()
go func() {
defer ws.Close()
for {
message, ok := <-client.send
if !ok {
ws.WriteMessage(websocket.CloseMessage, []byte{})
return
}
ws.WriteMessage(websocket.TextMessage, message)
}
}()
}
}
It’s time to run the server now and check for any errors.
go run main.go
When we update the target log file sample_data/sample_log.log, we can see the JavaScript console logs the output of the browser. Cool, looks like it worked. Now, let’s take it one step further. Let’s say, upon every client connection, we want the server to immediately send the last ‘n’ lines of the log file to the client. Instead of showing blank lines initially.
How can we archive this? Well, there are many ways to do this. To keep it simple. We can use the plain old buffer io.Scan() function from Go to write a readLastNlines() function as follows:
func readLastNLines(fileName string, n int) ([]string, error) {
if err != nil {
return nil, err
}
defer file.Close()
scanner := bufio.NewScanner(file)
lines := make([]string, 0)
for scanner.Scan() {
lines = append(lines, scanner.Text())
if len(lines) > n {
lines = lines[1:]
}
}
if scanner.Err() != nil {
return nil, scanner.Err()
}
return lines, nil
}
Create another go routine in handleWebSocketConnection() function to invoke initialRead(), which internally calls readLastNLines() to get the last n lines to send down the wire. But don’t forget about the go keyword before the initialRead() function call.
func (b *Broadcaster) initialRead(client *Client, filePath string, n int) {
// Send last n lines from file to the client
lines, err := readLastNLines(filePath, n)
if err != nil {
log.Println(err)
return
}
for _, line := range lines {
b.broadcast <- line
}
}
func handleWebSocketConnection(b *Broadcaster, filePath string, n int) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
ws, err := upgrader.Upgrade(w, r, nil)
if err != nil {
log.Println(err)
return
}
client := &Client{socket: ws, send: make(chan []byte)}
b.register <- client
go b.initialRead(client, filePath, n) // <-- this is added
go func() {
defer func() {
b.unregister <- client
ws.Close()
}()
for {
_, _, err := ws.ReadMessage()
if err != nil {
b.unregister <- client
ws.Close()
break
}
}
}()
go func() {
defer ws.Close()
for {
message, ok := <-client.send
if !ok {
ws.WriteMessage(websocket.CloseMessage, []byte{})
return
}
ws.WriteMessage(websocket.TextMessage, message)
}
}()
}
}
At last, we need to update the main() function for the handleWebSocketConnection() method signature.
func main() {
targetFile := "./sample_data/sample_log.log"
lastNLines := 20
// ...
router.HandleFunc("/ws", handleWebSocketConnection(broadcaster, targetFile, lastNLines))
// ...
}
Now, there you have it. You can read the complete source code in this GitHub repo.
Enjoy.
Bye now, and happy coding!
Streaming Log Files in Real-Time with GoLang and WebSockets: A ‘tail -f’ Simulation was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.