SoatDev IT Consulting
SoatDev IT Consulting
  • About us
  • Expertise
  • Services
  • How it works
  • Contact Us
  • News
  • June 8, 2023
  • Rss Fetcher

How to create a streamlined communication channel between the server and client sides

Photo by Artem Sapegin on Unsplash

Recently, I came across a programming task required to display the last ‘n’ lines of a log file on a web interface, echoing the functionality ‘tail -[n] -f.’ The tail -f -30 /var/log/nginx/error.log command that we’ve probably done millions of times on the terminal. At first, I thought this was a well-charted territory. I mean, it’s a common task, right? After doing some research, there aren’t many solutions lying around. As a problem-solver by nature, I decided to tackle it head-on.

To continuously feed data from the backend to the frontend, we must venture beyond traditional REST APIs. I explored the possibilities of Server-Side Events and File Streaming APIs, but their server-side implementations appeared rather complex. Therefore, I opted for WebSockets, providing a more streamlined communication channel between the server and client sides.

As I already had an existing web API server written in GoLang that serves up a couple of REST endpoints as a starting point. Please refer to this if you have questions about this is done.

Let’s add the required packages for hosting a WebSocket service.

go get github.com/gorilla/websocket

We need to upgrade an HTTP endpoint for WebSocket. We will need this method to upgrade a regular HTTP endpoint to a WebSocket endpoint later.

var upgrader = websocket.Upgrader{
ReadBufferSize: 1024,
WriteBufferSize: 1024,
}

Next, we would prepare two types of structs: one for the client object and another for a broadcaster object. The client object represents a single client. It holds a socket connection reference and the sending channel. The broadcaster object is responsible for managing multiple client connections.

type Client struct {
socket *websocket.Conn
send chan []byte
}

type Broadcaster struct {
clients map[*Client]bool
broadcast chan string
register chan *Client
unregister chan *Client
}

Let’s create a broadcaster producer function to create Broadcaster instance objects. A broadcaster keeps track of the WebSocket connections. It also keeps a channel to receive messages to broadcast to all connections.

ster() *Broadcaster {
return &Broadcaster{
broadcast: make(chan string),
register: make(chan *Client),
unregister: make(chan *Client),
clients: make(map[*Client]bool),
}
}

func (b *Broadcaster) run() {
for {
select {
case client := <-b.register:
b.clients[client] = true
case client := <-b.unregister:
if _, ok := b.clients[client]; ok {
delete(b.clients, client)
close(client.send)
}
case message := <-b.broadcast:
for client := range b.clients {
client.send <- []byte(message)
}
}
}
}

We can use this Go package to do the ‘tailing’ read of the file. So, we can include this package in our code. It will send the lines of text to the broadcaster, which sends them to the send channel — a Go routine that we will define.


func (b *Broadcaster) tailFile(filepath string) {
t, err := tail.TailFile(
filepath,
tail.Config{Follow: true, Location: &tail.SeekInfo{Offset: 0, Whence: 2}},
)
if err != nil {
log.Fatalf("tail file err: %v", err)
}

for line := range t.Lines {
if line.Text != "" {
b.broadcast <- line.Text
}
}
}

After setting up the broadcaster logic, we can hook it up in our main() routine. We add a static file route to test this feature with JavaScript easily.


import (
"encoding/json"
"log"
"net/http"

"github.com/gorilla/mux"
"github.com/gorilla/websocket"
"github.com/hpcloud/tail"
)

// ...

func main() {
broadcaster := newBroadcaster()
go broadcaster.run()
go broadcaster.tailFile("./sample_data/sample_log.log") // file path, last n number of lines

staticServer := http.FileServer(http.Dir("./public_html"))
router := mux.NewRouter()
// or just http.HandleFunc(...) for not using mux.
router.HandleFunc("/ws", handleWebSocketConnection(broadcaster))
router.Handle("/", staticServer) // we can also add a static file server to test it using JavaScript
// ...
log.Fatal(http.ListenAndServe(":8000", router))
// Original map
}

Put some index.html content under /public-html.

<html>
<head>
<title>WebSocket Tester</title>
</head>
<body>
<script>
var ws = new WebSocket('ws://localhost:8000/ws');
ws.onopen = function(event) {
console.log('Connection is open ...');
};

ws.onerror = function(err) {
console.log('err: ', err, err.toString());
};

// Event handler for receiving text from the server
ws.onmessage = function(event) {
console.log('Received: ' + event.data);
};

ws.onclose = function() {
console.log('Connection is closed...');
};
</script>
</body>
</html>

And we need to create the connection handler for WebSocket. Notice we are using two Go Routines here. They are under a for loop, which keeps them running until they break. The first Go routine is used to clean the WebSocket connection when we receive errors from the socket. The other is used to write messages we receive from the send channel, which was passed from the broadcaster we defined earlier.


func handleWebSocketConnection(b *Broadcaster, filePath string, n int) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
ws, err := upgrader.Upgrade(w, r, nil)
if err != nil {
log.Println(err)
return
}
client := &Client{socket: ws, send: make(chan []byte)}
b.register <- client

go func() {
defer func() {
b.unregister <- client
ws.Close()
}()

for {
_, _, err := ws.ReadMessage()
if err != nil {
b.unregister <- client
ws.Close()
break
}
}
}()

go func() {
defer ws.Close()
for {
message, ok := <-client.send
if !ok {
ws.WriteMessage(websocket.CloseMessage, []byte{})
return
}
ws.WriteMessage(websocket.TextMessage, message)
}
}()
}
}

It’s time to run the server now and check for any errors.

go run main.go

When we update the target log file sample_data/sample_log.log, we can see the JavaScript console logs the output of the browser. Cool, looks like it worked. Now, let’s take it one step further. Let’s say, upon every client connection, we want the server to immediately send the last ‘n’ lines of the log file to the client. Instead of showing blank lines initially.

How can we archive this? Well, there are many ways to do this. To keep it simple. We can use the plain old buffer io.Scan() function from Go to write a readLastNlines() function as follows:

func readLastNLines(fileName string, n int) ([]string, error) {
if err != nil {
return nil, err
}
defer file.Close()

scanner := bufio.NewScanner(file)
lines := make([]string, 0)

for scanner.Scan() {
lines = append(lines, scanner.Text())
if len(lines) > n {
lines = lines[1:]
}
}

if scanner.Err() != nil {
return nil, scanner.Err()
}

return lines, nil
}

Create another go routine in handleWebSocketConnection() function to invoke initialRead(), which internally calls readLastNLines() to get the last n lines to send down the wire. But don’t forget about the go keyword before the initialRead() function call.


func (b *Broadcaster) initialRead(client *Client, filePath string, n int) {
// Send last n lines from file to the client
lines, err := readLastNLines(filePath, n)
if err != nil {
log.Println(err)
return
}

for _, line := range lines {
b.broadcast <- line
}
}
func handleWebSocketConnection(b *Broadcaster, filePath string, n int) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
ws, err := upgrader.Upgrade(w, r, nil)
if err != nil {
log.Println(err)
return
}
client := &Client{socket: ws, send: make(chan []byte)}
b.register <- client

go b.initialRead(client, filePath, n) // <-- this is added

go func() {
defer func() {
b.unregister <- client
ws.Close()
}()

for {
_, _, err := ws.ReadMessage()
if err != nil {
b.unregister <- client
ws.Close()
break
}
}
}()

go func() {
defer ws.Close()
for {
message, ok := <-client.send
if !ok {
ws.WriteMessage(websocket.CloseMessage, []byte{})
return
}
ws.WriteMessage(websocket.TextMessage, message)
}
}()
}
}

At last, we need to update the main() function for the handleWebSocketConnection() method signature.

func main() {
targetFile := "./sample_data/sample_log.log"
lastNLines := 20
// ...
router.HandleFunc("/ws", handleWebSocketConnection(broadcaster, targetFile, lastNLines))
// ...
}

Now, there you have it. You can read the complete source code in this GitHub repo.

Enjoy.

Bye now, and happy coding!


Streaming Log Files in Real-Time with GoLang and WebSockets: A ‘tail -f’ Simulation was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

Previous Post
Next Post

Recent Posts

  • European leaders worry they’re too reliant on U.S. tech
  • Cartoonist Paul Pope is more worried about killer robots than AI plagiarism
  • Week in Review:  Meta reveals its Oakley smart glasses
  • 2 days left to save up to $210 on your TechCrunch All Stage pass
  • Minimize squared relative error

Categories

  • Industry News
  • Programming
  • RSS Fetched Articles
  • Uncategorized

Archives

  • June 2025
  • May 2025
  • April 2025
  • February 2025
  • January 2025
  • December 2024
  • November 2024
  • October 2024
  • September 2024
  • August 2024
  • July 2024
  • June 2024
  • May 2024
  • April 2024
  • March 2024
  • February 2024
  • January 2024
  • December 2023
  • November 2023
  • October 2023
  • September 2023
  • August 2023
  • July 2023
  • June 2023
  • May 2023
  • April 2023

Tap into the power of Microservices, MVC Architecture, Cloud, Containers, UML, and Scrum methodologies to bolster your project planning, execution, and application development processes.

Solutions

  • IT Consultation
  • Agile Transformation
  • Software Development
  • DevOps & CI/CD

Regions Covered

  • Montreal
  • New York
  • Paris
  • Mauritius
  • Abidjan
  • Dakar

Subscribe to Newsletter

Join our monthly newsletter subscribers to get the latest news and insights.

© Copyright 2023. All Rights Reserved by Soatdev IT Consulting Inc.