Depict memory usage in nodejs using memwatch and D3

27-10-2016
source code

Introduction

In this article I will explain how to depict memory usage in a nodejs application using memwatch-next and D3. The goal of depicting memory usage is to understand how much memory an application is using over time. This understanding in turns can lead to determine whether an application has a memory leak or not. As such I will show 2 charts from 2 different applications, one with memory leak and one without so we can understand the difference.

An application with memory leak

Listing 1 is showing an example of a memory leak.

var fs = require('fs');
var path = require('path');
var memwatch = require('memwatch-next');
var randomstring = require('randomstring');

var memwatchStats = [];
var largeObject = null;

function setLargeObject() {
    var largeObjectCopy = largeObject;
    var currentTime = (new Date()).toLocaleTimeString();
    var randomText = randomstring.generate(getRandomNumber(1,5));
    largeObject = {        
        longString: new Array(1000 * 1000).join(randomText),
        printTime: function () {
            console.log(currentTime);
        },
        printObject: function () {
            console.log(largeObjectCopy);
        }
    };
    largeObject.printTime();
}

function getRandomNumber(min, max) {
  return Math.random() * (max - min) + min;
}

setInterval(setLargeObject, 1000);
memwatch.on('stats', function (stats) {
    stats.time = Date.now();
    memwatchStats.push(stats);
});

setTimeout(function () {
    fs.writeFileSync(path.resolve('mem-stats','leak-data.json'), JSON.stringify(memwatchStats));
    console.log('end of processing');
    process.exit(0);
}, 120 * 1000);

The focus of this script is to generate a memory leak by creating a largeObject and referencing it in the printObject function. This will keep the object hanging in memory and not garbage collected.

I am storing the memwatch-next stats in an array then write that array to a file called leak-data.json before terminating the program. I am adding cuurent time to the stats object so I can use it later when depicting this json file into a chart.

After storing the stats, the program is terminated. You can control for how long the program will be running by changing the time for the setTimeout as shown in listing 2.

setTimeout(function () {
    fs.writeFileSync(path.resolve('mem-stats','leak-data.json'), JSON.stringify(memwatchStats));
    console.log('end of processing');
    process.exit(0);
}, 120 * 1000);

Here the program will run for 2 minutes for example.

An application without memory leak

Listing 3 shows the same program mentioned before but without causing a memory leak. The trick as you guessed it is to remove the printObject function. I have removed the rest of the program for brevity.

function setLargeObject() {
    var currentTime = (new Date()).toLocaleTimeString();
    var randomText = randomstring.generate(getRandomNumber(1,5));
    var largeObject = {
        longString: new Array(1000 * 1000).join(randomText),
        printTime: function () {
            console.log(currentTime);
        }
    };
    largeObject.printTime();
}

setTimeout(function() {
     fs.writeFileSync(path.resolve('mem-stats','no-leak-data.json'), JSON.stringify(memwatchStats));
     console.log('end of processing');
     process.exit(0);
 }, 120 * 1000);

Here I am storing the memwatch-next stats in a file called no-leak-data.json

Displaying memory usage in D3

After collecting memory usage from both programs and store them in json files, we can now display these usages in a chart.

I have downloaded an example to create a D3 line chart and refactored it a bit so I don't repeat my self as I am drawing 2 charts, one for each file.

Listing 4 shows the markup for view-chart.html. As you can see, I have moved most of the functionality of D3 into utils.js.

<!DOCTYPE html>

<head>
    <meta charset="utf-8">
    <link rel="stylesheet" href="style/style.css">
    <script src="scripts/d3.v3.min.js"></script>
    <script src="scripts/utils.js"></script>
</head>

<body>

    <h2>Memory usage chart for leak example</h2>
    <div id="leakChart"></div>
    <hr/>
    <h2>Memory usage chart for NONE leak example</h2>
    <div id="noLeakChart"></div>

    <script>

        var leakSvg = createSvg('leakChart');
        drawChart(leakSvg, 'leak-data.json');
        addXLable(leakSvg);
        addYLable(leakSvg);

        var noLeakSvg = createSvg('noLeakChart');
        drawChart(noLeakSvg, 'no-leak-data.json');
        addXLable(noLeakSvg);
        addYLable(noLeakSvg);

    </script>
</body>

To display this file you can use live-server. This is a light weight http server. You can install it globally and run it as shown in listing 5:

# install live-server globally
npm install -g live-server

# move to the project's directory then run
live-server .

live-server . will open a new browser window where you can open the view-chart.html file.

Listing 6 shows how I am using the json files to draw the chart. I am showing part of the drawChart function only for brevity. All the utility functions I am using are located in the utils.js in the scripts folder.

function drawChart(sgv, dataFileName) {
    d3.json("mem-stats/" + dataFileName, function (error, data) {
        data.forEach(function (d) {
            d.date = d.time;
            var memoryInMB = parseInt(d.current_base / 1000000);
            d.close = +memoryInMB;
        });

        // ...
    });
}

This function takes an svg object and a json file name. It then loops through the data in the json file and uses the current_base property for the data bit of the chart (y axis) and the time for the time bit of the chart (x axis). I am converting the current_base into megabytes (MB) and removing numbers after the decimal point for ease of reading.

Comparing the two charts

Image 1 shows the memory usage from the leak sample program running over a period of 2 minutes.

Leak Memory

Image 2 shows the memory usage from the NONE leak sample program running over a period of 2 minutes.

Leak Memory

The difference is quite clear. In image 1 you can see the memory usage being increased over time because our largeObject is not being garbage collected whereas in image 2, the memory usage is maintained and goes up when the program needs it then goes down again when resources are garbage collected.

Making sense of this example

You should easily adapt this technique in any of your applications. If you store the stats in a database over a longer period of time then you should be able to depict the memory usage for your application. Even better, you can select stats from certain periods of the day, for example in the peak time for your application usage then you can see how your application is using memory when under pressure.

Conclusion

In this example I have created two sample programs, one that does a memory leak and one that doesn't. I have collected the memory stats using memwatch-next then I have stored the stats in 2 json files. I then used D3 to depict the memory usage and compare the two charts.