HTTP Compression in Node.js: A Dive into Gzip, Deflate, and Brotli

To celebrate that the wonderful NPM package compression just added Brotli in version 1.8.0, we’ve written this article to review compression: the different types of HTTP compression, why use compression for an API, and how to implement it in Node.js. We also look at analytics on the common compression algorithms Deflate, Gzip, and Brotli.

Introduction

Optimizing data transfer is crucial for performance and user experience. You might have heard of the classic Walmart website study that 100ms website performance gain increased sales by 1%. This is also true for APIs that often transfer large amounts of JSON data between servers and clients. In general, no one likes slow…so let’s compress.

HTTP compression is a powerful technique that can significantly reduce bandwidth usage and improve response times, making it essential for both web applications and APIs.

For APIs, compression becomes increasingly important as:

• Internal and external APIs are becoming more widespread.
• Mobile clients need to minimize data usage.
• Real-time applications require faster response times.
• APIs often transfer repetitive or large JSON structures.
• Cloud providers charge for bandwidth usage.

When an API serves thousands or millions of requests daily, even small reductions in response size can lead to significant bandwidth savings and performance improvements. HTTP compression helps achieve this by automatically compressing response data before transmission and decompressing it at the client side. At Ayrshare, our social API extensively uses compression.

In this comprehensive guide, we’ll explore different compression algorithms, implement them in Node.js, and analyze their performance with real-world data. We’ll pay special attention to JSON data compression, as it’s particularly relevant for modern APIs.

Why Compression is Essential for Modern APIs

Before diving into the technical details, let’s understand why compression is crucial, especially for APIs:

Performance and Cost Benefits

• Reduced API Bandwidth: APIs often transfer large JSON payloads with repeated field names. Compression typically reduces these large payloads by 20-30%, directly lowering cloud bandwidth costs.
• Faster Response Times: While compression adds some minimal CPU overhead, the reduced transfer time leads to faster overall responses. Brotli is the fastest of the bunch.
• Larger Payload Capacity: Compression can reduce or eliminate the need for pagination by allowing larger data sets to be transferred in a single request, simplifying client implementations and reducing total API calls. Pagination often adds complexity and extra calls, so it’s preferable if you can efficiently return the entire data set in one go.

Infrastructure Impact

• Server Efficiency: Reduced network I/O typically outweighs compression CPU cost, leading to better resource utilization and lower infrastructure costs.
• Global Performance: Compression helps maintain consistent API performance across different network conditions and geographical locations.
• Cache Benefits: Compressed responses take up less cache space, allowing more unique responses to be cached and reducing backend load. We use Redis, so reducing storage requirements is a great thing.

Understanding Compression Algorithms

There are three widely used HTTP compression algorithms: deflate, gzip, and br (brotli). The most widely used is gzip, with deflate as the back-up. Brotli is the relatively new kid on the block, but also the best.

Deflate

Deflate is the core compression algorithm that forms the basis for gzip. It uses a combination of LZ77 and Huffman coding to achieve compression. While it can be used directly, it’s more commonly encountered as part of gzip. Is it the back-up if gzip isn’t available.

Gzip

Gzip is essentially Deflate with additional metadata and error checking. It adds:

• Headers with metadata.
• CRC32 checksum for error detection.
• Operating system information.
• Timestamps.

Brotli (br)

Developed by Google, Brotli is a newer compression algorithm that typically achieves better compression ratios than gzip and generally takes fewer CPU cycles. It uses a more sophisticated dictionary-based compression approach and is particularly effective for text-based content like JSON, HTML, CSS, and JavaScript. If you’re using a RESTful API implementation with JSON, then this is the winner.

Implementing Compression in Node.js

Setting Up Express with Compression

First, install the required packages of express and compression. While there are many NPM packages that offer compression, this one is the GOAT of Node compression…so recommended!

npm install express compression

As mentioned above, starting with release 1.8.0 of compression Brotli is supported. This is an awesome addition as you’ll see below.

The compression package can be used a express middleware. While there are lot of options such as setting the compression level, threshold of size to compress (default in files size is 1024 bytes of 1KB – nothing smaller will be compressed), and filters, you usually can just go with the defaults.

Basic implementation of the compression middleware:

const express = require('express');
const compression = require('compression');
const app = express();

// Enable compression
app.use(compression());

Client-Side Headers

To request compressed content, clients should send the appropriate Accept-Encoding header:

Accept-Encoding: gzip, deflate, br

The server will try to first use br (brotli) if available, followed by gzip, and finally deflate.

Servers will respond with the Content-Encoding header indicating the compression method used:

Content-Encoding: br

Make sure the caller can handle the compressed data. If using a browser and a standard API client like Postman, the decompression happens automatically. If making direct API calls you’ll need to handle the decompression.

Compression Performance Analysis

We conducted an experiment to compare the three compression algorithms using a large text dataset of Shakespeare’s entire works (5.3 MB of content). You can review the compression test code on GitHub.

Here are our findings:

Compression Algorithm Test Results

Compression Algorithm Findings

Compression Effectiveness

• Brotli achieved the best compression ratio at 64.26%.
• Gzip and Deflate showed identical compression ratios (61.66%).
• Brotli saved an additional 0.13 MB compared to gzip/deflate.

Processing Speed

• Brotli was surprisingly the fastest at 95.31ms.
• Deflate and gzip took roughly 2.5x longer.
• All algorithms processed 5.3 MB in under 300ms.

Size Reduction

• All algorithms provided significant size reduction.
• Brotli reduced the file to 1.89 MB.
• Gzip and Deflate reduced to 2.03 MB.

Overall, Brotli is the best choice.

Additional Info on the Compression Package

While there are lost of configuration options for the compression package, I recommend the level and threshold, followed by filter for further processing or logging.

Compression Configuration Examples

// Advanced compression configuration
app.use(compression({
  level: 6,                // Compression level (0-9) - 6 is a good level
  threshold: 1024,         // Only compress responses larger than 1 KB (default)
  filter: (req, res) => {
    // console.log("Log something.");
    return compression.filter(req, res);
  }
}));

Brotli is the Best Overall Choice

Our experiment demonstrates that modern compression algorithms can significantly reduce data transfer sizes, with Brotli leading in both compression ratio and processing speed. While gzip remains a reliable choice with broad support, implementing Brotli can provide additional benefits for compatible clients.

Consider your specific use case when choosing a compression algorithm:

• For maximum compression: Use Brotli.
• For broad compatibility: Use gzip.
• For minimal overhead: Use deflate.

Remember to test compression with your actual content, as results may vary based on data characteristics and patterns.

Additional Resources

• Express.js Compression Documentation
• MDN Web Docs: HTTP compression
• Brotli Specification

Understanding Our Test Implementation

Let’s examine the code we used to test these compression algorithms. You can also find the code at GitHub.

This implementation creates a test environment to measure compression performance of the different compression algorithms.

Complete Compression Test Code

const express = require('express');
const compression = require('compression');
const axios = require('axios');
const zlib = require('zlib');

// Create test data (for backup if fetch fails)
const generateTestData = (size = 5 * 1024 * 1024) => {
  const words = [
    "the", "be", "to", "of", "and", "a", "in", "that", "have", "I",
    "Shakespeare", "Hamlet", "Macbeth", "Romeo", "Juliet", "Othello",
    "wherefore", "thou", "thy", "thee", "hast", "doth", "forsooth"
  ];

  let result = '';
  while (result.length < size) {
    const word = words[Math.floor(Math.random() * words.length)] + ' ';
    result += word;
    if (Math.random() < 0.1) result += '\n';
  }
  return result;
};

// Format bytes into human-readable format
const formatBytes = (bytes) => {
  if (bytes === 0) return '0 Bytes';
  const k = 1024;
  const sizes = ['Bytes', 'KB', 'MB', 'GB'];
  const i = Math.floor(Math.log(bytes) / Math.log(k));
  return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i];
};

// Main test function for each compression algorithm
const testCompression = async (algorithm, testData) => {
  return new Promise((resolve, reject) => {
    const app = express();
    const port = 3000 + Math.floor(Math.random() * 1000);

    // Configure compression middleware
    app.use(compression({
      level: 6,
      threshold: 0,
      memLevel: 8,
      strategy: zlib.constants.Z_DEFAULT_STRATEGY,
      flush: zlib.constants.Z_NO_FLUSH,
      chunkSize: 16 * 1024,
      filter: () => true,
      algorithm
    }));

    // Test endpoint
    app.get('/test', (req, res) => {
      res.setHeader('Cache-Control', 'no-cache');
      res.setHeader('Content-Type', 'text/plain');
      res.setHeader('X-No-Compression', 'false');
      res.send(testData);
    });

    // Start server and run test
    const server = app.listen(port, async () => {
      try {
        const startTime = process.hrtime();

        const response = await axios.get(`http://localhost:${port}/test`, {
          headers: {
            'Accept-Encoding': algorithm,
            'Cache-Control': 'no-cache',
            'Accept': 'text/plain'
          },
          responseType: 'arraybuffer',
          decompress: false
        });

        const endTime = process.hrtime(startTime);
        const duration = (endTime[0] * 1000 + endTime[1] / 1000000).toFixed(2);

        resolve({
          algorithm: response.headers['content-encoding'] || 'none',
          originalSize: formatBytes(testData.length),
          compressedSize: formatBytes(response.data.length),
          compressionRatio: ((testData.length - response.data.length) / testData.length * 100).toFixed(2) + '%',
          timeTaken: duration + 'ms',
          savingsInMB: ((testData.length - response.data.length) / (1024 * 1024)).toFixed(2) + ' MB'
        });
      } catch (error) {
        reject(error);
      } finally {
        server.close();
      }
    });
  });
};

// Run all tests
const compareCompressions = async () => {
  console.log('Using test data...');
  const testData = generateTestData();
  console.log(`Test data size: ${formatBytes(testData.length)}`);

  try {
    const results = [];
    results.push(await testCompression('gzip', testData));
    results.push(await testCompression('deflate', testData));
    results.push(await testCompression('br', testData));

    console.table(results);
  } catch (error) {
    console.error('Test failed:', error.message);
  }
};

// Execute tests
compareCompressions();