Hello! Today we are talking about Node.js, which we often use as the front-end server for AI systems.

With the spread of AI models, we increasingly find ourselves handling very large data files. Model files for LLMs and training datasets can reach several gigabytes—sometimes tens or even hundreds of gigabytes.

Meanwhile, Node.js is widely adopted as a front-end server for web applications, and it is also remarkably useful for data management and as a bridge to AI back ends written in Python.

In this article, we walk through a problem we ran into while trying to handle a roughly 5 GB file with Node.js v20 LTS, and how we solved it.

How Node.js Buffer Size Limits Have Evolved

The buffer size limit in Node.js has changed significantly from version to version:

In theory, Node.js v20 LTS can handle buffers of up to 4 GB, but limits on I/O operations (reading and writing files) still remain. These limits come not from Node.js itself but from libuv, the asynchronous I/O library that runs underneath it.

The Problem We Actually Hit: A 5 GB AI Model File

In one of our projects, we tried to save a 5 GB AI model file via a Node.js v20 LTS instance serving as a model management server, using the following code:

save_file(target_dir, file_name, file_buffer) {
  try {
    // Create the destination directory if it does not exist
    if (!fs.existsSync(target_dir)) {
      fs.mkdirSync(target_dir, { recursive: true });
    }

    const file_path = path.join(target_dir, file_name);
    fs.writeFileSync(file_path, file_buffer);
    return true;
  } catch (error) {
    console.error(`File save error: ${error.message}\n${error.stack}`);
    return false;
  }
}

This produced the following error:

File save error: The value of "length" is out of range. It must be >= 0 && <= 4294967295. Received 5368709120

The error shows that while the buffer limit in Node.js v20 LTS is 4 GB, the file we were trying to handle was 5 GB (5,368,709,120 bytes).
We had written this code rather naively, but saving huge files this way is simply not acceptable.

As the error indicates, a 5 GB file cannot be processed in one go.

(5 GB is actually on the tame side. When someone inexperienced starts handling model data in the hundreds of gigabytes, code that used to work perfectly fine can suddenly start failing all at once.)

The Solution: Stream Processing and Asynchronous I/O

To solve this problem, we switched to an approach based on stream processing and asynchronous I/O:

async save_file(target_dir, file_name, input_data) {
  try {
    // Create the destination directory if it does not exist (async version)
    await fs.promises.mkdir(target_dir, { recursive: true });

    const file_path = path.join(target_dir, file_name);
    
    // Write the file using a stream
    const writeStream = fs.createWriteStream(file_path);
    
    // If the input is a Buffer
    if (Buffer.isBuffer(input_data)) {
      // Split into chunks and write
      const chunkSize = 1024 * 1024; // 1MB at a time
      for (let i = 0; i < input_data.length; i += chunkSize) {
        const chunk = input_data.slice(i, Math.min(i + chunkSize, input_data.length));
        writeStream.write(chunk);
      }
      writeStream.end();
    } 
    // If the input is a stream
    else if (typeof input_data.pipe === 'function') {
      input_data.pipe(writeStream);
    }
    // Otherwise (a string, etc.)
    else {
      writeStream.write(input_data);
      writeStream.end();
    }

    // Wait for completion or failure
    await new Promise((resolve, reject) => {
      writeStream.on('finish', resolve);
      writeStream.on('error', reject);
    });
    
    return true;
  } catch (error) {
    console.error(`File save error: ${error.message}\n${error.stack}`);
    throw error; // Use throw since this is an async method
  }
}

With this improved code, we were able to save the 5 GB model file without any issues.

The main improvements are as follows.

1. Stream processing
   By splitting the data into small chunks (1 MB) for processing, we avoided the buffer size limit.
2. Asynchronous processing
   async/await allows the server to keep responding to other requests while the file is being processed.
3. Progress reporting
   To monitor the transfer of large files, we also built in per-chunk progress reporting (omitted from the code example).

In short, when handling huge files, caching, streaming, and asynchronous processing are essential for improving stability.

Caution Still Needed on the Latest Node.js (v23 as of April 2025)

From Node.js v22 onward, buffers of up to 8 TB can be handled in theory, but limits still apply to actual I/O operations. When working with large files, we recommend adopting stream processing regardless of the version.

(Bonus) Going Further: Performance Optimization with Multiple Cores

Node.js runs on a single thread, so CPU-bound workloads cannot take full advantage of multi-core performance. The answer to this is the cluster module.

For a simple file save like the one in this article, writing to a single file is fundamentally an I/O-bound operation that gets serialized by the OS file system, so saving from multiple processes actually does not help much. Worse, if multiple processes write to the same file simultaneously, file system locks and seek pointer contention can occur, and performance may even degrade.

That said, when you need to perform some amount of processing on files, going multi-core can improve performance, so here is a quick introduction.

Basic usage of the cluster module

import cluster from 'node:cluster';
import http from 'node:http';
import { cpus } from 'node:os';
import process from 'node:process';

const numCPUs = cpus().length;

if (cluster.isPrimary) {
  console.log(`Primary process ${process.pid} running`);
  
  // Start one worker per CPU core
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork();
  }
  
  cluster.on('exit', (worker, code, signal) => {
    console.log(`Worker ${worker.process.pid} exited`);
    // Restart the worker as needed
    cluster.fork();
  });
} else {
  // Each worker starts an HTTP server on the same port
  http.createServer((req, res) => {
    res.writeHead(200);
    res.end('Hello World\n');
  }).listen(8000);
  
  console.log(`Worker ${process.pid} started`);
}

Combining Optimizations for Large-File Processing

When handling large files plus some kind of CPU-bound processing, combining stream processing with the cluster module enables even more efficient workloads:

1. Optimal use of CPU cores
   cluster module to start one process per CPU core
2. Stream processing
   Implement chunk-based stream processing inside each worker process
3. Load balancing
   Split large files across workers (for example, assigning each worker a byte range)

Summary

We have introduced stream processing for Node.js applications that handle large files such as AI models. Huge files can be handled efficiently by combining stream processing with asynchronous I/O operations.

See you next time!