Downloading an S3 File with Progress Tracking using AWS SDK for JavaScript

In this article, we’ll explore how to download a file from Amazon S3 (Simple Storage Service) using the AWS SDK for JavaScript while tracking the download progress. We’ll use the @aws-sdk/client-s3 package to interact with the S3 service and incorporate a progress callback to monitor the download progress.

Prerequisites

Before we begin, make sure you have the following:

• An AWS account with access to an S3 bucket.
• Node.js installed on your machine.

Setting Up the AWS SDK

To get started, let’s install the required package:

npm install @aws-sdk/client-s3

Next, create a new JavaScript file (e.g., s3FileDownloader.js) and import the necessary dependencies:

import { S3Client, GetObjectCommand } from '@aws-sdk/client-s3';

// Create an S3 client instance
const s3Client = new S3Client({
  region: 'YOUR_AWS_REGION',
  credentials: {
    accessKeyId: 'YOUR_ACCESS_KEY_ID',
    secretAccessKey: 'YOUR_SECRET_ACCESS_KEY',
  },
});

Replace ‘YOUR_AWS_REGION’, ‘YOUR_ACCESS_KEY_ID’, and ‘YOUR_SECRET_ACCESS_KEY’ with your own AWS region, access key ID, and secret access key, respectively.

Downloading the S3 File with Progress Tracking

We’ll define a function called downloadS3File that takes the bucket name, object key, and a progress callback as parameters. This function will initiate the S3 file download and provide progress updates using the callback.

const downloadS3File = async (bucketName, objectKey, progressCallback) => {
  const getObjectCommand = new GetObjectCommand({
    Bucket: bucketName,
    Key: objectKey,
  });

  try {
    const fileStream = await s3Client.send(getObjectCommand);

    fileStream.on('httpDownloadProgress', (progress) => {
      const percentCompleted = Math.round((progress.loaded / progress.total) * 100);
      progressCallback(percentCompleted);
    });

    return new Promise((resolve, reject) => {
      const chunks = [];
      fileStream.on('data', (chunk) => {
        chunks.push(chunk);
      });
      fileStream.on('end', () => {
        const fileContent = Buffer.concat(chunks);
        resolve(fileContent);
      });
      fileStream.on('error', (error) => {
        reject(error);
      });
    });
  } catch (error) {
    console.error('Error downloading S3 file:', error);
    throw error;
  }
};

The downloadS3File function performs the following steps:

1) Creates a GetObjectCommand with the specified bucket name and object key. 2) Sends the command using the s3Client instance, which returns a readable stream representing the file. 3) Attaches an httpDownloadProgress event listener to the stream, which calls the progress callback with the calculated percentage based on the loaded and total bytes. 4) Creates a promise to handle the file download process. 5) Listens for data events on the stream, accumulating the file chunks in an array. 6) Listens for end event to indicate the completion of the download. 7) Concatenates the file chunks into a Buffer and resolves the promise with the file content. 8) Handles any errors that occur during the download process.

To use this function, you can call it with the bucket name, object key, and a progress callback:

const bucketName = 'your-bucket';
const objectKey = 'your-file.json';

downloadS3File(bucketName, objectKey, (progress) => {
  console.log(`Download progress: ${progress}%`);
})
  .then((fileContent) => {
    console.log('Download completed successfully');
    // Use the downloaded file content here
    // For example, if it's a JSON file, parse it: JSON.parse(fileContent.toString('utf-8'));
  })
  .catch((error) => {
    console.error('Error downloading S3 file:', error);
  });

Replace ‘your-bucket’ and ‘your-file.json’ with the appropriate bucket name and object key.

The downloadS3File function will track the progress of the download and call the progress callback with the percentage completed. Once the download is complete, the promise will resolve with the file content, allowing you to perform further processing.

Conclusion

In this article, we learned how to download an S3 file with progress tracking using the AWS SDK for JavaScript. By incorporating a progress callback and utilizing the httpDownloadProgress event, we were able to provide real-time progress updates during the download process. This functionality can be useful for applications that require monitoring file download progress and providing user feedback.