Orphaned Files In S3: Prevention Strategies

Hey guys! Building a service with NestJS, TypeORM, PostgreSQL, and S3-compatible storage can be super cool. But, it also comes with its own set of challenges. One of the trickiest is dealing with orphaned files in your S3 bucket. Imagine this: a user uploads a movie poster, it gets successfully saved to your S3, but BOOM, the database insertion fails. What happens? You're left with a poster in S3 with no corresponding entry in your database – an orphaned file! This article dives into how to prevent these orphaned files, making sure your S3 and database stay synchronized.

Understanding the Problem: The Root of Orphaned Files

So, why do we get these orphaned files? It usually boils down to a mismatch between your storage system (S3) and your database. When you're creating a service like this, you're dealing with two separate, yet interconnected systems. Let's say a user uploads a poster, and the system will save the poster in S3 and then you create an entity to point to it in your PostgreSQL database. You want the storage and database to function smoothly. When there's a hiccup, a failure, your poster file ends up in S3 without a corresponding reference in your database. This can happen for a bunch of reasons, such as database connection issues, validation errors, or even unexpected server crashes during the save process. In simple words, the database, or the system as a whole, can't keep up with the files you stored on the storage. When this happens you got a bunch of unused files in S3.

The Risks of Orphaned Files

Having orphaned files isn't just an aesthetic issue; it's a potential problem that you have to solve. First, you're wasting storage space and thus increasing your costs. Second, it can make your S3 bucket a mess, making it harder to manage and debug. Third, these files might contain sensitive information that could compromise your security. They can create vulnerabilities when they aren't meant to be there. You don't want some old image to be in the storage, right? So it is super important to avoid it! Let's get our hands dirty in preventing orphaned files.

Strategy 1: Transactions for the Win

One of the best ways to avoid this mess is to use transactions. In essence, a transaction is a way of grouping multiple operations (in this case, saving the file to S3 and creating a database entry) into a single, atomic unit. Think of it as an all-or-nothing deal. If all parts of the transaction succeed, then everything is saved. If any part fails, the entire transaction is rolled back, and nothing is saved. This is the key to keeping your S3 and database in sync.

How to Implement Transactions with NestJS and TypeORM

Here's how you can set this up in your NestJS service, using TypeORM. First, you need to inject the DataSource into your service. Then, use the transaction method to wrap your S3 upload and database save operations.

import { Injectable } from '@nestjs/common';
import { InjectDataSource } from '@nestjs/typeorm';
import { DataSource } from 'typeorm';
import * as AWS from 'aws-sdk';

@Injectable()
export class MovieService {
  constructor(
    @InjectDataSource() private dataSource: DataSource,
  ) {}

  async createMovie(movieData: MovieCreateDto, file: Express.Multer.File) {
    return await this.dataSource.transaction(async (transactionalEntityManager) => {
      try {
        // 1. Upload the file to S3
        const s3Result = await this.uploadToS3(file);
        const posterPath = s3Result.Key;

        // 2. Create the movie entry in the database
        const movie = transactionalEntityManager.create(Movie, {
          ...movieData,
          posterPath,
        });
        await transactionalEntityManager.save(movie);

        return movie; // Or a success message
      } catch (error) {
        // If anything goes wrong, the transaction will roll back automatically
        console.error('Transaction failed:', error);
        throw error; // Rethrow the error to be handled by the controller
      }
    });
  }

  async uploadToS3(file: Express.Multer.File): Promise<AWS.S3.ManagedUpload.SendData> {
    const s3 = new AWS.S3();
    const params = {
      Bucket: 'your-s3-bucket-name',
      Key: `posters/${Date.now()}-${file.originalname}`, // Example: posters/1678886400000-movie-poster.jpg
      Body: file.buffer,
      ContentType: file.mimetype,
    };

    try {
      const result = await s3.upload(params).promise();
      return result;
    } catch (error) {
      console.error('S3 upload failed:', error);
      throw error; // Rethrow the error to be handled by the transaction
    }
  }
}

In this example, if the S3 upload fails, the database operation will not occur. Likewise, if the database insert fails, the entire transaction rolls back, and the file is not uploaded. Note how the S3 upload and database save happen within the transaction. This means they're linked: if one fails, both fail. Using transactions significantly reduces the risk of orphaned files. Remember to handle errors gracefully within the catch block, logging any issues that might arise.

Strategy 2: The Two-Phase Commit (Advanced)

If you're dealing with situations where the S3 and database interactions are complex, or if you're worried about network issues interrupting the transaction, you might want to explore the two-phase commit. This is a more advanced technique where you use an additional mechanism to ensure that both systems commit their changes together.

How It Works

1. Prepare Phase: The database prepares to commit the transaction, but doesn't yet do so. Similarly, you prepare the file for S3 (e.g., by generating a temporary key). This phase confirms that both systems can perform the operation. You are basically telling them to be ready.
2. Commit Phase: Once both systems report they're ready, you tell them to commit the changes. The database saves the data, and the file gets its final name in S3. You are telling them to execute their operations.

Implementing a Two-Phase Commit

Implementing a full two-phase commit is complex and usually requires specific libraries or database features. You can use tools like distributed transactions for PostgreSQL. The idea is to have a coordinator that manages the commit process. First, the coordinator asks each system (database and S3) if they are ready to commit (prepare phase). If both systems are ready, the coordinator tells them to commit. If any system fails during preparation, the coordinator tells both to roll back. It's a robust, but more complicated, solution.

Strategy 3: Eventual Consistency and Background Jobs

If you have a system where perfect immediate consistency isn't strictly required, you can leverage eventual consistency and use background jobs to clean up orphaned files. This approach doesn't prevent orphaned files from appearing initially, but it ensures they are eventually removed. This is the last resort solution if you couldn't avoid the orphaned files by using the previous methods.

Implementation

1. Upload the file to S3.
2. Insert the file information to your database.
3. Create an event to your queue (such as RabbitMQ, Redis Queue, or similar) to check if the file exists and has a corresponding entry. This event can be triggered after a certain amount of time. For example, 10-20 minutes.
4. Create a consumer. The consumer will then check if a file is in S3, but there's no entry in the database (or vice-versa). If a discrepancy is found, it can be fixed. The orphaned file is then deleted from S3, or the database entry is created.

Considerations

• You will need a job queue system like BullMQ, or similar.
• The database query for finding orphaned files must be optimized to avoid performance issues.
• The delay introduced by the queue must be considered and can be tuned based on your needs.

Strategy 4: Using a Dedicated File Management Service

Instead of handling file uploads and database entries directly in your application, consider using a dedicated file management service. Services like Cloudinary or Filestack offer features like file uploads, transformations, storage, and content delivery. They also provide APIs for managing files and integrating with your database. These services usually handle the file upload and database integration, reducing the chance of orphaned files.

Advantages

• Simplified Implementation: You don't need to write custom code for S3 integration, file validation, or image transformation. You use ready-to-go APIs and the implementation is simpler.
• Scalability: File management services are designed to handle a large amount of files and traffic.
• Built-in Features: They often come with features such as image optimization, CDN integration, and more.

Regular Audits and Monitoring: Staying Vigilant

No matter which strategies you choose, it is always super important to be vigilant, perform regular checks, and establish monitoring systems. Regular checks will help you catch any orphaned files that may have slipped through the cracks. It is always better to be careful and test your solution frequently.

Periodic Audits

• Database Integrity Checks: Implement scheduled jobs to check your database for entries that point to files that don't exist in S3, or vice-versa. This can be automated to run daily, weekly, or whatever frequency makes sense for your application.
• S3 Inventory Reports: Use S3 inventory reports to list all the files in your bucket. Then, compare the file list with your database records. This will help you find discrepancies.

Monitoring

• Implement Logging: Log all file upload and database operations. This will help you to monitor the process and troubleshoot any issues.
• Set Up Alerts: Create alerts that notify you if the number of orphaned files exceeds a certain threshold. This will help you catch problems early on.
• Monitoring Tools: Use monitoring tools such as Prometheus and Grafana to visualize and monitor your system's behavior.

Conclusion: Keeping Your System Clean

Preventing orphaned files in S3 is essential for building a reliable and cost-effective service. Transactions, two-phase commits, eventual consistency with background jobs, and using dedicated file management services are all effective strategies. Choose the strategy that best suits your needs. By using transactions to ensure atomic operations, setting up background jobs to reconcile discrepancies, and continuously monitoring your system, you can keep your S3 bucket clean and your application running smoothly. Remember to implement regular audits and monitoring to catch any issues that might slip through the cracks. Keep experimenting, improving, and refining your approach. Happy coding!