File Storage Providers for .NET

On almost every application I can recall working on has had to deal with uploading and downloading files.  Most with various requirements for storing or processing those files.  I've had to save files into a database column, upload to Amazon S3, queue for OCR processing, resize, encrypt and countless other things. Regardless of what I've been asked to do I always have to persist the file somewhere.  

When working with file persistence there are a few patterns I tend to follow in most of  my projects.

  1. Save files locally when developing the code.  I don't need some random picture I pulled off of my phone showing up in a clients S3 bucket.
  2. Store the file meta data in a database table with a field for the file path.
  3. I want to be able to use the same concepts across file services used by different projects.

Recently I started using a simple"IStorageProvider" interface for local and service based storage implementations across projects.

IStorageProvider Interface

The "IStorageProvider" interface is pretty simple it allows me to upload and retrieve a file.

public interface IStorageProvider
{
    bool HasPreSignedUrl { get; }

    Task<string> UploadFile(Stream fileStream, MediaItem item);

    Task<Stream> GetFileStream(string storagePath);

    Task<string> GetPreSignedUrl(MediaItem item, string storedPolicyName = null);
}
IStorageProvider
The PresignedUrl methods are provided for retrieving files from services like Amazon S3

The "MediaItem" model my look different across projects but will usually take a form similar to the following class.

public class MediaItem: BaseModel
{
  public string FileName { get; set; }
  
  public long FileSize { get; set; }
  
  public string ContentType { get; set; }
  
  public string StoragePath { get; set; }
}
MediaItem.cs

Using the Storage Provider

Using the storage provider is as simple as injecting "IStorageProvider" and uploading or retrieving a file.

The "ModelMediaController" shows an example of uploading multiple files in ASP.NET and using the storage provider to save the file contents.

public class MediaController
{
    protected IStorageProvider StorageProvider { get; }
    protected ApplicationDbContext DataContext { get; }

    public MediaController(ApplicationDbContext dataContext, IStorageProvider storageProvider) : base(context)
    {
      	DataContext = dataContext;
        StorageProvider = storageProvider;
    }
    
    [HttpGet("{id}")]
    [ActionName("GetMedia")]
    public async Task<IActionResult> GetMedia(Guid id)
    {
      var item = await DataContext.Media.SingleOrDefaultAsync(x => x.Id == id);

      if (item == null)
      {
        return NotFound();
      }

      if (StorageProvider.HasPreSignedUrl)
      {
        return Redirect(await StorageProvider.GetPreSignedUrl(item));
      }

      return File(await StorageProvider.GetFileStream(item.StoragePath), item.ContentType);
    }

 	[HttpPost]
	public async Task<IActionResult> AddMediaItem(MediaPostModel model)
   	{
        var results = new List<MediaViewModelWithUrl>();

        for(var i = 0; i < model.Files.Count; i++)
        {
          var file = model.Files[i];

          var entity = new MediaItem
          {        
            FileName = file.FileName.ToLower(), 
            ContentType = file.ContentType, 
            FileSize = file.Length       
          };

          await StorageProvider.UploadFile(file.OpenReadStream(), entity);

          DataContext.MediaItems.Add(entity);

          await DataContext.SaveChangesAsync();

          var path = Url.Action("GetMedia", "Media", new {id = entity.Id});
          var url =$"{ControllerContext.HttpContext.Request.Scheme}://{ControllerContext.HttpContext.Request.Host}{path}";

          results.Add(new MediaViewModel(entity, url));
        }

        return Ok(results);
  	}
}

When creating URL's to view a media item I have chosen to always send the user back to my server and then redirect to a presigned url if the storage provider supports it. The main reason I do this instead of sending a presigned url the first time is because I want to keep the presigned url lifetime short.  I have client applications that will pull down a list of media items to a mobile device and may not request the contents of the files until hours later and my presinged urls would have expired by then.

The supporting model classes can be pretty simple.

public class MediaPostModel
{
    [Required]
    public List<IFormFile> Files { get; set; }
}
public class MediaViewModel
{
    public MediaViewModel(MediaItem entity, string url)
    {
        //map entity properties
        Url = url;
    }

    public string Id { get; }
    public string FileName { get; }
    public string Url { get; }
    //... and more
}

Local Provider Implementation

Here's how I've implemented a local storage provider

public class LocalStorageProvider : IStorageProvider
    {
        private readonly ILogger<LocalStorageProvider> _logger;
        private readonly AppConfig _appConfig;
        
        public LocalStorageProvider(IOptions<AppConfig> options, ILogger<LocalStorageProvider> logger)
        {
            _logger = logger;
            _appConfig = options.Value;
        }

        public bool HasPreSignedUrl { get; } = false;

        public Task<Stream> GetFileStream(string storagePath)
        {
            var path = Path.Combine(_appConfig.LocalStoragePath, storagePath);
            if (!File.Exists(path))
            {
                throw new FileNotFoundException($"Item not found at storage location {storagePath}");
            }

            return  Task.FromResult<Stream>(File.OpenRead(path));
        }

        public Task<string> GetPreSignedUrl(MediaItem item, string storedPolicyName = null)
        {
            throw new System.NotImplementedException();
        }

        public async Task<string> UploadFile(Stream fileStream, MediaItem item)
        {
            var storageDirectory = Path.Combine(_appConfig.LocalStoragePath, item.Id.ToString());
            if (!Directory.Exists(storageDirectory))
            {
                Directory.CreateDirectory(storageDirectory);
            }
            
            var storagePath = Path.Combine(item.Id.ToString(), item.FileName);
            var path = Path.Combine(_appConfig.LocalStoragePath,storagePath);

            if (File.Exists(path))
            {
                _logger.LogInformation($"Deleting existing file at path {path}");
            }
            
            using (var writeStream = File.OpenWrite(path))
            {
                await fileStream.CopyToAsync(writeStream);
            }
            
            item.StoragePath = storagePath;
            return storagePath;
        }
    }
LocalStorageProvider.cs

S3StorageProvider

And my Amazon S3 provider

public class S3StorageProvider : IStorageProvider
{
    private AppConfig _appConfig;
    public S3StorageProvider(IOptions<AppConfig> options)
    {
    	_appConfig = options.Value;
    }

    public Task<Stream> GetFileStream(string storagePath)
    {
    	throw new System.NotImplementedException("Use PreSigned urls for accessing S3 content");
    }

    public bool HasPreSignedUrl { get; } = true;

    public Task<string> GetPreSignedUrl(MediaItem item, string storedPolicyName = null)
    {
        var client = new AmazonS3Client(RegionEndpoint.GetBySystemName(_appConfig.S3Region));
        var request = new GetPreSignedUrlRequest()
        {
        BucketName = _appConfig.S3Bucket,
        Key = item.StoragePath,
        Expires = DateTime.UtcNow.AddMinutes(5)
        };

        var url = client.GetPreSignedURL(request);

        return  Task.FromResult<string>(url);
    }

    public async Task<string> UploadFile(Stream fileStream, MediaItem item)
    {
        var storagePath = Path.Combine(item.Id.ToString(), item.FileName);

        var client = new AmazonS3Client(RegionEndpoint.GetBySystemName(_appConfig.S3Region));
        var transferUtility = new TransferUtility(client);

        await transferUtility.UploadAsync(fileStream, _appConfig.S3Bucket, storagePath);

        item.StoragePath = storagePath;
        return storagePath;
    }
}

Other Scenarios

Example code is rarely complete. So, here are some ways we could expand or improve on this topic.

  • Create a storage provider for another backend like Azure Blob Storage
  • Create a NuGet package for the interface and storage providers implementations to improve reuse across projects
  • Implement "GetPresignedUrl" for the local storage provider
  • Could this pattern be used when uploading large files in chunks?

Learn More

Cover image  by Maarten van den Heuvel on Unsplash