Limit the amount of bytes allowed to be buffered in memory for WebAPI Multipart posts (ASP.NET Host).

Topics: ASP.NET Web API
Apr 22, 2014 at 9:17 PM
I didn’t find an option through the public api (skip all below if you know of one :)

I looked at the source and found an option in MimeMultipartBodyPartParser constructor - maxMessageSize parameter.

Calling ReadAsMultipartAsync methods in HttpContentMultipartExtensions which ultimately call:

public static async Task<T> ReadAsMultipartAsync<T>(this HttpContent content, T streamProvider, int bufferSize, CancellationToken cancellationToken) where T : MultipartStreamProvider

Which instantiates a MimeMultipartBodyPartParser:

using (var parser = new MimeMultipartBodyPartParser(content, streamProvider))
{

(Which could also be instantiated like this)

public MimeMultipartBodyPartParser(
HttpContent content,
MultipartStreamProvider streamProvider,
long maxMessageSize,
int maxBodyPartHeaderSize)

}
...
Which leads to a later call to MimeMultipartBodyPartParser ::ParseBuffer, checks and throws upon any bad state including DataTooBig
if (_mimeStatus != MimeMultipartParser.State.BodyPartCompleted && _mimeStatus != MimeMultipartParser.State.NeedMoreData)
{ …
throw Error.InvalidOperation(Properties.Resources.ReadAsMimeMultipartParseError, bytesConsumed, data);
}

So as quick workaround I thought to simply copy HttpContentMultipartExtensions::ReadAsMultipartAsync and change it to accept maxMessageSize and use the appropriate MimeMultipartBodyPartParser constructor.
I will handle the Error.InvalidOperation Exception if the limit is reached (Although this might also result due to bad data but I can live with that)

Any thoughts?

Thanks,

-Itai
Apr 23, 2014 at 3:40 PM
So for the meantime I figured an alternative approach that's less intrusive.

Basically, it involves enforcing the limit at the MultipartStreamProvider output stream by relying on the fact that data is written to it in chunks (e.g bufferSize, 32k by default, configurable).

However there are limitations to this approach:
  1. The amount of actual received data that overflows will be more than the limit (chunk fragment).
  2. MemoryStream will probably auto resize before the limit is reached so the actual allocated memory will be greater than the specified limit. However as far as I’m aware it doubles on resize which may or may not be an issue regardless of any limit enforcement.
  3. It doesn’t track rewrites (e.g stream seeks). I didn’t see such behavior by the framework in this scenario.
  4. There is one Stream base method public Task WriteAsync(byte[] buffer, int offset, int count) that is not virtual and since the contract is Stream it would be possible to bypass the output stream enforcement if the framework starts calling this method on the output stream (currently it doesn’t).
Nevertheless, I tested this second approach and I think it’s a sufficient guard that’s better than nothing.

I’m going to open an issue on this (probably the first approach would be better), code below, feedback welcome.

Regards,

-Itai


// POST api/upload
public async Task<HttpResponseMessage> Post()
{
if (!Request.Content.IsMimeMultipartContent("form-data"))
{
    throw new HttpResponseException(HttpStatusCode.UnsupportedMediaType);
}

try
{
    var provider = await Request.Content.ReadAsMultipartAsync(new MultipartGuardedMemoryStreamProvider(1024*1024));
}                       
catch (Exception ex)
{
    if (ex.InnerException != null && ex.InnerException is GuardedMemoryStreamOverflowException)
    {
        // BLL Logic

        return Request.CreateResponse(HttpStatusCode.RequestEntityTooLarge); 
    }

    throw (ex);
}

return Request.CreateResponse(HttpStatusCode.OK);
}

public class MultipartGuardedMemoryStreamProvider : MultipartStreamProvider
{
readonly long _maxWritablePosition;

public MultipartGuardedMemoryStreamProvider(long maxWritablePosition)
{
    _maxWritablePosition = maxWritablePosition;
}

public override Stream GetStream(HttpContent parent, HttpContentHeaders headers)
{
    if (parent == null)
    {
        throw new ArgumentNullException("parent");
    }

    if (headers == null)
    {
        throw new ArgumentNullException("headers");
    }

    return new GuardedMemoryStream(_maxWritablePosition);
}
}

public class GuardedMemoryStream : MemoryStream
{
private readonly long _maxWritablePosition;

public GuardedMemoryStream(long maxWritablePosition)
{
    _maxWritablePosition = maxWritablePosition;
}

private void ThrowOnOverflow(int expectedWriteLength)
{
    if (Position + expectedWriteLength > _maxWritablePosition)
        throw new GuardedMemoryStreamOverflowException(Position, expectedWriteLength, _maxWritablePosition, Length);
}

public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
{
    ThrowOnOverflow(count);

    return base.WriteAsync(buffer, offset, count, cancellationToken);
}

public override void Write(byte[] buffer, int offset, int count)
{
    ThrowOnOverflow(count);

    base.Write(buffer, offset, count);
}

public override IAsyncResult BeginWrite(byte[] buffer, int offset, int count, AsyncCallback callback, object state)
{
    ThrowOnOverflow(count);

    return base.BeginWrite(buffer, offset, count, callback, state);
}

public override void WriteByte(byte value)
{
    ThrowOnOverflow(1);

    base.WriteByte(value);
}
}


public class GuardedMemoryStreamOverflowException : Exception
{
public long Position { get; private set; }
public int ExpectedWriteLength { get; private set; }
public long MaxWritablePosition { get; private set; }
public long Length { get; private set; }

public GuardedMemoryStreamOverflowException(long position, int expectedWriteLength, long maxWritablePosition, long length)
{
    Position = position;
    ExpectedWriteLength = expectedWriteLength;
    MaxWritablePosition = maxWritablePosition;
    Length = length;
}
}