5

Resolved

ReadAsMultipartAsync returns result before all files have been written

description

We trying to perform validation of the files which have been processed by ReadAsMultipartAsync, but sometimes on the attempts to open file for reading IOException is throwing, like this:

The process cannot access the file 'C:\Windows\TEMP\BodyPart_abd1efe9-b1d5-4956-b79b-0519837e9377' because it is being used by another process.","StackTrace":" at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.IO.File.OpenRead(String path)

at ...

at System.ComponentModel.DataAnnotations.ValidationAttribute.GetValidationResult(Object value, ValidationContext validationContext)
at System.Web.Http.Validation.Validators.DataAnnotationsModelValidator.Validate(ModelMetadata metadata, Object container)
at System.Web.Http.Validation.DefaultBodyModelValidator.ShallowValidate(ModelMetadata metadata, ValidationContext validationContext, Object container)
at System.Web.Http.Validation.DefaultBodyModelValidator.ValidateProperties(ModelMetadata metadata, ValidationContext validationContext)
at System.Web.Http.Validation.DefaultBodyModelValidator.ValidateNodeAndChildren(ModelMetadata metadata, ValidationContext validationContext, Object container)
at System.Web.Http.ModelBinding.FormatterParameterBinding.<>c__DisplayClass1.<ExecuteBindingAsync>b__0(Object model)
at System.Threading.Tasks.TaskHelpersExtensions.<>c__DisplayClass40.<ToAsyncVoidTask>b__3f()
at System.Threading.Tasks.TaskHelpers.RunSynchronously[TResult](Func`1 func, CancellationToken cancellationToken)

Example code:

var provider = new MultipartFormDataStreamProvider(_uploadPath);
return content.ReadAsMultipartAsync(provider)
 .ContinueWith(multiPartTask =>
      {
            if (multiPartTask.IsCanceled || multiPartTask.IsFaulted)
            {
                  Trace.TraceError("Failed multipart message reading: {0}", multiPartTask.Exception);
                  return null;
            }

             // ...
             // use files in the task Result
             // using (var fileStream = File.OpenRead(fileName)) {}

             return null;
      });
When processing time is delayed by something, eg Thread.Sleep, count of the io exceptions decreased, but it's not workaround.
How can we get access to the files, when it's actually written?

file attachments

comments

dtretyakov wrote Jul 17, 2012 at 11:59 AM

I suppose that this behaviour caused of next code segment in the HttpContentMultipartExtensions.cs, which does not waiting for a IO completition:

// Start async read/write loop
MultipartReadAsync(context);

HenrikN wrote Jul 17, 2012 at 2:09 PM

We have seen this also and it is a tricky issue. First check that there is no other process such as a virus control or indexing service that access the file.

However, even with these turned off we have seen that the file sometimes is in use. We believe there may be a race very deep in the stack way below us which we are not in control over and can't guarantee won't get triggered. We have brought this up with the .NET base team and the only workaround we have received is to have a loop with a try/catch that tries to access the file until it doesn’t hit an access denied exception upon access.

PS: I include somebody from the .NET base team in case they have an updated advice.

Hope this helps,

Henrik

dtretyakov wrote Jul 17, 2012 at 4:08 PM

Thanks for information, but we've found this issue on the Azure WebRole without any virus control, so it can't be cause of that exception. Also, in our case we trying to read file, so IOException caused by some process which has write operation permission.

We're looking forward for solving this issue by the .NET base team ;)

HenrikN wrote Jul 17, 2012 at 4:47 PM

In the meantime the best workaround is to have a try/catch loop that catches the exception and retries :)

nealt wrote Jul 20, 2012 at 5:16 PM

Added sample project. Load can be generated via LoadTest project (VS Ultimate) or by something like JMeter.

roncain wrote Jul 23, 2012 at 12:16 PM

The attachment demonstrates this is the same issue we saw intermittently, and we have been working with the framework team within Microsoft to address an async race condition we discovered in the IO layers below WebApi.

There is occasionally a small period of time immediately after the multipart code has closed the file before it is available to be open or deleted. We saw 1%-2% failure rate as the load increased. The most reliable workaround we have found is the one Henrik mentioned -- in the try/catch logic you have in your attachment, if you detect failure, delay a brief period and retry (allow for multiple retries). We generally saw recovery after a tiny delay (5-10ms) followed by a retry. In our testing, it recovered 100% of the time using the retry strategy.

Caution -- plan for the 99% case and do not delay unless failure occurs, or you will unnecessarily hurt performance.

It's unfortunate but apparently it has been there awhile in the async I/O layers. Sorry for the inconvenience.

zacharypierce wrote Aug 9, 2012 at 6:08 AM

Below is a simple function that implements the core portion of the current (entirely unsatisfactory) workaround of simply retrying to open the file after a delay. A more sophisticated version might allow the passing of some closure to be executed within the "using" block and more carefully targeted exception handling.

{{

/// <summary>
/// Attempts to open the file at <code>filePath</code> up to <code>maxRetries</code> times,
/// with a thread sleep time of <code>sleepPerRetryInMilliseconds</code> between retries
/// and returns true if the file was readable.
/// </summary>
/// <param name="filePath">A local file path</param>
/// <param name="maxRetries">The maximum number of attempted file reads</param>
/// <param name="sleepPerRetryInMilliseconds">The amount of time the current thread will sleep between retries</param>
/// <returns>true if the file specified was readable within the allowed number of retries, false otherwise</returns>
private static bool RetryUntilFileReadable(string filePath, int maxRetries, int sleepPerRetryInMilliseconds)
{
var retries = maxRetries;
var fileReadable = false;
while (!fileReadable && retries > 0)
{
    try
    {
        using (FileStream fileStream = File.OpenRead(filePath))
        {
            fileReadable = true;
        }
    }
    catch (Exception)
    {
        retries--;
        Thread.Sleep(sleepPerRetryInMilliseconds);
    }
}
return fileReadable;
}

}}

roncain wrote Aug 9, 2012 at 12:45 PM

I recommend a slight change to the approach, exactly as you suggested, based on what I discovered when trying different work-arounds. I recommend you pass in a delegate you call from within the loop and refactor the method to something like this:
bool TryIO(Func<object> callback, out object result) {} // retry count and delay omitted for brevity
where a true return indicates success, and the result 'out' param is what the Func returned when no exception occurred. You call 'callback' inside the Try/Catch and return true if no exception occurs.

The caller would do something like:
 FileStream fs = null;
 if (TryIO(() => File.Open("xyz"), out fs)) { using (fs) { fs.Read(...); } }
Why this approach? I found that doing additional I/O unrelated to what the caller was actually trying to do increased the failure rate slightly. We're talking something like .001% here, and the retries ultimately recover, but the number of retries increases, and that is an unnecessary perf hit.

The Func approach says "try to do exactly this one I/O operation until it succeeds or we max out the retries."

Again, sorry for the inconvenience. We spent significant effort trying to find a way in WebAPI framework code to protect callers from this issue. You can repro this without WebAPI. It may be small consolation, but the new async I/O methods in .Net 4.5 do not have this issue.

Final thought -- even though the async I/O uncovered this issue, it is still worth doing I/O asynchronously, even in your app code (i.e. don't do File.Open() and ReadAllText() -- use the async approach). Blocking the current thread doing synchronous I/O on the server will reduce the number of concurrent requests you can process.

HongmeiG wrote Aug 13, 2012 at 4:55 PM

We can add the workaround at our layer.

HenrikN wrote Aug 29, 2012 at 10:47 PM

This issue has been fixed with commit http://aspnetwebstack.codeplex.com/SourceControl/changeset/bdbcf01b84c9

The issue is due to a race condition which happens as a result of a race condition between asynchronously opened files and the traditional Begin/End asynchronous pattern (APM) for writing to the file. The result is that even though we think the file is closed before handing control back to the user, the file handle is still in use below us resulting in an IOException when accessed by the user.

The workaround is that we wrap the FileStream with a delegating stream that handles BeginWrite/EndWrite using the Task.Factory.FromAsync wrappers. Due to the way Task wraps APM Begin/End patterns this gets around the race condition.


** Closed by HenrikN 08/29/2012 3:47PM

HenrikN wrote Aug 29, 2012 at 10:48 PM

Meant to mark as fixed, not closed

benjoyce wrote Jan 6 at 5:08 PM

Hi all. I seem to be getting this error too but zacharypierce's solution, "RetryUntilFileReadable", seems to work for me. However; should this still be happening in 4.5?

I can pretty much reproduce this behaviour when needed so am happy to test alternative/official solutions.

Cheers,

Ben

Sign in to add a comment or to set email notifications