4

Closed

TransferEncodingChunked=true doesn't work on WebHost

description

When setting response.Headers.TransferEncodingChunked, the response doesn't get chunked, and a client (Fiddler) will fail trying to read the response.

Repro 1:
    public HttpResponseMessage Get()
    {
        HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
        response.Headers.TransferEncodingChunked = true;
        response.RequestMessage = Request;
        return response;
    }
Repro 2:
    public HttpResponseMessage Get()
    {
        HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
        response.Content = new LazyContent();
        response.Headers.TransferEncodingChunked = true;
        response.RequestMessage = Request;
        return response;
    }

    public class LazyContent : HttpContent
    {
        protected override Task SerializeToStreamAsync(Stream stream, TransportContext context)
        {
            stream.WriteByte(0);
            return Task.FromResult<object>(null);
        }

        protected override bool TryComputeLength(out long length)
        {
            length = 0;
            return false;
        }
    }
Closed Nov 15, 2013 at 9:47 PM by kichalla
Verified.

comments

kichalla wrote Jul 12, 2013 at 9:04 PM

Just FYI...WebHostBufferPolicySelector is the one which causes whether the responses are chunked or not. So the above scenario, you would need to override WebHostBufferPolicySelector to add a condition for not buffering LazyContent.

Following the code:
/// <summary>
        /// Determines whether the host should buffer the <see cref="HttpResponseMessage"/> entity body.
        /// </summary>
        /// <param name="response">The <see cref="HttpResponseMessage"/>response for which to determine
        /// whether host output buffering should be used for the response entity body.</param>
        /// <returns><c>true</c> if buffering should be used; otherwise a streamed response should be used.</returns>
        public virtual bool UseBufferedOutputStream(HttpResponseMessage response)
        {
            if (response == null)
            {
                throw Error.ArgumentNull("response");
            }

            // Any HttpContent that knows its length is presumably already buffered internally.
            HttpContent content = response.Content;
            if (content != null)
            {
                long? contentLength = content.Headers.ContentLength;
                if (contentLength.HasValue && contentLength.Value >= 0)
                {
                    return false;
                }

                // Content length is null or -1 (meaning not known).  
                // Buffer any HttpContent except StreamContent and PushStreamContent
                return !(content is StreamContent || content is PushStreamContent);
            }

            return false;
        }

davidmatson wrote Jul 16, 2013 at 11:46 PM

The problem actually repros regardless of the buffering policy. I've tried both of the following options:
config.Services.Replace(typeof(IHostBufferPolicySelector), new AlwaysBufferPolicySelector());
config.Services.Replace(typeof(IHostBufferPolicySelector), new NeverBufferPolicySelector());

private class NeverBufferPolicySelector : IHostBufferPolicySelector
{
    public bool UseBufferedInputStream(object hostContext)
    {
        return false;
    }

    public bool UseBufferedOutputStream(System.Net.Http.HttpResponseMessage response)
    {
        return false;
    }
}

private class AlwaysBufferPolicySelector : IHostBufferPolicySelector
{
    public bool UseBufferedInputStream(object hostContext)
    {
        return false;
    }

    public bool UseBufferedOutputStream(System.Net.Http.HttpResponseMessage response)
    {
        return false;
    }
}
Without that the following header set, the buffer policy selector does control chunking:
response.Headers.TransferEncodingChunked = true;

But with that header set, the response is not handled correctly with either of the buffer policies above.

kichalla wrote Jul 17, 2013 at 12:35 AM

The buffer policy of the host correctly/smartly handles the situation of whether a response should be chunked or not based on the existing logic…so ideally the user needn’t set the chunked encoding property explicitly…

For example:
if the content is already buffered (ex: downloading a file from the disk), then according to our policy, we do not send it chunked as we already know the content-length...in this case using chunked transfer encoding doesn’t make sense…If I remember correctly, we are supposed to send chunked encoding only when we do not know the content length upfront...for example, if we are streaming data from the service to the client as the data is received at the service from some other party (PushStreamContent example)…in case of PushStreamContent, we correctly set the chunked encoding header…

So, for any custom httpcontent that user creates, they would need to extend the buffer policy to either send data in chunked or non-chunked.

davidmatson wrote Jul 17, 2013 at 4:31 PM

Policy does handle most situations, but we shouldn't break the response just because the user sets a header value. If the user explicitly indicates chunking on a per-response basis, we should honor that header rather than ignore it or break the response.

yishaigalatzer wrote Sep 18, 2013 at 11:17 PM

How does this work for Owin host?

yishaigalatzer wrote Sep 18, 2013 at 11:45 PM

Kiran says this does work for Owin

DarrelMiller wrote Oct 15, 2013 at 10:03 PM

If you want to force users to use the IHostBufferPolicySelector to determine chunking, I guess I can live with that, but you shouldn't ever be returning the Transfer-Encoding: chunked if you are not actually chunking the response. Having fiddler fail on a response is not a reasonable solution.

damianh wrote Oct 28, 2013 at 10:43 AM

Works on owin when using HttpListener host, but not on SystemWeb host (IIS).

damianh wrote Oct 28, 2013 at 10:56 AM

(A) Problem seems to be in System.Web.Http.Owin.HttpMessageHandlerAdapter.FixUpContentLengthHeaders where is sets TransferEncodingChunked to true, but no actual chunking occurs, resulting in a 'hung' response: https://jabbrlive.blob.core.windows.net/jabbr-uploads/clipboard_3722.png

Commenting out the line: https://jabbrlive.blob.core.windows.net/jabbr-uploads/clipboard_ed95.png semi-worked... but resulted in a 'chunk' that is actually the entire body: https://jabbrlive.blob.core.windows.net/jabbr-uploads/clipboard_3027.png , when it should have looked like this (HttpListener host): https://jabbrlive.blob.core.windows.net/jabbr-uploads/clipboard_dd9e.png

Tratcher wrote Oct 28, 2013 at 11:00 PM

Yeah, the issue is in IIS, so Microsoft.Owin.Host.SystemWeb has the same problem. https://katanaproject.codeplex.com/workitem/148

davidmatson wrote Nov 13, 2013 at 6:00 PM

commit 39d1b2c7ce3951d12c9e6d5e892eb2be44ddbaac

davidmatson wrote Jan 24 at 6:38 PM

Fixed in changeset 39d1b2c7ce3951d12c9e6d5e892eb2be44ddbaac