$select in nightlies throwing EntityFramework stack overflow

Topics: ASP.NET Web API
Apr 29, 2013 at 5:51 PM
I'm using DTOs that get selected into from the database so the methods return IQueryable<SomeDTO> which is then being further selected upon by the http query string in the select.

It doesn't throw on it being invalid, but it does throw on a stack overflow that crashes the entire web api site.

What am I doing wrong? (I haven't changed anything and I'm using my own json formatter from previous samples to clean things up.)
Apr 29, 2013 at 6:30 PM
Geminiman wrote:
I'm using DTOs that get selected into from the database so the methods return IQueryable<SomeDTO> which is then being further selected upon by the http query string in the select.

It doesn't throw on it being invalid, but it does throw on a stack overflow that crashes the entire web api site.

What am I doing wrong? (I haven't changed anything and I'm using my own json formatter from previous samples to clean things up.)
Could you share more details and a repro if that is possible? Details like, if you are building an OData service, which formatter are you using, what is the query string, what is the backend database (EF or in-memory or some other thing)?

Thanks,
RaghuRam.
Apr 29, 2013 at 6:30 PM
raghuramn wrote:
Geminiman wrote:
I'm using DTOs that get selected into from the database so the methods return IQueryable<SomeDTO> which is then being further selected upon by the http query string in the select.

It doesn't throw on it being invalid, but it does throw on a stack overflow that crashes the entire web api site.

What am I doing wrong? (I haven't changed anything and I'm using my own json formatter from previous samples to clean things up.)
Could you share more details and a repro if that is possible? Details like, if you are building an OData service, which formatter are you using, what is the query string, what is the backend database (EF or in-memory or some other thing)?

Thanks,
RaghuRam.
Stack trace would help too.
Apr 29, 2013 at 6:33 PM
Backend database is SQl server 2012 with Entity Framework (latest nightly).

I'm not directly building an odata service. We perform a great deal of logic in the background on our queries and then pass those along for odata querying (i.e. $filter, $orderby etc.) We return DTOs instead of the actual entities.

We're using a custom MediaTypeFormatter that we built using json.net.
Apr 29, 2013 at 6:53 PM
Could you share a stack trace or a repro?
Also, do you customize json.net settings? like JsonSerializerSettings.ReferenceLoopHandling? If so, could you share your settings?

You could email it to me at ranadimi AT microsoft DOT com as well.
Apr 29, 2013 at 7:02 PM
System.ArgumentException: Object of type 'System.Data.Entity.Infrastructure.DbQuery1[System.Web.Http.OData.Query.Expressions.SelectExpandWrapper1[Tradepoint.API.DTO.Contacts.ContactDTO]]' cannot be converted to type 'System.Linq.IQueryable1[Tradepoint.API.DTO.Contacts.ContactDTO]'.
at System.Web.Http.ApiController.<InvokeActionWithExceptionFilters>d__a.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter
1.GetResult()
at System.Web.Http.Tracing.Tracers.HttpControllerTracer.<ExecuteAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at System.Web.Http.Tracing.ITraceWriterExtensions.<TraceBeginEndAsyncCore>d__21
1.MoveNext()
iisexpress.exe Information: 0 : Message='Will use new 'JsonNetFormatter' formatter', Operation=JsonNetFormatter.GetPerRequestFormatterInstance
iisexpress.exe Information: 0 : Message='Selected formatter='JsonNetFormatter', content-type='application/json'', Operation=DefaultContentNegotiator.Negotiate
iisexpress.exe Information: 0 : Operation=CorsMessageHandler.SendAsync, Status=500 (InternalServerError)
iisexpress.exe Information: 0 : Response, Status=500 (InternalServerError),


JsonSerializerSettings has formatting = indented and constructorhandling = AllowNonPublicDefaultConstructor. Otherwise nothing else is set.

It uses GetPerRequestFormatterInstance.

Here's the code for the writeToStreamAsync:
public override Task WriteToStreamAsync(Type type, object value, Stream writeStream, System.Net.Http.HttpContent content, TransportContext transportContext)
        {
            string callback;
            //This fixes a bug in javascript to do with returning json arrays
            //Instead of returning an array we simply return odata format which we'll need eventually anyhow. This lends itself to getting the count in the future for paging.
            // See here: http://haacked.com/archive/2009/06/25/json-hijacking.aspx
            var response = new BaseResponse<object>();

            if (value == null)
            {
                response.results = new object[] { };
            } 
            else if (value.GetType() == typeof(HttpError))
            {
                response.results = null;

                var errors = value as HttpError;

                response.errors = new ResponseError[] { new ResponseError
                {
                    ErrorType = (errors.FirstOrDefault(e => e.Key == "ExceptionType").Value ?? "Message").ToStringEx(),
                    Description = (errors.FirstOrDefault(e => e.Key == "ExceptionMessage").Value ?? errors.First().Value).ToStringEx()
                }};
                    
            } 
            else if (type == typeof(IQueryable) || (type.IsGenericType && (type.GetGenericTypeDefinition() == typeof(IQueryable<>) || type.GetGenericTypeDefinition() == typeof(DbQuery<>))))
            {

                response.results = ((IQueryable<object>) value).ToArray();
                //response.__InlineCount = _request.GetInlineCount();
            }
            else if (type == typeof(IEnumerable) || (type.IsGenericType && (type.GetGenericTypeDefinition() == typeof(List<>) || type.GetGenericTypeDefinition() == typeof(IEnumerable<>)))) 
            {

                response.results = ((IEnumerable<object>) value).ToArray();
                //response.__InlineCount = _request.getInlineCount();
            }
            else
            {
                var al = new List<object>();
                al.Add(value);
                response.results = al.ToArray();
            }


            return Task.Factory.StartNew(() =>
            {
                using (StreamWriter streamWriter = new StreamWriter(writeStream, encoding))
                {
                    callback = null;
                    var jsonp = IsJsonpRequest(_request, out callback);
                    if (jsonp)
                        streamWriter.Write(callback + "(");

                    using (JsonTextWriter jsonTextWriter = new JsonTextWriter(streamWriter))
                    {
                        var serializer = JsonSerializer.Create(_serializerSettings);
                        serializer.Serialize(jsonTextWriter, response);

                        if (jsonp)
                            streamWriter.Write(")");

                    }
                }
            });

        }
Apr 29, 2013 at 7:25 PM
I will take a look at this issue.

But, you mentioned in your original post that you are getting a stackoverflow and it is crashing the whole site. The stack trace that you showed is different. Am I missing something here? Or is this a different issue?
Apr 29, 2013 at 7:34 PM
I thought it was the same issue. Turns out that it's a bug in the nightlies for Entityframework when there is an order by clause. Not related to this.
Apr 29, 2013 at 7:38 PM
Great. stackoverflow's are scary :). I will look into this issue asap. someone else reported it on codeplex issue tracker as well. https://aspnetwebstack.codeplex.com/workitem/1020
Apr 30, 2013 at 12:01 AM
I assume you are asking for application/xml and not using ODataFormatter. We only support JsonFormatter and ODataFormatter with $select and $expand right now.

Getting XmlFormatter to work with $select and $expand is a much harder job and was not done right now.
May 1, 2013 at 1:55 PM
I'm asking for application/json and using my own formatter that inherits from MediaTypeFormatter.
May 1, 2013 at 2:20 PM
Also, I removed my formatter for testing and went back to the default jsonformatter. This is the output when spitting out an IQueryable:

{"Message":"An error has occurred.","ExceptionMessage":"Object of type 'System.Data.Entity.Infrastructure.DbQuery1[System.Web.Http.OData.Query.Expressions.SelectExpandWrapper1[Tradepoint.API.DTO.Contacts.ContactDTO]]' cannot be converted to type 'System.Linq.IQueryable1[Tradepoint.API.DTO.Contacts.ContactDTO]'.","ExceptionType":"System.ArgumentException","StackTrace":" at System.Web.Http.ApiController.<InvokeActionWithExceptionFilters>d__a.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\r\n at System.Web.Http.Tracing.Tracers.HttpControllerTracer.<ExecuteAsyncCore>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\r\n at System.Web.Http.Tracing.ITraceWriterExtensions.<TraceBeginEndAsyncCore>d__211.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()\r\n at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__0.MoveNext()"}

Here's the signature for the get:

public IQueryable<ContactDTO> ListPhoneProviders() {}

And here's the Uri: /listphoneproviders?$select=ID,Name
May 1, 2013 at 2:48 PM
Should also note that this is occuring before the WriteToStreamAsync method is being called, so my customizations to the output shouldn't be effecting it.
May 1, 2013 at 8:14 PM
Edited May 1, 2013 at 8:15 PM
I could not repro this. The only way I could get that exception is if I am asking for application/xml or I have configured my JsonFormatter to use DataContractJsonSerializer. $select and $expand is only supported with json.net formatter and ODataFormatter right now.


The piece of code that is throwing this exception is this,
value = MediaTypeFormatter.GetTypeRemappingConstructor(type).Invoke(new object[] { value });
which exists in those two code paths(i.e XmlFormatter or JsonFormatter with DCJS).


Is it possible for you to share a repro?
May 1, 2013 at 8:15 PM
Do you mean that any class that uses DataContract would cause this? All of my classes use DataContract.
May 1, 2013 at 8:19 PM
No. If you do this,
config.Formatters.JsonFormatter.UseDataContractJsonSerializer = true;
it is expected to fail.
May 1, 2013 at 8:21 PM
I don't do that, and it is using a JsonFormatter. Creating a test case is a major deal. The guy that is also having the same problem in the issues looks like he's reproing for you, but basically with all defaults, if I return IQueryable, it fails with this error. I am not using the ODataFormatter, nor am I using the EntityController stuff either. I'm using simple API Controller which I would expect to work.
May 1, 2013 at 8:40 PM
Problem is I am not able to repro any of these. And these seem to be very basic. I am really curious to see what I am missing here.
May 2, 2013 at 5:00 PM
Edited May 2, 2013 at 5:01 PM
(updated here too - https://aspnetwebstack.codeplex.com/workitem/1020)

Finally, I realized what the issue is. Big thanks to garchibald for helping me isolate the issue.

Looks like you are requesting that url from your browser(chrome?) and the browser sends an accept header that looks like this,
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Notice that it is asking for application/xml which we don't support right now with $select and $expand. Could you try the same url from fiddler or some other client that doesn't ask for xml. IE 10 should work too as it sends this,
Accept: text/html, application/xhtml+xml, */*
May 2, 2013 at 5:28 PM
Also fails in fiddler with:
User-Agent: Fiddler
Accept: application/json

And jquery with Accept: application/json.