OData - Eta?

Topics: ASP.NET Web API
Aug 16, 2012 at 6:22 PM

What's the word on getting this back in at least in beta?

At least there was some of it before and it was relatively easy to add $select into what was there. No in RTM it's completely screwed.

Coordinator
Aug 16, 2012 at 11:28 PM

Hi Gemini,

We did pull out a few items from Web API RTM so that we could give them a more complete treatment with the features that we felt most customers required. For OData, we did an out-of-band preview release of the bits (with a lot more functionality) on NuGet.

You can read more about it on Henrik's blog post:

http://blogs.msdn.com/b/henrikn/archive/2012/08/15/asp-net-web-api-released-and-a-preview-of-what-s-next.aspx

And you can read a lot more about it on Alex's blog post :)

http://blogs.msdn.com/b/alexj/archive/2012/08/15/odata-support-in-asp-net-web-api.aspx

Thanks,

Eilon

Aug 17, 2012 at 3:09 PM

Good that it's getting done. Bad the way it's implemented right now... (i.e. its' terminally useless in it's current state!)

1. It's way too geared to Entity Framework. It shouldn't care when returning results if there is a key or not as it dosn't matter. Only an idiot would return EntiyFramework root objects on a publicly facing website. Everyone else knows to use DTOs. Thus expecting that we're going to return the entire product object or even a filtered version is nuts. Please make sure that you make this thing not care about EF. There is no reason for it to have any dependency AT ALL on EF. That's the point to linq.

2. It doens't support $inlinecount=allpages. If you pass that in the query it blows up despite the docs on Odata. Also I shouldn't have to do anything in code for this to return. It should return automatically because it's passed and it has the non-filtered query. - This means that most client implementations will blow up.

3. ODataResult (which shouldn't be needed because your oddata stuff has the IQueryable, it has the next resource because it's known based on the request, and it has the count because it has the root IQueryable before any odata filtering, top etc.) blows up if you pass a simple IQueryable telling you that it can't map IEnumerable properly.

4. It STILL DOESN'T SUPPORT $select!!! The second most important odata operation after $filter and it still isn't supported!!!! (yes that's right folks, you're still returning SELECT * on everything!!!!)

5. It blows up if $format=json is passed which again is entirely standard and shouldn't throw. This means that most client implementations will blow up.

6. Doesn't appear to be inheritable to fix any of these MAJOR issues from the outside.

I understand that this was ported from the WCF RIA services, but this is a major step back from the old stuff that we had in the RC. At least that we could override and fix most of the problems and it worked with any IQueryable the way it should have. This is hooked to EF AND you can't override any of it from what I can tell (there is no protected elements to be able to hook into in the QueryableAttribute when you inherit from it)

Aug 17, 2012 at 4:02 PM

Compared to the RC, the Queryable attribute still works with any IQueryable. Please check out the ODataQueryableSample sample [1]:

This sample shows how to introduce OData queries in ASP.NET Web API using either the [Queryable] attribute or by using the ODataQueryOptions action parameter which allows the action to manually inspect the query before it is being executed. The CustomerController shows using [Queryable] attribute and the  OrderController shows how to use the ODataQueryOptions parameter.

The sample illustrates queries using $orderby, $skip, $top, any(), all(), and $filter

See [2] for more information about the samples.

Using the ODataQueryOptions allows you to get the query directly so that you can apply it yourself if you happen to not have an IQueryable handy.

In general, I would reiterate that this is an early preview of brand new bits. It isn't ported from WCF RIA services, but rather based on OData.lib which itself is a moving target. For example, the URI parsing is known not to be robust at the moment. That being the case, the best way to provide feedback is by logging issues, or even better to suggest direct code changes. The code is all there in the repository so nothing is hidden.

Henrik

[1] http://aspnet.codeplex.com/SourceControl/changeset/3e1332f46409
[2] http://blogs.msdn.com/b/henrikn/archive/2012/08/15/asp-net-web-api-sample-on-codeplex.aspx

Coordinator
Aug 17, 2012 at 4:07 PM
Edited Aug 17, 2012 at 4:08 PM

This is all excellent feedback. Thank you for trying out the bits!

This is a very early preview of the ASP.NET Web API OData support and it has many known limitations. You can find a detailed overview of what is currently working in the preview in this blog post. The purpose of getting this preview out early was to get great feedback like this so that we can make sure we are focusing on the right features and issues in preparation for shipping ASP.NET Web API OData later this year.

This code is actually not a port of the WCF RIA code. It is a new code base based on ODataLib and the new OData query parser in ODataLib.Contrib. We are now working in a tight partnership with the OData team to build first-class OData support for ASP.NET Web API.

Would you be willing to open issues in our Issue Tracker to track each of these problems and limitations? The source code for ASP.NET Web API OData is now part of the http://aspnetwebstack.codeplex.com open source project so you can follow our active development. I think you will find that we will make rapid progress on addressing many of these issues. We even have daily signed builds that you can try out as issues get addressed. 

Also, we are very open to accepting contributions so please feel free to send us a pull request! You can find instructions on how to contribute in the documentation on the site.

Thanks again!

Daniel Roth

 

Aug 17, 2012 at 4:10 PM

I've looked at the samples and ODataResult throws when passed a standard IQueryable that works just fine and can get results from it.

 

I understand it's an early preview, but it's going down the wrong direction IMHO.

1. I should be able to Inherit the queryableattribute and override it and inject code at any point so that I can handle whatever I wanted like you could with the old QueryableAttribute.

2. It should not throw on things that it doesn't recognize, especially because of #1 not being possible because otherwise you can inject your own filtering such as enabling $select like I did in an action filter to fix the Beta and RC.

3. There is absolutely no reason for ODataResult because: 1. You can get the count without it, and the count should be returned based on the $inlinecount property, nothing else. 2. You know the next Page URI by definition.  It is not required for it to know to return a full OData formatted result because that's done by the formatter. Thus this is redundant and just not a good idea no matter how you slice it.

4. Aside from the fact that ODataQueryOperations is also not inheritable so you can't make it parse other options that are passed, if you have [Queryable] at the top of the class it will throw on the other options that you might want to process anyhow so you can't use [Queryable] and then override something.

Aug 17, 2012 at 4:26 PM

I have added all of the issues that I have found so far.

I'm burried in a project (which this is causing major problems for because I'd worked around most of this stuff in the RC's QueraybleAttribute and while I knew this was coming i'd expected to be able to override and fix everything neccessary) ,but if I get time I may do a pull request and do some of this. I just don't want to be doing some of the major things like $inlinecount and the formatting of results if it isn't going to be used because I don't want to have proprietary code, I need it to work out of the box without having to build my own web stack every time there is an update.

Aug 17, 2012 at 4:41 PM

Dan, stupid question but where's the Microsoft.OData stuff in the repository? I did a git on the web api stack and it's not in there....

Aug 17, 2012 at 4:45 PM
Geminiman wrote:

Dan, stupid question but where's the Microsoft.OData stuff in the repository? I did a git on the web api stack and it's not in there....


http://aspnetwebstack.codeplex.com/SourceControl/changeset/897bf750a622 added those to the git repository.

Aug 17, 2012 at 4:48 PM
Edited Aug 17, 2012 at 4:51 PM
Geminiman wrote:

I have added all of the issues that I have found so far.

I'm burried in a project (which this is causing major problems for because I'd worked around most of this stuff in the RC's QueraybleAttribute and while I knew this was coming i'd expected to be able to override and fix everything neccessary) ,but if I get time I may do a pull request and do some of this. I just don't want to be doing some of the major things like $inlinecount and the formatting of results if it isn't going to be used because I don't want to have proprietary code, I need it to work out of the box without having to build my own web stack every time there is an update.


We have taken a dependency on the OData library (http://nuget.org/packages/Microsoft.Data.OData and http://nuget.org/packages/Microsoft.Data.OData.Contrib) to improve our OData support. As a part of the move we have removed our custom query parser (for $filter and $orderby) and replaced it with the one that the OData team is working on. The big change in this is that the old parser used to work straight out of CLR types where as the OData uri parser works with IEdmType's. We build an IEdmType implicitly out of your clr types. There are a couple of issues that we have found out in this translation with collection's and inheritance resulting in the errors that you are reporting. We are working on a fix and will soon push it to codeplex.

Also, we are working on getting daily builds out to public for these bits. Once that is done(very soon), you don't have to build webstack on your box.

 

Aug 17, 2012 at 4:59 PM

Thanks for the update. That will help, although I really don't think that ODataResult should be needed at all nor is desirable.

Coordinator
Aug 17, 2012 at 4:59 PM

Thank you for filing the issues. We are working on getting them resolved as soon as we can.

It is unfortunate that we weren't able to make these changes in our OData support publicly available on the CodePlex site sooner. Adding code to our open source project does take some time due to the requisite legal reviews (yes, I am blaming this on the lawyers :o). Fortunately, now that we are part of the open source project we can work in a fully transparent fashion.

We don't want you to have to write a bunch of proprietary code either, so please consider contributing any work you do to get your scenarios running. We can then support these scenarios directly in the framework and everyone in the community benefits. To help ensure that any pull request work you do makes it smoothly into the product please just be sure to follow the guidance on contributing in our docs (open an issue in issue tracker, create a discussion to let us know what you plan on doing, etc).

Daniel Roth

Aug 17, 2012 at 5:10 PM
Geminiman wrote:

I've looked at the samples and ODataResult throws when passed a standard IQueryable that works just fine and can get results from it.

 

I understand it's an early preview, but it's going down the wrong direction IMHO.

1. I should be able to Inherit the queryableattribute and override it and inject code at any point so that I can handle whatever I wanted like you could with the old QueryableAttribute.

We have noticed that everytime one wants to do some custom query composition having to extend QueryableAttribute is unnecessary. Now we have ODataQueryOptions for these scenarios. A sample can be found here.

2. It should not throw on things that it doesn't recognize, especially because of #1 not being possible because otherwise you can inject your own filtering such as enabling $select like I did in an action filter to fix the Beta and RC.

This was an explicit design choice that we made for QueryableAttribute to be complaint with the OData spec. The OData protocol spec says that $query-parameters are special and that the server should throw if it doesn't understand them. That said, we knew people are going to have custom query composition and introduced ODataQueryOptions for those scenarios. ODataQueryOptions doesn't throw on parameters it doesn't understand.

3. There is absolutely no reason for ODataResult because: 1. You can get the count without it, and the count should be returned based on the $inlinecount property, nothing else. 2. You know the next Page URI by definition.  It is not required for it to know to return a full OData formatted result because that's done by the formatter. Thus this is redundant and just not a good idea no matter how you slice it.

The ODataResult was meant for people wanting to implement custom server driven paging in their OData services. I will post a sample sometime soon. we will work on this feedback to improve this class.

4. Aside from the fact that ODataQueryOperations is also not inheritable so you can't make it parse other options that are passed, if you have [Queryable] at the top of the class it will throw on the other options that you might want to process anyhow so you can't use [Queryable] and then override something.

Agreed. ODataQueryOptions.ApplyTo should have been virtual. Also consider ODataQueryOptions.RawValues which contains an instance of ODataRawQueryOptions which contains raw values of all the $ odata parameters.


This is excellent feedback for the team. Thanks a lot for it and trying out the bits. Look out for updates to the code from the team in the next few days :)

Aug 17, 2012 at 6:20 PM

reghuramn: 

 

#1: That would be fine, but if you have Queryable on your class and there is one case you want to override then it will still throw an error in the QueryableAttribute that it's not implemented on that new $xxx filter. Further there needs to be a way to do this globally which is why overriding QueryableAttribute is so powerful because I don't want to have to do this constantly. (i.e. all of our objects have 3 fields. I want to be able to work on the interface version knowing these are there and I don't want to have to do it over and over again in code iwth the ODataqueryOptions on every get method.

#2: Yes but if Queryable is there as well it will throw. See above. Not to mention the other case. At the very least the Queryable attirbute should be not throwing if the method signature that it came from has ODataQueryOptions... but it would be much better if this was simply extendable as per above.

#3: The problem I see is that right now that's the only way to get inlinecount to work (doesn't really work but...) which isn't a good idea. I should be able to ignore this and just have $inlinecount=allpages in my request and volia it should work and return everything including the next page URI info because it doesn't need to be passed explicitly, you know what it is going to be automatically based on the Request URI.

#4: Excellent. If there was some way of plugging in an overriden ODataRowQueryOptions into the QueryableAttribute that would solve the need to inherit from QueryableAttribute and override the processing as well because then we could do mass rules automatically by the fact that we specified our implementation of ODataRowQueryOptions as the default, which could then be overridden per Method as necessary if we wanted to. Make sense?

Aug 17, 2012 at 9:01 PM

Ok, so I have $select and $inlinecount implemented.  This also defaults a specific structure to the resultset so that it returns: 

{

   results: [data],

   __count: <some number>

}

 

Where __count is optional. This allows specifying metadata and anything else you want to inject into the result, and it also prevents the javascript bug that causes a massive security hole (this should have been the default in web api from the very beginning btw because of that)

Note that my $select implementation is last because it has to materialize the data before it selects. This is a hack, I'm sure that someone at MS will be able to look at my code an in 3 seconds be able to figure out why it freaks out on materialization with the select applied to an ef query that wasn't previously materialized and fix it for me because obviously it's non-optimal right now because it's still doing a select * against the database, and just returning only the relevant results, which is better than nothing but... Anyhow, that's what I threw together this afternoon because not having select was killing speed in our app. (quadrupling or more data just running unit tests!)

Now since I've never used GIT before and I just did a clone, how do I push this up properly?

Aug 17, 2012 at 9:05 PM

The guidelines are here at http://aspnetwebstack.codeplex.com/wikipage?title=Contributing .

 

 

 

Aug 17, 2012 at 9:11 PM

Thanks... I think. Going to take me longer to make sure that I follow all of the rules than it did to put this stuff in!

BTW, I'm sure you're aware, but I don't think your implementation of ODataQueryOptions can ever work for $inlinecount... I can't find anywhere where I can get at the value in the pipeline to alter the data that's spit out in anything other than within the QueryableAttribute. Thus it would never work in that case.... which could be a serious issue.

Aug 17, 2012 at 9:12 PM
Geminiman wrote:

reghuramn: 

 

#1: That would be fine, but if you have Queryable on your class and there is one case you want to override then it will still throw an error in the QueryableAttribute that it's not implemented on that new $xxx filter. Further there needs to be a way to do this globally which is why overriding QueryableAttribute is so powerful because I don't want to have to do this constantly. (i.e. all of our objects have 3 fields. I want to be able to work on the interface version knowing these are there and I don't want to have to do it over and over again in code iwth the ODataqueryOptions on every get method.

I understand your scenario for extending the QueryableAttribute. And as i understand it, the only blocker for extending the QueryableAttribute is the validation logic that we have in place for $query parameters. The class is not sealed and the OnActionXXXMethods are overridable. We will consider opening up the validation method as well so that you can support your own parameters.

#2: Yes but if Queryable is there as well it will throw. See above. Not to mention the other case. At the very least the Queryable attirbute should be not throwing if the method signature that it came from has ODataQueryOptions... but it would be much better if this was simply extendable as per above.

You dont need the queryable attribute if you are using ODataQueryOptions. You use only one of them.

#3: The problem I see is that right now that's the only way to get inlinecount to work (doesn't really work but...) which isn't a good idea. I should be able to ignore this and just have $inlinecount=allpages in my request and volia it should work and return everything including the next page URI info because it doesn't need to be passed explicitly, you know what it is going to be automatically based on the Request URI.

I think you are getting confused between server driven paging ang client controlled paging. The scenario that you are explaining is more like client driven paging where in the client sends $skip and $top. Automatically figuring out the next page link is very easy in these cases. Things get more involved with server driven paging. Think of scenarios like SQL cursors. Your datastore is getting updated continuosly and the client sends a query. You send some results back and the next page link. Meanwhile, something got updated in between but the next page link should still return the same results as if the query was done before the update happened. ODataResult aims to solve that problem.

All that said, we are working on simplifying the code to support the straight forward paging scenarios without any extra steps. Something like the ResultLimit option we used to have on the QueryableAttribute earlier. Keep looking for them :)

#4: Excellent. If there was some way of plugging in an overriden ODataRowQueryOptions into the QueryableAttribute that would solve the need to inherit from QueryableAttribute and override the processing as well because then we could do mass rules automatically by the fact that we specified our implementation of ODataRowQueryOptions as the default, which could then be overridden per Method as necessary if we wanted to. Make sense?

Perfect. We will work on this feedback.

Thanks a lot once again for all the feedback.

Aug 17, 2012 at 9:17 PM

#2: The common case where this will occur is that the Class is decorated with QueryableAttribute, and then in one case you want to override. If you do this you'll get a not supported error even though you are by definition supporting it with the options.  So you'd have to go through and manually add the Queryable attribute to every method call in the class AND remember to put it on any new ones. Just for contacts in our system I have 15 gets to do with different security scenarios and data mappings to prevent security holes. That's a lot of times to remember to put it on.

 

#3: Understood. Never thought about the forward only case from the server handling the paging from a stored procedure. Was really happy to not have to use 'em anymore with EF except in really wild cases... especially now that TVFs are supported in EF 5 :)

Aug 17, 2012 at 9:22 PM
Geminiman wrote:

#2: The common case where this will occur is that the Class is decorated with QueryableAttribute, and then in one case you want to override. If you do this you'll get a not supported error even though you are by definition supporting it with the options.  So you'd have to go through and manually add the Queryable attribute to every method call in the class AND remember to put it on any new ones. Just for contacts in our system I have 15 gets to do with different security scenarios and data mappings to prevent security holes. That's a lot of times to remember to put it on.

 Interesting. Haven't thought about this scenario. Could you log an issue with the issue tracker to support this scenario ?

#3: Understood. Never thought about the forward only case from the server handling the paging from a stored procedure. Was really happy to not have to use 'em anymore with EF except in really wild cases... especially now that TVFs are supported in EF 5 :)

 

Aug 17, 2012 at 9:32 PM

done! Thanks!

Aug 23, 2012 at 11:28 PM

For $filter/$sort queries, how do we deal with our output having potentially different names or casings vs. the model?

For example, if my model has a "Name" property but my JSON outputs the property as "name" I cannot use $filter=startswith(name,'Joe') - which my consumers would expect (I get an ODataException)

Jason

Aug 24, 2012 at 12:15 AM

Interesting scenario Jason. We have had a couple of asks around this. Unfortunately, the query parser (from ODataLib.Contrib) is right now case sensitive. We will work with the odata team to see if they can provide a flag to be cas-insensitive while parsing the query. Could you please file an issue on the issue tracker for this ?

Aug 24, 2012 at 3:17 AM

The other scenario that will be an issue is when you implement $select. Basically it works perfectly if the original query does not contain a projection, however if you do a projection on a projection it throws an error. Obviously this case needs to happen, because often you'll have a DTO that was projected into and then it will need to be further selected from there to limit the information being retrieved.

Aug 24, 2012 at 4:41 PM

@raghuramn,

Added issue 366

http://aspnetwebstack.codeplex.com/workitem/366

Mar 14, 2013 at 8:38 AM
See also http://aspnetwebstack.codeplex.com/workitem/820 where the compatibility with Excel PowerPivot was addressed.