To repro:

public class Parent
{

public class Nest
{
    public string Name { get; set; }
}

]

Put Nest as action return type:
[QueryableAttribute]
public IEnumerable<Nest> Get()

Throw following exception:

System.InvalidOperationException occurred
Message=No EntitySet exists in the EdmModel with EntityClrType 'MvcWebApi.ODataTests.Models.Order+NestClass'.
Source=System.Web.Http.OData
StackTrace:

   at System.Web.Http.OData.ODataQueryContext..ctor(IEdmModel model, Type entityClrType) in C:\dd\DevDiv\Offcycle\WPT\WebStackRuntime\runtime\src\System.Web.Http.OData\OData\ODataQueryContext.cs:line 65

InnerException: