A recent DBA asked me the question “I thought inline SQL was bad, so why are we using it again?” LINQ may *smell* like inline SQL, but it is not. Let’s first take a look at some simple LINQ that is easy to read:

from d in Devices
where d.CZone == 4
&& d.Type == "X"
select d.Id

So how is this different than inline SQL? To be technical, you’re writing a query against a data model, with full intellisense. You’re also writing a provider agnostic query. This same query can be performed against SQL Server, Oracle, or even the Facebook API if there was a supporting framework in place. We now have a truly unified query architecture.

Let’s keep taking about this simple LINQ query, and see how you would write it if you didn’t want to use LINQ. Most developers before the days of LINQ would probably use a stored procedure. Stored procedures are great. They’re efficient, reusable, and easily updatable. Here is what it may look like:

Create Procedure GetMyStuff
As
Select Id
From Devices
Where CZone = 4
And Type = 'X'
Select Id
GO

A nice, simple SQL query. There are a few disadvantages that may not be immediately apparent:

• If you need a second, similar query, you have to either have to create and maintain two stored procedures. As an alternative, you could modify the stored procedure to operate differently based on a parameter. Both of these options are not idea, but LINQ does give us an alternative that I’ll discuss in a bit.
• You don’t get intellisense when you’re writing your code.
• You have to be concerned with two different “programming” paradigms, and also have to manually manage the translation in both directions.

Now, let’s take our example to the other end of the spectrum, which will help show where LINQ can really shine where straight SQL does not. This example is a query for a search page. I set up a simple ASPX page to demonstrate. Here is a sample of the user interface:

The user enters a number of search criteria, and the results are displayed. I literally coded this in under 5 minutes. If you’re used to using stored procedures to retrieve this type of data, think about how you would go about creating this. You have a couple of options that I’m aware of:

• Write a separate stored procedure for every combination of parameters. In this case that would be 7 stored procedures. This would certainly not be ideal.
• Write a single stored procedure that can handle each parameter as nullable parameters, and use “Where @Param Is Null Or Param = @Param”. This option is easy, but has some potential performance implications.
• Write a single stored procedure that can handle each parameter as nullable parameter, and “If” statements to handle each scenario. This would be time consuming an error prone.

In LINQ, we’re able to dynamically build up a query. For the search example, my LINQ looks like this:

var dataContext = new DataClassesDataContext();

var query = (IQueryable<Device>) dataContext.Devices;

if (txtCZone.Text.Length > 0)
    query = query.Where(device => device.CZone == int.Parse(txtCZone.Text));
if (txtUCZone.Text.Length > 0)
    query = query.Where(device => device.UCZone == int.Parse(txtUCZone.Text));
if (txtLZone.Text.Length > 0)
    query = query.Where(device => device.LZone == int.Parse(txtLZone.Text));

dgResults.DataSource = query.ToList();
dgResults.DataBind();

And of course we can use the query syntax instead (replacing lines 5-10 above):

if (txtCZone.Text.Length > 0)
	query = from device in query where device.CZone == int.Parse(txtCZone.Text) select device;
if (txtUCZone.Text.Length > 0)
	query = from device in query where device.UCZone == int.Parse(txtUCZone.Text) select device;
if (txtLZone.Text.Length > 0)
	query = from device in query where device.LZone == int.Parse(txtLZone.Text) select device;

The result is that the SQL code is specifically written to support only the parameters that the user has entered. No extra SQL, and no specific SQL to maintain. Remember that LINQ can be chained together without querying the underlying data. The actual querying of the data only occurs when enumerating the results, using an operation like “ToList()”.

To support paging we need to run 3 different types of underlying queries. Here is where LINQ is really going to shine. We can use the same base query for all of these operations, and not have to worry about the drastically different underlying SQL statements.

1. Result count – Simply by calling the “.Count()” method, we can retrieve the number of rows the query will return in total. The underlying SQL will be a simple and efficient Count operation.
2. Page n query – By utilizing Skip and Take, any page within the results can be queried. The work of generating a common table expression is handled for you.
3. First page query – If the underlying provider has an optimization for using the SQL TOP command, the first page of data you query will be able to avoid a common table expression. This has the advantage of being more efficient when the first (and often most common) page of results is displayed.

Real-world Results

I initially ran into this in a real application that was primarily used to search through a large table of records. It had originally used the stored procedure approach, and was causing the entire system to slow down to the point of being unusable. Thanks to LINQ, we were able to make the search usable. In fact, the results were drastic:

In addition to the improved performance, the code was easier to maintain. The stored procedure was extremely cluttered, had large where clauses, and even contained two nearly identical copies of the query. One for calculating the count, and one for paging support.

Conclusion

LINQ gives us much more than “inline SQL”. It gives us a unified query syntax, delayed execution, query expression building, and dynamically created SQL output. Additionally, the generated queries are optimized based on the exact query being performed instead of making generic SQL that is optimized for multiple scenarios.

SQL Query Complexity Disproportional to LINQ Complexity

Recall the example from my last post:

//Query Syntax:
from device in Devices
where device.Type != null
select device.DeviceId

//SQL Syntax:
SELECT [t0].[DeviceId]
FROM [Devices] AS [t0]
WHERE [t0].[Type] IS NOT NULL

In this case, LINQ to SQL has done something wonderful. It’s saved us from having to understand or worry about the translation of syntax between C# and SQL. Now, what happens when we write something a little more advanced, such as a nested group by?

from d in Devices
group d by d.CZone into czoneGroup
select new { Key = czoneGroup.Key, val = from d2 in czoneGroup
	group d2 by d2.LZone into lzoneGroup
	select lzoneGroup.Key }

And the corresponding SQL:

SELECT [t0].[CZone] AS [Key]
FROM [Devices] AS [t0]
GROUP BY [t0].[CZone]
GO

DECLARE @x1 Int = 3
SELECT [t0].[LZone]
FROM [Devices] AS [t0]
WHERE ((@x1 IS NULL) AND ([t0].[CZone] IS NULL)) OR ((@x1 IS NOT NULL) AND ([t0].[CZone] IS NOT NULL) AND (@x1 = [t0].[CZone]))
GROUP BY [t0].[LZone]
GO

DECLARE @x1 Int = 1
SELECT [t0].[LZone]
FROM [Devices] AS [t0]
WHERE ((@x1 IS NULL) AND ([t0].[CZone] IS NULL)) OR ((@x1 IS NOT NULL) AND ([t0].[CZone] IS NOT NULL) AND (@x1 = [t0].[CZone]))
GROUP BY [t0].[LZone]
GO

//Reminaing SQL removed....

What just happened? Our innocent nested join has turned into a monster! This is an example of a query that is simple to do in LINQ, but has no translation to a simple SQL statement. Instead of just bombing, the LINQ to SQL engine comes up with a solution that a user may not have written themselves. A typical SQL developer may have looked for a different approach.

Side note: In the nested group-by, notice that LINQ to SQL uses multiple queries. This differs from the Entity Framework approach, which uses outer joins to achieve the same effect.

Does it matter? The answer isn’t so simple. In this simplified example, the performance impact is minimal. Unfortunately, with a large amount of data in this type of query, you could start to experience terrible performance. I personally saw a nested query that was only a few lines of code turn into a 27 page SQL statement. The SQL statement was technically correct, but took seconds to execute, when it should have taken a fraction of a second.

One simple solution that I have found to be very effective, yet not intuitive, is breaking apart the initial query and forcing it to execute using the ToList() method. You’ll have to have a decent “where” clause to avoid excessive amounts of data being returned. Once we have the raw data, LINQ to objects will provide us the same set of tools to further manipulate our data. For instance, here is a modified version of the example presented earlier:

//Simple & fast initial query from the database
var rawData = (from d in Devices
where d.Location = 'B3').ToList();

//This operation happens "disconnected"
var results = from d in rawData
group d by d.CZone into czoneGroup
select new { Key = czoneGroup.Key, val = from d2 in czoneGroup
	group d2 by d2.LZone into lzoneGroup
	select lzoneGroup.Key };

The reason this works well is that it’s taking advantage of the strength of SQL Server, which is to query data, and the strength of .NET, which is to process and manipulate data.

LINQ Abstracting Away Problems it Can’t Solve

Here is a simplified version of a query I saw recently:

int sum = (from d in Devices
where 1 == 2 && d.CZone != null
select d.CZone.Value).Sum()

To make it extremely clear what I’m trying to accomplish, I put “1 == 2” in the “where” clause, so that no rows match the condition. The “Sum()” method returns the type that it’s acting on. For example, if you’re summing integers, the result is an integer. If you’re summing nullable integers, the result is a nullable integer. This is perfectly valid LINQ. This is effectively the SQL that is generated (I simplified it for clarity):

Select SUM(CZone)
From Devices
Where 1 = 2

Since the result of this SQL statement is NULL, it can’t be converted back to an integer. The exception is “InvalidOperationException: The null value cannot be assigned to a member with type System.Int32 which is a non-nullable value type.”

When the LINQ is translated to SQL, there is no such operation as converting a nullable value to a non-nullable, so the “.Value” operation is ignored. This would be fine if the sum function still expected a nullable return type, but it’s now expecting an integer. When it can’t find any rows to return, it tries to return NULL. Since it’s trying to package up a NULL value into a standard integer type, it has no choice but to throw an exception.

Conclusion

Getting started with LINQ is fairly straightforward, but you can’t forget the fact that whatever query you’re writing must be converted into a SQL statement, and the results must be converted back to data that is understandable to .NET. Every LINQ query you write should be checked with a tool such as LINQPad to ensure that the SQL is efficient, and matches what you expect.

Also keep in mind that when you upgrade your data provider, your queries can change. For example, converting a statement from LINQ to SQL to Entity Framework can generate different SQL queries, just as updating to a newer version of the same ORM can.

The first key point is to know what the heck LINQ is. LINQ itself is a number of separate features. One of these key features is being able to write SQL-like syntax (query syntax) in your code. At a basic level, that’s all you need to know for now.

LINQ (to objects)

First, we’re going to talk about LINQ to objects, which I typically just refer to as LINQ (possibly making the matter more confusing). It has absolutely nothing to do with SQL Server, Oracle, or any other kind of relational database. I’m talking about LINQ to objects, because I think that understanding it and contrasting it with LINQ to SQL is critical to understanding both.

For a moment, forget that LINQ exists. Let’s say that you wanted to filter a list of names, to only get names that start with the letter “J”. You could write the following “utility” function: (if you don’t understand “yield return”, see this post on that topic).

public static IEnumerable<string> GetNamesStartingWithJ(IEnumerable<string> names)
{
    foreach(var name in names)
        if(name.StartsWith("J"))
            yield return name;
}

A new feature in C# introduced in .NET 3.0, is a concept known as an extension method. This lets us turn my handy dandy static utility method into a method that can be called on a list of names. By changing the signature to this:

public static IEnumerable<string> GetNamesStartingWithJ(this IEnumerable<string> names)

I can then call it like this (Sweet!):

var myListOfNames = new List<T> {"Abe", "Jack", "Jason"};
var jNames = myListOfNames.GetNamesStartingWithJ();

We haven’t even talked about LINQ yet, but we’ve basically reinvented a portion of it. As an exercise for the reader, think about how you could use a Lambda parameter to pass in a filter criteria to create a ".Where” method. All the pieces are in place to re-create this form of LINQ yourself.

One actual new feature for LINQ is known as query syntax. Basically, it gives us an alternative way to write our query. It makes the code look more like SQL, and less like a long chain of extension methods.

Lambda Syntax:

var uppercaseJNames = names.Where(name => name.StartsWith("J")).Select(name => name.ToUpper());

Query Syntax (same query):

var uppercaseJNames = from name in names
	where name.StartsWith("J")
	select name.ToUpper();

In both of those examples, the exact same operations are occurring, and you get the result. The one you choose will most likely come down to personal preference. It’s also worth noting that some of the extension methods provided out of the box are not available in query syntax. You can either avoid the query syntax in those cases, or use a hybrid approach.

How is LINQ to SQL (and Entity Framework, etc) Different?

Now, I hope you understand that there isn’t really any magic going on in LINQ. Microsoft has simply given us a new set of easy to use tools that make working with sets a breeze.

LINQ to SQL is a different matter. Instead of executing code, you’re building an expression. An expression is simply a “picture” of what you’re trying to accomplish. It can interpreted in many different ways. To understand the underlying technology, you’ll have to read up on expression trees, which I’m intentionally keeping outside the scope of this post.

If we have a “picture” of a query, what happens to it when we want to “run” it? LINQ to SQL, Entity Framework, and other LINQ implementations look at your query, and basically translate it into something else. How about an example?:

//Query Syntax
var deviceIds = from device in Devices
where device.Type == "I"
select device.DeviceId

//Lambda Sytax (extension methods)
var deviceIds = Devices
   .Where (device => (device.Type == "I"))
   .Select (device => device.DeviceId)

//SQL
SELECT [t0].[DeviceId]
FROM [Devices] AS [t0]
WHERE [t0].[Type] = "I"

I’ve provided the query syntax and the lambda syntax. At the bottom is the resulting translation into a SQL statement.

In this last example, I’ll try to make it clear that your code is simply interpreted and translated:

//Query Syntax:
from device in Devices
where device.Type != null
select device.DeviceId

//SQL Syntax:
SELECT [t0].[DeviceId]
FROM [Devices] AS [t0]
WHERE [t0].[Type] IS NOT NULL

Notice that the C# operator “!=” translates in SQL to “IS NOT NULL”. This was handled automatically for us. Our expression did NOT get back all the rows and apply a conditional to it.

Why is this important? To use either technology effectively, you have to understand that when you’re working with objects, it’s simply a chain of methods, and often behaves as you would expect. When working with LINQ to SQL (or a related technology), the expression is evaluated, and might not execute like you expected.

Understanding the internal workings of these technologies will let us fully take advantage of all the wonderful features it has to offer. In upcoming posts, I’ll be warning you of some potential pitfalls related to how your queries are interpreted and translated. I’ll also be showing you how to get significant performance gains by using LINQ to SQL or Entity Framework efficiently (over traditional SQL based solutions). I’ll also be showing you how I write LINQ queries to query an AutoCAD document!

Related Posts:

• Using Yield Return for readability and performance
• Using “var” to simplify code and avoid redundancy

First, let’s take a look the code for a repository method that builds a query, executes the query, and returns the results in a list:

public IEnumerable<Cat> FindAllCats()
{
	var query = from c in db.Cats
		select c;

	return query.ToList();
}

The “ToList” is forcing the IQueryable<Cat> query to execute and put the results in a list immediately. However, we know that IQueryable<T> inherits from IEnumerable<T>, so what happens if we avoid the list creation completely?

public IEnumerable<Cat> FindAllCats()
{
	var query = from c in db.Cats
		select c;

	return query;
}

In this scenario, our method is returning the same interface, but the underlying type is now a LINQ database iterator instead of a List<T>.

Delaying execution can lead to multiple executions

If the code is not explicitly putting the results into a list, we’re actually passing back a form of an iterator. This works great if we only need to execute the query once. However, if we iterate through the list more than once, we actually end up executing our query multiple times. This can obviously lead to poor performance.

If you’re writing fast queries, you may not even notice if they’re being called too many times. However, there may be a worse problem lurking in your code. Each time you iterate through the enumerator, you’re getting a different set of objects. The same query is being made with the same results, but the objects are re-built each time. This leads to objects that are equivalent, but not the same. For example, you may get back Cat objects with the names “Bill” and “Ted”, but if you actually check them for equality using “==”, they will not be the same object instance. Correction: Scott points out in the comments that this isn’t necessarily the case. Keep in mind that it can still occur if projecting types and not working with the original objects.

Delaying execution may mean you no longer have a database connection when attempting to execute the query

If you delegate the task of initiating your query to another layer, you better be sure that the database connection is still around, and is in a queryable state. If you’re using the standard repository pattern and a short-lived database connection pattern, you may quickly run into problems when you try to iterate through the results of the enumerator you receive from your repository layer.

Conclusion

If you’re thinking about moving the execution of your queries to another layer, make sure you understand the consequences. You’ll need to weigh those consequences against the tiny benefit that you’ll receive from the delayed execution. There may be cases where delaying the execution or possibly avoiding it completely will improve your application, but those are probably very rare cases.

IEnumerable vs IQueryable

First, you need to understand the purpose of IEnumerable and IQueryable. IEnumerable defines a stream of objects that can be retrieved sequentially. It’s implemented for nearly every type of list, and it’s an integral part of LINQ. There are now methods included such as “Where” and “Select” that let us filter, sort, and manipulate lists of data in interesting yet simple ways. This can also be referred to as LINQ to objects.

IQueryable inherits from IEnumerable, and is designed to be translatable into a query. IQueryable is typically used to build an expression tree that represents the requested operations. The operations are not actually executed until the expression tree is evaluated and used.

As an example, let’s say I have a database with a table and entities called DateRange. Suppose I cast the DateRange entityset (which implements IQueryable<DateRange>) as IEnumerable<DateRange>. When I call LINQ expressions on that IEnumerable, the underlying query is run immediately, which effectively causes all of the data from that table to be retrieved. If I use IQueryable without casting, my operations get turned into SQL that gets executed when I actually try to iterate through the data (probably using ToList() or foreach). It’s obviously preferable to have the query run in SQL since it can more efficiently filter the data.

The Problem

As I mentioned earlier, I recently starting writing a data access method that started to contain some non-trivial logic. Whenever I see logic, I want to be able to unit test it! I ended up pulling out the logic into its own static method. This method takes in IEnumerable<DateRange>, which can also accept IQueryable<DateRange> (because of the inheritance, you’ll recall). Then, I simply use the AsQueryable method in IEnumerable. This ends up building the needed expression tree that can be translated into a SQL query, but it also lets me test against an in memory collection.

public static DateRange FindRelativeToDate(IEnumerable<DateRange> enumerable, DateTime reference, int periodOffset)
{
	//Build as an expression tree, if possible, otherwise enumerate
	var queryable = enumerable.AsQueryable();

	//Now put in all the logic

	var reference2 = reference.AddDays(-1);

	var initialRange = from bp in queryable
		 where reference >= bp.Start && reference2 < bp.End
		 select bp;

	var currentDateRange = initialRange.First();

	if(periodOffset == 0)
		return currentDateRange;

	var newRange = from bp in queryable
		select bp;

	if (periodOffset > 0)
	{
		newRange = newRange.Where(x => x.Start >= currentDateRange.Start);
		newRange = newRange.OrderBy(x => x.Start);
	}
	else
	{
		newRange = newRange.Where(x => x.End <= currentDateRange.End);
		newRange = newRange.OrderByDescending(x => x.Start);
	}

	return newRange.Skip(Math.Abs(periodOffset)).Take(1).FirstOrDefault();
}

In my unit test class, I can define a list of sample data. I took real data from the database to make the tests as close to reality as possible:

private static readonly List<DateRange> _DateRanges = new List<DateRange>
{
	new DateRange {Start = new DateTime(2009, 1, 1), End = new DateTime(2009, 1, 30)},
					new DateRange {Start = new DateTime(2009, 1, 31), End = new DateTime(2009, 3, 01)},
					new DateRange {Start = new DateTime(2009, 3, 2), End = new DateTime(2009, 3, 31)},
					new DateRange {Start = new DateTime(2009, 4, 1), End = new DateTime(2009, 4, 30)},
};

Now, I can easily test the static class I wrote (this is 1 of 9+ real tests):

[TestMethod]
public void FindRelativeToDate_MiddleOfDateRange_ContainingDateRange()
{
	var result = DateRangeRepository.FindRelativeToDate(_DateRanges, new DateTime(2009, 3, 15), 0);
	Assert.AreEqual(_DateRanges[2], result);
}

The only thing that is left to do is wire up the repository method so that it calls my static method. This is a thin wrapper layer that will actually get used in production. If you run profiler, you’ll see that the query expression is being evaluated and converted into an efficient SQL expression.

public DateRange FindRelativeToDate(DateTime reference, int periodOffset)
{
	var ctx = dbEntities;
	return FindRelativeToDate(ctx.DateRangeSet, reference, periodOffset);
}

Limitations/Conclusion

Unfortunately, this method of testing a repository doesn’t scale easily. If you start working with multiple entity sets that are combined with join operations, this technique is next to impossible to use. You’ll see the most benefit when working with a single entity type, and need to test logic in your repository method.

One thing you need to be aware of, is that LINQ to SQL and Entity Framework don’t implement every IQueryable/IEnumerable method. This means that you could potentially make calls on the in-memory collection that will then fail when you use the actual database. Fortunately, these problems can usually be detected fairly quickly.