A recent DBA asked me the question “I thought inline SQL was bad, so why are we using it again?” LINQ may *smell* like inline SQL, but it is not. Let’s first take a look at some simple LINQ that is easy to read:

from d in Devices
where d.CZone == 4
&& d.Type == "X"
select d.Id

So how is this different than inline SQL? To be technical, you’re writing a query against a data model, with full intellisense. You’re also writing a provider agnostic query. This same query can be performed against SQL Server, Oracle, or even the Facebook API if there was a supporting framework in place. We now have a truly unified query architecture.

Let’s keep taking about this simple LINQ query, and see how you would write it if you didn’t want to use LINQ. Most developers before the days of LINQ would probably use a stored procedure. Stored procedures are great. They’re efficient, reusable, and easily updatable. Here is what it may look like:

Create Procedure GetMyStuff
As
Select Id
From Devices
Where CZone = 4
And Type = 'X'
Select Id
GO

A nice, simple SQL query. There are a few disadvantages that may not be immediately apparent:

• If you need a second, similar query, you have to either have to create and maintain two stored procedures. As an alternative, you could modify the stored procedure to operate differently based on a parameter. Both of these options are not idea, but LINQ does give us an alternative that I’ll discuss in a bit.
• You don’t get intellisense when you’re writing your code.
• You have to be concerned with two different “programming” paradigms, and also have to manually manage the translation in both directions.

Now, let’s take our example to the other end of the spectrum, which will help show where LINQ can really shine where straight SQL does not. This example is a query for a search page. I set up a simple ASPX page to demonstrate. Here is a sample of the user interface:

The user enters a number of search criteria, and the results are displayed. I literally coded this in under 5 minutes. If you’re used to using stored procedures to retrieve this type of data, think about how you would go about creating this. You have a couple of options that I’m aware of:

• Write a separate stored procedure for every combination of parameters. In this case that would be 7 stored procedures. This would certainly not be ideal.
• Write a single stored procedure that can handle each parameter as nullable parameters, and use “Where @Param Is Null Or Param = @Param”. This option is easy, but has some potential performance implications.
• Write a single stored procedure that can handle each parameter as nullable parameter, and “If” statements to handle each scenario. This would be time consuming an error prone.

In LINQ, we’re able to dynamically build up a query. For the search example, my LINQ looks like this:

var dataContext = new DataClassesDataContext();

var query = (IQueryable<Device>) dataContext.Devices;

if (txtCZone.Text.Length > 0)
    query = query.Where(device => device.CZone == int.Parse(txtCZone.Text));
if (txtUCZone.Text.Length > 0)
    query = query.Where(device => device.UCZone == int.Parse(txtUCZone.Text));
if (txtLZone.Text.Length > 0)
    query = query.Where(device => device.LZone == int.Parse(txtLZone.Text));

dgResults.DataSource = query.ToList();
dgResults.DataBind();

And of course we can use the query syntax instead (replacing lines 5-10 above):

if (txtCZone.Text.Length > 0)
	query = from device in query where device.CZone == int.Parse(txtCZone.Text) select device;
if (txtUCZone.Text.Length > 0)
	query = from device in query where device.UCZone == int.Parse(txtUCZone.Text) select device;
if (txtLZone.Text.Length > 0)
	query = from device in query where device.LZone == int.Parse(txtLZone.Text) select device;

The result is that the SQL code is specifically written to support only the parameters that the user has entered. No extra SQL, and no specific SQL to maintain. Remember that LINQ can be chained together without querying the underlying data. The actual querying of the data only occurs when enumerating the results, using an operation like “ToList()”.

To support paging we need to run 3 different types of underlying queries. Here is where LINQ is really going to shine. We can use the same base query for all of these operations, and not have to worry about the drastically different underlying SQL statements.

1. Result count – Simply by calling the “.Count()” method, we can retrieve the number of rows the query will return in total. The underlying SQL will be a simple and efficient Count operation.
2. Page n query – By utilizing Skip and Take, any page within the results can be queried. The work of generating a common table expression is handled for you.
3. First page query – If the underlying provider has an optimization for using the SQL TOP command, the first page of data you query will be able to avoid a common table expression. This has the advantage of being more efficient when the first (and often most common) page of results is displayed.

Real-world Results

I initially ran into this in a real application that was primarily used to search through a large table of records. It had originally used the stored procedure approach, and was causing the entire system to slow down to the point of being unusable. Thanks to LINQ, we were able to make the search usable. In fact, the results were drastic:

In addition to the improved performance, the code was easier to maintain. The stored procedure was extremely cluttered, had large where clauses, and even contained two nearly identical copies of the query. One for calculating the count, and one for paging support.

Conclusion

LINQ gives us much more than “inline SQL”. It gives us a unified query syntax, delayed execution, query expression building, and dynamically created SQL output. Additionally, the generated queries are optimized based on the exact query being performed instead of making generic SQL that is optimized for multiple scenarios.

SQL Query Complexity Disproportional to LINQ Complexity

Recall the example from my last post:

//Query Syntax:
from device in Devices
where device.Type != null
select device.DeviceId

//SQL Syntax:
SELECT [t0].[DeviceId]
FROM [Devices] AS [t0]
WHERE [t0].[Type] IS NOT NULL

In this case, LINQ to SQL has done something wonderful. It’s saved us from having to understand or worry about the translation of syntax between C# and SQL. Now, what happens when we write something a little more advanced, such as a nested group by?

from d in Devices
group d by d.CZone into czoneGroup
select new { Key = czoneGroup.Key, val = from d2 in czoneGroup
	group d2 by d2.LZone into lzoneGroup
	select lzoneGroup.Key }

And the corresponding SQL:

SELECT [t0].[CZone] AS [Key]
FROM [Devices] AS [t0]
GROUP BY [t0].[CZone]
GO

DECLARE @x1 Int = 3
SELECT [t0].[LZone]
FROM [Devices] AS [t0]
WHERE ((@x1 IS NULL) AND ([t0].[CZone] IS NULL)) OR ((@x1 IS NOT NULL) AND ([t0].[CZone] IS NOT NULL) AND (@x1 = [t0].[CZone]))
GROUP BY [t0].[LZone]
GO

DECLARE @x1 Int = 1
SELECT [t0].[LZone]
FROM [Devices] AS [t0]
WHERE ((@x1 IS NULL) AND ([t0].[CZone] IS NULL)) OR ((@x1 IS NOT NULL) AND ([t0].[CZone] IS NOT NULL) AND (@x1 = [t0].[CZone]))
GROUP BY [t0].[LZone]
GO

//Reminaing SQL removed....

What just happened? Our innocent nested join has turned into a monster! This is an example of a query that is simple to do in LINQ, but has no translation to a simple SQL statement. Instead of just bombing, the LINQ to SQL engine comes up with a solution that a user may not have written themselves. A typical SQL developer may have looked for a different approach.

Side note: In the nested group-by, notice that LINQ to SQL uses multiple queries. This differs from the Entity Framework approach, which uses outer joins to achieve the same effect.

Does it matter? The answer isn’t so simple. In this simplified example, the performance impact is minimal. Unfortunately, with a large amount of data in this type of query, you could start to experience terrible performance. I personally saw a nested query that was only a few lines of code turn into a 27 page SQL statement. The SQL statement was technically correct, but took seconds to execute, when it should have taken a fraction of a second.

One simple solution that I have found to be very effective, yet not intuitive, is breaking apart the initial query and forcing it to execute using the ToList() method. You’ll have to have a decent “where” clause to avoid excessive amounts of data being returned. Once we have the raw data, LINQ to objects will provide us the same set of tools to further manipulate our data. For instance, here is a modified version of the example presented earlier:

//Simple & fast initial query from the database
var rawData = (from d in Devices
where d.Location = 'B3').ToList();

//This operation happens "disconnected"
var results = from d in rawData
group d by d.CZone into czoneGroup
select new { Key = czoneGroup.Key, val = from d2 in czoneGroup
	group d2 by d2.LZone into lzoneGroup
	select lzoneGroup.Key };

The reason this works well is that it’s taking advantage of the strength of SQL Server, which is to query data, and the strength of .NET, which is to process and manipulate data.

LINQ Abstracting Away Problems it Can’t Solve

Here is a simplified version of a query I saw recently:

int sum = (from d in Devices
where 1 == 2 && d.CZone != null
select d.CZone.Value).Sum()

To make it extremely clear what I’m trying to accomplish, I put “1 == 2” in the “where” clause, so that no rows match the condition. The “Sum()” method returns the type that it’s acting on. For example, if you’re summing integers, the result is an integer. If you’re summing nullable integers, the result is a nullable integer. This is perfectly valid LINQ. This is effectively the SQL that is generated (I simplified it for clarity):

Select SUM(CZone)
From Devices
Where 1 = 2

Since the result of this SQL statement is NULL, it can’t be converted back to an integer. The exception is “InvalidOperationException: The null value cannot be assigned to a member with type System.Int32 which is a non-nullable value type.”

When the LINQ is translated to SQL, there is no such operation as converting a nullable value to a non-nullable, so the “.Value” operation is ignored. This would be fine if the sum function still expected a nullable return type, but it’s now expecting an integer. When it can’t find any rows to return, it tries to return NULL. Since it’s trying to package up a NULL value into a standard integer type, it has no choice but to throw an exception.

Conclusion

Getting started with LINQ is fairly straightforward, but you can’t forget the fact that whatever query you’re writing must be converted into a SQL statement, and the results must be converted back to data that is understandable to .NET. Every LINQ query you write should be checked with a tool such as LINQPad to ensure that the SQL is efficient, and matches what you expect.

Also keep in mind that when you upgrade your data provider, your queries can change. For example, converting a statement from LINQ to SQL to Entity Framework can generate different SQL queries, just as updating to a newer version of the same ORM can.

• Easy to update (fix, improve) on-the-fly.
• Hard to work with data (multiple tables, arrays, etc) in VB6 and ASP, at least compared to .NET.

Given the circumstances, I don’t think that I would go back and change history if given the chance.

The problem is that the world has changed, and their architecture just doesn’t scale well. The biggest problem is in the area of reporting (which is now primarily .NET). Since all the data manipulation and logic is in the stored procedures, the database is forced to do all the work. In a small application with not a lot of data, this probably isn’t a big deal. The report runs for a few seconds and no one really experiences a slow down.

The problem arises when there is a lot of data. Easily millions of rows in dozens of tables. The current fix is to scale vertically, meaning that they throw more hardware (AKA money) at the problem.

The real solution to the problem is to scale horizontally. There are two main ways of doing this. The first is to add more database servers. The problem is that this isn’t really all that easy. The software to do this is getting better, but the fact remains that two different systems now have to stay synchronized. I’ve read articles about businesses that have scaled their business, and a common theme is that databases are one of the hardest parts of your architecture to scale.

The second aspect of scaling is to reallocate where your work is being done. You need to start thinking of the database as a…well….a database! The first and foremost purpose of a database is to store massive amounts of data, and allow you to quickly retrieve that data.

Databases are amazingly fast when you use them simply as a place to store data. If you design your database correctly, and set up indexes that are optimized for the ways you want to retrieve your data, there should be no reason to wait for your data. SQL Server can easily handles millions or even billions of rows, and query any of them almost instantaneously. Even with multiple queries being executed concurrently or in succession.

Consider the following diagrams:

The existing architecture is on the left, and the proposed structure is on the right. It’s not a big change, I’m simply suggesting that the business logic be moved from the stored procedures, into the code.

The first reaction that I usually receive from this suggestion is that the code is going to be ugly compared to the corresponding SQL. The reality is that the SQL is ridiculously complicated (at least in my experience). SQL is great for set based logic, but really starts to break down when trying to do object oriented or procedural operations.

The fact is that .NET is great at processing massive amounts of data. First of all, it’s incredibly fast. If you’re writing your code correctly, you’ll be amazed at how fast it can process data.

More importantly, if you’re using a programming language to manipulate your data instead of T-SQL, you can really start to break down the problem at hand. Databases are traditionally very bad at breaking a large problem into smaller problems. Sure, you can call other functions and stored procedures, but you can tell that it’s not the strongest feature of the database. A programming language like .NET lets you pass data around in any structure that you can conceive.

The other major advantage of processing the data in your code, is that you can now easily build testable code. Any code that you can easily test will have less bugs, and should be easier to maintain in the long run.

If your data is now being processed and organized in your code and not the database, you are probably ready to scale horizontally. It’s relatively easy to add more front end servers. Since they all hit a common database, there aren’t really any synchronization issues.

So does this really work? Of course. I’m not the first to think of it by any means. Take any of the largest websites on the Internet, and look at how they have designed a scalable architecture. Digg.com has always had database scalability issues with their MySql servers, so they try to minimize them as much as possible. Twitter uses ONE database server to handle thousands of requests per second. eBay.com doesn’t even do joins in their database. They would never be able to scale if they put that burden on a database that handles 26 billion SQL queries each day.

Microsoft has made matters worse by integrating the .NET CLR into SQL Server. You can now write .NET code that gets executed on your database server. This is a great tool, but it must be used with great care. This isn’t an excuse to write more code in your database.

In summary, databases are an expensive commodity, don’t abuse them! Be careful how much you do in the database, and ask yourself if you can move some processing into code, where it is more easily scaleable.

Just recently, Scott Hanselman came out with a podcast about website scaling. If you’re looking for more site scaling concepts, it’s worth a listen.

The problem is that SQL Server puts null values above non-null values when doing an "order by". To reverse this behavior, this was the most elegant and efficient solution that I found:

Select FooValue
From foos
Order by (Case When [Order] Is Null Then 1 Else 0 End), [Order]

I found information about the original problem here, and the solution was from Tim in the comments. Thanks!

1. Constrain everything humanly possible. If it’s an integer that wouldn’t normally be negative, add a ">= 0" constraint.
2. Constrain primarily where it’s necessary to maintain referential integrity.

Consider the following diagram. It’s a map of the flow of data from your user, which eventually makes its way into the database.

Since we’re getting input from a user, and they’re the one that can fix invalid data, we validate data at the top layer. There’s usually no getting around this. In fact, for the best user experience on the web, you’re going to perform some JavaScript validation. Then you’ll probably validate it again on the server, in case they have JavaScript disabled.

At this point, unless there is a bug in your code, you’re sure that the data is valid. You may not know if it’s referentially valid. Validating the input a third time in the database is probably overkill. It’s also a potential performance bottleneck.

Yes, there are many times when this doesn’t apply. For example, when multiple systems are interacting with the same database, and one counts on the data in a certain format. The only way to guarantee you get data in a format you expect is to constrain it at the database level.

In general, I avoid strict constraints at the database level. The biggest reason is that it requires your to synchronize all of your validators. They all have to agree on the same set of restrictions, or the code will fail. That goes against the LEAN and Agile philosophies. When I want to allow negative numbers in my integer field, it’s much easier to simply change it in my application. This is amplified if you have to talk to a DBA to make changes.

Another reason to avoid constraints is that they can’t always understand the data like the application can. For example, should a constraint attempt to ensure that valid email addresses are entered? If you’re storing a persons age, do you constrain it so that it can’t go above 500? 200? 100?

Now let’s assume that there is a bug in your application code, and you didn’t have a trusty constraint to stop it. You now have invalid data in your database. The good news is that you now have the potential to clean it up, or adapt your code to deal with it. The bad news is that if that value is used in a calculation that could have bad side effects, you could have big problems.

As with anything, there is no hard and fast rule for every situation, but we can at least make some general guidelines.

Think LEAN. Anything that doesn’t provide value to the customer is waste. Think Agile, requirements change, be adaptable.

(there is no emoticon to express my anger!)

The picture above shows how I felt when I started writing my first LINQ expression. The biggest problem was the fact that the latest version of ReSharper doesn’t support any .NET 3.0+ language features. Not only does it not support LINQ, it’s IntelliSense severely interrupts you while writing it. So much so that it makes it unusable.

I went ahead and downloaded the latest development build (Build 783). On their download page, it’s listed as "Works here". That wasn’t encouraging. It does work a little better with LINQ, but it’s still a steaming pile of you know what (dog poop for the not-so-smart). This further reinforces my love/hate relationship with ReSharper.

Anyway, I was eventually able to write some LINQ code. A great tool to get started is LinqPad, which is basically a query analyzer but with LINQ expressions. Writing LINQ is very difficult with a SQL background, because everything is backwards. You think you know what you’re doing, but you don’t.

Right now, we’re using NHibernate in the main project that I’ve been working on for the past couple of months. It’s amazing, but there are a couple of things that would be nice:

• Better optimization of queries – It looks like LINQ does an amazing job with this.
• Batched reads & writes – LINQ does batched writes, but lazy loading by default. Maybe not as big a deal as I think.
• Cross session saving – I spent hours battling with some code that loaded a complex object with relationships in one session, and then saved them in another. It appears that LINQ solves this, but I’ll have to run some tests to be sure.
• Less work generating mapping files and relationships.

One thing that is nice about LINQ to objects is the fact that it will generate all of the model classes, plus the glue that connects the model to the database. You can either use Visual Studio and drop the tables into a mapping file, or you can use SqlMetal to script the class generation.

One of the biggest questions I’m trying to answer write now, is how unit testing fits in with LINQ. We’re currently testing our data access layer by using an in memory SQLite database, which let’s us perform close to real world saves and loads. We also use interfaces for our data access methods, which makes it easy to create testable classes that can simply be supplied a database interface.

I’m also not sure if it even makes sense to put my LINQ queries in a data access layer. The code would almost seem trivial, and would just create a lack of flexibility. Ironically, it almost feels like you should use LINQ to query against your data access layer.

For now, there are more questions than answers. For now, I don’t plan on retrofitting my last project with LINQ, but I’m going to investigate if it will be a good foundation for the data access logic in my next project. Of course if I go that route, you’ll be sure to hear about it!

I added a trigger to the membership table so that rows automatically get inserted into my table. The problem is, the trigger violates the foreign key constraint! Here is the trigger code:

Create Trigger dbo.Trigger_CreateExtraUserRecord ON aspnet_Users
After Insert
As
Begin
    Set Nocount On
    Insert Into tfs_Users
    (MembershipUserId)
    Select UserId From inserted
End

As you can see, it’s an "After Insert" trigger, so the first insert will be done at this point (I have verified that).

It must be using some kind of transaction, and the foreign key is violated because it’s not committed.

The solution (not perfect, but it works), is to use this before the insert to disable the foreign key check:

Alter Table tfs_Users Nocheck Constraint All

And use this after:

Alter Table tfs_Users Check Constraint All

Instead of disabling all constraints, you could also specify the constraint name, which would obviously be better in most cases.

Does anyone have a better explanation for this behavior?