Lee Richardson - Blogs at Near Infinity

USB Device Not Recognized in USB 3.0

Lee Richardson — Fri, 01 Mar 2013 00:00:00 -0500

Unless you're having this problem:

For USB 3.0 ports on an Arduino-like stack you may safely stop reading. Nothing to see here just trying to fill out the Internet.

The Problem

This problem occurred for me on an ATMEL ATMEGA32U2 chip with the LUFA USB framework for AVRs and the HID USB driver. Everything worked fine for USB 2.0, but on modern PC's it would pop up "USB device not recognized".

In device manager the device showed up as Unknown device. Device status would look like this:

Windows has stopped this device because it has reported problems. (Code 43)

And in the event viewer it showed something like this:


Device USB\VID_0000&PID_0002\5&ea89d57&0&2 had a problem starting.

Driver Name: null
Class GUID: {00000000-0000-0000-0000-000000000000}
Service: 
Lower Filters: 
Upper Filters: 
Problem: 0x2B
Status: 0x0

The Solution

Despite a lot of fruitless packet monitoring and upgrading various packages and help from master-hacker Joe Ferner, solving this ultimately boiled down to compiling the GenericHid sample project from LUFA (which did work in USB 3.0) and then commenting out more and more of my project until it worked and then incrementally adding everything back via binary search.

The problem ended up being the amount of time that passed between the call to LUFA's

USB_Init();

And the USB calls in the main program loop. So essentially I was doing something like this:


int main(void);

  MCUSR &= ~(1 << WDRF);

  wdt_disable();

  clock_prescale_set(clock_div_1);

  USB_Init();

  _delay_ms(500); // <- DON'T DO THIS

  for (;;)

  {

    HID_Device_USBTask(&Generic_HID_Interface);

    USB_USBTask();

  }

}

Take that delay out and everything works.

Summary

The takeaway: move all of your initialization logic - anything that might take any time at all - prior to the USB_Init call. It might just save you a lot of pain down the road. That's it. Hope this helps someone, somewhere.

Google Play vs Windows Store: A Stark Contrast

Lee Richardson — Tue, 22 Jan 2013 00:00:00 -0500

It's a familiar story: anything-goes-wild-west vs bureaucracy-ridden-walled garden. But you probably already knew that. What surprised me is how incredibly stark the contrast is.

Getting an app in Android and on Windows 8 couldn't be more different: from fees, to technical support, to the app-submission process.

Setup for the Letdown

You'd think an app store coming from behind might try to get their app count up by accepting new submissions first and asking questions later. Couldn't be further from the truth.

My first Windows Store app took a total of 33 days from the first time I tried to submit through when it was finally accepted. My Google Play app? 4 days.

Let's be honest, 33 days is completely unacceptable. Why so long? Partly it's because they have standards. No really. Believe it or not they have live human beings manually test your app and provide a write-up of why it wasn't good enough for their marketplace. It's sort of like getting a free tester.

This approach actually makes sense for them. Classic Windows is similar to what Android is today. You can pretty much install anything from any random website. Google just centralizes their chaos. The result for classic Windows? Malware, Trojan viruses, countless unwanted browser plug-ins, and a slowly degrading computer experience over time as more crap gets installed. In short: a lack of trust in apps. If Microsoft allowed everything in they would be right back to the problem they had in the first place.

But 33 days? Really!?

In my case my first rejection actually came in just 3 days. It was a couple of minor issues. The app needed a privacy policy and apparently also a link to create a new account rather than just a log in screen (even though you could do some stuff without logging in). That second item was an excellent point I'd never even thought of. I'd incorrectly assumed most of my app users would be existing website users. There was a brief mention of what to do, but no link or in-app account creation.

My previously submitted Google Play app had these problems too, but they certainly didn't have human beings considering how users will interact with my app. Go manual testing! Go Microsoft!

But my initial excitement was just a set up for a big letdown. I resubmitted within a day and then waited.

And waited.

Customer Support: Microsoft++

How a company responds to problems can be more insightful than the fact that they have them. After all everyone screws up at some point, right?

The following was one of several interactions I had with Microsoft support.

Dear Andy,

It has now been 22 days since I resubmitted my app. It has been almost a month since my initial submission. It has been three days since you estimated it would be a day or two.

Is this going anywhere? Am I being quietly ignored for some reason? If there are reasons my app is being ignored could you please list them so that I could resolve the issues? Or if Microsoft simply does not want me developing apps for Windows 8 is there a facility for me to get my money back?

I am extremely frustrated by this process. If I were ever to get my app in the store and needed to get an update out quickly, might it take as long as the initial submission has been? Because a month turn-around to fix bugs is completely unacceptable: not just to me, but to every single user of the Windows Store.

I genuinely want Microsoft to succeed and will do what I can to help, but if this is how it's treating its developers, customers, and supporters then I'm sorry to say my expectations are low.

Sincerely,
- Lee

A bit too strong? I was getting angry. To date I had been at least impressed with their professional and quick responses.

However I was floored by their response to my angry e-mail.

Hi Lee,

Thank you for using the Microsoft Windows Store.

Thank you for getting back to me. I have been checking your account on a daily basis to see if your app has been passed. I can tell you I am looking into this issue for you. I haven't ignored you at all. On the contrary I am eager to help you get this issue resolved.

I understand that it is really frustrating for you and I am doing everything I can to have this issue resolved for you. I hope to have an answer for you soon. Again I am sorry that it is taking so long to get through the process.

If you have any more questions please reply and I will be happy to help you.

Thank you
Andy
Your Partner at Microsoft
Use what you know. Do what you've always imagine

I don't know about you, but I think sincere compassion is something I'm secretly hoping for in every customer service interaction. If I could just get a quick resolution to the problem I'd still give Microsoft a big win in my book. Sadly that wasn't to be.

By way of comparison Google had issues with my initial Google Play submission too. It gave me a nice opportunity to compare how the companies responded to adversity.

When Google accepted my money but wouldn't let me submit my app I got a response like this:

Hi Lee,

Thank you for your note. We apologize for the delay in your account registration. We have activated your account and you may begin uploading applications. Thank you for your interest in Google Play.

If we can assist you further, please let us know.

Regards,
The Google Play Team

Not bad. But not compassionate. No name of a person who wrote the note, no "it must be frustrating".

But whatever. It was a relatively quick resolution from Google. Two days to respond to me for a total of 4 days from time of initial submission until final app acceptance. I'll take an emotionless quick resolution over a drawn-out unresolved empathetic one any day.

Fees: Google FTW

As a quick aside do you know how much it costs to get an app on Android? It's $25 "once and only once. Just enough to keep the riffraff out. And Windows 8? $50" every .. single .. year.

Why should I have to pay another $50 fee next year just for the privilege of making a quick update to my free, no-advertisements, non-revenue generating app?

The chances are good I won't be renewing that fee. The chances are good a lot of developers won't. And that will lead to stale apps and a bad customer experience. Poorly done Microsoft. #win8 #fail.

Fail, Fail, and More Fail

So how does the Windows Store story end? 25 days from my initial submission I got this response:

Notes from Testers:

.D:: This app appears to be incomplete in its functionality to our reviewers. The app appears to have dependencies on non-Windows Store style APIs. In the app description as well as in the app, the users are asked to download additional software to be functional.

Rejected. Again. This time for a different reason. In their defense I should have read their submission guidelines. But couldn't they have told me this during the first rejection? Did it really require 25 days for this response?

I was crushed. They were asking for a massive rearchitecting that didn't make sense and maybe wasn't even possible. Google Play was beginning to look golden.

The problem was in how my app works. It shows the realtime status of a continuous integration server, but requires a desktop app to collect the data and upload it to "the cloud". The mobile app just gives a view into the cloud data and receives push notifications and live tile updates from a server. Seems reasonable right? Apparently the entire architecture violates Microsoft's submission guidelines.

My plan involved adding a single CI server that didn't require a desktop app to sneak by their policies while leaving the existing process all other CI servers. I'd use Travis, since it's guaranteed to never be behind a firewall. I had a low confidence the plan would work.

After a week of evenings followed by an all-nighter I resubmitted at 5 am EST (I'm a sick puppy, I know) and crossed my fingers that it wouldn't be another 22 days.

3.5 hours later, at 8:30 am, I got an e-mail from Microsoft. With that kind of turn-around it couldn't be good news.

It was. It passed. I was shocked.

They couldn't have run the automated tests in that amount of time let alone manually tested it. And what humans do business at like 7 am EST? Did they just take pity on me? Did Andy, my compassionate customer support representative come through for me? I still don't know.

Updates

One final comparison. Updates on the Windows Store seem to take around 8 hours. On Google Play they're seemingly instantaneous. In both stores customers have an update to install by the next day. No complaints there.

Summary

So who has the better process? Google. Hands down. They allow for a lean, minimum viable product style of development. Publish early and often, get feedback quickly, push the features your customers really want.

But I like elements of the Windows Store process. Friendly customer support people rock. Manual testing made my app better. But were their "improvements" the kinds of things real customers would prioritize highly? Maybe, maybe not.

I guess my ideal app store would cost $25 once; incorporate a manual tester that provides helpful suggestions not requirements; reject apps rarely if ever; give empathetic responses and quick resolutions when problems do happen; oh and turn submissions around in 3.5 hours or less every time.

That's not too much to ask for, is it?

Lee Richardson has been developing applications for public and private sector clients in the Washington DC area for over fourteen years. You can follow him on twitter @lprichar.

How To Get The Most From Twitter: Scoping Rules

Lee Richardson — Tue, 29 Nov 2011 10:36:04 -0500

Ever noticed a tweet disappear? As in someone else can't see something you posted? There's a common mistake even twitter pros make that causes this to occur.

Are you sure you're seeing all of the replies to your tweets -- even from people you don't follow? Do you ever find yourself missing important posts -- like those from your real life friends? Do you know which will reach a wider audience: new style retweets vs old style retweets?

This article will answer these and other twitter related questions. It will help beginner to intermediate twitter users get the most out of the service.

#1 Standard #Tweets

These appear simple. These show up in the timeline of your followers (@admirer and @vet in the example above). And they don't show up in the timeline of anyone who doesn't follow you (@effusive). If this was all there was to twitter it would be a complete waste of time.

The subtlety with this type of tweet is that your content may show up to someone that doesn't follow you but that is following a hashtag or a topic you mention. In the example above @hiro thows a hashtag onto the word aardvarks to indicate that it's special somehow. His tweet is then picked up by @spca who runs a constant search in a twitter client for the word "#aardvark".

The first time I realized how powerful this could be I had was at a conference. I was extremely bored by a terrible keynote. With nothing else to do I discovered the conference hashtag on twitter and started watching it. Suddenly I realized I wasn't alone. I was in a huge room full of people making fun of the presenter! It was like telepathy, or group consciousness or something. I've never felt alone or bored at a conference since.

Interesting statistic: As of June 2010, about 65 million tweets were posted each day, equaling about 750 tweets sent each second. (reference)

#2 Replies

Replies are more complicated. In the example above the guy on the left, @vet, replies to @hiro by prefixing his tweet with "@hiro". However, this probably doesn't work like you think. The tweet obviously shows up in @hiro's timeline, but it also shows up in @admirer's timeline because she follows both people mentioned in the tweet (@hiro and @vet). But the tweet does not show up to @effusive because, while he follows @vet, the author of the tweet, he doesn't follow @hiro, the person being mentioned.

Getting this scoping rule wrong is way too easy. For instance you can't just announce that someone did something great. If @hiro started a tweet with "@vet gave a great anti-aardvark presentation" then most of his followers (e.g. @effusive) wouldn't actually see the tweet! I've seen seasoned twitter users make this mistake.

As an interesting aside twitter can help track conversations. If you click a reply button in a twitter client or on the website twitter will keep track of which tweet you replied to and then help piece a conversation together. But if you simply start a new tweet with "@somone" without clicking reply, the above scoping rules still apply, but twitter won't help anyone reconstruct the conversation.

Finally a word on notifications: If someone mentions you (via a reply or anywhere in their tweet) and you follow them then you will get an e-mail and possibly an SMS notification. If someone that you do not follow replies to you, you will not get an email. For this reason you must either frequently check the "mentions and more" section of the website or use a column based twitter client, discussed in the final section on lists.

Interesting statistic: On average only 23% of tweets get a reply. (more)

Another one: 38% of tweets are conversational. In other words 38% of tweets start with an "@". If your percentage of replies is lower you probably aren't using twitter to its fullest. (more)

#3 Reply-All's

So what happens if you want to reply to someone, but still have it show up in the timelines of all of your followers? As long as your tweet doesn't start with an "@" symbol then you're fine. So the convention that has grown up is to prefix your tweet with a "." and then the "@someone". In the example above @effusive now sees the tweet where he didn't in a standard reply.

Interesting statistic: The highest usage rate of twitter to date occurred during the 2011 FIFA Women's World Cup Final between Japan and the United States when 7,196 tweets were published per second! (more)

#4 Old Style Retweet or "Retweet with Comment"

Prior to late 2009 if you saw a tweet you wanted to share with your followers you would retweet (RT) it by copy and pasting their tweet and prefixing it with "RT @author". The problem from a scoping perspective was that people would get duplicates. In the example above @admirer sees a duplicate from @vet when he retweets @hiro's tweet.

There are other problems with old style retweets unrelated to scoping. One problem is that you get long chains of "RT @person1 RT @person2 ...". Another problem is it was hard to follow a person but not all their retweets. The biggest problem is old style retweets take valuable characters away from your available 140, and sometimes require mangling the original.

Old style retweeting is still in use today primarily from people wish to retweet but add a comment, but also because it allows people to reach a wider audience (since new style retweets won't show up in searches or lists, more on this later).

Interesting statistic: Only 6% of tweets get retweeted (more)

#5 New Style Retweet

In November of 2009 twitter rolled out a new technique for retweeting aimed at solving the problems mentioned above. It caused a lot of confusion, but did solve the problems.

The end result of new style retweets from a scoping perspective is that sometimes people see tweets from people they don't follow. In the example above @effusive sees @hiro's tweet in his timeline even though he doesn't follow @hiro, because @vet performed a new-style retweet.

If you click the retweet button on the twitter website today it will perform a new style retweet. Some twitter clients like TweetDeck give you the option. The only way to perform an old style retweet or retweet with comment via the website is to copy and paste the message.

Interesting statistic: Less than half of tweets are posted using the web user interface. By far the most common twitter client is Tweetdeck with 8.48% of the market. (more)

#6 Direct Messages

Direct messages are simple. Prefix your tweet with "D somone" (no at symbol before the name) and they will be the only person that will see it. Just like e-mail except the recipient will get an e-mail and a text message if they've set that up on their phone. Note that neither people watching an included hashtag (e.g. @spca) nor people mentioned will see direct messages unless they are the recipient.

Interesting statistic: Six percent of all tweets are sent via SMS. (more)

#7 Lists

The last scoping topic worthy of mention is lists. This fantastic feature was added by twitter in late 2009. It allows you to organize the people you follow. For instance I have a private "Infrequent Posters" list that I try very hard to never miss a tweet from. Meanwhile I follow enough people that my main list is more like a stream that I dip into but don't get stressed out if I miss some of.

Lists work best when used in conjunction with a column based twitter clients (e.g. TweetDeck). These types of twitter clients are the only way to go if you want to become a twitter pro. If you've gotten to the point where you start missing content from important people (e.g. your real life friends), or you're missing responses to your tweets, or mentions from people you don't follow: then you probably need lists, but you absolutely need something like TweetDeck.

In regards to scoping, lists contain one subtlety that you need to be aware of. They do not support new style retweets. In the example above @vet posts a normal tweet. He then new-style retweets one of @hiro's tweets. @effusive follows @vet and has also placed him into a list. While the list helps @effusive not miss tweets from @vet, it does not end up displaying @hiro's tweet.

Note that this rule applies both to people who maintain their own lists as well as to people who follow lists maintained by other people (e.g. your company may maintain a public list of its twitter enabled employees).

Interesting Statistic: Pointless babble is the most common type of content on twitter. It represents 40.55% of all traffic. (more)

Conclusion

Twitter may appear to be fairly simple at first glance. In reality it has grown over the years to become very sophisticated. Sophisticated and incredibly powerful. But to fully tap that power you must understand the scoping rules and subtleties surrounding things like replies, retweets, and lists. Hopefully you're closer now.

Integrating JavaScript and C# with Script#

Lee Richardson — Tue, 23 Aug 2011 23:09:48 -0400

Have you ever had an enum in your server code that you wanted to access in client side code? Or wanted the safety of compile time errors when there are discrepancies between your server-side and client-side code? Or had a C# Data Transfer Object (DTO) that you wanted to enable Intellisense for in JavaScript? I hadn't found a good solution either ... until now.

Compiling C# to What!?

Compiling C# code into JavaScript may seem foreign, but Script# is a mature technology that is absolutely worth a look. Our team has been using it very happily for about three months. We've found a number of benefits including:

Consistently working Intellisense
Better encapsulation
Simpler Object Orientation
Easier deployment (Script# compiles multiple files to a single, optionally minified file)
Safer refactoring
Simpler unit testing
Type safety; and
Design time syntax checking (no more mistyping a variable and accidentally declaring it to the global scope)

Scott Hanselman covered most of these topics in last month's Hanselminutes episode: Script# Compiles to JavaScript. If you have a spare car ride I definitely recommend the listen. But what wasn't covered were some additional benefits provided by an off the beaten path approach to this great technology. The main benefit is tighter server side C# to client side JavaScript integration. A secondary benefit is less of a dependency on Script#.

To the Command Line

Typically to get going with Script# you:

Install the Script# Visual Studio plug-in
File -> New project
Select "Script Library" and
Compile to generate JavaScript

Nice and easy. The down side to this approach is you can't easily include files outside of your project. Specifically, you can't include data transfer objects or enum's from your server side code. Furthermore, all of your team members must install Script#. If instead you compile with Script#'s "ssc.exe" command you obtain more control and get less dependency.

How-To

The how-to looks something like this:

Download and install Script#
Add a Class Library (not a Script# project)
Project properties -> Build -> Advanced -> "Do not reference mscorlib"
Ideally move the Script# files (C:\Program Files (x86)\ScriptSharp) locally to the solution and check them into source control to a Libs or something similar
Remove all references, but add: ScriptSharp.dll, ScriptSharp.Web.dll, Script.Jquery
Edit your .csproj to manually reference Script#'s mscorlib (right click, Unload project, Edit MyProject.csproj)

<Reference Include="mscorlib, Version=0.7.0.0, Culture=neutral, PublicKeyToken=8fc0e3af5abcb6c4, processorArchitecture=MSIL">
<SpecificVersion>True</SpecificVersion>
<HintPath>..\Libs\ScriptSharp\v1.0\mscorlib.dll</HintPath>
</Reference>
Modify AssemblyInfo.cs and remove the following lines:

[assembly: ComVisible(false)]
[assembly: Guid("b5e2449f-193c-46d1-9023-9143618d8491")]
Modify AssemblyInfo.cs and add the following:

[assembly: ScriptAssembly("ScriptSharpDemoAssembly")
Ensure it compiles in Visual Studio
Create a batch script or PowerShell script that compiles using ssc.exe like this:

..\Libs\ScriptSharp\v1.0\ssc.exe ^
    /debug ^
    /out:MyScriptSharp.js ^
    /ref:"..\Libs\ScriptSharp\v1.0\Framework\mscorlib.dll" ^
    .\Properties\AssemblyInfo.cs ^
    .\Class1.cs

Notice the last two lines in the script are a list of the files that you want to include. The point of this exercise is that you can now include files outside of your main Script# project in that list.

Now for completeness if you put a simple static method in Class1.cs, something like this:

namespace MyScriptSharp
{
    public class Class1
    {
        public static string HelloWorld()
        {
            return "Hello World!";
        }
    }
}

Then run the batch file you should get something like this:

Type.registerNamespace('MyScriptSharp');

////////////////////////////////////////////////////////////////////////////////
// MyScriptSharp.Class1

MyScriptSharp.Class1 = function MyScriptSharp_Class1() {
}
MyScriptSharp.Class1.helloWorld = function MyScriptSharp_Class1$helloWorld() {
return 'Hello World!';
}

MyScriptSharp.Class1.registerClass('MyScriptSharp.Class1');

Obviously you could get these results faster with approach #1. But now you have a lot more control.

Summary

The main downside to this approach is maintaining the batch file is a bit of a hassle. But the upsides are that you can include any file from your server-side C# code. And any changes in that server-side code are automatically reflected in your JavaScript. And any breaking changes in your server-side code generate compile time errors in your client side code. And furthermore none of your team members need to install Script#. For our team it's an easy tradeoff. What about for yours? Please share your thoughts in the comments or on twitter.

Advanced Burn Up Charts

Lee Richardson — Mon, 11 Apr 2011 08:52:53 -0400

If you've read my previous posts on the subject you know that I love agile burn up charts for managing SCRUM style projects, particularly compared to burn down charts.

The problem is that burndown charts lack two essential pieces of information. First, how much work was actually accomplished during a given iteration (as opposed to how much work remains to be completed) and second how much total work the project contains (or if you prefer how much scope has increased each iteration).

What you might not know is how flexible they can be. In particular the example I gave last time has a problem. Can you spot it?

The problem is that the scope increase of 20 points in iteration six might very well have had zero impact on the actual deployment date. Those points might have all been low priority tasks that the team will work on after they deploy the initial 100 points. Wouldn't it be nice if the burn up chart could convey priority information as well?

So that's exactly what I did on my current project last week, and it appeared to be a big hit. And with the addition of some Excel trend lines the chart was able to convey a lot of insight into the project:

Obviously all that data is made up, but the chart still tells an interesting story:

The customer hasn't been adding high priority tasks, perhaps because there is a hard deadline approaching
Because of the customer's restraint in adding high priority tasks the team can be fairly confident that they will finish all the high priority tasks by Iteration 3
While the customer hasn't added high priority tasks they have still been able to be agile by adding normal and low priority tasks

What's really wonderful about this way of reporting is that it gives the customer the flexibility to add scope to future iterations while maintaining near term deadlines. In short it supports what agile is supposed to be about.

But why stop there? Depending on the story you need to tell or the scenario you need to evaluate perhaps you could incorporate team member contributions. For example:

In this example we can try to show what might happen if we remove a particular team member or if we'd never had a particular team member. What's unique about this way or reporting is that it's incorporating the backlog's increasing trend into account. While it's possible that the backlog may increase or decrease at a different rate with a different team makeup, it's a better scenario than ignoring the rate of increase of the backlog all together.

Perhaps we could incorporate other information. Maybe bug vs. feature information. Or planned vs. actual (e.g. task slippage) information. It seems like there is a lot of potential. Any other ideas out there? Feel free to post in the comments or via twitter. If there's enough interest I'd be happy to post another screencast similar to the last one I did on how to produce burn up and burn down with SharePoint and Excel except this time with trends and priorities.

So overall while these charts may be a little more complicated than a traditional burn down chart they would make an excellent talking point during a PowerPoint presentation, or as part of your iteration close out. And with a little training I bet just about any customer would learn to love the insight they provide.

Death to the DAO and How to Test LINQ

Lee Richardson — Thu, 22 Jul 2010 22:42:43 -0400

Occasionally I hear complaints that LINQ is hard to unit test. These complaints aren't about LINQ to objects, mind you, they're specific to the complexities of the flavors of LINQ that turn C# code into something else like SQL or CAML using expression trees. The most common technologies are LINQ to SQL, the Entity Framework, or in my case at the moment LINQ to SharePoint. In this post I'm going to propose a technique that makes testing LINQ not just easy, but downright elegant - assuming you're ok with extension methods - lots of extension methods. And assuming you're ready to kill your Data Access Objects (DAO) tier.

The Unit Testing Problem

Any architecture needs a place to put code that finds entities. For instance FindBySocialSecurityNumber(). In a traditional architecture we might put a method like this is in a DAO layer. If so our method will look something like this:

public class EmployeesDao {
    public Employee FindBySSN(Context ctx, string ssn) {
        return ctx.Employees.SingleOrDefault(e => e.Ssn == ssn);
    }
}

So how would we go about unit testing this?

One fairly typical solution would be to use an in-memory database. That approach works if our data store is a database, but it certainly doesn't work if the data store is something less traditional like SharePoint. But even if our store is a database, we'll still have the hassle of setting up the in-memory database.

Another solution might be to use a mock Context that returns an IQueryable. But wouldn't it be wonderful if we could avoid mocking all together?

Killing the DAO

The first question is why we even have a DAO tier to begin with. The original idea was that we wanted a place to put code specific to a particular data store. In other words we wanted to isolate the code that will need to be changed should the data store switch from SQL Server to Oracle. But isn't that exactly what LINQ does? I'd be pretty surprised if there wasn't a decent LINQ provider for just about any data store at this point that required more than minimal code changes. So why not embrace LINQ and reconsider alternatives to a DAO tier?

One alternative that I've been using for over a month now is to switch to extension methods. To give credit where it's due the idea originated with a conversation with fellow Near Infinity employee Joe Ferner. And I'm sure the idea isn't particularly original (please post in the comments if you know others that use this approach).

Using this technique our code changes from something like this:

var employeeDao = new EmployeesDao(); // or use IOC of course
employeeDao.FindBySSN(ctx, "111-11-1111");

To something like this:

ctx.Employees.FindBySSN("111-11-1111");

Among other things I find this far more aesthetically pleasing because each of the three elements to the statement represent a subsequent filtering of data. It's a more functional way of looking at things.

We could implement this off of the Employees property of the context if we have control over that (which I don't with spmetal). But if we implement this as an extension method like this:

public static class EmployeeExtensions {
    public static Employee FindBySSN(this IQueryable<Employee> employees, string ssn) {
        return employees.SingleOrDefault(e => e.Ssn == ssn);
    }
}

We now have something that's considerable easier to unit test.

Testing It

Once we've refactored our function as an extension method that filters down the corpus of entities, we can test the code using in-memory objects with a call to .AsQueryable(). For instance:

public void FindBySSN_OneSsnExists_EmployeeReturned() {
       var employees = new [] { new Employee { Ssn = "111-11-1111" } };
       var actual = employees.AsQueryable().FindBySSN("111-11-1111");
       Assert.IsNotNull(actual);
}

Notice we didn't have to mock anything.

Testability, but at What Cost?

This technique works great for the example above, but how does it scale to harder problems and what other downsides are there?

As far as scalability I've found this technique works great for every scenario I've run across in the month I've been doing it. It works for joins, aggregations, and even for inserts, update, and deletes.

As far as downsides the astute reader may be wondering about mockability. For instance what if we want to mock the call to FindBySSN and give it the exact Employee that will be returned. This scenario is admittedly harder. But what I've found is that far more often than not I don't really need to mock the types of things that used to live in the DAO tier. Instead I just mock the Employee object off of context to return in-memory objects and make my tests slightly larger in scope. Most of the time I find the larger scope increases the usefulness of the test. In the occasional case where I do really want to mock the "DAO" tier I use a technique described in this post by Daniel Cazzulino.

Conclusion

Obviously there is more to this architecture, for instance how do you handle insert and update operations? The short answer is it's easy, but I'll save that topic for a future post. For now why not give this approach a try? You weren't really happy with that useless old DAO tier anyway, were you? I say we eradicate it and never look back.

ASP.AJAX 4 Templates Part 4 - jQuery, and Manual JSON Object Manipulation

Lee Richardson — Sat, 10 Apr 2010 14:03:06 -0400

This is part 4 of a multi-part series exploring building client side AJAX applications with ASP.Net AJAX 4 Templating and the WCF Data Services (aka ADO.Net Data Services, aka oData, aka Astoria) in SharePoint 2010.

Part 1 - Exploring WCF Data Services in SharePoint 2010
Part 2 - Creating a read-only page with ASP.Net AJAX 4.0 Templates
Part 3 - Writing data back to SharePoint with ASP.Net AJAX 4.0 Templates
Part 4 - jQuery and Manual JSON Object Manipulation

In the previous posts in this series I've shown how you can access SharePoint data using WCF Data Services, how you can display that data using ASP.Net AJAX 4.0 Templates, and how you can write data back to SharePoint.

In this post I'll take writing data back to SharePoint a step further by showing how you can modify non-visible fields by modifying JSON objects directly. More specifically I'll enable dragging user story index cards on a virtual bulletin board using a jQuery plugin, save the X and Y coordinates back to JSON objects, and then save the updated records back to SharePoint.

Cards on a Corkboard

The first thing I'll do is style the cards and display them with absolute position. While I'm at it I'll also reference the draggable jQuery UI plugin. If you're following along at home just built a custom .js download from the jQuery UI site, drop it in your layouts directory and add a reference. So my PageHead content control now looks like this:

<asp:content
    id="PageHead"
    contentplaceholderid="PlaceHolderAdditionalPageHead"
    runat="server">

    <script type="text/javascript"
        src="../ajax/MicrosoftAjax.js"></script>
    <script type="text/javascript"
        src="../ajax/MicrosoftAjaxDataContext.js"></script>
    <script type="text/javascript"
        src="../ajax/MicrosoftAjaxTemplates.js"></script>
    <script type="text/javascript"
        src="../ajax/MicrosoftAjaxAdoNet.js"></script>
    <script type="text/javascript"
        src="../jquery-ui-1.8.custom/js/jquery-1.4.2.min.js"></script>
    <script type="text/javascript"
        src="../jquery-ui-1.8.custom/js/jquery-ui-1.8.custom.min.js">
    </script>
    <style type="text/css">
        .sys-template
        {
            display: none;
        }
        .userstorycard
        {
            border: 1px solid #777777;
            width: 200px;
            position: absolute;
            cursor: move;
        }
        .carddescription
        {
            font-size: 13px;
            background-image: url('card_bg.jpg');
            padding: 0px 5px 5px 5px;
        }
        .cardtitle
        {
            font-size: 15px;
            font-weight: bold;
            padding: 0px 0px 0px 0px;
            border-bottom: 1px solid red;
            background-color: White;
            padding: 0px 5px 0px 5px;
        }

...

The next thing to do is add X and Y fields to the user story list in SharePoint. And finally display the cards with absolute positioning:

Notice the sys:style attribute. The sys: part is the namespace you need to add for attributes when using templating that I mentioned in part three. Also notice the JavaScript string concatenation inside the binding. You can put in any JavaScript in there that you like. So now if you manually set X and Y values in SharePoint for a card the card shows in the right place. But we're still missing the ability to drag cards.

Making Cards Draggable with jQuery

In order to enable dragging support you're supposed to call the draggable() function on the DOM elements you want to make draggable. Simple right? You just call $(".userstorycard").draggable() which should find every element with a class of userstorycard and call draggable() on it.

But if you do this onLoad() then ASP.Net AJAX Templating engine hasn't had a chance to render the new DOM elements yet. Fortunately the DataView has the JavaScript equivalent of an OnRendered event that you can tie into:

And that's it for enabling dragging. Pretty simple. The result looks like this:

Ok, well you can't see the dragging support, but it's there, trust me. And it's pretty cool except for one thing.

Saving Non-Visible Properties

The problem is that every time you refresh the page the nice layout you've done disappears and everything goes back to being stacked one on top of the other in the upper left. My first thought was to create input type=hidden form elements on the page. This approach causes a lot more plumbing code than is necessary. The better way is that you can access the underlying in-memory JSON objects. Step one is to register an "event" for when the user stops dragging a card.

function onRendered() {
    // from http://jqueryui.com/demos/draggable/
    $(".userstorycard").draggable({
        stop: onDragStop
    });
}

Once you wire that up the onDragStop() function is where all the interesting stuff happens:

function onDragStop(event, ui) {
    var userStoryCard = ui.helper[0];
    var selectedUserStoryJsonObject =
        dataView.findContext(userStoryCard).dataItem;

    var newX = ui.position.left;
    var newY = ui.position.top;

    Sys.Observer.setValue(selectedUserStoryJsonObject, "X", newX);
    Sys.Observer.setValue(selectedUserStoryJsonObject, "Y", newY);
    dataContext.saveChanges();
}

The first line gets the DOM element that just the user just completed dragging. The second line retrieves the in memory JSON object represented by that DOM element. The next couple of lines get the X and Y that the user story card was dragged to relative to their parent DOM element. And then something odd happens.

Sys.Observer.setValue looks so complicated. Why couldn't we just call selectedUserStoryJsonObject.X = newX?

Well, if we'd done that the DataView wouldn't know about the change we just made. If we then tried to call dataContext.saveChanges() nothing would get sent back to the server. Sys.Observer.setValue notifies any interested parties, in this case the DataView, that a value has been changed. And this enables the functionality we saw in part two where the DataView only sends the relevant records back to SharePoint instead of the entire set of JSON objects.

And so, with this code in place, we can now move user story cards around on the page and when we refresh the page or come back days later our layout has persisted back into SharePoint.

Conclusion

In this post you've seen how to directly modify the in-memory JSON objects and send them back to SharePoint. Our application is finally getting interesting and I find it quite remarkable how little code it's taken. In the next post I'll show how to do master-detail scenario's for editing user story cards.

Client Side AJAX Applications in SharePoint 2010 - Part 3

Lee Richardson — Sun, 21 Mar 2010 22:17:25 -0400

Note: This is one of a multi-part series exploring building client side AJAX applications with ASP.Net AJAX 4 Templating and the WCF Data Services (aka ADO.Net Data Services, aka oData, aka Astoria) in SharePoint 2010.

In the previous two posts in this series I've shown how you can access SharePoint data using WCF Data Services and consume that data using ASP.Net AJAX 4 templates. In this post I'll update the previous example in order to show how to write data back to SharePoint.

Templates and Linking

Before we go into writing data back let's modify our example to link each item back to the SharePoint DispForm.aspx page. A naïve approach would look like this:

<ul id="userStoriesList" class="sys-template">
<li><a href="../../Lists/User%20Stories/DispForm.aspx?ID={{ ID }}">{{ Title }}</a></li>
</ul>

It turns out this approach doesn't work because the DataView won't look for the data binding syntax inside of attributes. The next approach that might make sense would now be to set href to a data binding syntax and inside there do some string concatenation like so:

<li><a href="{{ '../../Lists/User%20Stories/DispForm.aspx?ID=' + ID }}">{{ Title }}</a></li>

It turns out this doesn't work either, but for a different reason. Apparently the DataView can't natively convert some HTML attributes like href and src for reasons Bertrand La Roy describes. The solution is to prefix the attribute with the ASP.Net AJAX system namespace. Typically you would define the system namespace on the HTML body element. But with SharePoint you don't have access to this element. Fortunately you can declare this namespace on any element so the following code works:

<div xmlns:sys="javascript:Sys">
    <ul id="userStoriesList" class="sys-template">
        <li><a sys:href="{{ '../../Lists/User%20Stories/DispForm.aspx?ID=' + ID }}">{{ Title }}</a></li>
    </ul>
</div>

Great. Let's move on to updating data.

Live Binding Syntax

The {{ }} template syntax we've seen so far is what's known as one time binding. When you use this syntax the expression is evaluated only once when the template is rendered. What we want is one way live binding and/or two way live binding.

In order to understand the live bindings you need to understand that behind the scenes the DataView keeps an in memory copy of all of the JSON objects it downloads. It's possible for the data in these objects to change. And that's where one way live bindings come in. When you use the syntax { binding [FieldName] }, the DataView will automatically update the binding when the underlying JSON object changes.

Two way live binding happens automatically when you use the one way live binding syntax on HTML INPUT elements. For these scenarios the DataView automatically updates the in memory JSON objects. Then the DataView updates any one way live binding's.

So let's throw two DataViews onto the page, one with one way live bound hyperlinks, one with two way bound HTML INPUT textbox elements and see what happens:

Notice the input element with the sys:value bound to title and the new binding syntax in the text of the a href. The result looks like this:

It may not be all that pretty yet, but if you change the text in any of the text boxes the result is immediately updated in the hyperlinks on mouse out. But if you hop back to SharePoint or refresh the page you'll see that our data hasn't been updated.

Updating SharePoint, For Real This Time

Just because the DataView updated the in memory JSON objects it's bound to doesn't mean it sent a POST request back to ListData.svc. To accomplish that we need to call dataContext.saveChanges(). If you toss in a button below the table like this:

<button onclick="dataContext.saveChanges()">Save Changes</button>

And move the dataContext variable to a higher scope like this:

var dataContext;

function onLoad() {
dataContext = $create(Sys.Data.AdoNetDataContext, { serviceUri: "/demoprep/_vti_bin/ListData.svc" });

Then when you click the button the AdoNetDataContext figures out which JSON objects changed and sends just those objects back to ListData.svc. How do you know it only sends the rows that changed? If you look at Fiddler there will be a POST to ListData.svc/$batch with something like the following:

--batch_d425-6668-6b05
Content-Type: multipart/mixed;boundary=changeset_8a05-bdcd-0b3e
--changeset_8a05-bdcd-0b3e
Content-Type: application/http
Content-Transfer-Encoding: binary
MERGE http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3) HTTP/1.1
If-Match: W/"3"
Host: localhost
Accept: application/json
Accept-Charset: utf-8
Content-Type: application/json;charset=utf-8
{
"__metadata":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)",
"etag":"W/\"3\"",
"type":"Microsoft.SharePoint.DataService.UserStoriesItem"},
"ContentTypeID":"0x01080070E2807D369BD94FBD6C057D3110E6D3",
"Title":"Renamed Task",
"Predecessors":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Predecessors"}},
"Priority":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Priority"}},
"PriorityValue":"(2) Normal",
"Status":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Status"}},
"StatusValue":"Not Started",
"Complete":null,
"AssignedToID":null,
"TaskGroupID":null,
"Description":"<div></div>",
"StartDate":"\/Date(1266624000000)\/",
"DueDate":null,
"Points":8,
"Iteration":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Iteration"}},
"IterationID":1,
"ID":3,
"ContentType":"Task",
"Modified":"\/Date(1269128451000)\/",
"Created":"\/Date(1266679964000)\/",
"CreatedByID":1,
"ModifiedByID":1,
"Owshiddenversion":3,
"Version":"3.0",
"Attachments":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Attachments"}},
"Path":"/demoprep/Lists/User Stories"
}
--changeset_8a05-bdcd-0b3e--
--batch_d425-6668-6b05--

And that, as you may have noticed, is the JSON for just a single of the rows that we were displaying.

But how did the AdoNetDataContext know which objects were changed? Jonathan Carter explains this better than I could but basically when the data context downloads JSON objects it injects them with the ability to notify itself when they change. As Jonathan points out it's a neat trick that would not be possible in a statically typed language.

Summary

We've finally gotten to some of the core benefits of using ASP.Net AJAX 4 Templating. With very little JavaScript we were able to cleanly separate our UI from our data access code and write data back to the server. And the amount of HTTP traffic and server operations was kept to a bare minimum, keeping things as fast and responsive as possible. So all we're missing at this point is how to use it to make something real out of these technologies. And that will be the topic for the next post.

Client Side AJAX Applications in SharePoint 2010 - Part 2

Lee Richardson — Sun, 21 Mar 2010 20:36:54 -0400

Part 1 - Exploring WCF Data Services in SharePoint 2010
Part 2 - Creating a read-only page with ASP.Net AJAX 4.0 Templates
Part 3 - Writing data back to SharePoint with ASP.Net AJAX 4.0 Templates

In Part 1 I described how SharePoint 2010 exposes list data through ListData.svc and WCF Data Services. So far this is interesting, but the next logical question is how would you actually build a Web 2.0 application with the data? One answer is with a new technology called ASP.Net AJAX 4.0 Client Side Templating. I'll give you some background and then provide a walkthrough of how to build a SharePoint application page that reads and writes data to SharePoint through ListData.svc.

What is ASP.Net AJAX 4.0 Client Side Templating?

In Part 1 I talked about some of the problems with the two existing ways of writing Web 2.0 interfaces. ASP.Net AJAX 4.0 Client Side Templating addresses most of these issues. Specifically it:

Minimizes plumbing code
Cleanly separates presentation from data access code
Simplifies saving data back to the server via Live Data Bindings
Has no ViewState!
Uses lightweight JSON rather than HTML

Sounds good? Ready to see some code?

Show Me The Code

Note: This walkthrough assumes you haven't played with Visual Studio 2010 and SharePoint 2010, so please skip ahead to "No Really, Show Me the AJAX Templating Code" if you've already seen the sweet SharePoint and Visual Studio integration.

To get started open Visual Studio 2010. Select New Project and for the template navigate to SharePoint then 2010 select "Empty SharePoint Project" (p.s. see Visual Web Part in the list? Make sure you look that up if you haven't heard of it yet).

If you haven't been keeping up with what's new in SharePoint 2010 you should immediately notice that SharePoint integration is built into Visual Studio 2010 out of the box just based on the listed templates. But just wait, it gets much better.

The next screen asks what site you want to use for debugging. It's asking this because when you hit F5 ("Start Debugging") Visual Studio 2010 packages up your wsp file, deployes it to the URL you enter here, attaches to the w3wp process, and opens a browser into SharePoint, and you're immediately in debug mode!

If you aren't deeply impressed then you haven't developed in WSS 3.0. Now we aren't writing any server side code in this tutorial, but it's still extremely impressive how much work Microsoft has put into making SharePoint a first class citizen.

Once you have your site select Project -> Add New Item and select Application Page.

Visual Studio then plops out a page that's 100% ready to go including the correct Content controls for the SharePoint master page.

<%@ Assembly Name="$SharePoint.Project.AssemblyFullName$" %>
<%@ Import Namespace="Microsoft.SharePoint.ApplicationPages" %>
<%@ Register Tagprefix="SharePoint" Namespace="Microsoft.SharePoint.WebControls" Assembly="Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<%@ Register Tagprefix="Utilities" Namespace="Microsoft.SharePoint.Utilities" Assembly="Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<%@ Register Tagprefix="asp" Namespace="System.Web.UI" Assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" %>
<%@ Import Namespace="Microsoft.SharePoint" %>
<%@ Assembly Name="Microsoft.Web.CommandUI, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="ApplicationPage1.aspx.cs" Inherits="MyAppPage.Layouts.MyAppPage.ApplicationPage1" DynamicMasterPageFile="~masterurl/default.master" %>

<asp:Content ID="PageHead" ContentPlaceHolderID="PlaceHolderAdditionalPageHead" runat="server">

</asp:Content>

<asp:Content ID="Main" ContentPlaceHolderID="PlaceHolderMain" runat="server">
Hello World!
</asp:Content>

<asp:Content ID="PageTitle" ContentPlaceHolderID="PlaceHolderPageTitle" runat="server">
Application Page
</asp:Content>

<asp:Content ID="PageTitleInTitleArea" ContentPlaceHolderID="PlaceHolderPageTitleInTitleArea" runat="server" >
My Application Page
</asp:Content>

(ok, the "Hello World!" is mine). Now if you hit F5 or click Build -> "Deploy Solution" then as I mentioned SharePoint packages up the application page into a wsp file and deployes it out to SharePoint. Now if you navigate to your application page (in my case http://localhost/mysite/_layouts/MyAppPage/ApplicationPage1.aspx) then you should see something like this:

Amazed? You should be. We didn't write a line of evil looking CAML, didn't have to write any DOS batch files, didn't have to deploy to the GAC, we probably didn't even notice that a feature.xml file was created behind the scenes. Yet already we're deployed! Ok, enough exuberance, back to business.

No Really, Show Me the AJAX Templating Code

Before writing code you should probably create some lists to read and write to. As I mentioned in a previous post I like tracking user stories in SharePoint. So for my examples I have an Iterations list and a User Stories list, which is based off of the Tasks content type (automatically giving you a number of columns like Priority).

After creating some lists you need to ensure you have the latest ASP.Net Ajax Library. I suspect an updated copy will be included with the RTM copy of SharePoint 2010, but in my case I downloaded the latest copy and moved the entire Scripts directory into C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\LAYOUTS\ajax. Note that out of the box SharePoint 2010 already included some of these files in LAYOUTS, but it was missing others, so this step was necessary.

So here is the absolute simplest templating example that still works:

<asp:Content ID="PageHead" ContentPlaceHolderID="PlaceHolderAdditionalPageHead" runat="server">
    <script type="text/javascript" src="../ajax/MicrosoftAjax.js"></script>
    <script type="text/javascript" src="../ajax/MicrosoftAjaxDataContext.js"></script>
    <script type="text/javascript" src="../ajax/MicrosoftAjaxTemplates.js"></script>
    <script type="text/javascript" src="../ajax/MicrosoftAjaxAdoNet.js"></script>
</asp:Content>

<asp:Content ID="Main" ContentPlaceHolderID="PlaceHolderMain" runat="server">
    <script type="text/javascript">
        function pageInit() {
            var dataContext = $create(Sys.Data.AdoNetDataContext, {
                       serviceUri: "/demoprep/_vti_bin/ListData.svc"
                     });

            $create(Sys.UI.DataView,
                {
                    autoFetch: true,
                    dataProvider: dataContext,
                    fetchOperation: "UserStories",
                    fetchParameters: { $top: 20 }
                },
                null,
                null,
                $get("userStoriesList")
            );
        }
        Sys.Application.add_init(pageInit);
    </script>

    <ul id="userStoriesList" class="sys-template">
        <li>{{ Title }}</li>
    </ul>
</asp:Content>

The four script references in the PageHead Content control are necessary to define the required ASP.Net AJAX JavaScript classes you'll need for templating. But the Main content control is where things get interesting.

Skip over the JavaScript for now and notice the unordered list named userStoriesList. Inside of the li elements is a strange notation: {{ Title }}. This specifies a binding to the name of a SharePoint field. You could also have put in PriorityValue or any of the other columns specified in the JSON that I showed in Part 1.

As far as the JavaScript, the dataContext variable is a class of type AdoNetDataContext, which knows how to talk to WCF Data Services like ListData.svc. It inherits from a JavaScript class called DataContext which is extensible and knows how to talk with other protocols, but us SharePoint developers probably won't care about this most of the time.

The userStoriesTemplate variable is a class of type DataView. This is where the magic happens. DataViews pull data from a DataContext, duplicates the innerHtml of an associated DOM element for each row returned (e.g. userStoriesList), and replace the binding syntax with real data. Seems simple enough, but it is actually quite sophisticated when you get to some more advanced scenarios that I'll describe later.

One last thing to notice is the sys-template class. This is a prespecified CSS class that ASP.Net AJAX sets to display: block when it is finished rendering. It's a good idea to create a CSS class for sys-template with display: none so that end users don't see your template code while the page is loading. Obviously you should do this in a separate CSS class in LAYOUTS or if you're lazy you can put it in the PageHead content control like so:

<STYLE type="text/css">
       .sys-template { display: block; }
</STYLE>

<asp:Content ID="PageHead" ContentPlaceHolderID="PlaceHolderAdditionalPageHead" runat="server">
       <script type="text/javascript" src="../ajax/MicrosoftAjax.js"></script>
       <script type="text/javascript" src="../ajax/MicrosoftAjaxDataContext.js"></script>
       <script type="text/javascript" src="../ajax/MicrosoftAjaxTemplates.js"></script>
       <script type="text/javascript" src="../ajax/MicrosoftAjaxAdoNet.js"></script>
       <STYLE type="text/css">
              .sys-template { display: block; }
       </STYLE>
</asp:Content>

So now if you click If you click Build -> Deploy Solution you should see something like this:

Summary

Ok, this isn't super exciting yet. We certainly could have done this much easier with server side code. But bear with me, this has a lot of potential. For instance, how do you write data back to the server? How do you really make it AJAXy? I'll explore these questions in Part 3 - Writing data back to SharePoint with ASP.Net AJAX 4.0 Templates and subsequent posts in the series.

Client Side AJAX Applications in SharePoint 2010 - Part 1

Lee Richardson — Sun, 21 Mar 2010 18:37:06 -0400

If you haven't played with it yet, one of the things you'll immediately notice about SharePoint 2010 is that the UI is extremely AJAX centric. For instance modifying or adding list items is either done in-line with partial page refreshes or via lightbox popups. It's very pleasant to use.

One of the consequences of this Web 2.0 design is that the expectations of SharePoint end users will significantly increase. No longer will long delays and/or full page refreshes in your SharePoint based custom applications be acceptable. End users may not be able to clearly articulate it, but unless as developers we update our techniques to match the new technology our SharePoint based applications will begin to feel stale and will stick out from the rest of SharePoint 2010.

Fortunately Microsoft has stepped up the support it offers Web 2.0 developers in a big way. One of the most compelling features is in how SharePoint exposes all list data via REST/ATOM and JSON. This feature combined with the new templating feature of ASP.Net AJAX 4 allows developers to easily write fast, responsive user interfaces. I will explore these two technologies in the following series:

Why do we need yet another framework?

If you attempt to develop Web 2.0 applications in SharePoint with the techniques you have today there are really only two approaches: ASP.Net AJAX and server side controls or manually coding JavaScript, probably with jQuery or other JavaScript Libraries. If you take the ASP.Net AJAX approach you'll undoubtedly be using an UpdatePanel to get partial page refreshes and then writing ASP.Net code like you always have. There are several problems with this approach:

It sends the entire ViewState with every asynchronous request - this can result in slow, unresponsive apps (this can actually sneak up on you once you have a lot of production data as happened to me recently)
Even without a ViewState the UpdatePanel transmits large amounts of HTML (rather than lightweight JSON data)
It runs through the entire page lifecycle on updates, executing everything on the page even if it isn't relevant to what you're doing
ASP.Net AJAX doesn't have the simplicity of an ETag based HTTP caching like with a REST solution
Finally SharePoint just doesn't work well with UpdatePanel, I can't tell you how many issues I've had with various controls not working out of the box

And if you take the manual JavaScript rout you are faced with a number of different problems:

You have to write a lot of plumbing code

You have to write a service to return your data
You have to pay particular attention to security and permissions since you're exposing your API's to the world
You have to convert your data (ideally JSON) into HTML with probably string concatenation or document.createElement statements

Your UI code is not clearly separated from your data access code making your code hard to maintain
Saving data back to the server involves even more plumbing code

SharePoint 2010 combined with ASP.Net AJAX 4.0 solves these problems. It probably introduces new problems too, but I haven't run into too many thus far. So over the next couple of posts I hope to convince you that we do need yet another web framework and that Microsoft is headed in a great direction with these technologies.

So How Does it Work?

In WSS 3.0 you were able to access list data with the SOAP based Lists.asmx. This technique worked well for .Net desktop apps or server side application pages, but not so well with client side JavaScript. SharePoint 2010 exposes its list data with REST based ListData.svc (though WCF Data Services which I'll describe in more detail below). Once you install WCF Data Services on a machine with SharePoint 2010 you can simply navigate to http://[server]/[optional site]/_vti_bin/ListData.svc and see all of the lists you have permission to access. The result will look like this:

<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>
<service xml:base="http://localhost/mySite/_vti_bin/ListData.svc/"
    xmlns:atom="http://www.w3.org/2005/Atom"
    xmlns:app="http://www.w3.org/2007/app"
    xmlns="http://www.w3.org/2007/app">
<workspace>
    <atom:title>Default</atom:title>
    <collection href="Iterations">
      <atom:title>Iterations</atom:title>
    </collection>
    <collection href="UserStories">
      <atom:title>UserStories</atom:title>
    </collection>
</workspace>
</service>

You can then append the names of lists (e.g. http://localhost/site/_vti_bin/ListData.svc/UserStories) and immediately view the list items in the list as an ATOM feed. A typical result looks like this:

<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>
<feed xml:base="http://localhost/mySite/_vti_bin/ListData.svc/"
    xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices"
    xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"
    xmlns="http://www.w3.org/2005/Atom">
<title type="text">UserStories</title>
<id>http://localhost/mySite/_vti_bin/ListData.svc/UserStories</id>
<updated>2010-03-20T02:20:26Z</updated>
<link rel="self" title="UserStories" href="UserStories" />
<entry m:etag="W/"3"">
    <id>http://localhost/mySite/_vti_bin/ListData.svc/UserStories(2)</id>
    <title type="text">Blah</title>
    <updated>2010-03-19T15:13:28-05:00</updated>
    <author>
      <name />
    </author>
    <link rel="edit" title="UserStoriesItem" href="UserStories(2)" />
    <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Predecessors"
        type="application/atom+xml;type=feed"
        title="Predecessors"
        href="UserStories(2)/Predecessors" />
    <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Priority"
        type="application/atom+xml;type=entry"
        title="Priority"
        href="UserStories(2)/Priority" />
    <link
        rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Status"
        type="application/atom+xml;type=entry"
        title="Status"
        href="UserStories(2)/Status" />
    <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Iteration"
        type="application/atom+xml;type=entry"
        title="Iteration"
        href="UserStories(2)/Iteration" />
    <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Attachments"
        type="application/atom+xml;type=feed"
        title="Attachments"
        href="UserStories(2)/Attachments" />
    <category
        term="Microsoft.SharePoint.DataService.UserStoriesItem"
        scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
    <content type="application/xml">
      <m:properties>
        <d:ContentTypeID>0x01080070E2807D369BD94FBD6C057D3110E6D3</d:ContentTypeID>
        <d:Title>Do Something</d:Title>
        <d:PriorityValue>(2) Normal</d:PriorityValue>
        <d:StatusValue>Not Started</d:StatusValue>
        <d:Complete m:type="Edm.Double" m:null="true" />
        <d:AssignedToID m:type="Edm.Int32" m:null="true" />
        <d:TaskGroupID m:type="Edm.Int32" m:null="true" />
        <d:Description><div></div></d:Description>
        <d:StartDate m:type="Edm.DateTime">2010-02-20T00:00:00</d:StartDate>
        <d:DueDate m:type="Edm.DateTime" m:null="true" />
        <d:Points m:type="Edm.Double">3</d:Points>
        <d:IterationID m:type="Edm.Int32">3</d:IterationID>
        <d:ID m:type="Edm.Int32">2</d:ID>
        <d:ContentType>Task</d:ContentType>
        <d:Modified m:type="Edm.DateTime">2010-03-19T15:13:28</d:Modified>
        <d:Created m:type="Edm.DateTime">2010-02-20T15:32:26</d:Created>
        <d:CreatedByID m:type="Edm.Int32">1</d:CreatedByID>
        <d:ModifiedByID m:type="Edm.Int32">1</d:ModifiedByID>
        <d:Owshiddenversion m:type="Edm.Int32">3</d:Owshiddenversion>
        <d:Version>3.0</d:Version>
        <d:Path>/mySite/Lists/User Stories</d:Path>
      </m:properties>
    </content>
</entry>
</feed>

You can append various parameters to filter, sort, or paginate your data from this page. For example http://localhost/site/_vti_bin/ListData.svc/UserStories?$filter=(PriorityValue eq '(2) Normal')&$orderby=Title will return list items whose priority is normal sorted by title. See Query Expressions on MSDN for more details.

Now if you're interested you can take it a step further by viewing the data as JSON. To do this you'll need to modify the HTTP headers. Fortunately there is a tool from Microsoft for viewing, replaying, and modify HTTP sessions called Fiddler. After you install it open Fiddler from the add-in in your web browser (I used IE, but Firefox should work too). Then refresh the web page for a list (in my case ListData.svc/UserStories). Now in Fiddler drag the last request to the Request Builder, ensure "Automatically Authenticate" is checked in options, and replace "Accept: */*" with "Accept application/json". When you click Execute you should see something like:

{
"d" : {
"results": [
{
"__metadata": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)",
"etag": "W/\"3\"",
"type": "Microsoft.SharePoint.DataService.UserStoriesItem"
}, "ContentTypeID": "0x01080070E2807D369BD94FBD6C057D3110E6D3",
"Title": "Blah",
"Predecessors": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Predecessors"
}
}, "Priority": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Priority"
}
}, "PriorityValue": "(2) Normal", "Status": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Status"
}
}, "StatusValue": "Not Started",
"Complete": null,
"AssignedToID": null,
"TaskGroupID": null,
"Description": "<div></div>",
"StartDate": "\/Date(1266624000000)\/",
"DueDate": null,
"Points": 3,
"Iteration": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Iteration"
}
}, "IterationID": 3,
"ID": 2,
"ContentType": "Task",
"Modified": "\/Date(1269011608000)\/",
"Created": "\/Date(1266679946000)\/",
"CreatedByID": 1,
"ModifiedByID": 1,
"Owshiddenversion": 3,
"Version": "3.0",
"Attachments": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Attachments"
}
}, "Path": "/demoprep/Lists/User Stories"
}
]
}
}

That's SharePoint data as JavaScript Object Notation. Notice the etag and href's to associated list data. Cool.

But ListData.svc isn't just for reading data. To experiment with it try deleting a list item. To do this in fiddler set the HTTP verb to "DELETE", set the URL to a particular list item's URL (e.g. http://localhost/site/_vti_bin/ListData.svc/UserStories(2) where 2 is the ID for a list item), leave Request Headers empty, and hit Execute. Refresh the list of items and yours is gone. Permanently. The item isn't even in the recycle bin.

Security in WCF Data Services

So at this point you are undoubtedly concerned about security. Just to give some background ListData.svc is provided by a technology called WCF Data Services, which was formerly called ADO.Net Data Services, which was code named Astoria. And WCF Data Services is an implementation of the open data protocol (OData) which Scott Hanselman has an excellent podcast about.

This technology was originally designed for exposing database data. And in this scenario you do have to handle your own security. SharePoint 2010, however, takes things a step further by giving you WCF Data Services (nearly) out of the box while also handling your authorization security.

For instance if you modify the row level permissions for a list item to deny read permissions to a user then ListData.svc will not return that list item for that user. If you try to access the URL for that list item directly SharePoint returns a 404. If you try to modify or delete a list item you don't have permissions to modify then SharePoint returns a 401 Unauthorized.

Very cool. I find this especially nice because you don't have to worry about maintaining your security code. It's intimately tied to SharePoint 2010's security model which is already quite sophisticated.

Incidentally, security is not all that's included when you use these services. SharePoint also performs data validation and returns the appropriate HTTP error codes if you for instance violate the new uniqueness constraint.

Summary

SharePoint offers everything you've seen so far (nearly) out of the box and without a line of code. This sets the stage for writing some nice client side applications with technologies like ASP.Net AJAX 4.0 client site templating. Which is exactly what I intend to do in Part 2 - Creating a read-only page with ASP.Net AJAX 4.0 Templates.

[video] How To Create Burndown Charts For User Stories in SharePoint

Lee Richardson — Fri, 12 Feb 2010 16:46:16 -0500

I love tracking user stories in SharePoint. It's free (assuming you have access to a Windows Server 2003 or 2008 machine with WSS turned on) and is extremely flexible. Adding new custom columns is easy. Adding custom views is easy. Bulk editing is amazingly simple and powerful (assuming you're willing to use IE). And building a custom home page "portal" with all the states of your team's workflow (e.g. unassigned, stories I'm working on, my resolved stories, to peer test, etc) is easy and is intuitive to use.

This 13 minute screencast shows you how to create a user stories list in SharePoint and build burn up and burn down charts off of the data using Microsoft Excel. It includes the custom fields you'll need in SharePoint, the formula's you'll have to create in Excel, and shows tips along the way like how to simplify refreshing your reports when data changes.

Once you've watched the screencast if you want to implement this yourself here is the information you'll need:

Custom SharePoint Columns

Iteration (Choice column e.g. Iteration00, Iteration01, zBacklog)
Released (Date - the date the iteration was closed out)
Points (Number)
Created Date (for demo only, you should be able to use "Created" on your project)

Calculated Columns in Excel

Total Points to Date
- The total points on a specific date including backlog, current iteration and all completed stories
- =SUMIF([Created Date],CONCATENATE("<=",Table_owssvr_1[[#This Row],[Released]]),[Points])
Completed To Date
- Total points for stories released prior to current story's release date
- =SUMIF([Released],CONCATENATE("<=",Table_owssvr_1[[#This Row],[Released]]),[Points])
Backlog
- What remains to be completed at a point in time (e.g. don't include user stories added to the backlog in iteration 4 if this is for a task in iteration 03)
- =Table_owssvr_1[[#This Row],[Total Points to Date]]-Table_owssvr_1[[#This Row],[Completed to Date]]

Burndown Pivot Table Values

Row: Released
Filter: Released After [Some Date]
Values: Max(Backlog)

Burnup Pivot Table Values

Row: Released
Filter: Released After
Values
- Max(Total Points to Date)
- Max(Completed To Date)

Summary

I hope you found the presentation useful and if you are currently using SharePoint for tracking user stories please comment on your experiences.

Business Process Modeling for Software Developers

Lee Richardson — Wed, 23 Dec 2009 09:01:57 -0500

As a software developer I never thought I'd be saying this (I suppose eight years of working for the company that invented this technique might bias me), but you can not underestimate the value of business process modeling when starting a new project. This is especially true if it's a small project and you don't have the benefit of a dedicated requirements analyst.

As a consultant (or as a person who is learning someone else's business) your job is to understand your customers existing business better than they do themselves. And you need to be on the same page with others regarding how you are going to change their day to day functions. And that's what business process modeling is for.

You may ignore this document after the first week on the project, in fact I would encourage you to. But even if you throw it away immediately after creating it, the process of developing the document will still:

Flush out important questions
Show your customer you understand their world
Help document the project to other developers
Facilitate communication (especially with the people who pay the bills)
Generate user stories (requirements)
Identify the entities that can feed into an Entity Relationship Diagram (or database I suppose)
Aid making good choices for the decisions it be hard to undo later (like whether to manually code workflows or use Windows Workflow Foundation)
More clearly identify pain points and areas where your software can help end users; and
Identify metrics that can help determine project success from an ROI and product owner's perspective

Convinced this is a tool you need in your toolbelt yet? As long as the answer isn't "I only write code leave me alone" then check out this quick how-to:

Start with the AS-IS diagram

As soon as possible on a new project you should plan to interview your future end users. This shouldn't be an unstructured Q&A session. You should start with a rough draft of what you think their business process is and ask questions that identify the places where you were wrong.

In order to start the diagram you need to identify both the end users and existing systems and put those in "swimlanes" down the left side of a diagram. Swimlanes are important because they help you be explicit about who is performing what activities. Then flush out activities in the happy path in appropriate swimlanes like so:

As soon as you talk to end users (like the clerk in the example above) you will discover unhappy paths. What if the warehouse is out of a product? What if the payment was declined? What happens if a customer needs to return something?

Obviously you document these using standard activity diagram notation:

Identifying Pain Points/Opportunities

The interesting part about unhappy paths is how often they occur, how they affect the customers of your customer, and ultimately how they affect your customer's bottom line. For instance, how often is payment declined in the example above? Fairly frequently? Can you get an actual percentage? If it turns out it's because the original application failed to validate credit cards, then a small amount of programming effort can provide a large benefit to length of time until order fulfillment. Sometimes you can make the process more efficient without writing a line of code.

And at the end of your project if you can show that you've decreased the average length of time it takes to go through the process, ideally by showing less time was spent in unhappy paths, you will likely have made yourself one happy customer. This can be especially effective if your new system can show statistics (like percent of orders that are declined) and compare those to the old process.

Developing the TO-BE diagram

AS-IS process diagrams are completely useless unless you can effectively describe what the process will look like after you implement your new system. The process of developing these documents is pretty similar to the AS-IS diagram except of course you're trying to fix the pain points.

One technique I like for generating a backlog of user stories (tasks) is to put unique numbers with each step in the TO-BE process, then ensure there is at least one user story per step in the process. You may be surprised at how easily activities in your new project's swimlane map to user stories.

Summary

I'm a fan of writing code as soon as possible on a project, but if you fail to understand an end user's business process you may be writing the wrong code. Business process modeling can ensure you're writing the right code to solve the most important problems first. And that may be the difference between project success and project failure.

Ruby on Rails, a Microsoft Developer's Perspective

Lee Richardson — Wed, 18 Nov 2009 10:42:15 -0500

I suppose I'm a pretty hard core Microsoft developer. I've been doing Microsoft .Net C# development since the .Net framework was first released in 2002. Before that, starting in 1997 about when the technology was first released, I developed in Visual Basic with classic ASP (Active Server Pages). So that's twelve years of Microsoft development all told. Which is why the Ruby on Rails class I've been taking this week is so interesting. Ruby represents a significant branching out for me. So these are my initial impressions. I am a complete Ruby neophyte, so please keep all flames at least friendly in tone :).

Dynamic Typing

Let's get to the heart of this Ruby stuff. I thought I would passionately hate dynamic typing. I was getting ready to fill pages ranting about its evils. But that moment never came. Don't get me wrong the Visual Studio strongly typed intellisense no-reading-documentation-needed experience is far nicer than the NetBeans look-it-up-in-irb-with-.methods experience. But the feeling was more like mild to moderate annoyance rather than unbridled hatred. Like when I mistyped a variable and Ruby decided it was a new variable initialized as nil. Annoying, and hard to track down, but hey I'm a Ruby newbie, experts probably don't make these mistakes, right? Anyway, I would be interested to hear how non-trivial, well tested applications perform with refactoring (e.g. variables/fields in views?) and how bad things get during O&M.

Speed of Development

I figured speed of development would be Ruby's saving grace. It was that and more. The Ctrl-S-Alt-Tab-Ctrl-R-zero-delay development cycle is frickin' amazing! It made development feel more fun than it has in a long time. Part of that may be the slower-than-molasses-in-December speed of development I experience today on my (not 2010) SharePoint project. Regardless, even if I were working in an ASP.Net MVC project I suspect the extra speed of development in Rails by skipping the main compilation and JIT compilation steps would increase the enjoyment of a Ruby project enough to equal out the lack of strong typing.

Active Record vs LINQ

Perhaps I'm just not experienced enough with active record, but to me the syntax feels contrived, non-intuitive, and just kind of agitating. I mean it's cool that you can write

p = Person.find_all_by_first_name_and_last_name("Lee", "Richardson")

Ugly as sin, but cool. The alternative doesn't feel much better:

p = Person.all(:conditions => { :first_name => "Lee", :last_name => "Richardson" })

Uch. Of course I am biased. I'm passionate about LINQ. I love LINQ more than any single feature in Microsoft development. I think LINQ may be the most brilliant stroke of genius Microsoft (Anders Hejlsberg) has ever had. And I don't think I realized how passionately I felt about the technology until I didn't have it. So for me I'll stick with:

Person p = ctx.People.Where(p => p.FirstName == "Lee" && p.LastName == "Richardson");

Duck Typing

My mind was blown when the instructor showed us polymorphism without inheritance. I can't argue about how incredibly powerful it is. I just cringe to think of the potential misuse. But without any real Ruby experience I'll have to just leave it at "Wow!"

Mocking

It's pretty scary that you can override the functionality of any method anywhere in Ruby. But when the instructor showed us overriding DateTime.now for the purposes of mocking I had an "ah ha!" moment. Mocking DateTime.Now in C# is an awful experience that involves an intermediate class with a virtual "Now()" method. Ruby sure got C# on that one.

YAML

It's no secret that I passionately hate XML. And it's no secret that Microsoft passionately loves XML. Pity about the mismatch. Ruby on Rails really endeared itself to me when its designers recognized that XML is a travesty against humanity and used YAML Ain't Markup Language instead. Nice!

Interactive Ruby Console

The Interactive Ruby Console (IRB) is just wonderful. Now I can at least get that functionality today in C# with Joe Ferner's excellent Developer's Toolbox. What I can't get is the Interactive Rails Console (ruby script/console). Now that is awesome. I can't wait for C# 4.0 which I suspect will have this. For now RoR++, C#-- .

Tooling

I'm not sure how related this is to dynamic typing, but surprisingly I really, really missed Visual Studio. Perhaps RubyMine is better that NetBeans. I sure wasn't impressed with Komodo. I just felt like constantly switch between my IDE and various console windows, and a database viewer felt clumsy yet somehow necessary with Rails. It just felt so 1999.

Summary

There's no doubt about it, I could be very happy on a Ruby project. The no XML thing is just a perk, the fast development cycles and bringing fun back to coding again is truly awesome. But am I ready to give up Microsoft development full time just yet? No way. LINQ, strong typing, and believe it or not Visual Studio tip the scales back to about equal. Now if only Microsoft could somehow make compilation instantaneous.

SharePoint 2010 Expectations Meet Reality

Lee Richardson — Sat, 24 Oct 2009 01:26:55 -0400

The current version of SharePoint and I have a love hate relationship. Since I want to be able to look forward to a more joyful and fulfilling relationship with the tool/platform, I made a list of my biggest frustrations before I left for the SharePoint 2010 conference (#spc09). My objective was to return with most of the items checked off as resolved and to have a bunch of new functionality that I hadn't realized I needed. SharePoint 2010 easily fulfils the second promise with stuff like Business Data Services (Awesome!) and Visio Data Services (Amazing!). But I don't want to try to recap the whole conference so here's just how it compares to my initial list:

Testability

SharePoint is notoriously hard to unit test. The good news is Microsoft provided a whole session devoted to software development best practices that included a good chunk on unit testing. It was wonderful to see!

The bad news: SharePoint 2010 will not have public constructors and classes will still be sealed (final). The good news to the bad news: After discussing this issue with Chris Keyser I discovered that Peli de Halleux of Microsoft Research has a free tool called Pex that solves the whole problem. It allows what it calls detours, which allow you to mock non-virtual methods. This stuff is amazing and I'll blog more about it later.

Summary: Testability is better even for existing SharePoint installs -- so between Sporm and Pex this is a non-issue -- I'm happy.

No Referential Integrity

It kills me that if you create an employee list item that references a company list item and you delete the company list item then nothing happens. No cascading delete. No error message. Just an orphan record. SharePoint 2010 now supports both true referential integrity through errors and cascading deletes!

Summary: Awesome!

Validation is Terrible

Ever tried to validate by pattern in SharePoint (e.g. social security number)? How about range validate (e.g. date of birth can't be after today)? Or maybe validate one field against another? The answer is you can't. Not easily or consistently. If you handle validation in an event receiver then the user doesn't get the error until after a postback on a separate page. And custom field controls are a lot of work and they don't work in datasheet view.

SharePoint 2010 mostly solves these validation issues by allowing field level and list level validation. The error messages even show in datasheet view. The bad news is that pattern matching is not currently supported (e.g. you can't validate social security number).

Summary: Good stuff, need to see the final version to be extremely happy.

Poor Documentation

Ever noticed that blogs give you better details on SharePoint APIs than MSDN does? I don't even bother looking on MSDN for SharePoint content anymore. Since the hands on labs didn't contain any documentation I'll have to refer to what Steve Ballmer said during the Q&A session, which was that better documentation will be a priority.

Summary: high hopes, but TBD.

Views Don't Allow Precedence

If you've ever needed a view with A and (B or C) and ended up with (A and B) or C, you know where I'm coming from. As a developer you can implement precedence in CAML, but as an end user you're stuck. Sadly this issue is still not fixed in SharePoint 2010.

Summary: Fail.

Re-Deployment is Hard

Currently redeploying lists is hard; redeploying web parts without overwriting user settings is hard; and redeploying workflows is very hard.

Without documentation and intermittent Internet Joe Ferner and I had a hard time manually confirming that this is better. We also missed a key session on Thursday that talked about it. There were some tweets that looked promising, but I'll have to wait until Thursday's presentations are posted to be able to comment for sure.

Summary: Good I think, but still TBD.

Massive Duplication in CAML Instantiated Lists

Using CAML to instantiate a list is currently awful in SharePoint. It's like 5 pages of CDATA and HTML and XML with massive duplication. Sadly SharePoint 2010 doesn't solve the root problem: list instantiation looks exactly the same. The good news is that along with just about every other aspect of SharePoint 2010 development, Visual Studio makes this task mind numbingly easy to do. If you can just try not to look in that file that was auto-generated for you, then you should be fine.

Summary: Better.

Poor Usability

Once you get used to the current version of SharePoint the user interface is extremely consistent and many users really like it. But if you were to compare it to any modern Web 2.0 site it's a complete failure. SharePoint 2010 does an excellent job of implementing selective refresh/AJAX/Web 2.0. But to me it feels more complicated. Maybe this is because it introduces the ribbon, which I have never liked. Maybe it just is more complicated. In any event I'm not the best judge of usability, so I'll have to wait and see what others say.

Summary: Definitely better, but jury's still out.

Can't Reference a List AND Content Type

Suppose you have a Calendar list. The list supports two content types: Events, and Iterations (sprints). It's a nice architecture because you want to view iterations and events in the same calendar views. Now if you have a User Story list, wouldn't it be nice to have a lookup field that points only to Iterations? Sadly you couldn't do this before and you won't be able to for the foreseeable future.

Summary: Fail.

CAML for Queries

Technically this wasn't on my list because I use Sporm, but on behalf of my non-spormified software development brethren let me say that writing queries in CAML is awful.

SharePoint 2010 makes enormous strides in this area. I won't go into great detail, but you can now use LINQ on the server side and on the client side!

Summary: Awesome!

Summary

Lots of TBD, lots of awesomeness, still some fail. Worth upgrading? Absolutely. Will it still be a love hate relationship? Probably, but at the moment it's looking pretty darn good.

A list of ten current SharePoint failures (such as bad testability, lack of referential integrity, poor usability, etc), and how SharePoint 2010 resolves or fails to resolve each one.

Sporm - Out of the Depths of SharePoint's XML Hell

Lee Richardson — Sun, 13 Sep 2009 21:10:03 -0400

In my last post I described how a new open source tool called sporm significantly simplifies unit testing SharePoint. Making SharePoint unit testable is my absolute favorite feature of sporm because SharePoint is notoriously hard to unit test. But sporm provides other benefits as well and its ability to pull us out of the depths of verbose loosely typed XML hell and into LINQ excellence is next on my list of favorite features. So in this post I'll describe the pre-sporm technique of querying with CAML, how to query data using sporm, and finally how sporm supports SharePoint's unique architecture of allowing multiple content types per list and what that means to you.

Caml's Are Ugly

Warning: if you're new to SharePoint then what you're about to see may shock and upset you. If, like me, you hate both XML and loose typing then you will agree that CAML is awful, but bear with me I promise sporm will make it better. Much better.

CAML or Collaborative Application Markup Language is how one queries for data in SharePoint. A simple query might look like this:

Simple right? ;) In case you didn't catch the meaning from the slightly, uh verbose, query this asks for records with a First_Name of "Lee", a Last_Name starting with "Rich" and a "Dob" less than or equal January 1 2009.

There are a couple of things to note about this query:

Field names are loosly typed. If Dob were ever renamed to DateOfBirth the query would fail at runtime (probably during a demo to a customer, or if you're lucky during integration tests), but certainly not at compile time.
And is a binary operator. This forces explicit precedence and removes the need for parenthesis, but at the cost of readability.
Types must be explicitly defined. I guess this is necessary since it's XML, but somehow I just don't feel like this should be necessary.
It's XML. Ok obviously, but the point is you have to type everything twice. <BeginsWith> </BeginsWith>. Ouch, so verbose, so angley, I so hate XML.

Now, there are tools that make this better. U2U's free CAML Query Builder tool significantly improves the experience of querying SharePoint data.

But it makes you wonder if something is wrong when you need a tool to retrieve data from your data store. Do you typically use tools to assist when you're writing SQL? Probably not. But like I said hang in there, sporm make things better.

Unlinq my Caml

If you were to write the same query as above using sporm it would look like this:

IQueryable<Employee> employees = GetEmployees().Where(e =>
      e.FirstName == "Lee"
      && e.LastName.StartsWith("Rich")
      && e.Dob <= new DateTime(2009, 1, 1));

Just a little easier to read than CAML, right? A couple of things to note:

Fields are strongly typed. If you were to rename FirstName the compiler would catch every single instance at design time.
Operators are standard C# operators. &&, .StartsWith(), and <= are all familiar and concise and use a standard, known precedence.
Types are standard C# types. You never have to explicitly say that "Rich" is a string, it just is.
Your query is actual C# code. The lambda (closure) and the fact that it uses deferred execution might throw off a junior developer in some scenarios, but the query is readable and works exactly the same as if you were querying in memory objects or querying a database with LINQ to SQL or the Entity Framework.

Nice! What's beautiful about this is that sporm converts your C# directly into CAML using C# 3.0's expression trees feature. What sporm can't convert to CAML it executes in memory transparently to you. Sporm uses log4net and outputs its CAML queries to the console by default, so it is a good idea to watch the output if you're concerned about performance.

Now the following isn't relevant to the comparison with CAML, but I would be negligent if I didn't explain how the GetEmployees() function works.

SP != DB B/C of Content Types

First of all GetEmployees looks like this:

private IQueryable<Employee> GetEmployees() {
      return MySPDataContext
            .GetCurrent()
            .GetList<Employees>()
            .OfType<Employee>();
}

How it works is that the static GetCurrent() method retrieves sporm's context object, which knows how to query a SharePoint site, from either the web context or thread local storage; next the GetList() method tells sporm which list you want to query; and the OfType() method tells sporm which content type within the list you want to query. This last part is important because sporm supports SharePoint's ability to have multiple content types per list, which other LINQ providers like LINQ to SharePoint do not. But what are content types and why should you care?

SharePoint's List/Content Type architecture seems odd at first, but it allows interesting scenarios not available in a traditional database. For instance you might have a calendar list that contains multiple types of records (list items) in the same list. Your calendar list might contain some combination of the following three record types: meetings with unique fields like Organizer (a person); iterations with unique fields like DeployedToProduction (a Boolean); and actions with unique fields like RelatedEmployee (a reference to another list). This architecture allows SharePoint to view all three types of records in a single view: like a per month calendar view. The data might look like this:

Title	Start	End	ContentType	Organizer	Deployed To Production	Related Employee
Iteration16	1/12/09	1/19/09	Iteration		false
Stakeholder Demo	1/19/09 3 PM		Meeting	Lee Richardson
Tag Trunk	1/19/09		Action			Lee Richardson

Sporm's unique architecture supports this scenario by allowing you to retrieve iterations like this:

return MySPDataContext
      .GetCurrent()
      .GetList<Calendar>()
      .OfType<Iteration>();

Or get meetings from the same list like this:

return MySPDataContext
      .GetCurrent()
      .GetList<Calendar>()
      .OfType<Meeting>();

While you may only use multiple content types per list occasionally you can feel comfortable knowing that sporm will support you when you need it. The rest of the time you can arrange your architecture to accommodate your 90% scenario of one content type per list. I'll discuss the architecture I'm using on my current project in my next post.

For now I hope this has clarified some of the benefits of using sporm, and I hope that you'll consider using it on your next SharePoint project.

Unit Testing SharePoint - Past, Present, and Sporm

Lee Richardson — Wed, 09 Sep 2009 15:44:18 -0400

As I described in SharePoint: The Wild West of Software Development there is a serious problem when you develop for SharePoint: ensuring quality through unit testing is really, really hard. And that's where a new open source tool just released today called sporm (SharePoint Object Relational Mapper) comes in. While sporm provides many benefits besides simplified unit testing I wanted to focus on this topic first, because sporm's approach, which models the entity framework in the way it supports POCO's, is a unique feature not available with other SharePoint tools like LINQ to SharePoint.

I'll start by describing the unit testing problem, then provide a conventional solution with mocking, then finish with the elegance of a sporm based solution.

Simple Unit Testing Turns Ugly

I would bet anyone that attempts to develop a tiered solution with SharePoint ends up with entities that look something like this:

public class Employee {
      public SPListItem ListItem { get; private set; }

      /// <summary>
      /// The constructor requires that you pass in a list item
      /// that is used by your strongly typed properties
      /// </summary>
      public Employee(SPListItem listItem) {
            ListItem = listItem;
      }

      /// <summary>
      /// This provides your entities strong typing to hide
      /// the native weak typing in SharePoint
      /// </summary>
      public virtual string LastName {
            get { return (string)ListItem["LastName"]; }
            set { ListItem["LastName"] = value; }
      }
}

And if you have some function that you'd like to unit test like this:

/// <summary>
/// Typically returns "[LastName], [FirstName]" but skips
/// the comma if either is missing
/// </summary>
public string GetNameFormatted() {
      bool noFirst = string.IsNullOrEmpty(FirstName);
      bool noLast = string.IsNullOrEmpty(LastName);
      if (noFirst && noLast) return "";
      if (noFirst) return LastName;
      if (noLast) return FirstName;
      return string.Format("{0}, {1}", LastName, FirstName);
}

Now you face a major problem: LastName is tightly coupled with SPListItem. And SPListItem only contains internal constructors making it unmockable. So if you keep the tight coupling and pass an SPListItem into the constructor then you need a connection to SharePoint and an employee list (table) and you need to create an employee list item (record), then you test it, and finally you ought to clean up after yourself and delete the list item. Talk about a simple unit test turned fragile integration test. Yuck.

Mocking the Solution

So at this point you're undoubtedly thinking it's time to mock. To do this you'll have to make a public constructor that takes no arguments and mark your properties virtual. Then you can write a test like this:

[TestMethod]
public void TestGetNameFormatted_FirstAndLastExist_LastCommaFirst() {
      MockRepository mocks = new MockRepository();
      Employee mockEmployee = mocks.StrictMock<Employee>();
      Expect.Call(mockEmployee.FirstName).Return("Lee").Repeat.Any();
      Expect.Call(mockEmployee.LastName).Return("Richardson").Repeat.Any();
      mocks.ReplayAll();
      string actual = mockEmployee.GetNameFormatted();
      mocks.VerifyAll();
      Assert.AreEqual("Richardson, Lee", actual);
}

And then if you're like me you absolutely hate your unit tests. Because let alone that you had to modify your production code to accommodate unit testing, more importantly your unit tests are now unreadable. All of them. Even in simple methods without dependencies your unit tests contain almost as much plumbing code as actual test code. Enter sporm.

Unmocking with Sporm

Sporm is an open source tool released on codeplex that was developed by fellow Near Infinity employee Joe Ferner (who, incidentally, is one of the most brilliant developers I know). While this is a version 1.0 product Joe has been using it on his project and independently I've been using it on my project for several months now, so it is relatively mature.

So how it works is that at design time sporm will read from your SharePoint site and generate partial classes for each of your list items and each of your content types. The generated classes don't inherit from anything, giving you the ability architect your solution how you see fit. And the properties are auto-properties, making the classes just pure POCO. Here's an example if you're interested:

[SPOrmContentType(Name="Employee")]
public partial class Employee : ISPOrmContentType {
      public static class FieldTitles {
            public const string FirstName = "First Name";
      }

      public static class FieldStaticName {
            public const string FirstName = "FirstName";
      }

      [SPOrmIdField()]
      public virtual int Id { get; set; }

      [SPOrmField(Title = FieldTitles.FirstName, StaticName = FieldStaticName.FirstName)]
      public virtual string FirstName { get; set; }
}

Since generated entities look like that, your entities can now look like this:

public partial class Employee {
      public string GetNameFormatted() {
            bool noFirst = string.IsNullOrEmpty(FirstName);
            bool noLast = string.IsNullOrEmpty(LastName);
            ...
      }
}

Now if that isn't a picture of beauty to you then you haven't been working with SharePoint. But wait, I hear you asking, how does this have anything to do with SharePoint? How would an instance of an employee be able to retrieve the data in a SharePoint ListItem? The answer is in the DataContext. In your production code you write something like this:

public static Employee FindById(int id) {
      return MySPDataContext
            .GetCurrent()
            .GetList<TList>()
            .OfType<TContentType>()
            .FirstOrDefault(c => c.Id == id);
}

While it might not look so pretty at first you can hide these details in a base type and the nice thing is that it does handle SharePoint's ability to have multiple content types per list (which LINQ to SharePoint does not do, incidentally). But for now all you need to know is that when you return an Employee thorugh MySPDataContext it subtypes your Employee class and returns you a proxy at runtime. The proxy overrides each property to return you the appropriate value in the SPListItem. And if you instantiate an Employee directly (as you would in unit testing) then you can use it like a pure POCO. If you're familiar with the entity framework this should sound extremely similar. The result is that you can write your unit tests like this:

[TestMethod]
public void TestGetNameFormatted_FirstAndLastExist_CommaSeparate() {
      Employee employee = new Employee {
            FirstName = "Lee",
            LastName = "Richardson"
      };
      string actual = employee.GetNameFormatted();
      Assert.AreEqual("Richardson, Lee", actual);
}

No SharePoint dependencies. No mocking. Nothing but pure, unadulterated unit test. Now that a thing of beauty.

SharePoint: The Wild West of Software Development

Lee Richardson — Mon, 17 Aug 2009 21:27:42 -0400

Some argue that Microsoft developers lack rigor -- that techniques like unit testing and continuous integration are virtually unheard of in this space. That's rubbish. I would be shocked if the percentage of developers that track code coverage is significantly different for the .Net or Java spaces (not quite mainstream, but respectable). Mind you I'm talking .Net development in general. SharePoint, now that's another story.

The shadowy world of SharePoint is one in which source control is a distant memory, where no distinction is drawn between development, test and production environments, and where roaming bandits take over small towns with no fear of the law.

Why is this? How has it come to be? The answer is that for the most part SharePoint treats code and data as one in the same. End users can modify data stores: they can add, delete and modify tables (Lists) and columns (Fields) at will. They can open up SharePoint Designer and modify pages, customize workflows, rob banks, and run local Sherriff's out of town.

And that's why end users love SharePoint. It removes the slow, burdensome bureaucracy involved in doling out development resources from large centrally planned corporate or government IT departments and places that power directly in the hands of end users, allowing them to get their jobs done faster and to adapt to change quicker. And that is why I maintain that as software developers SharePoint, or a product like it, is in our future -- like it or not.

But the power given to end users makes rigor and good design extremely hard for developers because the production machine you deployed to last month may look completely different now that you're ready to redeploy. And Microsoft makes good design even harder with an inflexible API. Critical classes like SPSite (think SqlConnection) contain no public constructor, rendering them completely unmockable (unless you're willing to spend $450 per developer for some TypeMock Isolator magic). And vital classes like SPList (think DataSet) are marked final, crippling your ability to make nice strongly typed entities in your architectures.

So what is a Type-A, quality minded software developer to do?

The thing to keep in mind is that the challenges that SharePoint places on developers and architects simultaneously make good coding practices harder (but not impossible) while making them more important than ever.

Because that column you thoughtfully added as required may get reset to optional by an end user in production, you need to write more good tests that validate once solid assumptions. Because you can't inherit SPList, you'll need to encapsulate it, requiring more code per entity. Because you can't mock essential SharePoint classes you will need to write more integration tests instead of unit tests. Because you're writing integration tests your tests will be more fragile and you'll spend more time fixing the test code than the real code.

And because you're spending more time focusing on quality while your Wild West, short term focused competition in the SharePoint world slings code together, they will appear faster and better able to get applications out the door. It can be frustrating, because over time your well designed and well tested application will be more maintainable and will cost IT departments less. But end users frequently don't understand this, so as long term minded SharePoint developers we must focus on educating end users.

Combat the short term focused mindset by keeping everything in .wsp files, storing code in source control, avoiding the temptation to apply fixes in production, using continuous integration (yes, for SharePoint!), and most importantly tracking code coverage (yes, code coverage for SharePoint code!). Put quality metrics in status reports for management to see. Track defects and make your stats publicly available (since you'll have significantly fewer over time than code slingers, and you want end users asking for stats). In short remind the world that quality counts, even in the shadowy realms of SharePoint. If we work together we can bring law and order to the ... um, West.

You cannot grant a user a permission level that is not attached to a web

Lee Richardson — Tue, 04 Aug 2009 17:09:31 -0400

If you receive this error message that looks like this:

Microsoft.SharePoint.SPException: You cannot grant a user a permission level that is not attached to a web.
at Microsoft.SharePoint.SPRoleDefinitionBindingCollection.AddInternal(SPRoleDefinition roleDefinition)
at Microsoft.SharePoint.SPRoleDefinitionBindingCollection.Add(SPRoleDefinition roleDefinition)

When you're doing something like this:

private static SPRoleDefinition FindAddEditListItemsRoleDefinition(SPWeb site) {
    SPRoleDefinition definition = site.RoleDefinitions
        .Cast<SPRoleDefinition>()
        .FirstOrDefault(def => def.Name == ROLE_DEFINITION_TITLE);

    if (definition != null) return definition;

    definition = new SPRoleDefinition {
        Name = ROLE_DEFINITION_TITLE,
        Description = "Can view and edit list items only.",
        BasePermissions = SPBasePermissions.EditListItems
            | SPBasePermissions.ViewListItems
            | SPBasePermissions.ViewFormPages
            | SPBasePermissions.ViewPages

            | SPBasePermissions.Open
        };
    site.RoleDefinitions.BreakInheritance(true, true);
    site.RoleDefinitions.Add(definition);
    site.Update();
    return definition;
}

It may be because SharePoint doesn't allow you to use the SPRoleDefinition you just created and you have to return the one you just added. Stupid, but hopefully it will help someone somewhere.

    ...
    site.Update();
    // you can't just return definition because SharePoint gives you
    //    "You cannot grant a user a permission level that is not
    //    attached to a web" when you attempt to use it, so
    //    you need to return it again
    return site.RoleDefinitions
        .Cast<SPRoleDefinition>()
        .First(def => def.Name == ROLE_DEFINITION_TITLE);
}

IEnumerable.Count() is a Code Smell

Lee Richardson — Thu, 23 Jul 2009 17:31:20 -0400

Way too often I come across code like this:

IEnumerable<Product> products = GetProducts();
if (products.Count() == 0) {
      SomePlaceHolder.Visible = false;
} else {
      SomePlaceHolder.Visible = true;
      SomeRepeater.DataSource = products;
      SomeRepeater.DataBind();
}

What's wrong with this is that Count() is a LINQ extension method that literally iterates through every item in an enumerable. In other words it won't short circuit after it finds more than zero items -- it will continue iterating through every single item. And while that may be a fast operation in dev, it may be a really slow operation in production.

But it gets worse. Because of LINQ's deferred execution the results of .Count() are not cached. So when there is at least one product you are guaranteed to enumerate every single product twice: once for .Count() and once for .DataSource = products.

Do Any() Solutions Exist?

One possibility is to instead use IEnumerable.Any(). So you could rewrite the above code like this:

IEnumerable<Product> products = GetProducts();
bool anyProducts = products.Any();
SomePlaceHolder.Visible = anyProducts;
if (anyProducts) {
SomeRepeater.DataSource = products;
SomeRepeater.DataBind;
}

The results will be better regardless of if your data source is in memory or a database, but for illustration purposes LINQ to SQL outputs the following SQL if you run the code smell version when there is at least one product in the database:

SELECT COUNT(*) AS [value]
FROM [dbo].[Product] AS [t0]
SELECT [t0].[ProductId], [t0].[ProductName], ...
FROM [dbo].[Product] AS [t0]

And it outputs this SQL if you run the better version when there is at least one product:

SELECT
    (CASE
        WHEN EXISTS(
            SELECT NULL AS [EMPTY]
            FROM [dbo].[Product] AS [t0]
            ) THEN 1
        ELSE 0
     END) AS [value]
SELECT [t0].[ProductId], [t0].[ProductName]
FROM [dbo].[Product] AS [t0]

Looks a little uglier, but if you run the execution plan you've converted two table scans into an index scan and a table scan (well, depending on your schema of course). Regardless, much better execution-wise.

Undeferring deferred execution

Now what if your data source is a database and you really want to optimize the heck out of this and only perform one SQL statement? You could disable deferred execution by doing this:

IList<Product> products = GetProducts().ToList();
bool anyProducts = products.Any();
SomePlaceHolder.Visible = anyProducts;
if (anyProducts) {
SomeRepeater.DataSource = products;
SomeRepeater.DataBind;
}

The .ToList() function forces enumeration to happen immediately and cache the results in a list. This achieves the result of converting two queries into one, but be warned it also destroys one of the nicest benefits of LINQ's deferred execution: the ability to append .Sort() or .Skip().Take() (paging) statements at the very end and have them execute in your original query, in SQL, in the database, where they belong.

Summary

IEnumerable.Count() seems like a pleasant and friendly convenience function; it feels so similar to the innocuous List.Count -- but don't be fooled. It's a mistake. One you (or the person who maintains the system when you leave) may soon regret.

Secrets of SharePoint 2010 Exposed at TechEd 2009

Lee Richardson — Sun, 17 May 2009 22:42:01 -0400

In case you aren't aware, the details of SharePoint 2010 are under tight wraps. If you asked any of the presenters at Tech Ed the typical response was "No idea what you're talking about." But that doesn't mean presenters didn't occasionally slip up or say more than they probably should have.

Publically Stated SharePoint 2010 Info

Here's what has been publically stated about SharePoint 2010 thus far:

Requires at least Windows Server 2008 64 bit
Requires SQL Server 64 bit (2005 or 2008)
Will not support for IE 6
Public testing to occur first half of next year

This was announced at TechEd by Tom Rizzo, director of Microsoft SharePoint Server, as posted in this InfoWorld article and also on the SharePoint blog.

New SharePoint 2010 Secrets

An exciting slip up that has received no press thus far was from the presentation: Microsoft Visual Studio 2010 Overview for the Business Application Developer. If you check 1:00:33 of the presentation you'll see the first glimpse of a SharePoint 2010 Site Home Page:

I could tell something wasn't right when I saw this part of the demo, but fellow Near Infinity employee Joe Ferner put it together first when he saw the big red "Give Feedback" button in the top right. He tweeted about what came next:

"Sharepoint 2010 to have inline list ajaxy editing."

It looks really nice, and is very exciting since SharePoint is currently a little unwieldy. Check out this screenshot from 1:01:08:

So that was the most exciting slip up. Other exciting tidbits were:

SharePoint designer will support saving workflows to re-use for provisioning

I learned this one during the Q&A session of the "Automate Business Processes Using InfoPath Forms with Integrated SharePoint Designer Workflows... All without Coding!" talk. It's currently a pain point that workflows developed in SharePoint designer can't be reused in the same way workflows developed in Visual Studio can, so this will be a wonderful new feature.

BDC will probably support updating and inserting data

Apparently MOSS only shipped without this feature because it hadn't been thoroughly tested. That there has been plenty of time to test this feature by now, so it's a pretty reasonable assumption that it will be in the next version.

Finally this is pure speculation, but I'm guessing:

SharePoint 2010 to support ASP.Net Data Services API

Microsoft supports two AJAX approaches: client side and server side. The server side uses the UpdatePanel control where you code as usual and everything magically gets AJAXized. The client side version is part of the unreleased ASP.Net 4.0, but ties in tightly with JSON provided by the .Net Data Services API that was released in the .Net Framework 3.5 SP1.

The SharePoint team could have used the server side approach, but why not use the latest technology available, speed up UI significantly, and simultaneously allow web part developers a fantastic level of client side AJAX support? If they made the entire SharePoint API available via the .Net Data Services API it would enable some powerful fast AJAX web parts. It would be awesome.

Finally, the only other tidbit isn't a secret at all, but what I consider to the most exciting feature of Visual Studio 2010: it will support editing SharePoint solutions without editing XML. I hate XML, so that will be wonderful. So are you ready for SharePoint 2010 today? I sure am. Pity about the wait.

Code Access Security Cheat Sheet

Lee Richardson — Thu, 07 May 2009 09:36:58 -0400

I did short presentation on Code Access Security (CAS) a while ago and put together a cheat sheet to help remember the terms and how they fit together. It includes screenshots of the .NET Framework 2.0 Configuration tool (in Control Panel\Administrative Tools) and uses Entity Relationship Diagram (ERD) Notation. I had to pull it up recently and figured others might also find it helplful.

It describes the following terms:

Permission
PermissionSet
Code Group
Policy Level
Assembly Instance
Evidence; and
Evidence Type

It should print to 8.5 x 11. Click to enlarge:

Hope you find it useful.

Making SharePoint Title a Calculated Column

Lee Richardson — Mon, 27 Apr 2009 08:55:54 -0400

All SharePoint lists start with a user editable "Title" column that ECB menu's hang off of and that is the default field to display in associated child tables. This works well most of the time. For example if you have a list like Company, you can change the display name of Title to "Company Name," and then Employee records display that field for their parent relationship.

Where this doesn't work well is lists like Employee, where there is no single column that uniquely identifies a row to an end user. You could create a new "Full Name" calculated column and delete the Title column, but making Title a calculated column just feels like a cleaner solution.

I looked into making Title calculated where it is defined in CAML, but it looked pretty messy, and a true calculated column additionally suffers from the limitation that you can't pull from other lists. The solution I settled on was to populate it in ItemAdding and ItemUpdating. Of course you tie into those methods with something like:

SPList employeeList = site.Lists["Employee List"];
employeeList.EventReceivers.Add(
    SPEventReceiverType.ItemUpdating,
    "[four part assembly name]",
    "Namespace.EmployeeEventReceiver"
    );
employeeList.EventReceivers.Add(
    SPEventReceiverType.ItemAdding,
    "[four part assembly name]",
    "Namespace.EmployeeEventReceiver"
    );

The only reason this is worth documenting is because setting title is a little messy in the event receivers and there is very little documentation on how to do it.

public class EmployeeEventReceiver : SPItemEventReceiver {

public override void ItemAdding(SPItemEventProperties properties) {
    string fullName = GetFullName(properties);
    properties.AfterProperties["Title"] = fullName;
    base.ItemAdding(properties);
}

private static string GetFullName(SPItemEventProperties properties) {
    string firstName = properties.AfterProperties["First_Name"].ToString();
    string middleName = properties.AfterProperties["Middle_Name"].ToString();
    string lastName = properties.AfterProperties["Last_Name"].ToString();
    return Employee.GetFullName(firstName, middleName, lastName);
}

public override void ItemUpdating(SPItemEventProperties properties) {
    string fullName = GetFullName(properties);
    string internalTitleName = properties.ListItem
        .Fields["Title"].InternalName;
    properties.AfterProperties[internalTitleName] = fullName;
    base.ItemUpdating(properties);
}
}

The tricky part is getting the title field by internal name in ItemUpdating vs. getting it normally in ItemAdding. It's pretty easy once you work through it, but it took me longer than it should have. So hopefully this helps someone somewhere.

An Exciting Future: C# 4.0, Silverlight 3, MVC Dynamic Data, Live Mesh, VS 2010, ...

Lee Richardson — Mon, 08 Dec 2008 10:45:30 -0500

Last week I attended the Public Sector Developer Conference in Reston, Virginia. Summary: I can barely contain my excitement for just about everything Microsoft is doing right now for software development. In particular the first talk "What I've learned about Visual Studio 2010, .NET Framework 4.0 (and beyond), Silverlight 3 (and beyond)" was so exciting I almost gave a standing ovation.

What Marc learned about Visual Studio 2010, .NET Framework 4.0, Silverlight 3

So believe it or not this talk by Marc Schweigert (who is a fabulous presenter by the way) was a 90 minute distillation of everything that happened at PDC (which sadly I missed). Consequently it flew like a jet engine, gave barely a minute or two to every topic, and required either massive amounts of caffeine or a good amount of prerequisite reading. Fortunately I had both.

To summarize the presentation I'll just give my favorite upcoming technologies and sort it roughly in decreasing level of my excitement:

Technology: ASP.Net MVC Dynamic Data
Overview: Ruby on Rails for .Net
Impressions: As if ASP.Net MVC wasn't cool enough. This looks awesome! I want to try it on a real project.

Technology: ADO.Net Data Services (Astoria)
Overview: Expose LINQ ORM data (e.g. LINQ to Entities) to web services, now with better BLOB support
Impressions: Powerful, exciting (just a little scary)

Technology: Astoria Offline
Overview: Write disconnected apps that sync to ADO.Net Data Services
Impressions: Frikin' awesome. I can't wait to see/write disconnected apps.

Technology: ASP.Net AJAX - jQuery Support
Overview: ASP.Net will abandon its custom "roll your own" JavaScript framework and adopt a leading open source one .
Impressions: Amazing. I couldn't be happier.

Technology: Live Mesh Web Apps
Overview: Offline capable, in browser or on desktop apps (HTML/AJAX or Silverlight) that run "in the mesh"
Impressions: Wow, this is some really cool stuff, I'll definitely be keeping an eye on this technology

Technology: SharePoint Integration
Overview: Build web parts, wsp files, etc directly in VS without an add-on
Impressions: Fantastic! Awesome! This was a major point.

Technology: Task Parallel Library (TPL)
Overview: Simplify parallelization and avoid programming in threads by using delegates
Impressions: This looks absolutely amazing, I can't wait

Technology: Parallel LINQ
Overview: Parallelize LINQ queries
Impressions: So easy, so powerful, so exiting

Technology: In Process Side By Side Model
Overview: Run both 2.0 and 4.0 CLR code in the same process
Impressions: Wow. Minimize existing app upgrade pain; free developers to use new features in old code bases; free language designers more flexibility in refactoring the C# language.

Technology: C# 4.0
Overview: Named optional parameters and dynamic types
Impressions: Named parameters: cool, dynamic types: controversial, but I suspect a very good thing

Technology: F#
Overview: Functional programming language to ship by default with VS
Impressions: I'm playing with this now, it's very interesting, but I'm not prepared to comment yet

Technology: Managed Extensibility Framework
Overview: Massively simplify creating desktop app add-on frameworks.
Impressions: This has the potential to change desktop apps to very light containers that load functionality as necessary. I am hopeful and excited.

Technology: VS 2010 IDE Improvements
Overview: WPF & XAML Editor better, Multi-Monitor Support, Built on WPF
Impressions: Cool, looking forward to trying. I like the dog-food approach, although as I've pointed out earlier dog-food isn't everything.

Technology: ADO.Net Entity Framework V2
Overview: The DBMS agnostic LINQ based ORM now with N-tier improvement, DDL generation, caching
Impressions: DDL generation was interesting, but apparently won't support deltas, that's for another product, maybe Oslo. Caching support looks fabulous.

Technology: WPF
Overview: WPF to have more controls, better usability in VS, and simplify ribbon programming
Impressions: Cool, but .. am I the only person who hates the ribbon?

Technology: Silverlight 3
Overview: 3D & GPU support; will work as well as WPF in VS; H.264 support
Impressions: 3D = Very cool; VS WPF = Cool, although I may still program in XAML; and H dot two sixty what?

Technology: Web Development in VS 2010
Overview: JavaScript intellisence improvements; view-state improvement, Deployment simplified (e.g. Web.Production.Config & setup packages); new controls (chart)
Impressions: Very nice.

Technology: ASP.Net AJAX - Client Templates
Overview: Simplifying AJAX without an UpdatePanel
Impressions: I'm lazy so I'll probably stick with UpdatePanel & server based approach, but I'm very happy to see the new client based focus as an additional option

Technology: Workflow Foundation (WF)
Overview: More controls, faster, better debugging
Impressions: I haven't really had a need for it WF, but the enhancements sound good.

Technology: Oslo
Overview: Create DSLs that do things like help manage databases
Impressions: I fail to see what problem this solves, but Martin Fowler thinks it's cool, so I'll keep an eye on it.

Technology: SQL Data Services
Overview: Basically Amazon S3 (cloud stuff)
Impressions: In case you didn't know I love referential integrity, so I'm not a big fan. What did impress me is Microsoft will merge the LINQ serialization approach with ADO.Net Data Services. Nice benefit to multiple technologies within one company.

Summary

There's so much going on it's a little overwhelming and I'm sure I missed stuff, but you can see why I'm so excited. I vow to never miss a PDC again. Anyway I hope this post helped clear up the vast amount of exciting new technologies coming out soon.

Applications Can't Use SharePoint Master Pages

Lee Richardson — Wed, 12 Nov 2008 12:08:28 -0500

This is the story of stupid SharePoint problem and an ugly, kludgy, and embarrasing solution. On our project we have the need for an application that looks and feels like a SharePoint subsite. Specifically, we need it to inherit from SharePoint's masterpage. But we also need it to be a separate IIS application.

Problem: SharePoint dislikes Applications

We get the following error when we try to dynamically set the MasterPageFile in code:

System.ArgumentException: The virtual path '/_layouts/application.master' maps to another application, which is not allowed.

Several sites say we just can't do what we're trying to do. We considered a Virtual Path Provider, but decided not to even go down that path because SharePoint's master page undoubtedly has hooks into web.config and httpmodules and such, so we came up with the following hack:

Solution: An Ugly SharePoint Text Manipulation Hack

Deploy a nearly blank page in to /_layouts/ (we do this in our WebAppManifest.xml & WebAppSolution.ddf files). In PageBase (that all pages inherit from) we override Render to retrieve the blank page and do text manipulation to insert our content into it. Here's the code:

protected override void Render(HtmlTextWriter writer) {
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(GetUrlToBlankPage());
request.Credentials = CredentialCache.DefaultCredentials;
using (WebResponse webResponse = request.GetResponse()) {
    StreamReader streamReader = new StreamReader(webResponse.GetResponseStream());
    string pageHostHtml = streamReader.ReadToEnd();

    using (StringWriter stringWriter = new StringWriter())
    using (HtmlTextWriter htmlTextWriter = new HtmlTextWriter(stringWriter)) {
      base.Render(htmlTextWriter);

      string renderPageHtml = stringWriter.ToString();

      string title = GetTitleFromHtml(renderPageHtml);
      pageHostHtml.Replace("PAGE TITLE", title);

      string content = GetContentFromHtml(renderPageHtml);

      pageHostHtml = RemoveFormData(pageHostHtml);
      pageHostHtml = pageHostHtml.Replace("DO NOT MODIFY THIS PAGE", content);
    }
    writer.Write(pageHostHtml);
}
}

private static string GetContentFromHtml(string html) {
Match match = Regex.Match(html, "<body>(.*)</body>",
    RegexOptions.IgnoreCase | RegexOptions.Singleline);
return match.Success ? match.Groups[1].Captures[0].Value : "";
}

private static string GetTitleFromHtml(string html) {
Match match = Regex.Match(html, "<title>(.*)</title>", RegexOptions.IgnoreCase);
return match.Success ? match.Groups[1].Captures[0].Value : "";
}

private static string RemoveFormData(string html) {
string replaced = Regex.Replace(html, "<form[^>]*>", "",,
    RegexOptions.IgnoreCase | RegexOptions.Singleline);
replaced = replaced.Replace("</form>", "");
return replaced;
}

Conclusion

I'm sure there are plenty of optimizations that we can do (e.g. caching), but that's the basic idea. But this solution makes me feel so dirty. So please, please, dear reader, tell me there is a better way. I want to feel clean again.

Eight Miserable TFS Features

Lee Richardson — Thu, 06 Nov 2008 10:39:29 -0500

I'd prefer to post a positive, happy, or ideally emotion-agnostic technical post, but today Microsoft Visual Studio Team Foundation Server (TFS)'s source control pissed me off one too many times. I could go on for pages, but this list represents the top eight reasons why you should never pick Team Foundation Server for source control (in decreasing order of annoyance).

1. Can't see changes on get latest

If you like to publically humiliate co-workers when they violate coding standards, or even if you just want to keep an eye on what they're doing, then TFS is absolutely not for you. I'm completely spoiled by Tortoise SVN which allows you to

Get latest
See which files have changed; and
Quickly view the diff

I suppose I should be happy TFS at least allows #1.

2. TFS drops Solution Items

Solution Items are files and folders that exist outside of projects, but within solutions. For instance my current project has an etc directory that contains database scripts and such. First of all it would be nice if Visual Studio automatically recognized new items in the etc directory. Ironically you have to "add existing item" even though it already exists. Fine, whatever. But more importantly it would be nice if Visual Studio didn't periodically delete files said files from the solution. As convenient of a feature as that may sound, it has caused numerous problems. A "Blame" has determined that nearly every person on the project has activated this feature leading me to believe it is truly a bug. Brilliant.

3. Checked In Files Are Marked Read-only

Suppose you have a file you want to open outside of Visual Studio (gasp). With TFS if you don't already have Visual Studio open you have to start the IDE in order to check the file out to remove the read only bit. So ten minutes later you're ready to go. At least TFS does allow multiple check-out.

4. TFS server name in project file

TFS embeds the server name in every single one of your project files. The beauty of this approach becomes evident when one team member uses an IP address, while another uses the domain name or computer name. You end up fighting over the project files and automatically checking them out when all you want to do is view them.

5. Can't Access Source Control Outside of Visual Studio

Want to check in or get latest without opening Visual Studio? Yea, you can't. Hope you don't have any java (non-Microsoft) code on your project.

6. Reconciling, Painfully Slow

TFS actually does a decent job of auto merging conflicting files. And when it can't auto-merge conflicting files you can view each file side by side and click which change you want to keep. But the process is painful and slow. For instance if you forget to get latest before check in and there are conflicts you can resolve them, but your check in will always fail. Get latest seems to take much longer when there are conflicts. Auto merge is really slow. If there are un-reconcilable conflicts hitting the "resolve" button takes forever and must be done for each and every conflict. It's like it downloads the entire history of the file. And that wait is mandatory even if you simply want to discard either your or the server's changes.

7. *.vsmdi files

This is just a frustrating known bug where if you use TFS for unit testing (Team Foundation Server Test System) and TFS for source control, Visual Studio will either take this .vsmdi file which is essential file for unit testing and either duplicate it or corrupt it. There's a lot of ink spilled on this vsmdi bug and there are some workarounds, but it's a pain.

8. Moving a folder opens all files in the folder

This was the last straw and the impetus for writing this post. Today I wanted to move a "solution item" into a project. I opened the Source control Explorer (which is impossible to find in the first place) and right clicked, selected move, entered my destination, and TFS in its wisdom opened every file in the folder nearly bringing my machine to its knees. There are hundreds of small annoyances like this every day with TFS.

In Summary

TFS actually has some nice features that I like over SVN. Like the merge tool is pretty good, and being able to see all people who are currently working on a file is nice. But overall the cons heavily outweigh the pros. And if it's any indication how bad TFS is for source control (or how much developers hate it) Codeplex has even implemented a SVN to TFS bridge on their server.

It's ridiculous to me that Microsoft uses this tool internally and sells it and people buy it! I must be missing something. If so please post comments to this post. Because the entire thing boggles my mind.

Forget Burndown Use Burnup Charts

Lee Richardson — Thu, 23 Oct 2008 22:41:25 -0400

Agile projects traditionally use burndown charts to visually show work remaining over time. This could be for the current iteration or it could be for the duration of the project. Either way they can help managers (or the Project Owner in Scrum) track velocity, estimate either the project or iteration completion date, or find trends in past performance. But burndown charts have a major shortcoming: they fail to show what makes agile projects agile: new requirements. And that's where burnup charts come in. But first let's examine burndown charts.

The Problem with Burndown

A typical burndown for the life of a project might look like this:

It shows the project started with 100 points of work in the backlog; it's completed eight iterations; the team accomplishes about ten points an iteration; and if everything continues at the current velocity the project will complete all work within another two iterations. Great.

But what happened in iteration six? Very little appears to have been accomplished. Maybe the team all took a vacation. Maybe there was a major problem or the team incorrectly estimated complexity. Or perhaps a large set of new requirements were added to the backlog because the customer decided what they thought they wanted wasn't what they really needed: namely the exact scenario agile was designed for.

Burnup Charts

The problem is that burndown charts lack two essential pieces of information. First, how much work was actually accomplished during a given iteration (as opposed to how much work remains to be completed) and second how much total work the project contains (or if you prefer how much scope has increased each iteration). A burnup chart for the exact project above might look like this:

We can now clearly see that the team did not take a breather in iteration six. They continued to complete about ten points per iteration, but during the sixth iteration the scope increased by about twenty points.

One could imagine the opposite happening as well. Later in the project the team might delete old user stories that were envisioned during project inception and thus decrease the total scope. The burndown chart would incorrectly show such a scenario as a sudden increase in velocity.

Summary

Either way the burndown chart hides essential information. I propose we throw it away and show the slightly more complicated, but infinitely more useful burnup chart. After all you wouldn't want upper management thinking you were lazy in iteration six would you?

Silverlight: Cannot specify both Name and x:Name attributes

Lee Richardson — Mon, 11 Aug 2008 23:58:29 -0400

I received the following error while upgrading from Silverlight beta 1 to beta 2: Cannot specify both Name and x:Name attributes. I'm sure there can be many causes, but since there isn't much written on this, here is one cause.

If you have a custom control you can no longer put an x:Name on the root element if you also have an x:Name on an instance of the control. A crappy work-around is to declare the root node in code and set it to the parent of a child element. For instance:

<Canvas x:Class="MyNamespace.MyCustomControl"
   xmlns="http://schemas.microsoft.com/client/2007"
   xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
   x~~:Name="LayoutRoot"~~
   >
    <StackPanel x:Name="_stackPanel">

Canvas LayoutRoot;

/// <summary>
/// Constructor
/// </summary>
public ComboBox() {
    InitializeComponent();

   // set _controlBase after InitializeComponent
   //     so _stackPanel is initialized
   LayoutRoot = (Canvas)_stackPanel.Parent;
}

This is pretty fragile, so I'd love to hear any other solutions.

DEFCON Day One

Lee Richardson — Sat, 09 Aug 2008 10:17:29 -0400

It would probably be more interesting to blog about my numerous traveling mishaps (summary: never stay at Circus Circus however close it may be to your conference), but I thought a quick bullet point version of the first day at DEFCON 16 would be more relevant. Incidentally this trip was possible because of a generous training budget benefit provided by Near Infinity.

Making the DEFCON 16 Badge, Joe "Kingpin" Grand

The default firmware makes the badge function as a TV-B-Gone (it really turned on/off my hotel room TV ... cool)
You should be able to transfer files between SD card readers in badges (haven't tried)
Later in the day I soldered for my first time ever and added a mini-USB port to the card in the "Hardware Hacking Village" This was a rush, very cool. Thanks to Joe and Andrew for the assistance.

Deciphering Captcha Michael Brooks

It never occurred to me you could generate all possible outcomes of a captcha algorithm to break it, cool idea
Never read a script at a conference, however interesting the topic may be, I left after 5 minutes

Whitespace: A Different Approach to JavaScript Obfuscation, Kolisar

Problem: Your Cross Site Scripting Code needs to evade human and/or automatic detection
Solution: Embed JavaScript code in whitespace as tabs (1) and spaces (0)
This was a really cool demo

New Tool for SQL Injection and DNS Exfiltration, Robert Ricks

SQL Injection is still a widespread vulnerability
Once you get in you can do all kinds of things, their demo included viewing the database schema
They showed a tool that automatically exploits vulnerabilities really fast, very cool

Living in the RIA (Rich Internet Application) World, Alex Stamos, David Thiel & Justine Osborne

A review of security vulnerabilities in five new RIA technologies: Adobe AIR, MS Silverlight, Google Gears, Mozilla Prism, and HTML 5
All technologies significantly increased attack surface and had vulnerabilities
Surprisingly Silverlight seemed to come out on top (go MS) followed by Adobe AIR
Google Gears, Mozilla Prism, and HTML 5 all have a long way to go

Bringing Sexy Back: Breaking in with Style, David Maynor & Robert Graham

Penetration testing is generally boring work, just run a few tools
Banks and federal agencies need to do more serious and creative pen-testing
Scenario: Russian czar hires developers with a 1 million budget and wants to break in to your company
Approach 1: Put iPhone in box, attach external power supply, turn on, fed ex to target company, while box is in mail room TTY to phone bypass physical security, attack wireless network
Approach 2: "Spear Phishing" - Start fake company, get domain name, get SSL certificate, send "New 401(k) Provider" e-mail to target company, link to bogus website, sign malicious ActiveX control so there aren't warnings when they download
Wow, this talk was really cool, really eye opening

Keeping Secret Secrets Secret & Sharing Secret Secrets Secretly, Vic Vandal

As a presenter always err on the side of going too fast, and assuming your audience is smarter than you, this presentation sucked
Steganography is more than least significant bit image modification, it generally is about hiding things in plain sight such as hiding data after the EOF character in text files

Free Anonymous Internet Using Modified Cable Modems, Blake Self & Durandal

I didn't actually attend this, I'm hoping Joe or Andrew will blog about it, because man did it sound cool. I'll post a link if I see anything.

All in all this conference is absolutely amazing and eye opening. It really does give you a completely different perspective on the security industry.

A Major Silverlight PITA and Two Annoying 3.0 Limitations

Lee Richardson — Tue, 27 May 2008 08:52:54 -0400

Pardon my rant, but the thing I currently hate most about Silverlight (besides copious XML) is the Visibility property. Any sane framework would implement Visibility as a Boolean. Not Silverlight though. It's creators in undoubted infinite wisdom implemented it as an enumeration. The values of the enumeration? There are two: Visible and Collapsed. Hmmm.

Of course this causes superfluous verbosity in common everyday code:

button1.Visibility = makeVisible ? Visibility.Visible : Visibility.Collapsed;

Or worse when things get a little more complex:

// don't display the panel if its button's aren't visible
panel1.Visibility = !(button1.Visibility == Visibility.Visible && button2.Visibility == Visibility.Visible) ? Visibility.Visible : Visibility.Collapsed;

Clearly this was done to keep Silverlight compatible with the Windows Presentation Foundation (WPF) which has three values in its enumeration property (Visible, Hidden, and Collapsed). But that's just as ridiculous. Why WPF couldn't use two properties, Visible (Boolean) and NotVisibleBehavior (enumeration) is beyond me.

It's ok though, because .Net 3.0 gave me a cure to any Framework shortcomings: Extension Methods. A syntactic sugar cure for all my bitterness:

public static void SetVisible(this FrameworkElement element, bool visible) {
element.Visibility = visible ? Visibility.Visible : Visibility.Collapsed;
}

public static bool IsVisible(this Visibility visibility) {
return visibility == Visibility.Visible;
}

Fantastic, now my "complex" example becomes:

// don't display the panel if its button's aren't visible
panel1.SetVisible(!(button1.IsVisible() && button2.IsVisible()));

Still not quite as nice as a Boolean visible property, but certainly doable.

3.0 Limitation #1, By Ref Extension Methods

But wait. Isn't it best practice in Silverlight to use binding for these types of things? Separation of logic from presentation and all. So I should do:

And then:

public class DisplayStuff : INotifyPropertyChanged {
    public Visibility IsPanelVisible { get; private set; }

    public void UpdateStatus(bool makeVisible) {
        IsPanelVisible = makeVisible ? Visibility.Visible : Visibility.Collapsed;
        // make sure to notify the control that the property has changed
        PropertyChanged(this, new PropertyChangedEventArgs("IsPanelVisible"));
    }
}

And we can set the DataContext of some parent element to an instance of DisplayStuff and all the children including our panel magically databind. That's cool, but the ugliness is back (well, not as bad since I removed the buttons to simply the example, but you can pretend). This is because we extended FrameworkElement not Visibility. No problem, just extend Visibility right?

public static void SetVisible(this Visibility visibility, bool visible) {
visibility = visible ? Visibility.Visible : Visibility.Collapsed;
}

Except this doesn't work. Can you spot the problem?

It compiles. It runs. But the value of IsPanelVisible never changes. Oh yea, C# is pass by value by default. And now the .Net Framework 3.0 limitation. This isn't possible:

public static void SetVisible(this ref Visibility visibility, bool visible) {

You get "The parameter modifier 'ref' cannot be used with 'this'." Grr.

Limitation #2, By Ref Automatic Properties

Ok, so remove "this", and go back to C# 2.0 helper functions which extension methods are syntactic sugar for anyway:

public static void SetVisible(ref Visibility visibility, bool visible) {

And now our class can do:

ExtensionMethods.SetVisible(ref IsPanelVisible, makePanelVisible);

Right? Not so fast I'm afraid. Compile error. "A property or indexer may not be passed as an out or ref parameter". And I guess this is reasonable. You can't pass the address of a function, which is what a property is in the background. So you should pass the private variable that backs the property.

Except that I don't have one! I used an automatic property. And .Net doesn't let me access the private variable backing the automatic property. So I'm stuck!

And this is .Net 3.0 limitation #2. Automatic properties are wonderful until you try to do much with them. Why couldn't the framework notice that I'm using an automatic property and pass the variable that I can't access by ref to my function?

And now I find myself back in a .Net 2.0 world because all the features I like so much in 3.0 are more sugar than substance.

Conclusion

Allowing automatic properties to pass by reference or allowing access to the private member behind them would be nice. Allowing extension methods to change the instance they extend would be nice. But ultimately none of this would be a problem if Visible had been implemented as a Boolean. The way every other framework in the world does. </Complaining>

Expression Trees: Why LINQ to SQL is Better than NHibernate

Lee Richardson — Thu, 27 Mar 2008 16:59:40 -0400

In my last post I described how the Where() function works for LINQ to Objects via extension methods and the yield statement. That was interesting. But where things get crazy is how the other LINQ technologies, like LINQ to SQL use extension methods. In particular it's their use of a new C# 3 feature called expression trees that makes them extremely powerful. And it's an advantage that more traditional technologies like NHibernate will never touch until they branch out from being a simple port of a Java technology. In this post I'll explain the inherent advantage conferred on LINQ technologies by expression trees and attempt to describe how the magic works.

What's so Magic about LINQ to SQL?

LINQ to SQL (and it's more powerful unreleased cousin LINQ to Entities) is a new Object Relational Mapping (ORM) technology from Microsoft. It allows you to write something like the following:

IEnumerable<Product> products = northwindDataContext.Products.Where(
p => p.Category.CategoryName == "Beverages"
);

Which as you'd expect returns products from the database whose category is Beverages. But wait, aren't you impressed? If not read over that code again, you should be very impressed. In the background that C# code is converted into the following SQL:

SELECT [t0].[ProductID], [t0].[ProductName], ...
FROM [dbo].[Products] AS [t0]
LEFT OUTER JOIN [dbo].[Categories] AS [t1]
ON [t1].[CategoryID] = [t0].[CategoryID]
WHERE [t1].[CategoryName] = @p0

In other words it's pretty smart. It isn't just returning all products and filtering them in memory using the LINQ to Objects version of Where() I discussed previously.

Doing something like that using NHibernate Criteria would require something like this:

ICriteria c = session.CreateCriteria(typeof(Product));
c.Add(Expression.Eq("Category.CategoryName", "Beverages"));
IEnumerable<Product> products = c.List<Product>();

You could use HQL too, but both NHibernate options suffer from the same problem. Did you spot it?

The LINQ to SQL version is taking actual strongly typed C# code and somehow smartly converting it to useful SQL. The NHibernate version does the same thing, but always using a weakly typed alternative. In other words the column "CategoryName" in NHibernate is a string. If it or its data type change in NHibernate you won't find out until runtime. And that is the beauty of LINQ to SQL: you'll find more errors at compile time. And if you're like me you want the compiler to find your mistakes before the unit tests that you (or your fellow developers) may or may not have written do.

So you're probably now wondering if you can put strongly typed C# in your where clause and it somehow magically gets converted to SQL, what's the limit? If you put in a String.ToLower() or StartsWith() will it get converted to equivalent SQL? What about a loop or conditional? A function call? A recursive function call? At some point it has to break down and either return all products and filter them in memory or just fail right? Before answering those questions we need to understand what's going on.

Understanding the Magic

The Magic happens in a class called Expression<T>. Expression takes a generic argument that must be a delegate and is usually one of the built in Func methods. However the class can only be instantiated to a lambda expression. That's right, not a delegate or anonymous method, only a Lambda expression. So in my deferred execution post where I explained what Lambda expression are, I said they were essentially syntactic sugar for an anonymous methods. Well, the emphasis is on the essentially, because they really aren't sugar at all. When you assign a lambda expression to an Expression, the compiler, rather than generating the IL to evaluate the expression, generates IL that constructs an abstract syntax tree (AST) for the expression! You can then parse the tree and perform actions based on the code in the lambda expression.

Below is an example adapted from the .Net Developer's guide on MSDN that shows how this works:

// convert the lambda expression to an abstract syntax tree
Expression<Func<int, bool>> expression = i => i < 5;

ParameterExpression param = (ParameterExpression)expression.Parameters[0];
// this next line would fail if we change the Lambda expression much
BinaryExpression operation = (BinaryExpression)expression.Body;
ParameterExpression left = (ParameterExpression)operation.Left;
ConstantExpression right = (ConstantExpression)operation.Right;

Console.WriteLine("Decomposed expression: {0} => {1} {2} {3}",
      param.Name,
      left.Name,
      operation.NodeType,
      right.Value
      );

This outputs "Decomposed expression: i => i LessThan 5". The first line is the most important. It defines an Expression that takes a delegate with a single int parameter and a return type of bool. It then instantiates the Expression to a simple lambda expression. Incidentally this would also work if we defined our own Delegate:

public delegate bool LessThanFive(int i);

public static void DoStuff() {
Expression<LessThanFive> expression = i => i < 5;
}

It would, however, not work if we used an anonymous method:

Expression<Func<int, bool>> expression = delegate(int i) { return i < 5; };

While that looks legal it actually results in the compile time error "An anonymous method expression cannot be converted to an expression tree."

There is a lot of complexity in parsing the AST, far beyond the scope of this article. However, the MSDN does have a nice diagram that helps explain how the following slightly more complicated Lambda expression that determines if a string has more letters than a number:

Expression<Func<string, int, bool>> expression =
(str, num) => num > str.Length;

How Deep Does The Rabbit Hole Go?

So LINQ to SQL uses this Expression Tree technique to parse a plethora of possible code that you could throw at it and turn it into smart SQL. For instance check out a couple of the following conversions that LINQ to SQL will (or will not) perform:

p => p.Category.CategoryName.ToLower() == "beverages"

Results In:

SELECT [t0].[ProductID], ...
FROM [dbo].[Products] AS [t0]
LEFT OUTER JOIN [dbo].[Categories] AS [t1] ON [t1].[CategoryID] = [t0].[CategoryID]
WHERE LOWER([t1].[CategoryName]) = @p0

Not bad, huh? How about:

p => p.Category.CategoryName.Contains("everage")

That results in the following SQL snippet:

WHERE [t1].[CategoryName] LIKE @p0

And it sets @p0 to "%everage%". Pretty cool. Ok this will get it to fail though, right?

public static string GetCat() {
    return "Beverages";
}

IEnumerable<Product> products = northwindDataContext.Products.Where(
      p => p.Category.CategoryName == GetCat()
      );

It turns out that LINQ to SQL will look inside of other functions! Alright, there's no way it can do complicated conditionals:

p => p.Category.CategoryName ==
"Beverages" ? p.UnitsInStock < 5 : !p.Discontinued

This should only pick up Beverages that have fewer than 5 items in stock regardless of whether they are discontinued and any other products that aren't discontinued. Would you believe that it runs a single SQL statement:

SELECT [t0].[ProductID], ...
FROM [dbo].[Products] AS [t0]
LEFT OUTER JOIN [dbo].[Categories] AS [t1] ON [t1].[CategoryID] = [t0].[CategoryID]
WHERE (
    (CASE
        WHEN [t1].[CategoryName] = @p0 THEN
            (CASE
                WHEN [t0].[UnitsInStock] < @p1 THEN 1
                WHEN NOT ([t0].[UnitsInStock] < @p1) THEN 0
                ELSE NULL
             END)
        ELSE CONVERT(Int,
            (CASE
                WHEN NOT ([t0].[Discontinued] = 1) THEN 1
                WHEN NOT NOT ([t0].[Discontinued] = 1) THEN 0
                ELSE NULL
             END))
     END)) = 1

Wow, it sure isn't pretty, but it scales to multiple conditionals, and most importantly it didn't return all products and process them in memory. Not bad.

Conclusion

I asserted up front that using expression trees and the strong typing that comes with them is the reason LINQ to SQL is inherently better that NHibernate. I really can't make that claim without admitting one of LINQ to SQL's biggest shortcomings: It currently does not support multiple table inheritance. Ultimately, however, it's a short term fault since the forthcoming LINQ to Entities does. And I stand by my claim because from a long term perspective as long as technologies like NHibernate remain pure ports of Java code they will never realize the full benefits of equivelant LINQ technologies that take advantage of .Net's native strengths: like expression trees.

Note: Please post comments to my blogspot blog

How System.Linq.Where() Really Works

Lee Richardson — Fri, 14 Mar 2008 14:53:47 -0400

After writing my last blog entry on Deferred Execution in LINQ I had a conversation with Seth Schroeder who rightly pointed out among other things that I really didn't show how LINQ's deferred execution works internally. So in this post I wanted to implement my own LINQ Where() extension method based off of the one in the System.Linq namespace. So I'll show you the code, explain interesting parts of how it works including collection initializiers and extension methods, and then explain where the deferred execution behavior comes from (i.e. the yield statement). I will only explain in the context of LINQ to Objects since that's far simpler than other Linq's. I will implement a Where() like LINQ to SQL does in a later blog post (that's where things get really crazy).

Implementing MyWhere()

Let's start out with some code. The first question is does this compile?

using System;
using System.Collections.Generic;
using MyExtensionMethods;

namespace PlayingWithLinq {
    public class LinqToObjects {
        public static void DoStuff() {
            IList<int> ints = new List<int>() {9,8,7,6,5,4,3,2,1};

            IEnumerable<int> result = ints.MyWhere(i => i < 5);

            foreach (int i in result) {
                Console.WriteLine(i);
            }
        }
    }
}

namespace MyExtensionMethods {
    public static class ExtensionMethods {
        public static IEnumerable<TSource> MyWhere<TSource>(
            this IEnumerable<TSource> source,
            Func<TSource, bool> predicate
            ) {

            foreach (TSource element in source) {
                if (predicate(element)) {
                    yield return element;
                }
            }
        }
    }
}

Side note: putting two namespaces in on file is far from a best practice, but yes that is allowed.

Lambdas and Collection Initializers

If you're new to C# 3.5 then your first thought may be that:

IList<int> ints = new List<int>() {9,8,7,6,5,4,3,2,1};

is not allowed. Actually it is. It's the collection initializer syntax that I initially whined about in my post C# 3.0: The Sweet and Sour of Syntactic Sugar (ironically I actually like this syntax the more I use it.)

Your next thought may be that:

i => i < 5

is not legitimate. This is in fact a Lambda Expression, and as I explained in Deferred Execution, The Elegance of LINQ it conceptually compiles down to an anonymous method. Incidentally those that know Groovy (myself not included) or Lisp may know this as a closure since as we'll see later it can access local variables.

Extension Methods

Ok, the .Net Framework certainly has no MyWhere() function on the List object so this certainly wouldn't compile in C# 2. But that's where C# 3's Extension Methods come in. The "this" in:

MyWhere<TSource>(this IEnumerable<TSource> source,

says that MyWhere() can be applied to any generic IEnumerable. If you want to, you can still call MyWhere() normally:

IList<int> ints = new List<int>() {9,8,7,6,5,4,3,2,1};
ExtensionMethods.MyWhere(ints, i => i < 5);

And in fact this is what the compiler does in the background when you call MyWhere() off of an IEnumerable. But now with extension methods you don't have to.

But does MyWhere() now exist on all IEnumerable objects everywhere? No, it turns out you only get MyWhere() when you import the namespace it exists in (MyExtensionMethods). Incidentally unlike Groovy and Ruby there is no way to add an extension method to a class itself, only to instances.

Whose got the Func()?

The last two questionable parts of the code are the Func<TSource, bool> and the yield. Func is pretty easy. It's simply one of several new predefined delegates (method signatures) that comes with the .Net framework off of the System namespace. The two generic argument one above will match any function that returns the second generic argument and takes the first generic argument as a parameter. It looks like this:

delegate TResult Func<T, TResult>(T arg1);

So rather than using a Lambda expression in my initial example I could have been very explicit about the delegate instance (myFunc):

public static void DoStuff() {
      IList<int> ints = new List<int>() {9,8,7,6,5,4,3,2,1};

      Func<int, bool> myFunc = IsSmall;
      IEnumerable<int> result = ints.MyWhere<int>(myFunc);

      foreach (int i in result) {
            Console.WriteLine(i);
      }
}

public static bool IsSmall(int i) {
      return i < 5;
}

And that would have done the same thing. Notice I had to specify the generic type on the call to MyWhere() since the compiler can't infer the type in this example.

Yield

Now the really interesting part: yield. Yield is what makes deferred execution work. It actually was introduced with C# 2.0, but I don't think anyone really used it (I didn't know about it until recently). So because MyWhere() returns an IEnumerable (and because it isn't anonymous and doesn't have ref or out parameters) it is allowed to use the yield statement. When a method has a yield return (or yield break) statement, then execution of the method doesn't even begin until a calling method first iterates over the resulting IEnumerable. Execution then begins in the method and runs to the first yield statement, returns a result, and passes execution back to the caller. When the calling method iterates to the next value execution continues in the method where it left off until it gets to the next yield statement and then it passes execution back to the caller again and so on. Weird huh? Joshua Flanagan has a nice article that explains this in more detail along with some of the nice benefits like a smaller memory footprint.

So here's a quiz. What happens when you execute the following code?

IList<int> ints = new List<int>() {9,8,7,6,5,4,3,2,1};

IEnumerable<int> result = ints.MyWhere<int>(i => i < 4);

ints.Add(0);

foreach (int i in result) {
Console.WriteLine(i);
}

Without the yield you'd get the numbers 3 through 1 since you added 0 after the call to MyWhere(). But since the yield in MyWhere() (and the Where() in System.Linq) defers execution until the foreach statement, you actually get 3 through 0. Ready for a little more mind bending? How about this:

IList<int> ints = new List<int>() {9,8,7,6,5,4,3,2,1};

int j = 4;

IEnumerable<int> result = ints.MyWhere<int>(i => i < j);

ints.Add(0);
j = 3;

foreach (int i in result) {
Console.WriteLine(i);
}

Does the state of j get captured? My intuition would say yes. If so you'd expect 3 through 0. Well, the closure part of anonymous methods and lambdas work by keeping a reference to their calling object (this). So consequently they always get the most up to date value of a variable. So if your intuition works like mine you'd be wrong. You actually get the numbers 2 through 0. Crazy huh? And definitely something I hope I won't run into in someone's code (JetBrains ReSharper actually warns you if you do something crazy like this).

Conclusion

If this made sense then you should have a pretty solid grasp of how most of Linq to Objects works. Understanding extension methods, Func delegates, and yield statements should form the majority of what Linq does. Well, except for expression trees. But that's a topic for another post. Please post if this doesn't make sense or if I got it all wrong, I'd love to hear from you.

P.S. To comment on this article please use my public Blog.

Deferred Execution, the Elegance of LINQ

Lee Richardson — Wed, 20 Feb 2008 09:45:58 -0500

One of the things I love about LINQ is its deferred execution model. It's the type of thing that makes sense academically when you first read about it (e.g. in Part Three of Scott Gunthrie's LINQ to SQL series), but for me anyway, it took some time to understand enough to use effectively.

For instance the Daily RSS Download open source application that I wrote about last week needs to download entries (posts) that are newly published since the last download. While it isn't a complicated problem, my first attempt at a solution didn't use the power of LINQ correctly. I'll explain my naâ€¢ve solution in this post, describe how LINQ's deferred execution works (i.e. Lambda expressions), explain the problems with my solution, then give an the elegant solution that is only possible because of LINQ's deferred execution model. See if you can spot my error along the way.

Downloading the Latest Entries

Downloading the latest entries would be a ridiculously simple problem if there weren't multiple formats for RSS. But since the solution needs to support Atom and RSS 2.0 and 1.0 and potentially other future formats, the class structure should be set up appropriately:

The newspaper class primarily exists to enumerate feeds:

public class Newspaper {
    public void DownloadNow() {
      foreach (Feed objFeed in Settings.Feeds) {
      objFeed.DownloadRecentEntries(...);
      }
    }
}

The Feed class is abstract and during runtime is either an RssFeed or an AtomFeed. The relevant function Feed.DownloadRecentEntries() calls the abstract Feed.GetEntries() method, which returns a group of Entry objects.

public abstract class Feed {
    public abstract IEnumerable<Entry> GetEntries(XDocument rssfeed);

  public void DownloadRecentEntries(...) {
        XDocument xdocFeed = XDocument.Load(Url);
        IEnumerable<Entry> lstRecentPosts = GetEntries(xdocFeed);

        foreach (Entry objEntry in lstRecentPosts) {
            objEntry.Download(...)
        }
  }
}

The Feed classes, RssFeed and AtomFeed then implement GetEntries as follows:

publpublic class RssFeed : Feed {
    public override IEnumerable<Entry> GetEntries(XDocument rssfeed) {
        return from item in rssfeed.Descendants("item")
               where (DateParser.ParseDateTime(item.Element("pubDate").Value)
                   >= this.LastDownloaded)
                   || this.LastDownloaded == null
               select (Entry)new RssEntry(item, this);
    }
}

public class AtomFeed : Feed {
    public override IEnumerable<Entry> GetEntries(XDocument rssfeed) {
        return from item in rssfeed.Descendants(_atomNamespace + "entry")
               where (DateParser.ParseDateTime(
                   item.Element(_atomNamespace + "published").Value)
                    >= this.LastDownloaded)
                    || this.LastDownloaded == null
               select (Entry)new AtomEntry(item, this);
    }
}

Yes, that's all LINQ to XML in there. It looks a lot like SQL, but as you'll see in a second it's really just glorified syntactic sugar. Expressive though, isn't it? While the astute reader may have already spotted the inelegance of my solution, for those unfamiliar with LINQ, let's first describe what AtomFeed.GetEntries() does.

What is this Deferred Execution Stuff?

If you already understand LINQ and how delayed execution works feel free to skip this section. For everyone else it's important to understand that the following line:

from item in rssfeed.Descendants("item")
               where (DateParser.ParseDateTime(item.Element("pubDate").Value)
                   >= this.LastDownloaded)
                   || this.LastDownloaded == null
               select (Entry)new RssEntry(item, this);

Is actually just syntactic sugar for the following set of statements:

rssfeed
    .Descendants(_atomNamespace + "entry")
    .Where( item => (DateParser.ParseDateTime(
        item.Element(_atomNamespace + "published").Value)
        >= this.LastDownloaded)
        || this.LastDownloaded == null)
    .Select( item => (Entry)new AtomEntry(item, this));

Now XDocument.Descendants() returns IEnumerable which most definitely does not have a Where() function on it. And if you look at the return type of Where() in this context, it returns an IEnumerable which definetly does not have a Select() method on it. That's because Where() and Select() are extension methods, meaning you can attach them on to just about anything. They're new to C# 3.0 and are beyond the scope of this article.

But more important for the topic of deferred execution is the => operator, which is a Lambda expression and is also new to C# 3.0. The best way to understand them is that they are essentially syntactic sugar for an anonymous method (e. (e.g. a type safe function pointer to code). So we could again rewrite our code as follows:

rssfeed
    .Descendants(_atomNamespace + "entry")
    .Where(delegate(XElement item) {
        return (DateParser.ParseDateTime(
            item.Element(_atomNamespace + "published").Value)
            >= this.LastDownloaded) || this.LastDownloaded == null; })
    .Select(delegate(XElement item) {
        return (Entry)new AtomEntry(item, this); });

Back in familiar territory yet? If not you probably aren't familiar with C# 2.0. In the background the compiler takes the anonymous methods above and turns them into methods on the current class and instantiates new delegates of the correct type that points to them and passes them to the Select() and Where() methods.

The The key thing to note is that the arguments for select and where are delegates, and so when those delegates are executed is beyond our control. In fact if you put a Console.WriteLine or a breakpoint inside of the AtomEntry constructor, it won't get called until the resulting IEnumerable is enumerated, specifically the following line in the first code sample:

foreach (Entry objEntry in lstRecentPosts) {

So that's delayed execution. But understanding how it works and how to use it are completely different things.

The Inelegant Solution

Getting back to my code sample you may have picked up that my where clause is the mistake. I implemented it like this because RSS and Atom have different field names for the published date. But the way I wrote it I'd have to make two changes if I wanted to change which entries to download. Ok, big deal, I'm extremely unlikely to make changes to that where clause right? Or I wasn't until I wanted functionality to set some defaults based on the average length of posts prior to downloading posts. Basically:

public static Feed CreateFeed(string strUrl, int intDisplayOrder) {
    IEnumerable<Entry> lstRecentEntries = feed.GetEntries(rssfeed);
    double intAveragePostSize = lstRecentEntries.Average(
        i => i.Description.Length);
    // if the feeds posts are typically small then include the
    // description field in the summary and download the content
    // for the main article from the link
    if (intAveragePostSize < 1000) {
        ...
    } else {
        ...
    }
}

Except this now ties me to the were clause, when what I'd really like to do is just get the average post size for the last couple of posts. The problem is that GetEntries() isn't generic enough.

The Elegant Solution

The The solution is then to normalize out (excuse the database terminology) the where clause into the two methods that use GetEntries(). So GetEntries() becomes simple:

public override IEnumerable<Entry> GetEntries(XDocument rssfeed) {
return from item in rssfeed.Descendants(_atomNamespace + "entry")
select (Entry)new AtomEntry(item, this);
}

And then Feed.CreateFeed() and Feed.DownloadRecentEntries() become more complicated

public abstract class Feed {
    public abstract IEnumerable<Entry> GetEntries(XDocument rssfeed);

    public static Feed CreateFeed(string strUrl, int intDisplayOrder) {
        IEnumerable<Entry> lstEntries = feed.GetEntries(rssfeed);
        // get the five most recent posts
        IEnumerable<Entry> lstRecentEntries =
            from entry in lstEntries.Take(5)
            select entry;
        double intAveragePostSize = lstRecentEntries.Average(
            i => i.Description.Length);
        if (intAveragePostSize < 1000) {
            ...
        } else {
            ...
        }
    }

    public void DownloadRecentEntries(...) {
        XDocument xdocFeed = XDocument.Load(Url);
        IEnumerable<Entry> lstEntries = GetEntries(xdocFeed);
        // get newly published posts
        IEnumerable<Entry> lstRecentPosts = from entry in lstEntries
            where (entry.Published >= this.LastDownloaded)
                || this.LastDownloaded == null
            select entry;

        foreach (Entry objEntry in lstRecentPosts) {
            objEntry.Download(...)
        }
    }
}

Note that we now have a second LINQ statement that runs against the results of the LINQ statement in GetEntries(). But since nothing's been executed yet we're just building out the statement that we will eventually run when the resulting IEnumerable if enumerated. So we've now spread our LINQ statements across an inheriting and a base class, and in process we've made GetEntries() extremely generic.

Conclusion

So what's the big deal? The big deal is that we can spread our data access statements across multple classes and because of deferred execution we don't need to worry about the performance of generic methods that are closer to the data that don't contain a "where" clause. This may not be a huge deal in this example, but it becomes extremely powerful when the user interface tier can tack on "order by" statements or "filters" BEFORE anything is executed against your data store. And that, for me, is at the heart of the beauty of LINQ.

Daily RSS Download

Lee Richardson — Thu, 14 Feb 2008 11:29:00 -0500

I published Daily RSS Download, my first open source project on CodePlex* today. It's not going to change the world, but if you have a need for it there is a decided lack of decent products that perform this functionality. In this post I'll give a little background about why I wrote it and explain what it does and how to use it. Besides needing this functionality I also wrote it to learn LINQ to Objects and LINQ to XML, but I'll cover the more interesting implementation details in a later post.

Why I Wrote It

For Christmas I received an IRex Iliad which is an e-book reader combined with a Wacom tablet. It's an awesome product that allows reading PDF's (among other formats) and writing on them. It's pricey, but the ability to jot notes on technical documents (in addition to recipes and guitar tablature, etc) as you read is invaluable for me. I now read about twice as much as I did before. It supports Wi-Fi, and in particular can connect to a computer on a regular basis to download files you put in a specific directory.

So theoretically it could download a customized newspaper every morning for me, right? I could have today's world news, national news, local news, technical news, weather, and my RSS feeds like Scott Gunthrie all in one place while I eat my cereal! And then I could cancel my Washington Post subscription and after about 7.5 years I would have recouped the costs of the Iliad. Sweet.

The problem is that the product doesn't come with any way to download RSS feeds. Well, you can use software from MobiPocket, but it's a pain to setup, and use, and I couldn't figure out how to have it automatically run on a daily basis. And furthermore it can't grab the real content from the website if the RSS feed only contains an abstract (e.g. washingtonpost.com). I searched and there was some software out there, but none of it did what I liked. And of course none of it generated a manifest.xml file which is an Iliad specific file that links HTML pages together and gives names to groups of content (i.e. grouping the files in a directory to make a "book" called "My Daily News for February 13").

So what a great opportunity to write it myself and learn LINQ to XML and LINQ to Objects in the process.

What It Does

The end result (or the index page anyway) looks something like this:

The images are local, the links go to a full page of content, and on the Iliad, because Daily RSS Download generates a manifest.xml, the next and previous buttons can move you to the next or previous article and you can see at a glance how many articles there are.

If you want to recreate the screenshot above, first head over to the Releases page of Daily RSS Download, where you can download the msi and install the application. When you open "Daily RSS Download Config" you can view a home page like this:

You can type in an RSS or Atom URL and click Add Feed. The application will try to connect to the website, download the title, and set some configuration options based on the average length of posts (specifically if you put in a feed from the washingtonpost.com website it will detect that the average post size is small and determine that it should download the main content from the website).

You can click on any of the feeds you've added and you'll get a Feed Settings page like below:

The fields are mostly self explanatory, but here are three of the more interesting settings:

Summary Source Values:

This setting determines where the abstract (summary) on the index page should come from. There are three options:

No Summary - Does not display a summary on the index.html page. This is what Scott Gunthrie's feed was set to in the first screenshot.

Extract from the content - This takes the first 300 characters from the main content as the summary. This was set for the washingtonpost.com feed in the first screenshot (although Use the RSS description field would actually have been more appropriate).

Use the RSS description field - This uses the entire description field from the RSS (or Atom) feed. This is what the weatherbug feed was set to in the first screenshot. Obviously this is a bad choice for a Scott Gunthrie type of RSS entry since he posts everything in the description field.

Content Source Values:

This setting determines where the main content page should get it's value. There are thee options:

No content, summary only - If you set a feed to this, then Daily RSS Download won't generate a content file. This would be a good choice for the weather feed in the example.

Use the RSS description field - The content file will be created from the RSS description field. This would be a good choice for a Scott Gunthrie type of feed.

Download from the referenced web page -Daily RSS Download will download the page referenced by the RSS or Atom feed. This would be a good option for a washingtonpost.com type of feed.

Content Start/End Markers

These are regular expressions that are used if you set content source to download the referenced web page. You can leave them blank or you can set them if you want to try to strip out header, footer, navigation bars, etc. The content start marker in the screenshot:

\<div id=\"article_body\"[^\>]*\>

Says match '<div id="article_body"' up through to the next '>'. Both markers are exclusive (the thing your matching on won't be included in the results).

Customizing the CSS

So that's it for the general settings and use. You can click "Download Now" on the main config page to download your feeds, and you can set it up to run on a recurring basis (it will only download new content) by setting a recurring task to run "DailyRssDownload.exe DownloadNow". The only other thing of interest is to make the content more pretty.

The generated HTML is CSS customizable, so in order to get the two column look above (and/or make it look pretty on an Iliad) you can customize the CSS as below:

h1
{
      margin-top: 0px;
      /* A pretty linux script font since the Iliad has a linux kernel */
      font-family: Zapf Chancery;
      font-size: 30pt;
      margin-bottom: 0px;
}
h2
{
}
.NewsHeader
{
      border-bottom: solid 1px black;
      text-align: center;
}
.DailyRss_Date
{
      text-align: center;
}
.DailyRss_Feed
{
}
.DailyRss_Entry
{
}
.DailyRss_EvenEntry
{
}
.DailyRss_OddEntry
{
}

/* LEFT COLUMN */
#ScottGusBlog
{
      float: left;
      width: 49%;
      border-right: solid 1px gray;
}
#washingtonpostcom-TodaysHighlights
{
      clear: both;
      float: left;
      width: 49%;
      border-right: solid 1px gray;
}

/* RIGHT COLUMN */
#WeatherBugLocalWeatherfor20190
{
      margin-left: 50%;
}

So basically just use the old float left, width 50%, margin-left 50% trick to get the pretty two-column look (without tables).

Conclusion

I hope you find the Daily RSS Download open source project useful. Please feel free to submit suggestions, feature requests, defects or preferably defects AND patches on the project's CodePlex home page.

* In case you aren't aware CodePlex is an open source project hosting website from Microsoft. It's similar to Source Forge, except there is no approval process for new projects and it integrates nicely with Visual Studio.

Create Data Disaster: Avoid Unique Indexes (Mistake 3 of 10)

Lee Richardson — Thu, 30 Aug 2007 19:54:53 -0400

I really enjoyed Seth Schroeder's critique of the last post in my ten part data modeling mistake series: Surrogate vs Natural Primary Keys. His argument regarding data migration in particular sheds light on a major shortcoming of using surrogate keys: they lead data modelers to a false sense of security regarding the uniqueness of data. Specifically if modelers ignore uniqueness constraints they allow duplicate data. And as Seth points out this has a nasty side effect of disallowing any clear way to compare data between systems. But there are other problems too.

So, in this post I'll address the uniqueness problem introduced with surrogate keys by way of an example, I'll provide two how-to's, one implementing uniqueness in Visio and one in NHibernate, I'll explain the difference between unique indexes and unique constraints, and finally I'll provide reasons why unique indexes might be overlooked, specifically by providing a critique of ORM tools.

Surrogate Keys = Data Disaster?

So as mentioned above the biggest problem with surrogate keys is they lull junior data modelers or lazy developers into thinking they don't need to worry about indexes. But they do; and it's as vital as implementing referential integrity. And for the same reason: data integrity.

As an example, imagine you're modeling a simple Country table. You could of course use CountryName as the primary key, but as you know from my post on surrogate keys, you would have problems with varchar join speed (assuming you disagree with Seth that it's a premature optimization) and to a lesser extent cascading updates (since country names do occasionally change).

Introducing a surrogate key (CountryId) resolves these issues, but you also remove an inherent advantage that natural keys have: they require uniqueness in country names. In other words you can now have two New Zealand's and the system wouldn't stop you.

What's the big deal? Country seems like a pretty benign table to have duplicates, right? Your users from New Zealand simply have an extra list item in their drop down to pick from and some pick one and some pick the other.

For Country one problem comes in reporting. Consider delivering a revenue by Country report. Your report probably lists New Zealand twice and a quick scan by an exec sees half of the actual revenue for that country that they should. And as a result numerous innocent sheep are slaughtered ... uh, or something.

Another major problem could come in syncing data with other systems. How do those systems know which record to use?

As you can imagine the problem is even worse with major entities like Customer, Order, Product, or something more scary like Airline Flights. And the longer the system stays in production, the more production data the system collects, the more duplicates rack up, and the more time and money that will be required to clean up the data when the problem is finally identified. In short the bigger the data disaster.

How To #1: Visio

So the solution is to add at least one unique constraint (or index) to every single table. In other words if you have a table without a uniqueness constraint chances are very good you''ve done something wrong.

The good news is that it's pretty easy to implement once you agree it's necessary. If you're modeling with Microsoft Visio this is a six step process:

Select the table.
Select the "Indexes" category.
Click New.
No need to enter a name, just click OK.
Select either "Unique constraint only" or "Unique index only" (more on this decision later).
Double click the column(s) to add.

Then when you generate or update your database Visio puts in DBMS specific uniqueness constraints. And voila, problem solved.

Unique Constraints vs Unique Indexes

The question will come up when using Visio or perhaps using various DBMS's including SQL Server whether to use a unique constraint or unique index. The short answer is that most people use unique constraints, but ultimately they're the same thing so it doesn't matter.

In case you're interested in the details though here's a quick rundown of the differences:

Unique Constraint

A logical construct.
Defined in the ANSI SQL standard.
Intent: data integrity.
Usually part of a table definition.

Unique Index

A physical DBMS implementation.
Not specified in ANSI SQL.
Intent: performance.
Usually external to a table definition.

But since most DBMS's implement unique constraints as unique indexes, it doesn't really matter which you choose.

How To #2: NHibernate

Since I have the pleasure of learning the NHibernate ORM tool on my current project, I thought I'd also describe the same technique with a different tool. Basically you can either set the Unique attribute to true to obtain uniqueness in one column, or set the unique-key attribute to obtain uniqueness among multiple columns. If you use NHibernate mapping attributes you write:

[Property(NotNull = true, Length = 100, Unique = true)]

public virtual string CountryName {

get { return _strCountryName; }

set { _strCountryName = value; }

}

Which generates the following hbm:

Which NHibernate turns into the following DDL:

create table Country (

CountryId NUMBER(10,0) not null,

CountryName NVARCHAR2(100) not null <b>unique</b>,

primary key (CountryId)

)

So quick quiz: was that a unique index or unique constraint it generated? If you answered who cares you're right. However if you answered a unique constraint you're also right.

The Problem with ORM

Obviously ignorance of the problem and shortsightedness are two causes for systems going into production without unique indexes, but I'd like to point out a third. While Object Relational Mapping (ORM) tools like NHibernate are extremely convenient for generating database schemas, modeling database tables with classes and generating DDL can lead developers to a false sense of purpose.

This can occur because ORM tools focus entirely on the world of objects and classes. In this world data's persistence is irrelevant. It exists for the purposes of a single operation, and consequently long term data persistence issues like data integrity are deemphasized. In fact, it would be easy to lose perspective of the fact that there is a database at all.

Don't get me wrong, the benefits you get like mandatory surrogate keys, DBMS neutrality, lazy loading, and minimal data access code are wonderful. Just don't forget that tags like NHibernate's unique and unique-key exist. And are very necessary.

Conclusion

To sum it up don't allow the convenience of surrogate keys to lull you into a false sense of security regarding the importance of keys. It's a trap. And it will be a disastrous one if you aren't careful.

Surrogate Keys - Data Modeling Mistake 2 of 10

Lee Richardson — Thu, 16 Aug 2007 08:08:45 -0400

In case you're new to the series I've compiled a list of ten data modeling mistakes that I see over and over that I'm tackling one by one. I'll be speaking about these topics at the upcoming IASA conference in October, so I'm hoping to generate some discussion to at least confirm I have well founded arguments.

The last post in this series Referential Integrity was probably less controversial than this one. After all, who can argue against enforcing referential integrity? But as obvious as surrogate keys may be to some, there is a good deal of diversity of opinion as evidenced by the fact that people continue to not use them.

I intend to address this topic by way of a fairly ubiquitous example that should draw out all of the arguments. I'll investigate the options for primary keys in a Person table. I'll provide four possible options and explain why each of them is a bad choice. I'll then give four arguments against surrogate keys, which I will then shoot down. So without further ado:

Contender 1: Name and Location

Of course you can't use name as a primary key for a person, because two people can have the same name and primary keys must be unique. But all too often I've seen databases with multiple, sometime numerous, natural (or business-related) primary keys. These databases combine a field like name with enough other fields such that the likelihood of uniqueness is approaching certainty.

In the case of person this would be equivalent to First and Last Name (wouldn't want to violate first normal form by combining those into one field, but that's a whole other topic), zip code, and we ought to throw address line 1 in to be safe. This is known as either a compound, composite, or multicolumn index.

Now our chances of uniqueness are close enough to certain to not warrant discussion, so let's jump right to space and performance. There are three major problems with this approach.

Con 1: Primary key size. The primary key index for person becomes enormous. The database must now catalog four large (probably all varchar) fields. This increases the size of the index which increases overhead for insert, delete and update operations and can even decreases read speed because of the increased disk I/O.

Con 2: Foreign key size. If you have a child table like PhoneNumber then, as the diagram above shows, the foreign key becomes four columns. Those four columns take up a lot of space again. And now a common query like "Get all phone numbers for a person" involves a full table scan, or, if you throw an index on them you end up with another huge index. In fact, you most likely end up propagating huge indexes and vast amounts of data all over the place like some evil data-cancer.

Con 3: Asthetics. It just isn't pretty. Having four column foreign keys all over the place increases the amount of code you need to write in stored procedures, middle tier, and presentation tier. Even intellisense won't help you with this one.

Contender 2: Social Security Number

The most obvious choice for a natural key for a person object is social security number, right? Obviously it depends on what type of data person is, but regardless you'll probably face the following four problems with this primary key candidate:

Con 4: Optionality. The social security administration specifies that U.S. citizens are not required to provide social security numbers in many circumstances. While employment is one of these circumstances, consumers of non-governmental services are definitely not. You can deny service if your consumer won't provide the number, but is your CEO prepared to turn away business based on a data modeling decision you make?

Con 5: Applicability. Only U.S. citizens have a social security number. Your system might only cater to U.S. citizens now, but will it always?

Con 6: Uniqueness. The social security administration "is adamant" that the numbers are not recycled, even after someone dies. But eventually the numbers will run out. If you visit the slate article cited above, it calculates this date as in the next century. But the math fails to include the fact that location information is encoded in the number which significantly limits the permutations. I don't know what the real number is, but the point is: you're gambling with how long until a conflict occurs. And even if the time argument fails to sway you, just think of who is assigning the numbers. How much do you trust a government office to not make an occasional mistake?

Con 7: Privacy. Does your application use primary keys in the user interface tier to uniquely identify records? Does it pass primary keys between pages or use them to identify rows in a combo box? You certainly wouldn't store such a thing in a cookie or pass it across the wire unencrypted right? Social security information is sensitive information and privacy zealots care very much how you handle their data. Primary keys are necessarily are closer to end users and harder to hide than regular fields. It just isn't the type of data to take a chance on.

Contender 3: E-mail

So e-mail is a pretty likely choice right? It's a relatively safe assumption that no two people share an e-mail (maybe). And anyone with a computer has one right? So there should be no uniqueness, privacy or optionality/applicability problems. But how about this:

Con 8: Accidental Denormalization. Do you have more than one e-mail address? Doesn't everyone? Imagine what a pain it would be if Evite only allowed you one e-mail address per person (ok, well if you didn't know it does allow you to consolidate accounts for those of us with multiple e-mail addresses). Even if your system only stores one e-mail address per person now, just think what a pain it would be to change the database to allow N e-mail addresses per person.

No. Wait. Really. Think about it...

Yea ... yuck.

Contender 4: Username

If your users log in with a username, that's a likely candidate for a primary key right? But what if they want to update their username (perhaps it was based on a last name that changed). This leads us to:

Con 9: Cascading Updates. If you have a natural key that might change you'll need to implement some type of cascading updates (whether your DBMS supports it or you write code by hand). In other words, change the username in the person table and you have to change the username foreign key in all child records of the invoices, comments, sales, certifications, defects, or whatever other tables you track. It may not happen often, but when it does it sure will wreak havoc on your indexes. Imagine rebuilding even 10% of your indexes at once because of one operation. It's just unnecessary.

Con 10: Varchar join speed. I left this to last because it applies to all of contenders thus far and is by far the most compelling argument against natural keys. Nine out of ten natural keys are varchar fields. Even an employee number as generated by another system may have a significant zero. It's a fact: joining across tables with varchars is always slower than joining across tables with integers. How much? According to Peter Zaitsev who runs a MySql performance blog it's 20% to 600% slower. And that's for one join. How many joins do you think comprise an average user interaction? Five? Ten? Twenty? It could very likely make a significant difference to your end user.

And The Winner Is

So surrogate keys win right? Well, let's review and see if any of the con's of natural key's apply to surrogate keys:

Con 1: Primary key size - Surrogate keys generally don't have problems with index size since they're usually a single column of type int. That's about as small as it gets.
Con 2: Foreign key size - They don't have foreign key or foreign index size problems either for the same reason as Con 1.
Con 3: Asthetics - Well, it's an eye of the beholder type thing, but they certainly don't involve writing as much code as with compound natural keys.
Con 4 & 5: Optionality & Applicability - Surrogate keys have no problems with people or things not wanting to or not being able to provide the data.
Con 6: Uniqueness - They are 100% guaranteed to be unique. That's a relief.
Con 7: Privacy - They have no privacy concerns should an unscrupulous person obtain them.
Con 8: Accidental Denormalization - You can't accidentally denormalize non-business data.
Con 9: Cascading Updates - Surrogate keys don't change, so no worries about how to cascade them on update.
Con 10: Varchar join speed - They're generally int's, so they're generally as fast to join over as you can get.

For every natural key con I see a surrogate key pro. But not everyone agrees. Here are some arguments against them.

Disadvantage 1: Getting The Next Value

Some have argued that getting the next value for a surrogate keys is a pain. Perhaps that's true in Oracle with its sequences, but generally it just takes a couple minutes research, or you can use ORM tools to hide the details for you.

Disadvantage 2: Users Don't Understand Them

One argument I uncovered is if users were to perform ad-hoc queries on the database they wouldn't be able to understand how to use surrogate keys.

Bunk. Bunk, bunk, bunk. End users shouldn't be fiddling in databases any more than airline customers should be fiddling in airplane engines. And if they are savvy enough to try, then let them learn to perform joins like the pros do.

Disadvantage 3: Extra Joins

Suppose you have users table with a social security number natural primary key, and a phone number child table with social security as a foreign key.

If your user enters a social security number on a log in screen you could theoretically get their phone numbers without accessing the users table. In a surrogate key world you would have to look up the surrogate key in the person table before getting their phone numbers.

While this is true, I have found that with most CRUD applications there are few times when this scenario comes up. The vast majority of queries involve already known surrogate keys. So while this argument may be true in some situations, it just isn't true enough of the time to be significant.

Disadvantage 4: Extra Indexes

I find this to be the most persuasive argument against natural keys. If your person object would normally have a natural key on social security number, then in surrogate-world you should have a unique index on social security number in addition to your primary key index on the surrogate key. In other words, you now have two indexes instead of one. In fact, if you have N indexes per table in natural key world, you'll always have N + 1 indexes in surrogate key world.

While the additional indexes do indeed add indexes, which increase database size, and slow insert and update performance, you could offset some of that expense by converting your old natural key, social security number for example, to a clustered index.

Or you could just relax in the knowledge that there are pro's and con's to every architectural decision and for surrogate keys the pro's outweigh the con's.

Summary

So now if some well meaning DBA argues to use natural keys on your next project you should have ten arguments against them, which will double as ten arguments for surrogate keys, and you should be prepared with rebuttals for four arguments against surrogate keys. Whew, that was a lot. But I assure you, if you use surrogate keys today it will definitely make your life easier in the long run.

Sync Your Database to SharePoint Using SSIS

Lee Richardson — Mon, 30 Jul 2007 18:47:01 -0400

Just a heads up that DevX just published an article of mine today. The article is entitled Sync Your Database to SharePoint Using SSIS. The article covers how to import and export SharePoint list items using Collaborative Application Markup Language (CAML), SharePoint's web services API, and SQL Server Integration Services.

The latter half of the article is a fairly detailed how-to, but the former half covers what SharePoint lists are, what SSIS is, and why you would want to use them all together. I hope you find the article useful, and feel free to comment here if you have thoughts on the article.

Referential Integrity - Data Modeling Mistake 1 of 10

Lee Richardson — Wed, 18 Jul 2007 19:02:03 -0400

In my mind data models are like the foundations of a house. Whether you use ORM or a more traditional modeling tool, they form the base of the entire rest of your project. Consequently, every decision you make (or don't make) regarding your data model during the design phase(s) of your project will significantly affect the duration of your project and the maintainability and performance of your application.

You could de-emphasize up-front planning, but every correction you make to the data model once code has been written on top of it will introduce significant delays to the project as developers refactor data access, business logic, and user interface tiers. That's why mistakes made during design are expensive, and it would behoove any architect (or project manager) to be well aware of the repercussions of data model decisions and minimize mistakes before construction begins.

After years of working with or maintaining applications based on poorly designed data models, and after years of modeling my own databases from scratch I've seen and made a lot of mistakes. So, I've compiled ten of the most common ones and the arguments for and against them.

I'll be speaking on this topic in the upcoming IASA conference in October, and so I wanted to vet these ideas with the community. I know there are strong feelings on these topics, so please help me out by commenting if you feel I've missed something or am off base.

I'll start with Mistake #1: Not Using Referential Integrity in this post. I'll give four common reasons for avoiding referential integrity and then rebuff them. I'll then cover the more controversial Mistake #2 Not Using Surrogate Keys in my next post.

Mistake #1 - Not using referential integrity

I've heard a lot of excuses for not using referential integrity, but I've never been swayed by one of them. If you have a record with a foreign key field you should be 100% certain that it will always refer to the primary key of an existing record in one and only one foreign table. The last thing you want to do is write large amounts of conditional logic because you aren't 100% certain that you aren't dealing with orphaned data. Nonetheless, here are some almost compelling arguments I've heard for not using it:

Reason #1: Project Too Small

If your project or database is only a few tables and a couple lines of code then you don't need referential integrity right? Wrong, numerous projects start small, get big, and have major problems because of it. It doesn't take much extra time to put in constraints. Avoid the urge to be lazy.

Reason #2: Accidental Oversight

Numerous applications I've seen forget a relationship or two. This is borne of writing and executing database creation statements by hand and is the reason that data modeling tools exist. When you visualize your database in a model it's hard to miss a relationship. So use a modeling tool and keep it in sync with your database, you won't regret it.

Incidentally I like Microsoft Visio for data modeling because you can change your schema during development and Visio won't delete your data. This enables you to keep your data model in sync with the database for the entire lifetime of the database. There are other benefits too, if you're interested see my article on data modeling in Microsoft Visio.

Reason #3: Maximize Insert Speed

It's a fact: indexes and constraints slow down insert and update operations. If your application is heavy on writing and light on reading, then you could argue referential integrity isn't for you. This argument is often combined with the "Only one application ever uses my database" argument.

There are two problems with this. One problem comes when either a well meaning DBA modifies data by hand and messes up the state of the database, or more realistically when there's a bug in the application that accidentally orphans data. Orphaned data may not affect your application, but a well designed solution should plan for the future. When that data warehouse project finally gets around to importing data from your database, what do they do with the orphaned data? Ignore it? Try to integrate it? Who knows? If you've been in this position, you'll know what I mean when I say the responsible architect's name (or their app) will be synonymous with a curse word.

The second problem is that even if a database without referential integrity don't end up with orphaned data, a second application that might want to integrate can still never be 100% certain that foreign keys refer to existing records. It comes down to designing for the future.

The answer to speed is to build your database with referential integrity, drop or disable your constraints and indexes before a bulk load, and re-enable them after the bulk load. It will increase the duration of your bulk load operation over not using constraints at all, but it will be much faster than leaving them enabled and checking them for each insert. So use referential integrity: the pros outweigh the cons.

Reason #4: Mutually exclusive relationships

Too often I've seen databases with a foreign key that relates to one of five tables based on the value of a char(1) field. The space conserving mindset that comes up with this implementation is admirable, but it produces far too many negative side effects.

What happens when the char(1) field gets out of sync with the foreign key field? What happens when someone deletes the foreign record or changes its primary key? More orphaned data happens.

The solution is to use five fields that each refer to a single table. You may have more nullable fields that take up more space in the database, but it's worth it in the long run.

Conclusion

Well, hopefully I've convinced you to avoid the urge to be a lazy data modeler, design for the future, use a data modeling tool, and drop constraints during bulk load operations. In short, always use referential integrity. But if not, hopefully you'll at least understand when people curse your name several years from now. :)

An Entity Relationship Diagram Example

Lee Richardson — Thu, 07 Jun 2007 00:38:24 -0400

It seems like a dying art, but I still strongly feel that Entity Relationship Diagrams (ERD) should be the starting point of all software development projects. Since they are for me anyway, I wanted a place to refer colleagues to for how to read these diagrams, and an Entity Relationship Diagram Example seemed like a great place to start.

The Example: A Resource Management Application

Consider that we're writing a resource management application. The first step to creating an ERD is always to identify the nouns (entities). In this case let's start with:

Company
Employee
Project; and
Technology Project (which are a specific type of Project that perhaps require special fields like "number of entities")

Here's the Example Entity Relationship Diagram I'll decipher piece by piece in this article (click to enlarge):

(note that I'm now using singular names since my somewhat controversial decision to switch to naming entities in the singular)

To read the notations of an Entity Relationship Diagram:

An Entity Relationship Diagram conveys a lot of information with a very concise notation. The important part to keep in mind is to limit what you're reading using the following technique:

Choose two entities (e.g. Company and Employee)
Pick one that you're interested in (e.g. how a single Company relates to employees)
Read the notation on the second entity (e.g. the crow's feet with the O above it next to the Employee entity).

The set of symbols consist of Crow's feet (which Wikipedia describes as looking like the forward digits of a bird's claw), O, and dash, but they can be combined in four distinct combinations. Here are the four combinations:

Zero through Many (crow's feet, O)
One through Many (crow's feet, dash)
One and Only One (dash, dash)
Zero or One (dash, O)

Zero through Many

If, as in the diagram above, the notation closest to the second entity is a crow's feet with an O next to it, then the first entity can have zero, one, or many of the second entity. Consequently the diagram above would read: "A company can have zero, one, or many employees".

This is the most common relationship type, and consequently many people ignore the O. While you can consider the O optional, I consider it a best practice to be explicit to differentiate it from the less common one through many relationship.

One through Many

If, as the next diagram shows, the notation closest to the second entity is a crow's feet with a dash, then the first entity can have one through many of the second entity. More specifically it may not contain zero of the second entity. The example above would thus read (read bottom to top): "A Project can have one through many Employees working on it."

This is an interesting combination because it can't (and for various reasons probably shouldn't if it could) be enforced by a database. Thus, you will only see these in logical, but not a physical, data models. It is still useful to distinguish, but your application will need to enforce the relationship in business rules.

One and Only One (onne)

If the notation closest to the second entity contains two dashes it indicates that the first entity can have one and only one of the second. More specifically it cannot have zero, and it cannot have more than one. The example would thus read: "An Employee can have one and only one Company."

This combination is the most common after zero through many, and so frequently people consider the second dash optional. In fact, some ignore both dashes, but I would highly recommend at least using one for clarity so as not to confuse the notation with "I'll fill in the relationship details later".

Zero or One

A zero or one relationship is indicated by a dash and an O. It indicates that the first entity can have zero or one of the second, but not more than one. The relationship in the example above would thus read: "A Project can have zero or one Technology Project."

The zero or one relationship is quite common and is frequently abbreviated with just an O (however it is most commonly seen in a many-to-many relationship rather than the one-to-one above, more on this later).

Relationship Types

Having examined the four types of notation, the discussion wouldn't be complete without a quick overview of the three relationship types. These are:

One to Many
Many to Many
One to One

One-to-Many

A one-to-many (1N) is by far the most common relationship type. It consists of either a one through many or a zero through many notation on one side of a relationship and a one and only one or zero or one notation on the other. The relationship between Company and Employee in the example is a one-to-many relationship.

Many-to-Many

The next most common relationship is a many-to-many (NM). It consists of a zero through many or one through many on both sides of a relationship. This construct only exists in logical data models because databases can't implement the relationship directly. Physical data models implement a many-to-many relationship by using an associative (or link or resolving) table via two one-to-many relationships.

The relationship between Employee and Project in the example is a many to many relationship. It would exist in logical and physical data models as follows:

One-to-One

Probably the least common and most misunderstood relationship is the one-to-one. It consists of a one and only one notation on one side of a relationship and a zero or one on the other. It warrants a discussion unto itself, but for now the Project to Technology Project relationship in the example is a one to one. Because these relationships are easy to mistake for traditional one-to-many relationships, I have taken to drawing a red dashed line around them. The red dashed line is not standard at all (although a colleague, Steve Dempsey uses a similar notation), but in my experience it can help eliminate confusion.

Conclusion

I hope you've found this a useful example for deciphering and verifying entity relationship diagrams. As always please add any comments, disagreements, thoughts or related resources.

Export Visio Database Table Names to Excel

Lee Richardson — Thu, 10 May 2007 18:23:07 -0400

If you use the Enterprise Architect edition of Microsoft Visio for data modeling regularly, then there is a good chance that at some point you've wanted to export just the table names into Excel. You might want to do this to map logical ERD entities to physical data model tables, track project status by entity, or track overlap between database versions.

Regardless, it turns out to be non-trivial to perform this export, particularly if you are unable to generate to a database to retrieve the table names. The trick is to use the reporting feature of Visio, but there are many reports and report options, and you will need one that is table-based to get the data into Excel easily.

Note: If you are unfamiliar with the capabilities of Microsoft Visio as a data modeling tool you may wish to take a look at my Data Modeling in Microsoft Visio Tutorial.

Export Procedure

1. This may seem a little unusual, but if you don't have any comments in any of your tables (which really shouldn't be the case), you will need to add comments for at least one of your tables. Without this step Visio will not display tables in a grid format in the report.

2. Now select the somewhat obscure "Report" item off of the "Database" menu.

3. Only the "Table Report" provides the ability to layout database tables in a grid. Select it and click Finish.

4. Under "Predefined logical/physical reports" Click the button labeled "Default To: General Report" and change it to "Default To: Database Report." This will remove tables formatted per page from the end of the report.

5. Under the "Attributes" tab select "Deselect All" then select the "Table stats summary" option.

6. Click "Export to RTF," save the file somewhere, and open it with Microsoft Word.

7. (optional) If you have any new lines in the notes field you may have to replace them with spaces. Just do a search and replace for "^l" and replace with " ".

8. Now you're ready to copy and paste.

And you're done! Hopefully this tutorial will make life easier for you next time you need to export table names from Visio to Microsoft Excel.

CAML: Nested Too Deep

Lee Richardson — Fri, 04 May 2007 13:04:38 -0400

I discovered an interesting error recently while working with Microsoft's Collaborative Application Markup Language (CAML) that, surprisingly, had received no ink. Partly what surprises me about this is that the error may require you to rewrite large sections of your code if you haven't previously considered this SharePoint limitation. I'll start with some context, but first of all the error is:

"Some part of your SQL statement is nested too deeply. Rewrite the query or break it up into smaller queries,"

So let's back up and describe what CAML is for everyone who isn't yet experiencing this problem.

CAML is an XML based query language that is similar to, and serves many of the same purposes as SQL and DDL. However, CAML is used primarily to interact with SharePoint lists. You can retrieve data using something like:

<Where><Eq><FieldRef Name='Title'/><Value Type='Text'>hello</Value></Eq></Where>

Which is equilivant to WHERE Title = "hello" in a SQL statement. At this point I'm sure you're asking why exchange 80 characters for 21? I suppose the answer is related to how CAML is primarily used in web services requests. Regardless, the nested too deeply error occurs when you try to insert large amounts of data into a list.

The insert syntax is actually all batch based, so if you have 100 list items to insert into a list, you build them all into one big CAML statement and then send it across the wire with a call to Lists.UpdateListItems(). Your CAML statement will look something like the following:

<Batch OnError="Continue">
    <Method ID="1" Cmd="New">
        <Field Name="Title">Hello<Field>
        <Field Name="Document">5</Field>
   </Method>
   <Method ID="2" Cmd="New">
        <Field Name="Title" >World</Field>
        <Field Name="Document">5</Field>
   </Method>
</Batch>

The problem comes when you want to insert something like 19,642 list items into a list. SharePoint complains with "Some part of your SQL statement is nested too deeply. Rewrite the query or break it up into smaller queries," and suddenly you're stuck with an enormous CAML statement that you need to break up into smaller batches. How small? Well, for me 300 worked and 500 didn't, so I set my batch size to 300.

This means that if you are currently experiencing this problem, then have fun rewriting your code to use smaller batches. And, if you're lucky enough to be reading this before you experience the error, then prepare yourself now for this possibility.

Or, if performance isn't an issue and you don't want to or can't rewrite the code that built your CAML then feel free to use this code to batch up CAML batches into smaller batches.

/// <summary>

/// Breaks a larg CAML query into smaller batches to avoid the error "Some part of your SQL statement is nested too deeply. Rewrite the query or break it up into smaller queries."

/// </summary>

/// <param name="listService">The SharePoint list service to execute the CAML against.</param>

/// <param name="strListName">The name of the list to execute the CAML against.</param>

/// <param name="elementLargeBatch">The CAML batch list of commands to be broken up.</param>

/// <param name="intBatchSize">The size of batches to use. If unsure use 300, it seems to work fairly well.</param>

/// <returns>The results of all batched queries.</returns>

public static XmlNode ExecuteLargeQuery(

Lists listService,

string strListName,

XmlElement elementLargeBatch,

int intBatchSize

) {

// calculate useful information

int intMethodCount = elementLargeBatch.ChildNodes.Count;

int intBatchCount = (int)Math.Ceiling((double)intMethodCount / (double)intBatchSize);

// prepare xml documents for batches and results

XmlDocument xmlDocBatch = new XmlDocument();

XmlDocument xmlDocResults = new XmlDocument();

XmlElement elementResults = xmlDocResults.CreateElement("Results");

// for each batch

for (int intCurrentBatch = 0; intCurrentBatch < intBatchCount; intCurrentBatch++) {

int intMethodStart = intCurrentBatch * intBatchSize;

int intMethodEnd = Math.Min(intMethodStart + intBatchSize - 1, intMethodCount - 1);

XmlElement elementSmallBatch = ListHelper.CreateBatch(xmlDocBatch);

// for each method in the batch

for (int intCurrentMethod = intMethodStart; intCurrentMethod <= intMethodEnd; intCurrentMethod++) {

XmlElement element = (XmlElement)elementLargeBatch.ChildNodes[intCurrentMethod];

elementSmallBatch.AppendChild(xmlDocBatch.ImportNode(element, true));

}

// execute the batch

XmlNode nodeBatchResult = listService.UpdateListItems(strListName, elementSmallBatch);

// add the results of the batch into the results xml document

foreach (XmlElement elementResult in nodeBatchResult.ChildNodes)

elementResults.AppendChild(xmlDocResults.ImportNode(elementResult, true));

// clean up

xmlDocBatch.RemoveAll();

}

return elementResults;

}

Useful MSDN References

Incidentally, these MSDN references might be useful if you're interested in more info:

Introduction to Collaborative Application Markup Language (CAML)

CAML Reference: Query Schema

CAML Reference: Batch Element

C# 3.0: The Sweet and Sour of Syntactic Sugar

Lee Richardson — Wed, 25 Apr 2007 13:51:53 -0400

I've just started reading the blog of Scott Guthrie who is a general manager at Microsoft and is currently writing about the new C# 3.0 (code named Orcas). The post I felt strongly about describes three new language features including:

Automatic Properties
Object Initializers, and
Collection Initializers

All three features are purely syntactic sugar: they add nothing of real value to the language (unlike generics in C# 2.0 for example). One features is wonderful, while two are terrible.

The Sweetest Syntactic Sugar Ever

Automatic properties are fabulous. I've always grumbled about the number of lines required to create simple properties. Anything that can turn:

public class Person {
    private string _firstName;

    public string FirstName {
        get { return _firstName; }
        set { _firstName = value; }
    }
}

Into:

public class Person {
public string FirstName { get; set; }
}

Gets my vote.

Sickeningly Sweet

Perhaps I'm a purest, but with more language conveniences comes: a steeper learning curve because of a larger language, and decreased readability because of the numerous ways of doing the same thing. Picture reading this in someone else's code:

Person p = new Person("Lee", "Richardson") {
    Phone = "111-111-1111",
    Age = 29,
    Company = new Company("Near Infinity") {
        Phone = "222-222-2222"
    }
}

Who would write this crap? Everyone else, now that Microsoft lets them. I vote for the verbose, the readable, the single approach:

Company company = new Company("Near Infinity");
company.Phone = "222-222-2222";

Person person = new Person("Lee", "Richardson");
person.Phone = "111-111-1111";
person.Age = 29;
person.Company = company;

Hmmm, and by the way did you count the number of lines?

Conclusion

So do the differences between my thoughts on automatic properties contradict with my thoughts on object initializers and collection initializers? They don't. Basically a little sugar is good, but too much ruins the dish.

Parameter passing in C#

Lee Richardson — Mon, 23 Apr 2007 17:34:18 -0400

The topic of parameter passing in C# keeps coming up, and so I thought this post might be useful. It's been my most popular post on my rapid application development blog, so I thought I'd repost here. Please kick the post if you like it.

So there I was, merrily browsing the Internet when I came across this fantastic site on topics such as:

Implementing a singleton pattern in C#,
Type initializers,
Static constructors,
Delegates and events,
And as my title suggests: Parameter passing in C#.

In short, all those things you rarely need to know to get your job done, but that separate mediocre developers from good ones. The site is by Jon Skeet and the articles are informative, well researched, well explained, well written. Here's the C# part of his site:

http://www.yoda.arachsys.com/csharp/

The article that caught my attention made sense to me, but being a very visual person I couldn't help but think that some pictures could really help illustrate the points. So without further ado, I illustrated the article. You probably don't need to read the article to understand this post - but you should:

http://www.yoda.arachsys.com/csharp/parameters.html.

Note: you can click the images to get a clearer view.

1. Value Types

Notice that the values live inside the box which will not be the case for reference types. Also, the assignment operation copies the value inside of the box, this is important to compare with reference types.