Are you sure you're seeing all of the replies to your tweets -- even from people you don't follow? Do you ever find yourself missing important posts -- like those from your real life friends? Do you know which will reach a wider audience: new style retweets vs old style retweets?

This article will answer these and other twitter related questions. It will help beginner to intermediate twitter users get the most out of the service.

#1 Standard #Tweets

These appear simple. These show up in the timeline of your followers (@admirer and @vet in the example above). And they don't show up in the timeline of anyone who doesn't follow you (@effusive). If this was all there was to twitter it would be a complete waste of time.

The subtlety with this type of tweet is that your content may show up to someone that doesn't follow you but that is following a hashtag or a topic you mention. In the example above @hiro thows a hashtag onto the word aardvarks to indicate that it's special somehow. His tweet is then picked up by @spca who runs a constant search in a twitter client for the word "#aardvark".

The first time I realized how powerful this could be I had was at a conference. I was extremely bored by a terrible keynote. With nothing else to do I discovered the conference hashtag on twitter and started watching it. Suddenly I realized I wasn't alone. I was in a huge room full of people making fun of the presenter! It was like telepathy, or group consciousness or something. I've never felt alone or bored at a conference since.

Interesting statistic: As of June 2010, about 65 million tweets were posted each day, equaling about 750 tweets sent each second. (reference)

#2 Replies

Replies are more complicated. In the example above the guy on the left, @vet, replies to @hiro by prefixing his tweet with "@hiro". However, this probably doesn't work like you think. The tweet obviously shows up in @hiro's timeline, but it also shows up in @admirer's timeline because she follows both people mentioned in the tweet (@hiro and @vet). But the tweet does not show up to @effusive because, while he follows @vet, the author of the tweet, he doesn't follow @hiro, the person being mentioned.

Getting this scoping rule wrong is way too easy. For instance you can't just announce that someone did something great. If @hiro started a tweet with "@vet gave a great anti-aardvark presentation" then most of his followers (e.g. @effusive) wouldn't actually see the tweet! I've seen seasoned twitter users make this mistake.

As an interesting aside twitter can help track conversations. If you click a reply button in a twitter client or on the website twitter will keep track of which tweet you replied to and then help piece a conversation together. But if you simply start a new tweet with "@somone" without clicking reply, the above scoping rules still apply, but twitter won't help anyone reconstruct the conversation.

Finally a word on notifications: If someone mentions you (via a reply or anywhere in their tweet) and you follow them then you will get an e-mail and possibly an SMS notification. If someone that you do not follow replies to you, you will not get an email. For this reason you must either frequently check the "mentions and more" section of the website or use a column based twitter client, discussed in the final section on lists.

Interesting statistic: On average only 23% of tweets get a reply. (more)

Another one: 38% of tweets are conversational. In other words 38% of tweets start with an "@". If your percentage of replies is lower you probably aren't using twitter to its fullest. (more)

#3 Reply-All's

So what happens if you want to reply to someone, but still have it show up in the timelines of all of your followers? As long as your tweet doesn't start with an "@" symbol then you're fine. So the convention that has grown up is to prefix your tweet with a "." and then the "@someone". In the example above @effusive now sees the tweet where he didn't in a standard reply.

Interesting statistic: The highest usage rate of twitter to date occurred during the 2011 FIFA Women's World Cup Final between Japan and the United States when 7,196 tweets were published per second! (more)

#4 Old Style Retweet or "Retweet with Comment"

Prior to late 2009 if you saw a tweet you wanted to share with your followers you would retweet (RT) it by copy and pasting their tweet and prefixing it with "RT @author". The problem from a scoping perspective was that people would get duplicates. In the example above @admirer sees a duplicate from @vet when he retweets @hiro's tweet.

There are other problems with old style retweets unrelated to scoping. One problem is that you get long chains of "RT @person1 RT @person2 ...". Another problem is it was hard to follow a person but not all their retweets. The biggest problem is old style retweets take valuable characters away from your available 140, and sometimes require mangling the original.

Old style retweeting is still in use today primarily from people wish to retweet but add a comment, but also because it allows people to reach a wider audience (since new style retweets won't show up in searches or lists, more on this later).

Interesting statistic: Only 6% of tweets get retweeted (more)

#5 New Style Retweet

In November of 2009 twitter rolled out a new technique for retweeting aimed at solving the problems mentioned above. It caused a lot of confusion, but did solve the problems.

The end result of new style retweets from a scoping perspective is that sometimes people see tweets from people they don't follow. In the example above @effusive sees @hiro's tweet in his timeline even though he doesn't follow @hiro, because @vet performed a new-style retweet.

If you click the retweet button on the twitter website today it will perform a new style retweet. Some twitter clients like TweetDeck give you the option. The only way to perform an old style retweet or retweet with comment via the website is to copy and paste the message.

Interesting statistic: Less than half of tweets are posted using the web user interface. By far the most common twitter client is Tweetdeck with 8.48% of the market. (more)

#6 Direct Messages

Direct messages are simple. Prefix your tweet with "D somone" (no at symbol before the name) and they will be the only person that will see it. Just like e-mail except the recipient will get an e-mail and a text message if they've set that up on their phone. Note that neither people watching an included hashtag (e.g. @spca) nor people mentioned will see direct messages unless they are the recipient.

Interesting statistic: Six percent of all tweets are sent via SMS. (more)

#7 Lists

The last scoping topic worthy of mention is lists. This fantastic feature was added by twitter in late 2009. It allows you to organize the people you follow. For instance I have a private "Infrequent Posters" list that I try very hard to never miss a tweet from. Meanwhile I follow enough people that my main list is more like a stream that I dip into but don't get stressed out if I miss some of.

Lists work best when used in conjunction with a column based twitter clients (e.g. TweetDeck). These types of twitter clients are the only way to go if you want to become a twitter pro. If you've gotten to the point where you start missing content from important people (e.g. your real life friends), or you're missing responses to your tweets, or mentions from people you don't follow: then you probably need lists, but you absolutely need something like TweetDeck.

In regards to scoping, lists contain one subtlety that you need to be aware of. They do not support new style retweets. In the example above @vet posts a normal tweet. He then new-style retweets one of @hiro's tweets. @effusive follows @vet and has also placed him into a list. While the list helps @effusive not miss tweets from @vet, it does not end up displaying @hiro's tweet.

Note that this rule applies both to people who maintain their own lists as well as to people who follow lists maintained by other people (e.g. your company may maintain a public list of its twitter enabled employees).

Interesting Statistic: Pointless babble is the most common type of content on twitter. It represents 40.55% of all traffic. (more)

Conclusion

Twitter may appear to be fairly simple at first glance. In reality it has grown over the years to become very sophisticated. Sophisticated and incredibly powerful. But to fully tap that power you must understand the scoping rules and subtleties surrounding things like replies, retweets, and lists. Hopefully you're closer now.

Compiling C# to What!?

Compiling C# code into JavaScript may seem foreign, but Script# is a mature technology that is absolutely worth a look. Our team has been using it very happily for about three months. We've found a number of benefits including:

• Consistently working Intellisense
• Better encapsulation
• Simpler Object Orientation
• Easier deployment (Script# compiles multiple files to a single, optionally minified file)
• Safer refactoring
• Simpler unit testing
• Type safety; and
• Design time syntax checking (no more mistyping a variable and accidentally declaring it to the global scope)

Scott Hanselman covered most of these topics in last month's Hanselminutes episode: Script# Compiles to JavaScript. If you have a spare car ride I definitely recommend the listen. But what wasn't covered were some additional benefits provided by an off the beaten path approach to this great technology. The main benefit is tighter server side C# to client side JavaScript integration. A secondary benefit is less of a dependency on Script#.

To the Command Line

Typically to get going with Script# you:

1. Install the Script# Visual Studio plug-in
2. File -> New project
3. Select "Script Library" and
4. Compile to generate JavaScript

Nice and easy. The down side to this approach is you can't easily include files outside of your project. Specifically, you can't include data transfer objects or enum's from your server side code. Furthermore, all of your team members must install Script#. If instead you compile with Script#'s "ssc.exe" command you obtain more control and get less dependency.

How-To

The how-to looks something like this:

1. Download and install Script#
2. Add a Class Library (not a Script# project)
3. Project properties -> Build -> Advanced -> "Do not reference mscorlib"
   
4. Ideally move the Script# files (C:\Program Files (x86)\ScriptSharp) locally to the solution and check them into source control to a Libs or something similar
5. Remove all references, but add: ScriptSharp.dll, ScriptSharp.Web.dll, Script.Jquery
6. Edit your .csproj to manually reference Script#'s mscorlib (right click, Unload project, Edit MyProject.csproj)
   
   <Reference Include="mscorlib, Version=0.7.0.0, Culture=neutral, PublicKeyToken=8fc0e3af5abcb6c4, processorArchitecture=MSIL">
     <SpecificVersion>True</SpecificVersion>
     <HintPath>..\Libs\ScriptSharp\v1.0\mscorlib.dll</HintPath>
   </Reference>
   
   
7. Modify AssemblyInfo.cs and remove the following lines:
   
   [assembly: ComVisible(false)]
   [assembly: Guid("b5e2449f-193c-46d1-9023-9143618d8491")]
   
   
8. Modify AssemblyInfo.cs and add the following:
   
   [assembly: ScriptAssembly("ScriptSharpDemoAssembly")
   
   
9. Ensure it compiles in Visual Studio
10. Create a batch script or PowerShell script that compiles using ssc.exe like this:
    
    ..\Libs\ScriptSharp\v1.0\ssc.exe ^
        /debug ^
        /out:MyScriptSharp.js ^
        /ref:"..\Libs\ScriptSharp\v1.0\Framework\mscorlib.dll" ^
        .\Properties\AssemblyInfo.cs ^
        .\Class1.cs
    
    

Notice the last two lines in the script are a list of the files that you want to include. The point of this exercise is that you can now include files outside of your main Script# project in that list.

Now for completeness if you put a simple static method in Class1.cs, something like this:

namespace MyScriptSharp
{
    public class Class1
    {
        public static string HelloWorld()
        {
            return "Hello World!";
        }
    }
}

Then run the batch file you should get something like this:

Type.registerNamespace('MyScriptSharp');

////////////////////////////////////////////////////////////////////////////////
// MyScriptSharp.Class1

MyScriptSharp.Class1 = function MyScriptSharp_Class1() {
}
MyScriptSharp.Class1.helloWorld = function MyScriptSharp_Class1$helloWorld() {
    return 'Hello World!';
}

MyScriptSharp.Class1.registerClass('MyScriptSharp.Class1');

Obviously you could get these results faster with approach #1. But now you have a lot more control.

Summary

The main downside to this approach is maintaining the batch file is a bit of a hassle. But the upsides are that you can include any file from your server-side C# code. And any changes in that server-side code are automatically reflected in your JavaScript. And any breaking changes in your server-side code generate compile time errors in your client side code. And furthermore none of your team members need to install Script#. For our team it's an easy tradeoff. What about for yours? Please share your thoughts in the comments or on twitter.

The problem is that burndown charts lack two essential pieces of information. First, how much work was actually accomplished during a given iteration (as opposed to how much work remains to be completed) and second how much total work the project contains (or if you prefer how much scope has increased each iteration).

What you might not know is how flexible they can be. In particular the example I gave last time has a problem. Can you spot it?

The problem is that the scope increase of 20 points in iteration six might very well have had zero impact on the actual deployment date. Those points might have all been low priority tasks that the team will work on after they deploy the initial 100 points. Wouldn't it be nice if the burn up chart could convey priority information as well?

So that's exactly what I did on my current project last week, and it appeared to be a big hit. And with the addition of some Excel trend lines the chart was able to convey a lot of insight into the project:

Obviously all that data is made up, but the chart still tells an interesting story:

• The customer hasn't been adding high priority tasks, perhaps because there is a hard deadline approaching
• Because of the customer's restraint in adding high priority tasks the team can be fairly confident that they will finish all the high priority tasks by Iteration 3
• While the customer hasn't added high priority tasks they have still been able to be agile by adding normal and low priority tasks

What's really wonderful about this way of reporting is that it gives the customer the flexibility to add scope to future iterations while maintaining near term deadlines. In short it supports what agile is supposed to be about.

But why stop there? Depending on the story you need to tell or the scenario you need to evaluate perhaps you could incorporate team member contributions. For example:

In this example we can try to show what might happen if we remove a particular team member or if we'd never had a particular team member. What's unique about this way or reporting is that it's incorporating the backlog's increasing trend into account. While it's possible that the backlog may increase or decrease at a different rate with a different team makeup, it's a better scenario than ignoring the rate of increase of the backlog all together.

Perhaps we could incorporate other information. Maybe bug vs. feature information. Or planned vs. actual (e.g. task slippage) information. It seems like there is a lot of potential. Any other ideas out there? Feel free to post in the comments or via twitter. If there's enough interest I'd be happy to post another screencast similar to the last one I did on how to produce burn up and burn down with SharePoint and Excel except this time with trends and priorities.

So overall while these charts may be a little more complicated than a traditional burn down chart they would make an excellent talking point during a PowerPoint presentation, or as part of your iteration close out. And with a little training I bet just about any customer would learn to love the insight they provide.

The Unit Testing Problem

Any architecture needs a place to put code that finds entities. For instance FindBySocialSecurityNumber(). In a traditional architecture we might put a method like this is in a DAO layer. If so our method will look something like this:

public class EmployeesDao {
    public Employee FindBySSN(Context ctx, string ssn) {
        return ctx.Employees.SingleOrDefault(e => e.Ssn == ssn);
    }
}

So how would we go about unit testing this?

One fairly typical solution would be to use an in-memory database. That approach works if our data store is a database, but it certainly doesn't work if the data store is something less traditional like SharePoint. But even if our store is a database, we'll still have the hassle of setting up the in-memory database.

Another solution might be to use a mock Context that returns an IQueryable. But wouldn't it be wonderful if we could avoid mocking all together?

Killing the DAO

The first question is why we even have a DAO tier to begin with. The original idea was that we wanted a place to put code specific to a particular data store. In other words we wanted to isolate the code that will need to be changed should the data store switch from SQL Server to Oracle. But isn't that exactly what LINQ does? I'd be pretty surprised if there wasn't a decent LINQ provider for just about any data store at this point that required more than minimal code changes. So why not embrace LINQ and reconsider alternatives to a DAO tier?

One alternative that I've been using for over a month now is to switch to extension methods. To give credit where it's due the idea originated with a conversation with fellow Near Infinity employee Joe Ferner. And I'm sure the idea isn't particularly original (please post in the comments if you know others that use this approach).

Using this technique our code changes from something like this:

var employeeDao = new EmployeesDao(); // or use IOC of course
employeeDao.FindBySSN(ctx, "111-11-1111");

To something like this:

ctx.Employees.FindBySSN("111-11-1111");

Among other things I find this far more aesthetically pleasing because each of the three elements to the statement represent a subsequent filtering of data. It's a more functional way of looking at things.

We could implement this off of the Employees property of the context if we have control over that (which I don't with spmetal). But if we implement this as an extension method like this:

public static class EmployeeExtensions {
    public static Employee FindBySSN(this IQueryable<Employee> employees, string ssn) {
        return employees.SingleOrDefault(e => e.Ssn == ssn);
    }
}

We now have something that's considerable easier to unit test.

Testing It

Once we've refactored our function as an extension method that filters down the corpus of entities, we can test the code using in-memory objects with a call to .AsQueryable(). For instance:

public void FindBySSN_OneSsnExists_EmployeeReturned() {
       var employees = new [] { new Employee { Ssn = "111-11-1111" } };
       var actual = employees.AsQueryable().FindBySSN("111-11-1111");
       Assert.IsNotNull(actual);
}

Notice we didn't have to mock anything.

Testability, but at What Cost?

This technique works great for the example above, but how does it scale to harder problems and what other downsides are there?

As far as scalability I've found this technique works great for every scenario I've run across in the month I've been doing it. It works for joins, aggregations, and even for inserts, update, and deletes.

As far as downsides the astute reader may be wondering about mockability. For instance what if we want to mock the call to FindBySSN and give it the exact Employee that will be returned. This scenario is admittedly harder. But what I've found is that far more often than not I don't really need to mock the types of things that used to live in the DAO tier. Instead I just mock the Employee object off of context to return in-memory objects and make my tests slightly larger in scope. Most of the time I find the larger scope increases the usefulness of the test. In the occasional case where I do really want to mock the "DAO" tier I use a technique described in this post by Daniel Cazzulino.

Conclusion

Obviously there is more to this architecture, for instance how do you handle insert and update operations? The short answer is it's easy, but I'll save that topic for a future post. For now why not give this approach a try? You weren't really happy with that useless old DAO tier anyway, were you? I say we eradicate it and never look back.

Part 1 - Exploring WCF Data Services in SharePoint 2010
Part 2 - Creating a read-only page with ASP.Net AJAX 4.0 Templates
Part 3 - Writing data back to SharePoint with ASP.Net AJAX 4.0 Templates
Part 4 - jQuery and Manual JSON Object Manipulation

In the previous posts in this series I've shown how you can access SharePoint data using WCF Data Services, how you can display that data using ASP.Net AJAX 4.0 Templates, and how you can write data back to SharePoint.

In this post I'll take writing data back to SharePoint a step further by showing how you can modify non-visible fields by modifying JSON objects directly. More specifically I'll enable dragging user story index cards on a virtual bulletin board using a jQuery plugin, save the X and Y coordinates back to JSON objects, and then save the updated records back to SharePoint.

Cards on a Corkboard

The first thing I'll do is style the cards and display them with absolute position. While I'm at it I'll also reference the draggable jQuery UI plugin. If you're following along at home just built a custom .js download from the jQuery UI site, drop it in your layouts directory and add a reference. So my PageHead content control now looks like this:

<asp:content
    id="PageHead"
    contentplaceholderid="PlaceHolderAdditionalPageHead"
    runat="server">

    <script type="text/javascript"
        src="../ajax/MicrosoftAjax.js"></script>
    <script type="text/javascript"
        src="../ajax/MicrosoftAjaxDataContext.js"></script>
    <script type="text/javascript"
        src="../ajax/MicrosoftAjaxTemplates.js"></script>
    <script type="text/javascript"
        src="../ajax/MicrosoftAjaxAdoNet.js"></script>
    <script type="text/javascript"
        src="../jquery-ui-1.8.custom/js/jquery-1.4.2.min.js"></script>
    <script type="text/javascript"
        src="../jquery-ui-1.8.custom/js/jquery-ui-1.8.custom.min.js">
    </script>
    <style type="text/css">
        .sys-template
        {
            display: none;
        }
        .userstorycard
        {
            border: 1px solid #777777;
            width: 200px;
            position: absolute;
            cursor: move;
        }
        .carddescription
        {
            font-size: 13px;
            background-image: url('card_bg.jpg');
            padding: 0px 5px 5px 5px;
        }
        .cardtitle
        {
            font-size: 15px;
            font-weight: bold;
            padding: 0px 0px 0px 0px;
            border-bottom: 1px solid red;
            background-color: White;
            padding: 0px 5px 0px 5px;
        }

The next thing to do is add X and Y fields to the user story list in SharePoint. And finally display the cards with absolute positioning:

Notice the sys:style attribute. The sys: part is the namespace you need to add for attributes when using templating that I mentioned in part three. Also notice the JavaScript string concatenation inside the binding. You can put in any JavaScript in there that you like. So now if you manually set X and Y values in SharePoint for a card the card shows in the right place. But we're still missing the ability to drag cards.

Making Cards Draggable with jQuery

In order to enable dragging support you're supposed to call the draggable() function on the DOM elements you want to make draggable. Simple right? You just call $(".userstorycard").draggable() which should find every element with a class of userstorycard and call draggable() on it.

But if you do this onLoad() then ASP.Net AJAX Templating engine hasn't had a chance to render the new DOM elements yet. Fortunately the DataView has the JavaScript equivalent of an OnRendered event that you can tie into:

function onLoad() {
    dataContext = $create(
        Sys.Data.AdoNetDataContext,
        { serviceUri: "/demoprep/_vti_bin/ListData.svc" }
        );

    dataView = $create(
        Sys.UI.DataView,
        {
            autoFetch: true,
            dataProvider: dataContext,
            fetchOperation: "UserStories",
            fetchParameters: { $top: 20 }
        },
        {
            rendered: onRendered
        },
        null,
        $get('userStories')
    );
}

function onRendered() {
    // from http://jqueryui.com/demos/draggable/
    $(".userstorycard").draggable();
}

And that's it for enabling dragging. Pretty simple. The result looks like this:

Ok, well you can't see the dragging support, but it's there, trust me. And it's pretty cool except for one thing.

Saving Non-Visible Properties

The problem is that every time you refresh the page the nice layout you've done disappears and everything goes back to being stacked one on top of the other in the upper left. My first thought was to create input type=hidden form elements on the page. This approach causes a lot more plumbing code than is necessary. The better way is that you can access the underlying in-memory JSON objects. Step one is to register an "event" for when the user stops dragging a card.

function onRendered() {
    // from http://jqueryui.com/demos/draggable/
    $(".userstorycard").draggable({
        stop: onDragStop
    });
}

Once you wire that up the onDragStop() function is where all the interesting stuff happens:

function onDragStop(event, ui) {
    var userStoryCard = ui.helper[0];
    var selectedUserStoryJsonObject =
        dataView.findContext(userStoryCard).dataItem;

    var newX = ui.position.left;
    var newY = ui.position.top;

    Sys.Observer.setValue(selectedUserStoryJsonObject, "X", newX);
    Sys.Observer.setValue(selectedUserStoryJsonObject, "Y", newY);
    dataContext.saveChanges();
}

The first line gets the DOM element that just the user just completed dragging. The second line retrieves the in memory JSON object represented by that DOM element. The next couple of lines get the X and Y that the user story card was dragged to relative to their parent DOM element. And then something odd happens.

Sys.Observer.setValue looks so complicated. Why couldn't we just call selectedUserStoryJsonObject.X = newX?

Well, if we'd done that the DataView wouldn't know about the change we just made. If we then tried to call dataContext.saveChanges() nothing would get sent back to the server. Sys.Observer.setValue notifies any interested parties, in this case the DataView, that a value has been changed. And this enables the functionality we saw in part two where the DataView only sends the relevant records back to SharePoint instead of the entire set of JSON objects.

And so, with this code in place, we can now move user story cards around on the page and when we refresh the page or come back days later our layout has persisted back into SharePoint.

Conclusion

In this post you've seen how to directly modify the in-memory JSON objects and send them back to SharePoint. Our application is finally getting interesting and I find it quite remarkable how little code it's taken. In the next post I'll show how to do master-detail scenario's for editing user story cards.

Part 1 - Exploring WCF Data Services in SharePoint 2010
Part 2 - Creating a read-only page with ASP.Net AJAX 4.0 Templates
Part 3 - Writing data back to SharePoint with ASP.Net AJAX 4.0 Templates
Part 4 - jQuery and Manual JSON Object Manipulation

In the previous two posts in this series I've shown how you can access SharePoint data using WCF Data Services and consume that data using ASP.Net AJAX 4 templates. In this post I'll update the previous example in order to show how to write data back to SharePoint.

Templates and Linking

Before we go into writing data back let's modify our example to link each item back to the SharePoint DispForm.aspx page. A naïve approach would look like this:

<ul id="userStoriesList" class="sys-template">
       <li><a href="../../Lists/User%20Stories/DispForm.aspx?ID={{ID}}">{{ Title }}</a></li>
</ul>

It turns out this approach doesn't work because the DataView won't look for the data binding syntax inside of attributes. The next approach that might make sense would now be to set href to a data binding syntax and inside there do some string concatenation like so:

<li><a href="{{ '../../Lists/User%20Stories/DispForm.aspx?ID=' + ID }}">{{ Title }}</a></li>

It turns out this doesn't work either, but for a different reason. Apparently the DataView can't natively convert some HTML attributes like href and src for reasons Bertrand La Roy describes. The solution is to prefix the attribute with the ASP.Net AJAX system namespace. Typically you would define the system namespace on the HTML body element. But with SharePoint you don't have access to this element. Fortunately you can declare this namespace on any element so the following code works:

<div xmlns:sys="javascript:Sys">
    <ul id="userStoriesList" class="sys-template">
        <li><a sys:href="{{ '../../Lists/User%20Stories/DispForm.aspx?ID=' + ID }}">{{ Title }}</a></li>
    </ul>
</div>

Great. Let's move on to updating data.

Live Binding Syntax

The {{ }} template syntax we've seen so far is what's known as one time binding. When you use this syntax the expression is evaluated only once when the template is rendered. What we want is one way live binding and/or two way live binding.

In order to understand the live bindings you need to understand that behind the scenes the DataView keeps an in memory copy of all of the JSON objects it downloads. It's possible for the data in these objects to change. And that's where one way live bindings come in. When you use the syntax { binding [FieldName] }, the DataView will automatically update the binding when the underlying JSON object changes.

Two way live binding happens automatically when you use the one way live binding syntax on HTML INPUT elements. For these scenarios the DataView automatically updates the in memory JSON objects. Then the DataView updates any one way live binding's.

So let's throw two DataViews onto the page, one with one way live bound hyperlinks, one with two way bound HTML INPUT textbox elements and see what happens:

<script type="text/javascript">
    function pageInit() {
        var dataContext = $create(Sys.Data.AdoNetDataContext, { serviceUri: "/demoprep/_vti_bin/ListData.svc" });

        $create(Sys.UI.DataView,
                {
                    autoFetch: true,
                    dataProvider: dataContext,
                    fetchOperation: "UserStories",
                    fetchParameters: { $top: 20 }
                },
                null,
                null,
                $get("userStoriesList")
            );

        $create(Sys.UI.DataView,
                {
                    autoFetch: true,
                    dataProvider: dataContext,
                    fetchOperation: "UserStories",
                    fetchParameters: { $top: 20 }
                },
                null,
                null,
                $get("userStoriesTable")
            );
    }
    Sys.Application.add_init(pageInit);
</script>

<div xmlns:sys="javascript:Sys">
    <ul id="userStoriesList" class="sys-template">
        <li><a sys:href="{{ '../../Lists/User%20Stories/DispForm.aspx?ID=' + ID }}">{ binding Title }</a></li>
    </ul>

    <table>
        <thead>
            <tr>
                <td>Title</td>
            </tr>
        </thead>
        <tbody id="userStoriesTable" class="sys-template">
            <tr>
                <td><input type="text" sys:value="{binding Title}" /></td>
            </tr>
        </tbody>
    </table>
</div>

Notice the input element with the sys:value bound to title and the new binding syntax in the text of the a href. The result looks like this:

It may not be all that pretty yet, but if you change the text in any of the text boxes the result is immediately updated in the hyperlinks on mouse out. But if you hop back to SharePoint or refresh the page you'll see that our data hasn't been updated.

Updating SharePoint, For Real This Time

Just because the DataView updated the in memory JSON objects it's bound to doesn't mean it sent a POST request back to ListData.svc. To accomplish that we need to call dataContext.saveChanges(). If you toss in a button below the table like this:

<button onclick="dataContext.saveChanges()">Save Changes</button>

And move the dataContext variable to a higher scope like this:

var dataContext;

function onLoad() {
       dataContext = $create(Sys.Data.AdoNetDataContext, { serviceUri: "/demoprep/_vti_bin/ListData.svc" });

Then when you click the button the AdoNetDataContext figures out which JSON objects changed and sends just those objects back to ListData.svc. How do you know it only sends the rows that changed? If you look at Fiddler there will be a POST to ListData.svc/$batch with something like the following:

--batch_d425-6668-6b05
Content-Type: multipart/mixed;boundary=changeset_8a05-bdcd-0b3e
--changeset_8a05-bdcd-0b3e
Content-Type: application/http
Content-Transfer-Encoding: binary
MERGE http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3) HTTP/1.1
If-Match: W/"3"
Host: localhost
Accept: application/json
Accept-Charset: utf-8
Content-Type: application/json;charset=utf-8
{
"__metadata":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)",
"etag":"W/\"3\"",
"type":"Microsoft.SharePoint.DataService.UserStoriesItem"},
"ContentTypeID":"0x01080070E2807D369BD94FBD6C057D3110E6D3",
"Title":"Renamed Task",
"Predecessors":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Predecessors"}},
"Priority":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Priority"}},
"PriorityValue":"(2) Normal",
"Status":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Status"}},
"StatusValue":"Not Started",
"Complete":null,
"AssignedToID":null,
"TaskGroupID":null,
"Description":"<div></div>",
"StartDate":"\/Date(1266624000000)\/",
"DueDate":null,
"Points":8,
"Iteration":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Iteration"}},
"IterationID":1,
"ID":3,
"ContentType":"Task",
"Modified":"\/Date(1269128451000)\/",
"Created":"\/Date(1266679964000)\/",
"CreatedByID":1,
"ModifiedByID":1,
"Owshiddenversion":3,
"Version":"3.0",
"Attachments":{"__deferred":{"uri":"http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(3)/Attachments"}},
"Path":"/demoprep/Lists/User Stories"
}
--changeset_8a05-bdcd-0b3e--
--batch_d425-6668-6b05--

And that, as you may have noticed, is the JSON for just a single of the rows that we were displaying.

But how did the AdoNetDataContext know which objects were changed? Jonathan Carter explains this better than I could but basically when the data context downloads JSON objects it injects them with the ability to notify itself when they change. As Jonathan points out it's a neat trick that would not be possible in a statically typed language.

Summary

We've finally gotten to some of the core benefits of using ASP.Net AJAX 4 Templating. With very little JavaScript we were able to cleanly separate our UI from our data access code and write data back to the server. And the amount of HTTP traffic and server operations was kept to a bare minimum, keeping things as fast and responsive as possible. So all we're missing at this point is how to use it to make something real out of these technologies. And that will be the topic for the next post.

Part 1 - Exploring WCF Data Services in SharePoint 2010
Part 2 - Creating a read-only page with ASP.Net AJAX 4.0 Templates
Part 3 - Writing data back to SharePoint with ASP.Net AJAX 4.0 Templates

In Part 1 I described how SharePoint 2010 exposes list data through ListData.svc and WCF Data Services. So far this is interesting, but the next logical question is how would you actually build a Web 2.0 application with the data? One answer is with a new technology called ASP.Net AJAX 4.0 Client Side Templating. I'll give you some background and then provide a walkthrough of how to build a SharePoint application page that reads and writes data to SharePoint through ListData.svc.

What is ASP.Net AJAX 4.0 Client Side Templating?

In Part 1 I talked about some of the problems with the two existing ways of writing Web 2.0 interfaces. ASP.Net AJAX 4.0 Client Side Templating addresses most of these issues. Specifically it:

• Minimizes plumbing code
• Cleanly separates presentation from data access code
• Simplifies saving data back to the server via Live Data Bindings
• Has no ViewState!
• Uses lightweight JSON rather than HTML

Sounds good? Ready to see some code?

Show Me The Code

Note: This walkthrough assumes you haven't played with Visual Studio 2010 and SharePoint 2010, so please skip ahead to "No Really, Show Me the AJAX Templating Code" if you've already seen the sweet SharePoint and Visual Studio integration.

To get started open Visual Studio 2010. Select New Project and for the template navigate to SharePoint then 2010 select "Empty SharePoint Project" (p.s. see Visual Web Part in the list? Make sure you look that up if you haven't heard of it yet).

If you haven't been keeping up with what's new in SharePoint 2010 you should immediately notice that SharePoint integration is built into Visual Studio 2010 out of the box just based on the listed templates. But just wait, it gets much better.

The next screen asks what site you want to use for debugging. It's asking this because when you hit F5 ("Start Debugging") Visual Studio 2010 packages up your wsp file, deployes it to the URL you enter here, attaches to the w3wp process, and opens a browser into SharePoint, and you're immediately in debug mode!

If you aren't deeply impressed then you haven't developed in WSS 3.0. Now we aren't writing any server side code in this tutorial, but it's still extremely impressive how much work Microsoft has put into making SharePoint a first class citizen.

Once you have your site select Project -> Add New Item and select Application Page.

Visual Studio then plops out a page that's 100% ready to go including the correct Content controls for the SharePoint master page.

<%@ Assembly Name="$SharePoint.Project.AssemblyFullName$" %>
<%@ Import Namespace="Microsoft.SharePoint.ApplicationPages" %>
<%@ Register Tagprefix="SharePoint" Namespace="Microsoft.SharePoint.WebControls" Assembly="Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<%@ Register Tagprefix="Utilities" Namespace="Microsoft.SharePoint.Utilities" Assembly="Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<%@ Register Tagprefix="asp" Namespace="System.Web.UI" Assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" %>
<%@ Import Namespace="Microsoft.SharePoint" %>
<%@ Assembly Name="Microsoft.Web.CommandUI, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="ApplicationPage1.aspx.cs" Inherits="MyAppPage.Layouts.MyAppPage.ApplicationPage1" DynamicMasterPageFile="~masterurl/default.master" %>

<asp:Content ID="PageHead" ContentPlaceHolderID="PlaceHolderAdditionalPageHead" runat="server">

</asp:Content>

<asp:Content ID="Main" ContentPlaceHolderID="PlaceHolderMain" runat="server">
Hello World!
</asp:Content>

<asp:Content ID="PageTitle" ContentPlaceHolderID="PlaceHolderPageTitle" runat="server">
Application Page
</asp:Content>

<asp:Content ID="PageTitleInTitleArea" ContentPlaceHolderID="PlaceHolderPageTitleInTitleArea" runat="server" >
My Application Page
</asp:Content>

(ok, the "Hello World!" is mine). Now if you hit F5 or click Build -> "Deploy Solution" then as I mentioned SharePoint packages up the application page into a wsp file and deployes it out to SharePoint. Now if you navigate to your application page (in my case http://localhost/mysite/_layouts/MyAppPage/ApplicationPage1.aspx) then you should see something like this:

Amazed? You should be. We didn't write a line of evil looking CAML, didn't have to write any DOS batch files, didn't have to deploy to the GAC, we probably didn't even notice that a feature.xml file was created behind the scenes. Yet already we're deployed! Ok, enough exuberance, back to business.

No Really, Show Me the AJAX Templating Code

Before writing code you should probably create some lists to read and write to. As I mentioned in a previous post I like tracking user stories in SharePoint. So for my examples I have an Iterations list and a User Stories list, which is based off of the Tasks content type (automatically giving you a number of columns like Priority).

After creating some lists you need to ensure you have the latest ASP.Net Ajax Library. I suspect an updated copy will be included with the RTM copy of SharePoint 2010, but in my case I downloaded the latest copy and moved the entire Scripts directory into C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\LAYOUTS\ajax. Note that out of the box SharePoint 2010 already included some of these files in LAYOUTS, but it was missing others, so this step was necessary.

So here is the absolute simplest templating example that still works:

<asp:Content ID="PageHead" ContentPlaceHolderID="PlaceHolderAdditionalPageHead" runat="server">
    <script type="text/javascript" src="../ajax/MicrosoftAjax.js"></script>
    <script type="text/javascript" src="../ajax/MicrosoftAjaxDataContext.js"></script>
    <script type="text/javascript" src="../ajax/MicrosoftAjaxTemplates.js"></script>
    <script type="text/javascript" src="../ajax/MicrosoftAjaxAdoNet.js"></script>
</asp:Content>

<asp:Content ID="Main" ContentPlaceHolderID="PlaceHolderMain" runat="server">
    <script type="text/javascript">
        function pageInit() {
            var dataContext = $create(Sys.Data.AdoNetDataContext, {
                       serviceUri: "/demoprep/_vti_bin/ListData.svc"
                     });

            $create(Sys.UI.DataView,
                {
                    autoFetch: true,
                    dataProvider: dataContext,
                    fetchOperation: "UserStories",
                    fetchParameters: { $top: 20 }
                },
                null,
                null,
                $get("userStoriesList")
            );
        }
        Sys.Application.add_init(pageInit);
    </script>

    <ul id="userStoriesList" class="sys-template">
        <li>{{ Title }}</li>
    </ul>
</asp:Content>

The four script references in the PageHead Content control are necessary to define the required ASP.Net AJAX JavaScript classes you'll need for templating. But the Main content control is where things get interesting.

Skip over the JavaScript for now and notice the unordered list named userStoriesList. Inside of the li elements is a strange notation: {{ Title }}. This specifies a binding to the name of a SharePoint field. You could also have put in PriorityValue or any of the other columns specified in the JSON that I showed in Part 1.

As far as the JavaScript, the dataContext variable is a class of type AdoNetDataContext, which knows how to talk to WCF Data Services like ListData.svc. It inherits from a JavaScript class called DataContext which is extensible and knows how to talk with other protocols, but us SharePoint developers probably won't care about this most of the time.

The userStoriesTemplate variable is a class of type DataView. This is where the magic happens. DataViews pull data from a DataContext, duplicates the innerHtml of an associated DOM element for each row returned (e.g. userStoriesList), and replace the binding syntax with real data. Seems simple enough, but it is actually quite sophisticated when you get to some more advanced scenarios that I'll describe later.

One last thing to notice is the sys-template class. This is a prespecified CSS class that ASP.Net AJAX sets to display: block when it is finished rendering. It's a good idea to create a CSS class for sys-template with display: none so that end users don't see your template code while the page is loading. Obviously you should do this in a separate CSS class in LAYOUTS or if you're lazy you can put it in the PageHead content control like so:

<STYLE type="text/css">
       .sys-template { display: block; }
</STYLE>

<asp:Content ID="PageHead" ContentPlaceHolderID="PlaceHolderAdditionalPageHead" runat="server">
       <script type="text/javascript" src="../ajax/MicrosoftAjax.js"></script>
       <script type="text/javascript" src="../ajax/MicrosoftAjaxDataContext.js"></script>
       <script type="text/javascript" src="../ajax/MicrosoftAjaxTemplates.js"></script>
       <script type="text/javascript" src="../ajax/MicrosoftAjaxAdoNet.js"></script>
       <STYLE type="text/css">
              .sys-template { display: block; }
       </STYLE>
</asp:Content>

So now if you click If you click Build -> Deploy Solution you should see something like this:

Summary

Ok, this isn't super exciting yet. We certainly could have done this much easier with server side code. But bear with me, this has a lot of potential. For instance, how do you write data back to the server? How do you really make it AJAXy? I'll explore these questions in Part 3 - Writing data back to SharePoint with ASP.Net AJAX 4.0 Templates and subsequent posts in the series.

One of the consequences of this Web 2.0 design is that the expectations of SharePoint end users will significantly increase. No longer will long delays and/or full page refreshes in your SharePoint based custom applications be acceptable. End users may not be able to clearly articulate it, but unless as developers we update our techniques to match the new technology our SharePoint based applications will begin to feel stale and will stick out from the rest of SharePoint 2010.

Fortunately Microsoft has stepped up the support it offers Web 2.0 developers in a big way. One of the most compelling features is in how SharePoint exposes all list data via REST/ATOM and JSON. This feature combined with the new templating feature of ASP.Net AJAX 4 allows developers to easily write fast, responsive user interfaces. I will explore these two technologies in the following series:

Part 1 - Exploring WCF Data Services in SharePoint 2010
Part 2 - Creating a read-only page with ASP.Net AJAX 4.0 Templates
Part 3 - Writing data back to SharePoint with ASP.Net AJAX 4.0 Templates
Part 4 - jQuery and Manual JSON Object Manipulation

Why do we need yet another framework?

If you attempt to develop Web 2.0 applications in SharePoint with the techniques you have today there are really only two approaches: ASP.Net AJAX and server side controls or manually coding JavaScript, probably with jQuery or other JavaScript Libraries. If you take the ASP.Net AJAX approach you'll undoubtedly be using an UpdatePanel to get partial page refreshes and then writing ASP.Net code like you always have. There are several problems with this approach:

• It sends the entire ViewState with every asynchronous request - this can result in slow, unresponsive apps (this can actually sneak up on you once you have a lot of production data as happened to me recently)
• Even without a ViewState the UpdatePanel transmits large amounts of HTML (rather than lightweight JSON data)
• It runs through the entire page lifecycle on updates, executing everything on the page even if it isn't relevant to what you're doing
• ASP.Net AJAX doesn't have the simplicity of an ETag based HTTP caching like with a REST solution
• Finally SharePoint just doesn't work well with UpdatePanel, I can't tell you how many issues I've had with various controls not working out of the box

And if you take the manual JavaScript rout you are faced with a number of different problems:

• You have to write a lot of plumbing code
• You have to write a service to return your data
• You have to pay particular attention to security and permissions since you're exposing your API's to the world
• You have to convert your data (ideally JSON) into HTML with probably string concatenation or document.createElement statements
• Your UI code is not clearly separated from your data access code making your code hard to maintain
• Saving data back to the server involves even more plumbing code

SharePoint 2010 combined with ASP.Net AJAX 4.0 solves these problems. It probably introduces new problems too, but I haven't run into too many thus far. So over the next couple of posts I hope to convince you that we do need yet another web framework and that Microsoft is headed in a great direction with these technologies.

So How Does it Work?

In WSS 3.0 you were able to access list data with the SOAP based Lists.asmx. This technique worked well for .Net desktop apps or server side application pages, but not so well with client side JavaScript. SharePoint 2010 exposes its list data with REST based ListData.svc (though WCF Data Services which I'll describe in more detail below). Once you install WCF Data Services on a machine with SharePoint 2010 you can simply navigate to http://[server]/[optional site]/_vti_bin/ListData.svc and see all of the lists you have permission to access. The result will look like this:

<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>
<service xml:base="http://localhost/mySite/_vti_bin/ListData.svc/"
    xmlns:atom="http://www.w3.org/2005/Atom"
    xmlns:app="http://www.w3.org/2007/app"
    xmlns="http://www.w3.org/2007/app">
  <workspace>
    <atom:title>Default</atom:title>
    <collection href="Iterations">
      <atom:title>Iterations</atom:title>
    </collection>
    <collection href="UserStories">
      <atom:title>UserStories</atom:title>
    </collection>
  </workspace>
</service>

You can then append the names of lists (e.g. http://localhost/site/_vti_bin/ListData.svc/UserStories) and immediately view the list items in the list as an ATOM feed. A typical result looks like this:

<?xml version="1.0" encoding="iso-8859-1" standalone="yes"?>
<feed xml:base="http://localhost/mySite/_vti_bin/ListData.svc/"
    xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices"
    xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"
    xmlns="http://www.w3.org/2005/Atom">
  <title type="text">UserStories</title>
  <id>http://localhost/mySite/_vti_bin/ListData.svc/UserStories</id>
  <updated>2010-03-20T02:20:26Z</updated>
  <link rel="self" title="UserStories" href="UserStories" />
  <entry m:etag="W/&quot;3&quot;">
    <id>http://localhost/mySite/_vti_bin/ListData.svc/UserStories(2)</id>
    <title type="text">Blah</title>
    <updated>2010-03-19T15:13:28-05:00</updated>
    <author>
      <name />
    </author>
    <link rel="edit" title="UserStoriesItem" href="UserStories(2)" />
    <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Predecessors"
        type="application/atom+xml;type=feed"
        title="Predecessors"
        href="UserStories(2)/Predecessors" />
    <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Priority"
        type="application/atom+xml;type=entry"
        title="Priority"
        href="UserStories(2)/Priority" />
    <link
        rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Status"
        type="application/atom+xml;type=entry"
        title="Status"
        href="UserStories(2)/Status" />
    <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Iteration"
        type="application/atom+xml;type=entry"
        title="Iteration"
        href="UserStories(2)/Iteration" />
    <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Attachments"
        type="application/atom+xml;type=feed"
        title="Attachments"
        href="UserStories(2)/Attachments" />
    <category
        term="Microsoft.SharePoint.DataService.UserStoriesItem"
        scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
    <content type="application/xml">
      <m:properties>
        <d:ContentTypeID>0x01080070E2807D369BD94FBD6C057D3110E6D3</d:ContentTypeID>
        <d:Title>Do Something</d:Title>
        <d:PriorityValue>(2) Normal</d:PriorityValue>
        <d:StatusValue>Not Started</d:StatusValue>
        <d:Complete m:type="Edm.Double" m:null="true" />
        <d:AssignedToID m:type="Edm.Int32" m:null="true" />
        <d:TaskGroupID m:type="Edm.Int32" m:null="true" />
        <d:Description>&lt;div&gt;&lt;/div&gt;</d:Description>
        <d:StartDate m:type="Edm.DateTime">2010-02-20T00:00:00</d:StartDate>
        <d:DueDate m:type="Edm.DateTime" m:null="true" />
        <d:Points m:type="Edm.Double">3</d:Points>
        <d:IterationID m:type="Edm.Int32">3</d:IterationID>
        <d:ID m:type="Edm.Int32">2</d:ID>
        <d:ContentType>Task</d:ContentType>
        <d:Modified m:type="Edm.DateTime">2010-03-19T15:13:28</d:Modified>
        <d:Created m:type="Edm.DateTime">2010-02-20T15:32:26</d:Created>
        <d:CreatedByID m:type="Edm.Int32">1</d:CreatedByID>
        <d:ModifiedByID m:type="Edm.Int32">1</d:ModifiedByID>
        <d:Owshiddenversion m:type="Edm.Int32">3</d:Owshiddenversion>
        <d:Version>3.0</d:Version>
        <d:Path>/mySite/Lists/User Stories</d:Path>
      </m:properties>
    </content>
  </entry>
</feed>

You can append various parameters to filter, sort, or paginate your data from this page. For example http://localhost/site/_vti_bin/ListData.svc/UserStories?$filter=(PriorityValue eq '(2) Normal')&$orderby=Title will return list items whose priority is normal sorted by title. See Query Expressions on MSDN for more details.

Now if you're interested you can take it a step further by viewing the data as JSON. To do this you'll need to modify the HTTP headers. Fortunately there is a tool from Microsoft for viewing, replaying, and modify HTTP sessions called Fiddler. After you install it open Fiddler from the add-in in your web browser (I used IE, but Firefox should work too). Then refresh the web page for a list (in my case ListData.svc/UserStories). Now in Fiddler drag the last request to the Request Builder, ensure "Automatically Authenticate" is checked in options, and replace "Accept: */*" with "Accept application/json". When you click Execute you should see something like:

{
"d" : {
"results": [
{
"__metadata": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)",
"etag": "W/\"3\"",
"type": "Microsoft.SharePoint.DataService.UserStoriesItem"
}, "ContentTypeID": "0x01080070E2807D369BD94FBD6C057D3110E6D3",
"Title": "Blah",
"Predecessors": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Predecessors"
}
}, "Priority": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Priority"
}
}, "PriorityValue": "(2) Normal", "Status": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Status"
}
}, "StatusValue": "Not Started",
"Complete": null,
"AssignedToID": null,
"TaskGroupID": null,
"Description": "<div></div>",
"StartDate": "\/Date(1266624000000)\/",
"DueDate": null,
"Points": 3,
"Iteration": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Iteration"
}
}, "IterationID": 3,
"ID": 2,
"ContentType": "Task",
"Modified": "\/Date(1269011608000)\/",
"Created": "\/Date(1266679946000)\/",
"CreatedByID": 1,
"ModifiedByID": 1,
"Owshiddenversion": 3,
"Version": "3.0",
"Attachments": {
"__deferred": {
"uri": "http://localhost/demoprep/_vti_bin/ListData.svc/UserStories(2)/Attachments"
}
}, "Path": "/demoprep/Lists/User Stories"
}
]
}
}

That's SharePoint data as JavaScript Object Notation. Notice the etag and href's to associated list data. Cool.

But ListData.svc isn't just for reading data. To experiment with it try deleting a list item. To do this in fiddler set the HTTP verb to "DELETE", set the URL to a particular list item's URL (e.g. http://localhost/site/_vti_bin/ListData.svc/UserStories(2) where 2 is the ID for a list item), leave Request Headers empty, and hit Execute. Refresh the list of items and yours is gone. Permanently. The item isn't even in the recycle bin.

Security in WCF Data Services

So at this point you are undoubtedly concerned about security. Just to give some background ListData.svc is provided by a technology called WCF Data Services, which was formerly called ADO.Net Data Services, which was code named Astoria. And WCF Data Services is an implementation of the open data protocol (OData) which Scott Hanselman has an excellent podcast about.

This technology was originally designed for exposing database data. And in this scenario you do have to handle your own security. SharePoint 2010, however, takes things a step further by giving you WCF Data Services (nearly) out of the box while also handling your authorization security.

For instance if you modify the row level permissions for a list item to deny read permissions to a user then ListData.svc will not return that list item for that user. If you try to access the URL for that list item directly SharePoint returns a 404. If you try to modify or delete a list item you don't have permissions to modify then SharePoint returns a 401 Unauthorized.

Very cool. I find this especially nice because you don't have to worry about maintaining your security code. It's intimately tied to SharePoint 2010's security model which is already quite sophisticated.

Incidentally, security is not all that's included when you use these services. SharePoint also performs data validation and returns the appropriate HTTP error codes if you for instance violate the new uniqueness constraint.

Summary

SharePoint offers everything you've seen so far (nearly) out of the box and without a line of code. This sets the stage for writing some nice client side applications with technologies like ASP.Net AJAX 4.0 client site templating. Which is exactly what I intend to do in Part 2 - Creating a read-only page with ASP.Net AJAX 4.0 Templates.

This 13 minute screencast shows you how to create a user stories list in SharePoint and build burn up and burn down charts off of the data using Microsoft Excel. It includes the custom fields you'll need in SharePoint, the formula's you'll have to create in Excel, and shows tips along the way like how to simplify refreshing your reports when data changes.

Once you've watched the screencast if you want to implement this yourself here is the information you'll need:

Custom SharePoint Columns

• Iteration (Choice column e.g. Iteration00, Iteration01, zBacklog)
• Released (Date - the date the iteration was closed out)
• Points (Number)
• Created Date (for demo only, you should be able to use "Created" on your project)

Calculated Columns in Excel

• Total Points to Date
  • The total points on a specific date including backlog, current iteration and all completed stories
  • =SUMIF([Created Date],CONCATENATE("<=",Table_owssvr_1[[#This Row],[Released]]),[Points])
• Completed To Date
  • Total points for stories released prior to current story's release date
  • =SUMIF([Released],CONCATENATE("<=",Table_owssvr_1[[#This Row],[Released]]),[Points])
• Backlog
  • What remains to be completed at a point in time (e.g. don't include user stories added to the backlog in iteration 4 if this is for a task in iteration 03)
  • =Table_owssvr_1[[#This Row],[Total Points to Date]]-Table_owssvr_1[[#This Row],[Completed to Date]]

Burndown Pivot Table Values

• Row: Released
• Filter: Released After [Some Date]
• Values: Max(Backlog)

Burnup Pivot Table Values

• Row: Released
• Filter: Released After
• Values
  • Max(Total Points to Date)
  • Max(Completed To Date)

Summary

I hope you found the presentation useful and if you are currently using SharePoint for tracking user stories please comment on your experiences.

As a consultant (or as a person who is learning someone else's business) your job is to understand your customers existing business better than they do themselves. And you need to be on the same page with others regarding how you are going to change their day to day functions. And that's what business process modeling is for.

You may ignore this document after the first week on the project, in fact I would encourage you to. But even if you throw it away immediately after creating it, the process of developing the document will still:

• Flush out important questions
• Show your customer you understand their world
• Help document the project to other developers
• Facilitate communication (especially with the people who pay the bills)
• Generate user stories (requirements)
• Identify the entities that can feed into an Entity Relationship Diagram (or database I suppose)
• Aid making good choices for the decisions it be hard to undo later (like whether to manually code workflows or use Windows Workflow Foundation)
• More clearly identify pain points and areas where your software can help end users; and
• Identify metrics that can help determine project success from an ROI and product owner's perspective

Convinced this is a tool you need in your toolbelt yet? As long as the answer isn't "I only write code leave me alone" then check out this quick how-to:

Start with the AS-IS diagram

As soon as possible on a new project you should plan to interview your future end users. This shouldn't be an unstructured Q&A session. You should start with a rough draft of what you think their business process is and ask questions that identify the places where you were wrong.

In order to start the diagram you need to identify both the end users and existing systems and put those in "swimlanes" down the left side of a diagram. Swimlanes are important because they help you be explicit about who is performing what activities. Then flush out activities in the happy path in appropriate swimlanes like so:

As soon as you talk to end users (like the clerk in the example above) you will discover unhappy paths. What if the warehouse is out of a product? What if the payment was declined? What happens if a customer needs to return something?

Obviously you document these using standard activity diagram notation:

Identifying Pain Points/Opportunities

The interesting part about unhappy paths is how often they occur, how they affect the customers of your customer, and ultimately how they affect your customer's bottom line. For instance, how often is payment declined in the example above? Fairly frequently? Can you get an actual percentage? If it turns out it's because the original application failed to validate credit cards, then a small amount of programming effort can provide a large benefit to length of time until order fulfillment. Sometimes you can make the process more efficient without writing a line of code.

And at the end of your project if you can show that you've decreased the average length of time it takes to go through the process, ideally by showing less time was spent in unhappy paths, you will likely have made yourself one happy customer. This can be especially effective if your new system can show statistics (like percent of orders that are declined) and compare those to the old process.

Developing the TO-BE diagram

AS-IS process diagrams are completely useless unless you can effectively describe what the process will look like after you implement your new system. The process of developing these documents is pretty similar to the AS-IS diagram except of course you're trying to fix the pain points.

One technique I like for generating a backlog of user stories (tasks) is to put unique numbers with each step in the TO-BE process, then ensure there is at least one user story per step in the process. You may be surprised at how easily activities in your new project's swimlane map to user stories.

Summary

I'm a fan of writing code as soon as possible on a project, but if you fail to understand an end user's business process you may be writing the wrong code. Business process modeling can ensure you're writing the right code to solve the most important problems first. And that may be the difference between project success and project failure.

Dynamic Typing

Let's get to the heart of this Ruby stuff. I thought I would passionately hate dynamic typing. I was getting ready to fill pages ranting about its evils. But that moment never came. Don't get me wrong the Visual Studio strongly typed intellisense no-reading-documentation-needed experience is far nicer than the NetBeans look-it-up-in-irb-with-.methods experience. But the feeling was more like mild to moderate annoyance rather than unbridled hatred. Like when I mistyped a variable and Ruby decided it was a new variable initialized as nil. Annoying, and hard to track down, but hey I'm a Ruby newbie, experts probably don't make these mistakes, right? Anyway, I would be interested to hear how non-trivial, well tested applications perform with refactoring (e.g. variables/fields in views?) and how bad things get during O&M.

Speed of Development

I figured speed of development would be Ruby's saving grace. It was that and more. The Ctrl-S-Alt-Tab-Ctrl-R-zero-delay development cycle is frickin' amazing! It made development feel more fun than it has in a long time. Part of that may be the slower-than-molasses-in-December speed of development I experience today on my (not 2010) SharePoint project. Regardless, even if I were working in an ASP.Net MVC project I suspect the extra speed of development in Rails by skipping the main compilation and JIT compilation steps would increase the enjoyment of a Ruby project enough to equal out the lack of strong typing.

Active Record vs LINQ

Perhaps I'm just not experienced enough with active record, but to me the syntax feels contrived, non-intuitive, and just kind of agitating. I mean it's cool that you can write

p = Person.find_all_by_first_name_and_last_name("Lee", "Richardson")

Ugly as sin, but cool. The alternative doesn't feel much better:

p = Person.all(:conditions =>
    { :first_name => "Lee", :last_name => "Richardson" })

Uch. Of course I am biased. I'm passionate about LINQ. I love LINQ more than any single feature in Microsoft development. I think LINQ may be the most brilliant stroke of genius Microsoft (Anders Hejlsberg) has ever had. And I don't think I realized how passionately I felt about the technology until I didn't have it. So for me I'll stick with:

Person p = ctx.People.Where(p =>
    p.FirstName == "Lee" && p.LastName == "Richardson");

Duck Typing

My mind was blown when the instructor showed us polymorphism without inheritance. I can't argue about how incredibly powerful it is. I just cringe to think of the potential misuse. But without any real Ruby experience I'll have to just leave it at "Wow!"

Mocking

It's pretty scary that you can override the functionality of any method anywhere in Ruby. But when the instructor showed us overriding DateTime.now for the purposes of mocking I had an "ah ha!" moment. Mocking DateTime.Now in C# is an awful experience that involves an intermediate class with a virtual "Now()" method. Ruby sure got C# on that one.

YAML

It's no secret that I passionately hate XML. And it's no secret that Microsoft passionately loves XML. Pity about the mismatch. Ruby on Rails really endeared itself to me when its designers recognized that XML is a travesty against humanity and used YAML Ain't Markup Language instead. Nice!

Interactive Ruby Console

The Interactive Ruby Console (IRB) is just wonderful. Now I can at least get that functionality today in C# with Joe Ferner's excellent Developer's Toolbox. What I can't get is the Interactive Rails Console (ruby script/console). Now that is awesome. I can't wait for C# 4.0 which I suspect will have this. For now RoR++, C#-- .

Tooling

I'm not sure how related this is to dynamic typing, but surprisingly I really, really missed Visual Studio. Perhaps RubyMine is better that NetBeans. I sure wasn't impressed with Komodo. I just felt like constantly switch between my IDE and various console windows, and a database viewer felt clumsy yet somehow necessary with Rails. It just felt so 1999.

Summary

There's no doubt about it, I could be very happy on a Ruby project. The no XML thing is just a perk, the fast development cycles and bringing fun back to coding again is truly awesome. But am I ready to give up Microsoft development full time just yet? No way. LINQ, strong typing, and believe it or not Visual Studio tip the scales back to about equal. Now if only Microsoft could somehow make compilation instantaneous.

Testability

SharePoint is notoriously hard to unit test. The good news is Microsoft provided a whole session devoted to software development best practices that included a good chunk on unit testing. It was wonderful to see!

The bad news: SharePoint 2010 will not have public constructors and classes will still be sealed (final). The good news to the bad news: After discussing this issue with Chris Keyser I discovered that Peli de Halleux of Microsoft Research has a free tool called Pex that solves the whole problem. It allows what it calls detours, which allow you to mock non-virtual methods. This stuff is amazing and I'll blog more about it later.

Summary: Testability is better even for existing SharePoint installs -- so between Sporm and Pex this is a non-issue -- I'm happy.

No Referential Integrity

It kills me that if you create an employee list item that references a company list item and you delete the company list item then nothing happens. No cascading delete. No error message. Just an orphan record. SharePoint 2010 now supports both true referential integrity through errors and cascading deletes!

Summary: Awesome!

Validation is Terrible

Ever tried to validate by pattern in SharePoint (e.g. social security number)? How about range validate (e.g. date of birth can't be after today)? Or maybe validate one field against another? The answer is you can't. Not easily or consistently. If you handle validation in an event receiver then the user doesn't get the error until after a postback on a separate page. And custom field controls are a lot of work and they don't work in datasheet view.

SharePoint 2010 mostly solves these validation issues by allowing field level and list level validation. The error messages even show in datasheet view. The bad news is that pattern matching is not currently supported (e.g. you can't validate social security number).

Summary: Good stuff, need to see the final version to be extremely happy.

Poor Documentation

Ever noticed that blogs give you better details on SharePoint APIs than MSDN does? I don't even bother looking on MSDN for SharePoint content anymore. Since the hands on labs didn't contain any documentation I'll have to refer to what Steve Ballmer said during the Q&A session, which was that better documentation will be a priority.

Summary: high hopes, but TBD.

Views Don't Allow Precedence

If you've ever needed a view with A and (B or C) and ended up with (A and B) or C, you know where I'm coming from. As a developer you can implement precedence in CAML, but as an end user you're stuck. Sadly this issue is still not fixed in SharePoint 2010.

Summary: Fail.

Re-Deployment is Hard

Currently redeploying lists is hard; redeploying web parts without overwriting user settings is hard; and redeploying workflows is very hard.

Without documentation and intermittent Internet Joe Ferner and I had a hard time manually confirming that this is better. We also missed a key session on Thursday that talked about it. There were some tweets that looked promising, but I'll have to wait until Thursday's presentations are posted to be able to comment for sure.

Summary: Good I think, but still TBD.

Massive Duplication in CAML Instantiated Lists

Using CAML to instantiate a list is currently awful in SharePoint. It's like 5 pages of CDATA and HTML and XML with massive duplication. Sadly SharePoint 2010 doesn't solve the root problem: list instantiation looks exactly the same. The good news is that along with just about every other aspect of SharePoint 2010 development, Visual Studio makes this task mind numbingly easy to do. If you can just try not to look in that file that was auto-generated for you, then you should be fine.

Summary: Better.

Poor Usability

Once you get used to the current version of SharePoint the user interface is extremely consistent and many users really like it. But if you were to compare it to any modern Web 2.0 site it's a complete failure. SharePoint 2010 does an excellent job of implementing selective refresh/AJAX/Web 2.0. But to me it feels more complicated. Maybe this is because it introduces the ribbon, which I have never liked. Maybe it just is more complicated. In any event I'm not the best judge of usability, so I'll have to wait and see what others say.

Summary: Definitely better, but jury's still out.

Can't Reference a List AND Content Type

Suppose you have a Calendar list. The list supports two content types: Events, and Iterations (sprints). It's a nice architecture because you want to view iterations and events in the same calendar views. Now if you have a User Story list, wouldn't it be nice to have a lookup field that points only to Iterations? Sadly you couldn't do this before and you won't be able to for the foreseeable future.

Summary: Fail.

CAML for Queries

Technically this wasn't on my list because I use Sporm, but on behalf of my non-spormified software development brethren let me say that writing queries in CAML is awful.

SharePoint 2010 makes enormous strides in this area. I won't go into great detail, but you can now use LINQ on the server side and on the client side!

Summary: Awesome!

Summary

Lots of TBD, lots of awesomeness, still some fail. Worth upgrading? Absolutely. Will it still be a love hate relationship? Probably, but at the moment it's looking pretty darn good.

Caml's Are Ugly

Warning: if you're new to SharePoint then what you're about to see may shock and upset you. If, like me, you hate both XML and loose typing then you will agree that CAML is awful, but bear with me I promise sporm will make it better. Much better.

CAML or Collaborative Application Markup Language is how one queries for data in SharePoint. A simple query might look like this:

<Query>
  <Where>
    <And>
      <And>
        <Eq>
          <FieldRef Name='First_Name' />
          <Value Type='Text'>Lee</Value>
        </Eq>
        <BeginsWith>
          <FieldRef Name='Last_Name' />
          <Value Type='Text'>Rich</Value>
        </BeginsWith>
      </And>
      <Leq>
        <FieldRef Name='Dob' />
        <Value Type='DateTime'>2009-01-01T00:00:00Z</Value>
      </Leq>
    </And>
  </Where>
</Query>

Simple right? ;) In case you didn't catch the meaning from the slightly, uh verbose, query this asks for records with a First_Name of "Lee", a Last_Name starting with "Rich" and a "Dob" less than or equal January 1 2009.

There are a couple of things to note about this query:

• Field names are loosly typed. If Dob were ever renamed to DateOfBirth the query would fail at runtime (probably during a demo to a customer, or if you're lucky during integration tests), but certainly not at compile time.
• And is a binary operator. This forces explicit precedence and removes the need for parenthesis, but at the cost of readability.
• Types must be explicitly defined. I guess this is necessary since it's XML, but somehow I just don't feel like this should be necessary.
• It's XML. Ok obviously, but the point is you have to type everything twice. <BeginsWith> </BeginsWith>. Ouch, so verbose, so angley, I so hate XML.

Now, there are tools that make this better. U2U's free CAML Query Builder tool significantly improves the experience of querying SharePoint data.

But it makes you wonder if something is wrong when you need a tool to retrieve data from your data store. Do you typically use tools to assist when you're writing SQL? Probably not. But like I said hang in there, sporm make things better.

Unlinq my Caml

If you were to write the same query as above using sporm it would look like this:

IQueryable<Employee> employees = GetEmployees().Where(e =>
      e.FirstName == "Lee"
      && e.LastName.StartsWith("Rich")
      && e.Dob <= new DateTime(2009, 1, 1));

Just a little easier to read than CAML, right? A couple of things to note:

• Fields are strongly typed. If you were to rename FirstName the compiler would catch every single instance at design time.
• Operators are standard C# operators. &&, .StartsWith(), and <= are all familiar and concise and use a standard, known precedence.
• Types are standard C# types. You never have to explicitly say that "Rich" is a string, it just is.
• Your query is actual C# code. The lambda (closure) and the fact that it uses deferred execution might throw off a junior developer in some scenarios, but the query is readable and works exactly the same as if you were querying in memory objects or querying a database with LINQ to SQL or the Entity Framework.

Nice! What's beautiful about this is that sporm converts your C# directly into CAML using C# 3.0's expression trees feature. What sporm can't convert to CAML it executes in memory transparently to you. Sporm uses log4net and outputs its CAML queries to the console by default, so it is a good idea to watch the output if you're concerned about performance.

Now the following isn't relevant to the comparison with CAML, but I would be negligent if I didn't explain how the GetEmployees() function works.

SP != DB B/C of Content Types

First of all GetEmployees looks like this:

private IQueryable<Employee> GetEmployees() {
      return MySPDataContext
            .GetCurrent()
            .GetList<Employees>()
            .OfType<Employee>();
}

How it works is that the static GetCurrent() method retrieves sporm's context object, which knows how to query a SharePoint site, from either the web context or thread local storage; next the GetList() method tells sporm which list you want to query; and the OfType() method tells sporm which content type within the list you want to query. This last part is important because sporm supports SharePoint's ability to have multiple content types per list, which other LINQ providers like LINQ to SharePoint do not. But what are content types and why should you care?

SharePoint's List/Content Type architecture seems odd at first, but it allows interesting scenarios not available in a traditional database. For instance you might have a calendar list that contains multiple types of records (list items) in the same list. Your calendar list might contain some combination of the following three record types: meetings with unique fields like Organizer (a person); iterations with unique fields like DeployedToProduction (a Boolean); and actions with unique fields like RelatedEmployee (a reference to another list). This architecture allows SharePoint to view all three types of records in a single view: like a per month calendar view. The data might look like this:

Sporm's unique architecture supports this scenario by allowing you to retrieve iterations like this:

return MySPDataContext
      .GetCurrent()
      .GetList<Calendar>()
      .OfType<Iteration>();

Or get meetings from the same list like this:

return MySPDataContext
      .GetCurrent()
      .GetList<Calendar>()
      .OfType<Meeting>();

While you may only use multiple content types per list occasionally you can feel comfortable knowing that sporm will support you when you need it. The rest of the time you can arrange your architecture to accommodate your 90% scenario of one content type per list. I'll discuss the architecture I'm using on my current project in my next post.

For now I hope this has clarified some of the benefits of using sporm, and I hope that you'll consider using it on your next SharePoint project.

I'll start by describing the unit testing problem, then provide a conventional solution with mocking, then finish with the elegance of a sporm based solution.

Simple Unit Testing Turns Ugly

I would bet anyone that attempts to develop a tiered solution with SharePoint ends up with entities that look something like this:

public class Employee {
      public SPListItem ListItem { get; private set; }

      /// <summary>
      /// The constructor requires that you pass in a list item
      /// that is used by your strongly typed properties
      /// </summary>
      public Employee(SPListItem listItem) {
            ListItem = listItem;
      }

      /// <summary>
      /// This provides your entities strong typing to hide
      /// the native weak typing in SharePoint
      /// </summary>
      public virtual string LastName {
            get { return (string)ListItem["LastName"]; }
            set { ListItem["LastName"] = value; }
      }
}

And if you have some function that you'd like to unit test like this:

/// <summary>
/// Typically returns "[LastName], [FirstName]" but skips
/// the comma if either is missing
/// </summary>
public string GetNameFormatted() {
      bool noFirst = string.IsNullOrEmpty(FirstName);
      bool noLast = string.IsNullOrEmpty(LastName);
      if (noFirst && noLast) return "";
      if (noFirst) return LastName;
      if (noLast) return FirstName;
      return string.Format("{0}, {1}", LastName, FirstName);
}

Now you face a major problem: LastName is tightly coupled with SPListItem. And SPListItem only contains internal constructors making it unmockable. So if you keep the tight coupling and pass an SPListItem into the constructor then you need a connection to SharePoint and an employee list (table) and you need to create an employee list item (record), then you test it, and finally you ought to clean up after yourself and delete the list item. Talk about a simple unit test turned fragile integration test. Yuck.

Mocking the Solution

So at this point you're undoubtedly thinking it's time to mock. To do this you'll have to make a public constructor that takes no arguments and mark your properties virtual. Then you can write a test like this:

[TestMethod]
public void TestGetNameFormatted_FirstAndLastExist_LastCommaFirst() {
      MockRepository mocks = new MockRepository();
      Employee mockEmployee = mocks.StrictMock<Employee>();
      Expect.Call(mockEmployee.FirstName).Return("Lee").Repeat.Any();
      Expect.Call(mockEmployee.LastName).Return("Richardson").Repeat.Any();
      mocks.ReplayAll();
      string actual = mockEmployee.GetNameFormatted();
      mocks.VerifyAll();
      Assert.AreEqual("Richardson, Lee", actual);
}

And then if you're like me you absolutely hate your unit tests. Because let alone that you had to modify your production code to accommodate unit testing, more importantly your unit tests are now unreadable. All of them. Even in simple methods without dependencies your unit tests contain almost as much plumbing code as actual test code. Enter sporm.

Unmocking with Sporm

Sporm is an open source tool released on codeplex that was developed by fellow Near Infinity employee Joe Ferner (who, incidentally, is one of the most brilliant developers I know). While this is a version 1.0 product Joe has been using it on his project and independently I've been using it on my project for several months now, so it is relatively mature.

So how it works is that at design time sporm will read from your SharePoint site and generate partial classes for each of your list items and each of your content types. The generated classes don't inherit from anything, giving you the ability architect your solution how you see fit. And the properties are auto-properties, making the classes just pure POCO. Here's an example if you're interested:

[SPOrmContentType(Name="Employee")]
public partial class Employee : ISPOrmContentType {
      public static class FieldTitles {
            public const string FirstName = "First Name";
      }

      public static class FieldStaticName {
            public const string FirstName = "FirstName";
      }

      [SPOrmIdField()]
      public virtual int Id { get; set; }

      [SPOrmField(Title = FieldTitles.FirstName, StaticName = FieldStaticName.FirstName)]
      public virtual string FirstName { get; set; }
}

Since generated entities look like that, your entities can now look like this:

public partial class Employee {
      public string GetNameFormatted() {
            bool noFirst = string.IsNullOrEmpty(FirstName);
            bool noLast = string.IsNullOrEmpty(LastName);
            ...
      }
}

Now if that isn't a picture of beauty to you then you haven't been working with SharePoint. But wait, I hear you asking, how does this have anything to do with SharePoint? How would an instance of an employee be able to retrieve the data in a SharePoint ListItem? The answer is in the DataContext. In your production code you write something like this:

public static Employee FindById(int id) {
      return MySPDataContext
            .GetCurrent()
            .GetList<TList>()
            .OfType<TContentType>()
            .FirstOrDefault(c => c.Id == id);
}

While it might not look so pretty at first you can hide these details in a base type and the nice thing is that it does handle SharePoint's ability to have multiple content types per list (which LINQ to SharePoint does not do, incidentally). But for now all you need to know is that when you return an Employee thorugh MySPDataContext it subtypes your Employee class and returns you a proxy at runtime. The proxy overrides each property to return you the appropriate value in the SPListItem. And if you instantiate an Employee directly (as you would in unit testing) then you can use it like a pure POCO. If you're familiar with the entity framework this should sound extremely similar. The result is that you can write your unit tests like this:

[TestMethod]
public void TestGetNameFormatted_FirstAndLastExist_CommaSeparate() {
      Employee employee = new Employee {
            FirstName = "Lee",
            LastName = "Richardson"
      };
      string actual = employee.GetNameFormatted();
      Assert.AreEqual("Richardson, Lee", actual);
}

No SharePoint dependencies. No mocking. Nothing but pure, unadulterated unit test. Now that a thing of beauty.

The shadowy world of SharePoint is one in which source control is a distant memory, where no distinction is drawn between development, test and production environments, and where roaming bandits take over small towns with no fear of the law.

Why is this? How has it come to be? The answer is that for the most part SharePoint treats code and data as one in the same. End users can modify data stores: they can add, delete and modify tables (Lists) and columns (Fields) at will. They can open up SharePoint Designer and modify pages, customize workflows, rob banks, and run local Sherriff's out of town.

And that's why end users love SharePoint. It removes the slow, burdensome bureaucracy involved in doling out development resources from large centrally planned corporate or government IT departments and places that power directly in the hands of end users, allowing them to get their jobs done faster and to adapt to change quicker. And that is why I maintain that as software developers SharePoint, or a product like it, is in our future -- like it or not.

But the power given to end users makes rigor and good design extremely hard for developers because the production machine you deployed to last month may look completely different now that you're ready to redeploy. And Microsoft makes good design even harder with an inflexible API. Critical classes like SPSite (think SqlConnection) contain no public constructor, rendering them completely unmockable (unless you're willing to spend $450 per developer for some TypeMock Isolator magic). And vital classes like SPList (think DataSet) are marked final, crippling your ability to make nice strongly typed entities in your architectures.

So what is a Type-A, quality minded software developer to do?

The thing to keep in mind is that the challenges that SharePoint places on developers and architects simultaneously make good coding practices harder (but not impossible) while making them more important than ever.

Because that column you thoughtfully added as required may get reset to optional by an end user in production, you need to write more good tests that validate once solid assumptions. Because you can't inherit SPList, you'll need to encapsulate it, requiring more code per entity. Because you can't mock essential SharePoint classes you will need to write more integration tests instead of unit tests. Because you're writing integration tests your tests will be more fragile and you'll spend more time fixing the test code than the real code.

And because you're spending more time focusing on quality while your Wild West, short term focused competition in the SharePoint world slings code together, they will appear faster and better able to get applications out the door. It can be frustrating, because over time your well designed and well tested application will be more maintainable and will cost IT departments less. But end users frequently don't understand this, so as long term minded SharePoint developers we must focus on educating end users.

Combat the short term focused mindset by keeping everything in .wsp files, storing code in source control, avoiding the temptation to apply fixes in production, using continuous integration (yes, for SharePoint!), and most importantly tracking code coverage (yes, code coverage for SharePoint code!). Put quality metrics in status reports for management to see. Track defects and make your stats publicly available (since you'll have significantly fewer over time than code slingers, and you want end users asking for stats). In short remind the world that quality counts, even in the shadowy realms of SharePoint. If we work together we can bring law and order to the ... um, West.