RemoteLINQ - How to make your LINQ span the globe

Code You Write To Make It Work

   1:  IEnumerable<int> numbers = Enumerable.Range(0, 500);
   2:  RemoteContext remoteContext = new RemoteContext(
   3:     new[] { this.GetType().Assembly }, 
   4:     new[] { "localhost", "someremotehost" },
   5:     7776);
   6:  RemoteOptions remoteOptions = new RemoteOptions();
   7:  IEnumerable<int> results = numbers.AsRemotable(remoteContext, remoteOptions).Select(i => DoSomeWork(i));

Let's walk through the code.

• Line 1, gives us an enumerable of numbers from 0 to 500 for processing.
• Line 2, creates the RemoteContext. This is really just a client which has a list of server connections to which we can send work.
• Line 3, is a list of assemblies that contain the necessary code to run your LINQ statement. The client will actually send a copy of the dll to the RemoteLINQ server so that the server will have a copy of the "DoSomeWork" (line 7) method so that it can execute it.
• Line 4, is the list of servers to divvy the work across.
• Line 5, is just the port that the client will try to connect to the server on.
• Line 6, contains any options to direct how the work gets split across the servers, etc.
• Line 7, is your LINQ statement. Notice the "AsRemotable". This is where all the magic starts. This will actually return a IRemoteEnumerable which knows how to take anything to the right of it and run it remotely.

How it works

Lets start out with what lambda expression look like in compiled code. When we compile lambda expressions they are moved into a new method with a compiler generated name. If you reflect on a class with lambdas and get all the methods on that type you will see a bunch of method names which you didn't create. Those are your lambdas. The fact that they are first class methods allows us from the outside, with reflection of course, to call those methods. So when we call "Select" on an IRemoteEnumerable, RemoteLINQ is taking the lambda expression passed in, and storing information about it into a serializable object (assembly, declaring type, method name, etc.). This will allow us to send that information to the server, who can then call the method without all the other stuff around it.

On the server side of things, when we start up a RemoteLINQ server, it will create a worker thread for each processor in the machine. The worker thread will wait quietly until new work arrives on the socket connection. When work arrives we take the serialized object I mentioned before and find the assembly, type, and method and execute that method with the work given to us. In this case the work will be one of the numbers from 0 to 500. The client will handle load balancing the servers by choosing the server with the least amount of pending work per processor. The amount of work queued up on the server is sent back after each unit of work is completed so that the client will have the most up to date picture of what the servers are doing to make the best guess as to where to send the next piece of work.

That's about it. You now have a way to split complex LINQ tasks across multiple machine with just a single call on your LINQ statement.

Download The Code