June 2008 - Posts

At the beginning of this month, we released the first CTP of Velocity, an early preview of our distributed object cache solution. You can download it here. Notice it's a very early preview so things will definitely change moving forward. This post introduces how to install and use Velocity.

 

Introduction

But first... what's in a name? Multi-tiered distributed applications are common-sense nowadays and with cloud computing within reach the need to build scalable distributed services has never been bigger. One of the core aspects in enabling those scenarios is to have intelligent caching of objects, not only to reduce the number of accesses to the underlying data source but also to boost availability by employing scale out techniques. Obviously, developers want to be able to do all of this without having to worry about the complexities that this brings, having to deal with load balancing and availability themselves. That's where Velocity comes into play.

The core idea is very straightforward: we have a cache that behind the scenes is distributed and replicated across a bunch of machines called the cluster. Storing data in the distributed cache is as easy as calling some Add or Put method, and retrieving it is as easy as calling Get. With some creative stealing from the documentation we end up with the following picture:

image

An important thing to emphasize is the fact the cache clients deal with regular .NET objects all the time and don't have to worry about storing those objects. Indeed, .NET serialization takes care of the rest. There are more concepts to it such as cache eviction policies (when objects are removed from the cache, such as least-recently used or LRU), the distribution mechanism where simple clients just contact the cluster "in the cloud" through any cache host and get redirected to whatever host the object is available on versus routing clients that have awareness of object placement through a routing table. Other important pieces include the supported concurrency models and associated locking mechanisms but let's not go there in this introductory post.

 

Installation

Installing Velocity is fairly straightforward. Just run the MSI. After a while you'll see the following window:

image

This is where you configure the "cache host". Under Cluster Configuration Share you can enter the UNC path to the cluster's configuration share. This is the place where an XML file is kept that makes sure that configuration settings are consistent across all hosts in the cluster. Create this folder and grant the Everyone account full access to it; this is known issue in the CTP which by no means will become the final design. Also notice that currently in the first CTP this is a single point of failure: if the share goes down, the cluster can die. Obviously this will be addressed in subsequent releases. For now, we'll just specify a local path and enter a new name under "Cluster Name". The Cluster Size is self-explanatory and for the purpose of this introductory sample, we'll stick with a one-host cluster (a degenerate case of a cluster if you will...).

Next, two ports are being specified: the service port and the cluster port. The defaults are just fine here. Basically the service port number is what clients connect to in order to talk to the cache host. The cluster port on the other hand is used by the servers in the cluster to talk to one another (there's another port, called the arbitration port which is listening on cluster+1, i.e. 22235 in the sample above).

Last but not least, there's the Max Server Memory setting which defaults to half of the available physical memory. I've reduced it to 256 MB since I'm using my main dev machine as my playground but obviously in production scenarios boxes will get dedicated to the distributed cache cluster, where it makes sense to boost this.

After clicking Save & Close, setup will ask you to open the required ports on the firewall, which can be done easily by allowing the DistributedCache.exe program (that runs as a service) through the firewall.

image 

Doing so isn't hard at all. Geeks can go the netsh way as illustrated below. Alternatively one can use the Firewall Settings in the Control Panel (click to enlarge):

image image

With this, setup has completed.

 

Configuration

Before we can start to use the service, we need to make a few changes to its configuration. Let's take a look at the configuration share's file structure first:

image

The XML file contains the configuration that's shared by all hosts in the cluster. In addition the ConfigStore file is a SQL CE 3.5 database that keeps additional information about partitions, nodes, regions, etc which you can find more information about in the CTP's documentation. Notice that you won't find the cached data here since we're talking about an in memory distributed cache. Geeks can investigate what goes inside this little database, but we'll instead just focus on the XML file:

image

In here you can find the list of hosts and caches that are part of the cluster. We'll take a look again at this file in just a minute when we've altered the cluster configuration. In order to configure the service, Velocity comes with a command-line driven tool named originally the "Administration Tool" which you can find through the start menu (click to enlarge).

image

In the configuration steps below, we're adding a cache to the cluster after investigating the hosts and caches that are part of the cluster. Once we've added the cache, the cluster is started which puts all the hosts online by starting their Windows Services.

image

The service is called DistributedCacheService which is kept in the cacheHostName property in the XML file:

image

Taking a look back the XML file, you'll notice a new cache configuration entry has been added:

image

The new cache has an eviction type of Least Recently Used (LRU) which means that the - as the name implies - least recently used objects get evicted from the cache when necessary. In addition, objects expire from the cache after a time-to-live of 10 minutes. The type of the cache is set to partitioned allowing objects to be distributed across the hosts in the cluster.

 

Using it

After covering the server side, we should take a look at the client side picture of using Velocity. Here's the simple program we want to use:

namespace VelocityDemo
{
    class Program
    {
        static void Main(string[] args)
        {
            CacheFactory factory = new CacheFactory();
            Cache cache = factory.GetCache("Test");

            cache.Put("Name", "Bart De Smet");

            string name = (string)cache.Get("Name");
            Console.WriteLine(name);
        }
    }
}

Notice how we get access to the cache called Test through a factory object, after which we simply use a Put method to add an item (key, value pair) to the cache which subsequently can be retrieved (on any client connected to the cache) using the corresponding Get method. Obviously more complex serializable objects will be stored in the cache but this simply shows the main idea.

In order to make this work we need to add a couple of references to the project. Notice that in the CTP the number of client-side assemblies that need to be references on the client isn't optimized in any way, so here it goes (click to enlarge):

image

Core ones are CacheBaseLibrary and ClientLibrary. All of these carry the System.Data.Caching namespace:

image

But wait a minute, how can the client know where to find the cache retrieved through the factory? Indeed, it can't:

image

In order to make things work, the exception hints you to create a client configuration file. The documentation that comes with the CTP contains such a configuration file, so I won't paste it here but the key takeaway here is that mirrored nature of client-side and server-side files:

image

Basically, the client just needs to point at a host in the "distributed caching cloud" to gain access to the cache (depending on the type of client, things look a little different - in the sample above the deployment type is for a simple client - more information can be found in the documentation).

Once you've run the program you can take a look at the cluster's cache by means of the show cachestats command in the administration tool:

image

To see the cache-behavior in action, try caching the current time (DateTime.Now) and get it back. Here's a piece of sample code illustrating this (using indexers instead of Get/Put calls):

while (true)
{
    object data = DateTime.Now;
    Console.WriteLine("Add to cache: {0}", data);
    cache["Time"] = data;

    data = cache["Time"];
    while (data != null)
    {
        Console.WriteLine("Retrieved from cache: {0}", data);
        Thread.Sleep(10000);
        data = cache["Time"];
    }

    Console.WriteLine();
}

For sake of this demo, I've lowered the TTL to 1 minute (stop the cluster using stop cluster, change the TTL value in the config file and restart the cluster using start cluster). Here's a sample output:

image

You can clearly see how the data becomes null when the object has been evicted from the cache - this is an important thing: an application should never assume a cache hit will occur and be prepared to handle a cache miss. If such a miss occurs, the data in the cache can be refreshed if the client feels the need for it. Notice there's a method called ResetObjectTimeout that can be used to reset the TTL counter for an object specified through its key value.

 

More stuff

There's more to Velocity than just the stuff covered in the post. For example, there's the concept of regions that allows objects to be located on a specific node, offering additional search capabilities to find cached objects at the price of scalability across hosts. To handle concurrency when dealing with cached objects, Velocity supports optimistic and pessimistic locking semantics. And finally for ASP.NET applications, there's integration with session state. Obviously there's much more to come in upcoming CTPs but there's more than enough to explore already in this CTP.

Enjoy playing with Velocity (but as usual, keep in mind the CTP quality)!

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

In the last couple of posts in the series, we've been establishing the interop baseline ready to be used by the query provider layer on top, which we're about to build now. Since the goal of this blog series is to live an IQueryable-free live, for reasons explained earlier including demo-simplification goals, this post will introduce a series of "fluent classes" that provide a limited query pattern.

 

The query provider

At the entry-point for our API, we provide a query provider class. Its sole role (at least for now) is to provide access to the underlying MSI database through an MsiConnection object introduced previously. If one wants to support update tracking on the objects retrieved through the provider, the query provider context is the best place to store this tracking information since it can keep a global view on the different tables and their underlying relationships. Since MSI has some notion of relationships between tables, such an implementation could be a good idea but obviously one can avoid all sorts of update tracking whatsoever by directly updating records in the database; after all we don't have a huge price to pay for the "connection" since the MSI database is a local file (i.e. no network traffic cost etc). However, if the end-user expects transactional semantics when updating multiple records as a a way to ensure database consistency, the global view of a context object is a plus as well (although one has to put the provided consistency mechanisms of the underlying database in the equation, see MSDN on the Windows Installer Database to get an idea about the capabilities of the MSI database engine).

To summarize this discussion, we'll provide a query provider object as the entry-point, from which we'll derive to provide access to the various tables. Users of the API can derive from the context themselves to provide access to other custom tables as well. Here's how it looks:

public abstract class MsiQueryProvider : IDisposable
{
     private MsiConnection _conn;

     protected MsiQueryProvider(string file)
     {
          _conn = new MsiConnection(file);
     }

     public void Dispose()
     {
          if (_conn != null)
          {
               _conn.Dispose();
               _conn = null;
          }
     }
}

 

Representing tables

Next we have to talk about tables. MSI tables are fairly simple; they just have a name and there's not much more interesting metadata (apart from the fields they hold of course) associated to it. One caveat though is the fact the names are singular (like Property) so it might be tempting to introduce some metadata through custom properties to create a mapping. That would work, but to simplify manners we'll just have the name of the entity class stand for the table's name. Where does the entity class come into play? A generic parameter does the trick:

public class MsiTable<T> : IEnumerable<T>
{
    ...
}

Given the following entity class:

public class Property
{
     [MsiColumn("Property")]
     public string Name { get; set; }

     public string Value { get; set; }
}

we can construct a database context for our MSI database as follows:

public class MyMsi : MsiQueryProvider
{
     public MyMsi(string file) : base(file)
     {
          Properties = new MsiTable<Property>(this);
     }

     public MsiTable<Property> Properties { get; private set; }
}

which can be used like:

var ctx = new MyMsi("c:\\temp\\demo.msi");
var res = from p in ctx.Properties where p.Name == "ProductCode" select p.Value;

Notice the table-object implements the IEnumerable interface which obviously simply queries the entire table. Since it queries the table, internally it can use the same querying infrastructure as derived queries (which we'll talk about next).

 

The built-in query pattern

C# and VB provide query comprehensions built-in into the language, which translate in chains of method calls. I've been showing this a lot, but one more time for the query above:

var res = from p in ctx.Properties where p.Name == "ProductCode" select p.Value;

becomes

var res = ctx.Properties.Where(p => p.Name == "ProductCode").Select(p => p.Value);

But one could well write other queries using orderby (translates into OrderBy* and ThenBy* method calls), or leaving out certain parts (e.g. no where clause). Since we don't want to support all query operators (dropping the I in IQueryable, just keeping someting queryable) we'll stick with a fluent pattern that allows enough expressiveness and plays nicely together with the built-in comprehensions that are used typically (where, select, orderby). Depending how flexible you want to make a query provider, you can provide more or less of those operators with the ultimate end of the spectrum being a full IQueryable<T> (trading compile-time errors for runtime exceptions in case of not-supported operators).

What we're going to build in this blog series is in fact a little state machine that accumulates query information (you could therefore - not surprisingly - call it a query accumulator) that supports those three basic operators (restriction using where, ordering using orderby and projection using select) in different orders:

image

Each of those classes inherits from a common query base class that looks like this:

abstract class MsiQueryBase<T> : IEnumerable<T>
{
     protected QueryData _query;

     protected MsiQueryBase(QueryData query)
     {
          _query = query;
     }

     IEnumerator IEnumerable.GetEnumerator()
     {
          return GetEnumerator();
     }

     public IEnumerator<T> GetEnumerator()
     {
          return _query.Execute<T>();
     }
}

The accumulative nature lies in the QueryData class we'll outline below. Each time the query gets extended with an additional piece of information, a new QueryData object is created. The reason for this is to keep intermediate query objects available so that multiple views can be constructed. This is more of a general design choice than one that's crucial for the LINQ to MSI provider (since our provider is so limited this "viewing" mechanism would be less useful anyway). With views I mean things like:

var expensive = from p in products where p.UnitPrice > 10 select p;
var notTooExpensive = from p in expensive where p.UnitPrice < 100 select p;

The hidden piece of crucial information here is the fact the chain of method invocations silently continues, conceptually like:

var notTooExpensive = products.Where(p => p.UnitPrice > 10).Select(p => p).Where(p.UnitPrice < 100).Select(p => p);

If the second query were to modify the query data encapsulated by the first query, iterating the first query would yield the results of the second one. Or in other words, how immutability is a must in this case. Notice that IQueryable<T> doesn't have this problem because of the way expression trees are generated (or better: composed). With IQueryable, the first query would be an expression tree which the second query simply refers to. We could either mimic this behavior by composing all pieces of query information into an expression tree, or make our internal query data representation immutable. We'll stick with the last approach as shown further.

Also note that every select operation results in what's called a closed query. Why closed? Because the projection (possibly) looses all track of the original entity information, through anonymous types used in the projection. It's sort of a semi-permeable membrane which can cause everything that follows to be "lost in translation". An example:

var res = (from p in products select new { Name = p.ProductName, Price = p.UnitPrice }).Where(p => p.Price > 100);

If the target query language allows nesting of queries, there's no problem. For example in SQL one can write:

SELECT Price FROM (SELECT ProductName AS Name, UnitPrice AS Price FROM Products) WHERE Price > 100

but lots of other query languages don't allow such flexibility (and indeed, if we were to try this in MSI's SQL implementation, it wouldn't work). In order to reflect this restriction, we make the close the query once a projection has been carried out. If grouping were supported, that's typically subject to analogous restrictions of course again depending on the underlying data store's flexibility.

 

Query data

The query data object we're going to use to represent the pieces of a query encountered during the chaining of query operators is shown below. It simply consists of a predicate (from Where), a set of orderings (from OrderBy* and ThenBy* calls) and a projection (from Select) as well as the original entity type and a reference to the query provider in order to gain access to the underlying data store. Alternatively, one could mimic the Queryable behavior by constructing an expression tree on the fly using Expression.Call, specifying the current expression tree as the left-hand side (the "this" operand if you will) and the passed-in parameters (like the predicate parameter for Where) as the call operands. I'll come back to this later in this series; for now, we'll construct a simple object like this:

internal class QueryData : ICloneable
{
     public MsiQueryProvider Provider;
     public Expression Where;
     public List<OrderClause> Order;
     public Expression Select;
     public Type EntityType;

     public QueryData()
     {
          Order = new List<OrderClause>();
     }

     public object Clone()
     {
          return new QueryData() { Provider = Provider, EntityType = EntityType, Where = Where, Order = new List<OrderClause>(Order), Select = Select };
     }

     public IEnumerator<T> Execute<T>()
     {
          yield break;
     }
}

Remember C#'s public is only as public as the container, so don't blame me for public fields (or feel free to add automatic property syntax on my behalf). Notice the internal object isn't immutable by itself, something you can debate about of course (or feel free to modify the constructor and add automatic property syntax with private setters on my behalf - just a matter of distributing work between the writer and the readers :-)), but we keep control over it ourselves (the "circle of trust"). Also notice that Expression objects are immutable by their design. Obviously, Execute will be where the real work is to be done, but we keep that for a future post.

 

Back to the Table<T>

Now that we have the query object and the idea of the fluent pattern, we can return to the MsiTable<T> object:

public class MsiTable<T> : IEnumerable<T>
{
     private QueryData _query;

     public MsiTable(MsiQueryProvider provider)
     {
          _query = new QueryData() { Provider = provider, EntityType = typeof(T) };
     }

     IEnumerator IEnumerable.GetEnumerator()
     {
          return GetEnumerator();
     }

     public IEnumerator<T> GetEnumerator()
     {
          return _query.Execute<T>();
     }

     public MsiQuery<T> Where(Expression<Func<T,bool>> predicate)
     {
          QueryData query = (QueryData)_query.Clone();
          query.Where = predicate;

          return new MsiQuery<T>(query);
     }

     public MsiOrderedQuery<T> OrderBy<K>(Expression<Func<T,K>> orderClause)
     {
          QueryData query = (QueryData)_query.Clone();
          query.Order.Add(new OrderClause { Mapper = orderClause, Descending = false });

          return new MsiOrderedQuery<T>(query);
     }

     public MsiOrderedQuery<T> OrderByDescending<K>(Expression<Func<T,K>> orderClause)
     {
          QueryData query = (QueryData)_query.Clone();
          query.Order.Add(new OrderClause { Mapper = orderClause, Descending = true });

          return new MsiOrderedQuery<T>(query);
     }

     public MsiClosedQuery<T, R> Select<R>(Expression<Func<T,R>> projection)
     {
          QueryData query = (QueryData)_query.Clone();
          query.Select = projection;

          return new MsiClosedQuery<T, R>(query);
     }
}

The constructor is straightforward, and so is the implementation of IEnumerable<T>, simply asking the query data to execute the query. Actually this code is in common with the MsiQueryBase<T> class, so you could use it as the base class here if you want. However, and this is again philosophical, a table by itself is not a query - it just so happens to be that internally we fetch results using the same query infrastructure upon enumeration of the table, which votes against using MsiQueryBase<T> as the base class here.

More interesting are the Where, OrderBy* and Select methods. The most important thing here is the signature:

public MsiQuery<T> Where(Expression<Func<T,bool>> predicate)
public MsiOrderedQuery<T> OrderBy<K>(Expression<Func<T,K>> orderClause)
public MsiOrderedQuery<T> OrderByDescending<K>(Expression<Func<T,K>> orderClause)
public MsiClosedQuery<T, R> Select<R>(Expression<Func<T,R>> projection)

All are little expression trees, representing the lambda passed in as the argument. These are captured in a clone of the QueryData object and wrapped inside another MsiQueryBase<T>-derived object, as depicted above in our state machine diagram. To distinguish between ascending and descending ordering, an OrderClause with a Descending property either set to false or true is added to the list of order clauses. This is an ordered list because obviously primary and n-ary (where n >= 2, represented by subsequent ThenBy* calls, see below) orderings are by themselves ordered.

 

Query, OrderedQuery and ClosedQuery

Finally we've arrived with the other query classes. Starting with MsiQuery which represents a query capturing a predicate (Where):

public class MsiQuery<T> : MsiQueryBase<T>
{
     internal MsiQuery(QueryData query) : base(query)
     {
          _query = query;
     }

     public MsiOrderedQuery<T> OrderBy<K>(Expression<Func<T,K>> orderClause)
     {
          QueryData query = (QueryData)_query.Clone();
          query.Order.Add(new OrderClause { Mapper = orderClause, Descending = false });

          return new MsiOrderedQuery<T>(query);
     }

     public MsiOrderedQuery<T> OrderByDescending<K>(Expression<Func<T,K>> orderClause)
     {
          QueryData query = (QueryData)_query.Clone();
          query.Order.Add(new OrderClause { Mapper = orderClause, Descending = true });

          return new MsiOrderedQuery<T>(query);
     }

     public MsiClosedQuery<T, R> Select<R>(Expression<Func<T,R>> projection)
     {
          QueryData query = (QueryData)_query.Clone();
          query.Select = projection;

          return new MsiClosedQuery<T, R>(query);
     }
}

Actually there's nothing new in here. All methods have been seen before in exactly the same shape as before. This method-level cloning is the result from trading IQueryable's endless composability for compile-time verified limited composition in our fluent pattern. Obviously, there are ways to "share" the common log (which in this case results in 3 shared lines of code per method). The reader should feel free to perform this refactoring. The core take-away is the limited number of methods per query class, in this particular case there's no Where method (since an object of this type itself is the result of a Where-application in the depicted state machine). If we were to support multiple Where-calls we'd have to AND them together, which wouldn't be a problem since Where and OrderBy*/ThenBy* operations are mutually commutative (i.e. you can put them everywhere in the chain of method calls, as long as you don't pass a Select boundary which introduces tricky nested queries as outlined above), but we deliberately choose to restrict the implementation in the scope of this series.

The MsiOrderedQuery<T> class is a little different since two new methods, ThenBy and ThenByDescending are introduced. However, those are no rocket science either:

public class MsiOrderedQuery<T> : MsiQueryBase<T>
{
     internal MsiOrderedQuery(QueryData query) : base(query)
     {
          _query = query;
     }

     public MsiClosedQuery<T, R> Select<R>(Expression<Func<T,R>> projection)
     {
          QueryData query = (QueryData)_query.Clone();
          query.Select = projection;

          return new MsiClosedQuery<T, R>(query);
     }

     public MsiOrderedQuery<T> ThenBy<K>(Expression<Func<T,K>> orderClause)
     {
          QueryData query = (QueryData)_query.Clone();
          query.Order.Add(new OrderClause { Mapper = orderClause, Descending = false });

          return new MsiOrderedQuery<T>(query);
     }

     public MsiOrderedQuery<T> ThenByDescending<K>(Expression<Func<T,K>> orderClause)
     {
          QueryData query = (QueryData)_query.Clone();
          query.Order.Add(new OrderClause { Mapper = orderClause, Descending = true });

          return new MsiOrderedQuery<T>(query);
     }
}

And finally, there's our closed query object, its closed nature being reflected by the lack of further query operators.

public class MsiClosedQuery<T, R> : MsiQueryBase<R>
{
     internal MsiClosedQuery(QueryData query) : base(query)
     {
          _query = query;
     }
}

In fact, this isn't the end of everything: the object still is an IEnumerable`1, so the user can continue to query using LINQ to Objects if System.Linq is in scope. The careful reader will have noticed this from the very beginning when MsiQueryBase<T> was introduced. For LINQ to MSI, we can see this as a feature but for "remotable" kinds of LINQ providers, I'd consider this a bug in the provider implementation. Why? The compiler would silently fall back to LINQ to Objects operators and you could well take a "bad start":

var res = from p in products join c in categories on p.Category equals c ...

Assume products and categories both represent a remote object but the provider doesn't support a GroupJoin operation. If the objects still derive from IEnumerable<T>, the LINQ to Objects operators will be in scope and you've silently created a local LINQ query that will suck down all objects from both tables for a client-side join. In LINQ to MSI such problems are not that much of a deal since the "remote database" isn't really remote; it's just a local file. And obviously, any logging mechanism on the provider (.Log property typically) will reveal the query sent to the database, so if that shows a "SELECT * fetch all" kind of query, it should trigger suspicion. IQueryable's AsEnumerable acts as an explicit boundary for remote to local execution and solves this problem.

There are ways around this problem of course (apart from buying in to the whole IQueryable stuff). For example, you could require explicit execution by means of an Execute method that returns the enumerator - a little less user-friendly but it rescues the mission anyway. Notice I said to return an "enumerator", not a memory-persisted list with all the pre-fetched results - this leaves room for lazy fetching of results through a data reader object which can be stopped at any moment without wasting network resources (e.g. you might still have a client-side Take(5) operation which would only cause at most five calls to *DataReader.Read()).

 

Known limitations

The query structure for our LINQ to MSI implementation supports Where, OrderBy*, ThenBy* and Select operators. MSI supports the following:

SELECT [DISTINCT]{column-list} FROM {table-list} [WHERE {operation-list}] [ORDER BY {column-list}]

which apparently only leaves out the DISTINCT operator (which is trivial to implement). However, there's one more restriction hidden in the FROM clause: {table-list}. We won't support joins in this implementation (for now) although MSI has limited support for them:

Only inner joins are supported and are specified by a comparison of columns from different tables. Circular joins are not supported. A circular join is a SQL query that links three or more tables together into a circuit.

I'll come back to this in a later post, outlining what it would take to implement support for this.

 

Next time...

...we'll start the translation battle. Stay tuned!

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

A rather unexpected intermezzo in this series. Why? The first rule of blogging is that blog readers are always right, and this time it was no different. Although I pointed out yesterday "We're just doing raw interop here without more fancy SafeHandle stuff or so, but feel free to fine-tune the interop layer." there's no way for me to answer for example Steven's question:

Bart, Can you explain why to chose to implement finalizer methods on your classes instead of using a from SafeHandle inherited class to close the handle?

apart from the lazy answer "It's only a sample, dude!". But I definitely agree, quality needs to come first especially when you have the luxury to sit down and write the code in a relaxed atmosphere (as opposed to on-the-fly coding in an on-stage setting at some conference, not that such a setting wouldn't be relaxed of course...). I'm not completely guilty though, since I promised myself to fix this fit-and-finish issue by the time I'd publish the sources for this series. I was under the impression to have written about SafeHandles somewhere in the Whidbey timeframe but couldn't find a trace of it, so likely it never grew out of the draft stage. Therefore, this looks like an ideal opportunity to talk a bit about SafeHandles.

 

What's a SafeHandle?

The pattern I used in my yesterday's post is an old-fashioned one. The IDisposable part of it is still relevant and allows you to dispose off resources in a controllable fashion, e.g. by means of a using block in C# or VB. The reason I call it controllable is the fact you don't rely on the garbage collector to clean up the object, using the finalizer (the thing with destructor syntax in C# which was a little unfortunate choice of syntax). The fact I still have a finalizer implemented is the old-fashioned part of it. Why is that? In short, there are too little guarantees provided by finalizers (and the runtime having the task of calling them) to ensure reliability in all circumstances. During the v2.0 "Whidbey" timeframe, the .NET Framework team fixed this and there was the lucky coincidence of SQL Server 2005 "Yukon" looking into running managed code inside the server for what later became SQLCLR, by hosting the CLR inside the server process which obviously needs to provide strong guarantees about the stability and overall behavior of the CLR and managed code being run by it.

This whole reliability requirement isn't just realized through one single feature. Indeed, there are many pieces working together to establish this:

  • Host Protection Attributes - categorizes functionality of frameworks so that hosts like SQL Server can allow or disallow certain categories of code to be executed (e.g. no WinForms in SQLCLR)
  • Critical Finalizers - ensure that the finalizer gets executed properly, even under extreme circumstances (guaranteed to run during app domain unload, which means app domains can't leak handles and can be used as recycle units in case the application - e.g. SQL Server - gets into troubles)
  • Constrained Execution Regions - regions of code with guarantees that no asynchronous exceptions will be thrown by the runtime (the runtime ensures those will only happen before or after the region); a CER also puts strong restrictions on the developer about what can be used inside the region (which makes it hard to use them correctly directly), therefore providing more control over what specific pieces of code do

The core problem with old-fashioned IntPtr-based interop is the fact handles could be leaked in case asynchronous exceptions occur (an example is a rude thread abort causing ThreadAbortException). In addition to this, there's also a security risk by means of so-called recycle attacks. To understand this issue, you need to know that handles are just integer numbers kept on a per-process basis, pointing at some object maintained by the system. Once a handle is freed, that number becomes available again to point at some other resource, so once a new handle is obtained by some other function call it might well get the same number. If the old handle value is still kept around, incorrectly assuming it still points at the original resource, it now points at a possibly security-sensitive resource. This is another argument why it's a good practice (amongst others) to pair every call to CloseHandle with a zero-out assignment on the handle being closed (operating on an invalid handle also has the chance to surface obscure problems earlier than operating on a stale handle value that in the meantime might point to something else).

Finally, SafeHandles are implemented using Critical Finalizers and hence provide a way to ensure proper clean-up. The essence of a SafeHandle is to act as a wrapper for an IntPtr but one which the CLR knows about, so that the runtime can do the right thing to release the handle (not surprisingly there's a ReleaseHandle method).

I can't possibly cover all details on those features here and others have done it already, so if you want to learn more, here are some good pointers:

 

MsiSafeHandle

Here's a dump of an MsiSafeHandle implementation, deriving from SafeHandleZeroOrMinusOneIsInvalid (meaning that IntPtr values wrapped by it are invalid when they are -1 or 0, something implemented inside the IsInvalid property getter).

[SecurityPermission(SecurityAction.InheritanceDemand, UnmanagedCode = true)]
[SecurityPermission(SecurityAction.Demand, UnmanagedCode = true)]
internal class MsiSafeHandle : SafeHandleZeroOrMinusOneIsInvalid
{
     private MsiSafeHandle() : base(true) /* take ownership of the handle */
     {
     }

     [ReliabilityContract(Consistency.WillNotCorruptState, Cer.Success)]
     override protected bool ReleaseHandle()
     {
         return 0 == MsiInterop.MsiCloseHandle(handle);
     }
}

The CLR knows that whenever this class is used in an interop signature, it can create an instance of this class, adding the IntPtr handle value to it. When the runtime (or the application itself by calling Dispose on the object explicitly) decides the handle needs to be cleaned up, it calls ReleaseHandle which we implement by calling our interop function. Notice the ReliabilityContract attribute as well, which is part of CER (System.Runtime.ConstrainedExecution namespace), exposing the guarantees the method implementation makes (something that can't be enforced by the runtime and is still the responsibility of the developer).

 

Revised interop signatures

For our interop class, some changes are required to use the MsiSafeHandle class instead of IntPtr and to attribute the CloseHandle method with a CER attribute:

[SuppressUnmanagedCodeSecurity]
internal static class MsiInterop
{
     [DllImport("msi.dll", EntryPoint="MsiOpenDatabaseW", CharSet=CharSet.Unicode, ExactSpelling=true)]
     public static extern uint MsiOpenDatabase(string databasePath, int persist, out MsiSafeHandle database);

     [DllImport("msi.dll", EntryPoint="MsiDatabaseOpenViewW", CharSet=CharSet.Unicode, ExactSpelling=true)]
     public static extern uint MsiDatabaseOpenView(MsiSafeHandle database, string query, out MsiSafeHandle view);

     [DllImport("msi.dll", EntryPoint="MsiViewExecute", CharSet=CharSet.Unicode, ExactSpelling=true)]
     public static extern uint MsiViewExecute(MsiSafeHandle view, MsiSafeHandle parameters);

     [DllImport("msi.dll", EntryPoint="MsiViewFetch", CharSet=CharSet.Unicode, ExactSpelling=true)]
     public static extern uint MsiViewFetch(MsiSafeHandle view, out MsiSafeHandle record);

     [DllImport("msi.dll", EntryPoint="MsiRecordGetStringW", CharSet=CharSet.Unicode, ExactSpelling=true)]
     public static extern uint MsiRecordGetString(MsiSafeHandle record, uint field, StringBuilder value, ref int bufferSize);

     [DllImport("msi.dll", EntryPoint="MsiRecordGetInteger", CharSet=CharSet.Unicode, ExactSpelling=true)]
     public static extern int MsiRecordGetInteger(MsiSafeHandle record, uint field);

     [DllImport("msi.dll", EntryPoint="MsiCloseHandle", CharSet=CharSet.Unicode, ExactSpelling=true)]
     [ReliabilityContract(Consistency.WillNotCorruptState, Cer.Success)]
     public static extern uint MsiCloseHandle(IntPtr handle);
}

Notice the MsiCloseHandle function is the only one which still takes an IntPtr. This is required because we still need to call this function directly inside our MsiSafeHandle class. However, other than in that particular spot, direct calls to MsiCloseHandle shouldn't be made anymore, since the handle will always be wrapped inside an MsiSafeHandle which implements IDisposable, ultimately calling into ReleaseHandle to do the heavy work.

 

Using it

I'll just give one sample here, based on our MsiConnection class. Other classes can be changed completely analogously:

public sealed class MsiConnection : IDisposable
{
     private string _fileName;
     private MsiSafeHandle _database;
     private bool _disposed;

     public MsiConnection(string fileName)
     {
          if (!File.Exists(fileName))
          {
               throw new FileNotFoundException(fileName + " not found.");
          }

          _fileName = fileName;
     }

     public void Open()
     {
          CheckDisposed();

          uint res = MsiInterop.MsiOpenDatabase(_fileName, 0, out _database);
          if (res != 0)
          {
               throw new Exception("Failed to open database. Error code = " + res);
          }
     }

     internal MsiSafeHandle Handle
     {
          get { CheckDisposed(); return _database; }
     }

     public void Dispose()
     {
          Dispose(true);
          GC.SuppressFinalize(this);
     }

     private void Dispose(bool disposing)
     {
          if (!_disposed && _database != null && !_database.IsInvalid)
          {
               _database.Dispose();
          }

          _disposed = true;
     }

     private void CheckDisposed()
     {
          if (_disposed)
              throw new ObjectDisposedException("MsiConnection");
     }
}

Notice we still are IDisposable, nothing's changed there. However, now we're storing MsiSafeHandle instead of an IntPtr which gets disposed by our Dispose method (but also automatically during critical finalization, so we get the guarantees we've always wanted). Depending on the class design you prefer for the Msi* classes, the helper Dispose method would either be private or protected virtual. In this case, I'm sealing the class which means the helper method should be private. If you were to choose otherwise, the helper method should be protected virtual and be decorated with a Demand for unmanaged code security permissions.

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

PattThis should really be the least interesting part of this series. But still, without any physical data access layer, there's no way to build abstractions on top of it. So, in this post of this series we'll take a look at some very simple MSI interop, giving us a basic data provider for MSI databases, mirrored after the typical structure of .NET data providers like System.Data.SqlClient.

 

Interop signatures

As all of you know, interop of native components in the world of the managed code CLR isn't that hard: System.Runtime.InteropServices is your friend. More specifically, we'll be importing stuff from msi.dll using DllImports. To see what's available there, you can either open up msi.h from the Windows SDK or run dumpbin /exports on the dll, or in a less geeky way you can simple go to MSDN to get some information. Ultimately you'll come up with the following signatures that are relevant to data access, just focusing on read operations for now:

internal static class MsiInterop
{
     [DllImport("msi.dll", EntryPoint="MsiOpenDatabaseW", CharSet=CharSet.Unicode, ExactSpelling=true)]
     public static extern uint MsiOpenDatabase(string databasePath, int persist, out IntPtr database);

     [DllImport("msi.dll", EntryPoint="MsiDatabaseOpenViewW", CharSet=CharSet.Unicode, ExactSpelling=true)]
     public static extern uint MsiDatabaseOpenView(IntPtr database, string query, out IntPtr view);

     [DllImport("msi.dll", EntryPoint="MsiViewExecute", CharSet=CharSet.Unicode, ExactSpelling=true)]
     public static extern uint MsiViewExecute(IntPtr view, IntPtr parameters);

     [DllImport("msi.dll", EntryPoint="MsiViewFetch", CharSet=CharSet.Unicode, ExactSpelling=true)]
     public static extern uint MsiViewFetch(IntPtr view, out IntPtr record);

     [DllImport("msi.dll", EntryPoint="MsiRecordGetStringW", CharSet=CharSet.Unicode, ExactSpelling=true)]
     public static extern uint MsiRecordGetString(IntPtr record, uint field, StringBuilder value, ref int bufferSize);

     [DllImport("msi.dll", EntryPoint="MsiRecordGetInteger", CharSet=CharSet.Unicode, ExactSpelling=true)]
     public static extern int MsiRecordGetInteger(IntPtr record, uint field);

     [DllImport("msi.dll", EntryPoint="MsiCloseHandle", CharSet=CharSet.Unicode, ExactSpelling=true)]
     public static extern uint MsiCloseHandle(IntPtr handle);
}

These functions are pretty simple to understand: using MsiOpenDatabase one opens a database, which can be used to define a query view using MsiDatabaseOpenView that subsequently can be executed using MsiViewExecute to yield results using MsiViewFetch, each of which can be accessed on a per-column basis using MsiRecordGetString and MsiRecordGetInteger. Finally, MsiCloseHandle is used to close any obtained handles. We're just doing raw interop here without more fancy SafeHandle stuff or so, but feel free to fine-tune the interop layer.

 

Respecting well-known patterns

Patterns deserve respect. Most of the time it are random designs that haven proven successful and received a career lifetime achievement award because of that, giving them the right to use the title "Pattern" (with capital P). I'd like to promote the design of data providers in .NET to this status. Essentially there are three players in the core pattern:

  • Connection - maintains the connection-oriented access to an underlying data store
  • Command - used to execute an arbitrary possibly parameterized command on the data store
  • DataReader - fetches results from a command in a sequential forward-only fashion

If we'd be really respectful, we'd go all the way implementing the System.Data.IDb* interfaces but since this is a sample only, we reserve ourselves the right to take some shortcuts and ignore concepts like transactions (although MsiOpenDatabase has such notions!). Obviously, the reader is free to extend this data provider at will to embrace those interfaces end-to-end providing meaningful implementations where applicable.

 

The connection

Nothing fancy here: just a call to MsiOpenDatabase, a way to obtain the handle for use in the command class and some clean-up using IDisposable and a finalizer. Ignore the fact I'm not subclass System.Exception for the purpose of this sample.

public class MsiConnection : IDisposable
{
     private string _fileName;
     private IntPtr _database;
     private bool _disposed;

     public MsiConnection(string fileName)
     {
          if (!File.Exists(fileName))
          {
               throw new FileNotFoundException(fileName + " not found.");
          }

          _fileName = fileName;
     }

     public void Open()
     {
          CheckDisposed();

          if (_database != IntPtr.Zero)
          {
               throw new InvalidOperationException("Database already opened.");
          }

          uint res = MsiInterop.MsiOpenDatabase(_fileName, 0, out _database);
          if (res != 0)
          {
               throw new Exception("Failed to open database. Error code = " + res);
          }
     }

     public void Close()
     {
          CheckDisposed();

          if (_database != IntPtr.Zero)
          {
               MsiInterop.MsiCloseHandle(_database);
               _database = IntPtr.Zero;
          }
     }

     internal IntPtr Handle
     {
          get { CheckDisposed(); return _database; }
     }

     public void Dispose()
     {
          Dispose(true);
          GC.SuppressFinalize(this);
     }

     private void Dispose(bool disposing)
     {
          if (!_disposed && _database != IntPtr.Zero)
          {
               MsiInterop.MsiCloseHandle(_database);
               _database = IntPtr.Zero;
               _disposed = true;
          }
     } 

     private void CheckDisposed()
     {
          if (_disposed)
               throw new ObjectDisposedException();
     }

     ~MsiConnection()
     {
          Dispose(false);
     }
}

 

The command

Just like the connection maps to the concept of an MSI connection, the command maps to the concept of an MSI view. Again we implement IDisposable but for the rest, everything should be self-expanatory:

public class MsiCommand : IDisposable
{
     private IntPtr _view;
     private string _command;
     private MsiConnection _conn;
     private bool _disposed;

     public MsiCommand(string command, MsiConnection conn)
     {
          _command = command;
          _conn = conn;
     }

     public MsiDataReader ExecuteReader()
     {
          CheckDisposed();

          if (_conn.Handle == IntPtr.Zero)
          {
               throw new Exception("Database connection not opened.");
          }

          uint res = MsiInterop.MsiDatabaseOpenView(_conn.Handle, _command, out _view);
          if (res != 0)
          {
               throw new Exception("Failed to create command. Error code = " + res);
          }

          return new MsiDataReader(this);
     }

     internal IntPtr Handle
     {
          get { CheckDisposed(); return _view; }
     }

     public void Dispose()
     {
          Dispose(true);
          GC.SuppressFinalize(this);
     }

     private void Dispose(bool disposing)
     {
          if (!_disposed && _view != IntPtr.Zero)
          {
               MsiInterop.MsiCloseHandle(_view);
               _view = IntPtr.Zero;
               _disposed = true;
          }
     } 

     private void CheckDisposed()
     {
          if (_disposed)
               throw new ObjectDisposedException();
     }

     ~MsiCommand()
     {
          Dispose(false);
     }
}

 

The data reader

Last but not least, the reader. Again IDisposable, but this time the core functionality is a little more involved. Just a little though. We'll execute the command "view" and fetch results which we keep around on a per-record basis for subsequent use in GetString and GetInteger to access the field values in the MSI table's current row.

public class MsiDataReader : IDisposable
{
     private MsiCommand _command;
     private IntPtr _record;
     private bool _disposed;

     internal MsiDataReader(MsiCommand command)
     {
          _command = command;
     }

     public bool Read()
     {
          CheckDisposed();

          if (_command.Handle == IntPtr.Zero)
          {
               throw new Exception("Command not ready.");
          }

          uint res;

          if (_record == IntPtr.Zero)
          {
               res = MsiInterop.MsiViewExecute(_command.Handle, IntPtr.Zero);
               if (res != 0)
               {
                    throw new Exception("Failed to run command. Error code = " + res);
               }
          }
          else
          {
               MsiInterop.MsiCloseHandle(_record);
          }

          res = MsiInterop.MsiViewFetch(_command.Handle, out _record);
          if (res == 259)
          {
               return false;
          }

          if (res != 0)
          {
               throw new Exception("Failed to get record. Error code = " + res);
          }

          return true;
     }

     public string GetString(uint column)
     {
          CheckDisposed();

          if (_record == IntPtr.Zero)
          {
               throw new Exception("No record fetched.");
          }

          column++;

          StringBuilder sb = new StringBuilder("", 1);
          int len = 0;

          uint res = MsiInterop.MsiRecordGetString(_record, column, sb, ref len);
          if (res == 234)
          {
               sb.Capacity = len++;
               res = MsiInterop.MsiRecordGetString(_record, column, sb, ref len);
          }

          if (res != 0)
          {
               throw new Exception("Failed to read string. Error code = " + res);
          }

          return sb.ToString();
     }

     public int GetInteger(uint column)
     {
          CheckDisposed();

          if (_record == IntPtr.Zero)
          {
               throw new Exception("No record fetched.");
          }

          column++;

          int res = MsiInterop.MsiRecordGetInteger(_record, column);

          return res;
     } 

     public void Dispose()
     {
          Dispose(true);
          GC.SuppressFinalize(this);
     }

     private void Dispose(bool disposing)
     {
          if (!_disposed && _record != IntPtr.Zero)
          {
               MsiInterop.MsiCloseHandle(_record);
               _record = IntPtr.Zero;
               _disposed = true;
          }
     } 

     private void CheckDisposed()
     {
          if (_disposed)
               throw new ObjectDisposedException();
     }

     ~MsiDataReader()
     {
          Dispose(false);
     }
}

That's it. I told you it wasn't too bad and now all the plumbing should be done, ready for us to build the LINQ implementation on top of this.

 

Sample

To finish up this post, let's take a look at a simple sample:

class Program
{
     static void Main()
     {
          using (var conn = new MsiConnection(@"C:\temp\PowerShell_Setup_x86.msi"))
          {
               using (var cmd = new MsiCommand("SELECT Property, Value FROM Property", conn))
               {
                    conn.Open();

                    using (var reader = cmd.ExecuteReader())
                    {
                         while (reader.Read())
                         {
                              Console.WriteLine("{0} = {1}", reader.GetString(0), reader.GetString(1));
                         }
                    }
               }
          } 
     }
}

and here's the output:

image

I'll post the code of the entire provider by the end of this series. Enjoy!

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

In a reaction to my post on LINQ to MSI yesterday, Hal wrote this:

I don't know enough about the dev side to know if this is a stupid question or not but here goes: Would I be able to take advantage of LINQ to MSI (or LINQ in general from a wider point-of-view) from within PowerShell?  I know someone made an MSI snapin but I seem to recall it being a pretty simple thing.  Having the ability for admins to query and work with MSI packages seems like it could be awfully useful, and the point of not learning yet another SQL variant rings true for everyone, not just developers.  :)

Obviously not a stupid question at all (there are only stupid answers, to start with a cliché for once :-)). Having LINQ capabilities in PowerShell is definitely something I have given a thought - actually it has been one of my fun projects a few months back but there are quite some challenges associated with this. However, Hal's comment made me think about it a bit more, so I mixed in another piece of magic called "Dynamic LINQ". Let's take a look at an experimental journey through "LINQ in PowerShell" integration.

 

There's no such thing as a pipe...

Well, at least a unique definition for it. Pipelines are found in various corners of the computer science landscape but two major implementations stand out:

  • First of all there's the pipe model employed by shells with typical samples in the UNIX world, the DOS shell and obviously Windows PowerShell. This kind of pipe model works in a push-like eager fashion: the source at the left provides data that's pushed through the pipe that acts as a filtering (e.g. PS where) and transformation mechanism (e.g. PS select). The fact data appears at the end of it is the effect of pushing data into it at the front. It's typically eager because sending the chained set of commands that form the pipeline to the command processor triggers execution immediately, in a left-to-right fashion.
  • On the other side, there's a lazy variant of the pipe model used by the LINQ query comprehension monadic model. Data flows in at the left again but it doesn't start to flow till the pipeline participant on the right sucks data out of it. So ultimately a chain of query operators pull data from the source starting all the way from the right. This laziness makes LINQ stand out since no more data fetching work is done than strictly needed (e.g. if you do a Take(5) on a sequence of 1,000 items, no more than 5 items will be fetched).

Two different models of pipelines that prove hard to unify. However, on the other side when thinking about LINQ in PowerShell it would be handy to leverage existing idioms rather than creating a whole new language in the language for querying although that would definitely work too as long as it feels natural enough. Dynamic LINQ provides a middle-ground: the operators (Where, Select, OrderBy, etc) are still implemented as method calls while their arguments are based on a textual expression language.

 

Dynamic LINQ

Scott blogged about Dynamic LINQ a while ago in his Dynamic LINQ (Part 1: Using the LINQ Dynamic Query Library) post. In essence, Dynamic LINQ talks to any IQueryable-capable LINQ implementation and allows you to write things like:

nw.Products.Where("UnitPrice > 100").Select("New(ProductName As Name, UnitPrice As Price)");

The string islands in here are converted into expression trees at runtime and executed against the IQueryable data source. Notice dynamic types will be generated on the fly as well, in the sample above a type with two properties Name and Price created by the projection. This moves the boundary of runtime versus compile time a bit further: the expression strings become expression trees which in turn get translated into the back-end language by the provider being targeted (SQL, LDAP, CAML, WS calls, etc).

What you need to get started with this post, is the following download: C# Dynamic Query Library (included in the \LinqSamples\DynamicQuery directory)

 

Encapsulating a data source: Northwind strikes again

Moving the boundary from compile-time to run-time is a good thing in quite some cases but extremes are rarely the best choice. All the way statically typed won't work here: PowerShell works with an interpreter and lots of runtime supporting features (such as ETS) - a good thing it does. You don't sell compilers to IT Pros. Now, for our LINQ mission, complete dynamic typing wouldn't work that well either: we're going to access some data store which has rich data with specific type information. We better encapsulate this so that our PowerShell scripts can take advantage of it. For example, a SQL database table produces a strongly-typed entity class, which is precisely what LINQ to SQL's sqlmetal tool (or equivalent designer) does.

However, I do agree that for some types of data sources, a more ad hoc access mechanism is more appropriate that for others. Ad hoc here doesn't just point at the query capability (after all we want to realize just that) but also at the burden you have to go through to get access to data. I'd categorize SQL under the ad hoc extremes: SQL has been so easy to access (SQL syntax, SQLCMD or OSQL tool) that it's a pity we'll have to create an entity type upfront to access any table whatsoever. But still, if you want ad hoc data access, there's still regular SQL (which you'll use for dynamism by trading runtime types representing the data). On the other side there are things like AD where the schema changes rarely and the entities could be part of a library that exposes all of the entities that ship with AD. Once that one's loaded you have virtually any (and strongly-typed) ad hoc data capability through LINQ. After all, it depends on the flux of the data source. Requiring new types every time a SQL database schema changes is definitely overhead but as mentioned for things like AD and e.g. MSI (which has fixed tables) that would be less of a stumble block.

Let's go for a LINQ to SQL sample anyway despite all of this philosophical fluff, so create a new Class Library project in VS and add a LINQ to SQL Classes file to it:

image

Drag and drop all of the tables from the Northwind database from the Server Explorer to the designer and compile the assembly. Data access made easy - that's what LINQ's all about!

image

This can actually already be used from PowerShell:

image

Since the context object is just an object, you can create an instance of it like this:

[System.Reflection.Assembly]::LoadFile("C:\temp\LINQthroughPowerShell\Northwind\bin\Debug\Northwind.dll")
$ctx = new-object Northwind.NorthwindDataContext

Just load the DLL and use new-object. As you can see in the screenshot above, everything what we need is available. But...

 

Breaking eagerness

IEnumerables make PowerShell pipes tick (amongst other "streams" of objects provided by participants in the pipeline). That's too eager. Let's show you: in the session above, type $ctx and see what happens:

image

Oops, all data is in the console already. Handy but wasteful. Why does this happen? Tables in LINQ to SQL are of type Table<T> (where T is the entity type) which are IQueryable<T> and thus IEnumerable<T>. PowerShell, eager as it is, enumerates over IEnumerables to get their results (which makes sense in this context). If you turn on logging on the LINQ to SQL data context, you'll see precisely what happens:

image

So, how can we break eagerness? We don't have such a thing like a lazy pipeline, so let's create one by having two markers: a cmdlet that establishes a "lazy context" and one that terminates it. Everything in between flowing through the pipe won't be an IEnumerable but a "captured IEnumerable", in our case more specifically an IQueryable, which we rewrite throughout the pipe by adding LINQ operators to it through Dynamic LINQ. I assume readers of this blog are familiar with cmdlet development; if not, check out my Easy Windows PowerShell cmdlet development and debugging post.

Below is the class that will encapsulate the IQueryable to suppress eager evaluation, creating our object traveling through the lazy context:

public class LinqQuery
{
    private IQueryable _queryable;

    internal LinqQuery(IQueryable queryable)
    {
        _queryable = queryable;
    }

    public string Expression
    {
        get
        {
            return _queryable.Expression.ToString();
        }
    }

    internal IQueryable Query
    {
        get
        {
            return _queryable;
        }
    }
}

and to establish a lazy context, we'll provide a New-Query cmdlet:

[Cmdlet("New", "Query")]
public class NewQueryCmdlet : Cmdlet
{
    [Parameter(Mandatory = true, Position = 0)]
    public IQueryable Input { get; set; }

    protected override void ProcessRecord()
    {
        WriteObject(new LinqQuery(Input));
    }
}

And finally, to end the context, triggering evaluation, we'll have:

[Cmdlet("Execute", "Query")]
public class ExecuteQueryCmdlet : Cmdlet
{

     [Parameter(Mandatory = true, Position = 0)]
    public LinqQuery Input { get; set; }

    protected override void
ProcessRecord()
    {
        WriteObject(Input.Query);
    }
}

This last one is interesting in that it returns the IQueryable, which by means of the eager pipeline triggers execution (since LINQ providers reach out to the server fetching results upon calling GetEnumerator).

 

Dynamic LINQ query operator cmdlets

This barely needs any explanation whatsoever because of the simplicity of the Dynamic LINQ library. We just start by importing the namespace:

using System.Linq.Dynamic;

And start writing our first cmdlet for Where:

[Cmdlet("Where", "LinqObject")]
public class WhereCmdlet : Cmdlet
{
    [Parameter(Mandatory = true, ValueFromPipeline = true)]
    public LinqQuery Input { get; set; }

    [Parameter(Mandatory = true, Position = 0)]
    public string Predicate { get; set; }

    protected override void ProcessRecord()
    {
        WriteObject(new LinqQuery(Input.Query.Where(Predicate)));
    }
}

Notice we taking in a lazy LinqQuery object from the pipeline and emit a new LinqQuery object in ProcessRecord. This makes a LinqQuery object immutable and one can define a LinqQuery object by means of the pipeline for later reuse (e.g. have a query object that establishes a view on data, and then write multiple queries on top of that). The Predicate parameter takes a string that represents the expression language based predicate for the query. Below you can see all of the extension methods brought in scope by Dynamic LINQ:

image

So, we'll create a cmdlet for all of those, which is absolutely straightforward. Actually, this could be done purely declaratively with my Method Invocation Cmdlet mechanism too, but let's not go there.

 

Putting it to the test

Assume all cmdlets have been written (either do it yourself or download the code below). Time to do a sample:

image

It's that easy. And notice the SQL query being sent to the server, precisely what we were looking for, only fetching what we requested. I did split up the query across some lines to show the expression being generated behind the scenes, which is just a normal chain of methods calls captured by a dynamically generated expression tree at runtime. Without this splitting, writing a query is a one-liner:

image

The query syntax is different than PowerShell-like syntax (e.g. > instead of -lt) but that's merely a syntax issue which would be easy to change. And the cmdlet names are pretty long but there are aliases (lwhere, lsort, ltake, lselect for instance).

 

Implicit more intelligent lazy scoping

Actually what we established above is somewhat equivalent to an explicit Dispose call carried out by a using block. In this case, we have a LinqQuery object created by new-query and disposed off by execute-query. The latter one we can make implicit if we assume that the end of a pipeline should trigger evaluation. That's debatable since it doesn't allow to keep a query object across multiple invocations. Depending on personal taste around explicitness, you might like this behavior and provide a "defer-query" opt-out cmdlet. A simple way to do this intelligent auto-evaluation is by using the PSCmdlet baseclass' MyInvocation property:

public abstract class LazyCmdlet : PSCmdlet
{
    [Parameter(Mandatory = true, ValueFromPipeline = true)]
    public LinqQuery Input { get; set; }

    protected abstract LinqQuery Process();

    protected override void  ProcessRecord()
    {
        LinqQuery result = Process();

        if (MyInvocation.PipelinePosition < MyInvocation.PipelineLength)
        {
            WriteObject(result);
        }
        else
        {
            WriteObject(result.Query);
        }
    }
}

Instead of having the Dynamic LINQ cmdlets override ProcessRecord directly, we let them implement Process and depending on the position in the invocation chain, our base class either returns the query object (avoiding eager expansion by the pipeline) or the IQueryable inside it, making it expand and fetch/yield results. Here's the corresponding class diagram:

image

and with some aliases you can now write:

image

 

Download it

If you want to play with this, you can get the code here: LinqThroughPowerShellProvider.cs. It doesn't include the Dynamic LINQ codebase, which you can get from C# Dynamic Query Library (included in the \LinqSamples\DynamicQuery directory).

Build instructions:

  1. Create a new Class Library project.
  2. Add a reference to System.Management.Automation.dll (from %programfiles%\Reference Assemblies).
  3. Add a reference to System.Configuration.Install.dll.
  4. Add LinqThroughPowerShellProvider.cs to it.
  5. Add Dynamic.cs form the Dynamic LINQ library to it.
  6. Build it.

Install instructions:

  1. Open an elevated prompt and go to the bin\Debug build output folder.
  2. Execute installutil -i <name of the dll>

Run instructions:

  1. Open Windows PowerShell.
  2. Execute add-pssnapin LTP
  3. Play around with samples :-). You can use any LINQ provider with IQueryable support (e.g. LINQ to SQL, AD, SharePoint, etc).

Have fun!

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

Introduction

Lately I've been delivering talks entitled "LINQ to Anything", to be repeated this summer at TechEd Africa. The goal of those talks is to focus on LINQ from the extensibility point of view, in other words: how to write query providers like LINQ to AD or LINQ to SharePoint (amongst many others I should give credits like LINQ to Amazon). Obviously, I'm always looking to improve the content of the talk and as a firm believer in the "small is beautiful" approach, smaller samples that pinpoint the core of LINQ providers are always welcome.

This time around, I decided to trade the IQueryable stuff for the manual implementation approach of the query pattern, something not employed in LINQ to AD or LINQ to SharePoint, although the former one will get a rewrite this summer that employs this methodology. But why this change? Well, although some providers definitely benefit from the whole range of query operators, I'd say there's some kind of break-even analysis to be made when considering the IQueryable path. IQueryable is a funny interface since it's not the interface you'd expect. How's that? Just take a glimpse of it:

image

This interface is somewhat unique because it offers an asymmetric world view. Interfaces always have this tension between implementers (who want just a handful of methods to implement) and users (who want as much flexibility as possible). Abstract base classes are often used as an escape valve, offering convenience overloads, but this conflicts with the single inheritance philosophy limiting its usefulness. Interfaces with code, providing defaults for some methods (such as convenience overloads, all falling back to the most flexible variant) are one direction considered by language and runtime teams. However, back to IQueryable's asymmetric approach. In the diagram above, you see precisely five members to be implemented, but when you looks at the consumer side you'll see more:

image

What you're seeing here are all the extension methods brought in scope by importing the System.Linq namespace. In fact, IQueryable is one of the first interfaces that powers itself by relying on a paired set of extension methods being available. First, this avoids requiring people to implement all of the query operators manually; there are way too many of them and chances to mess it up are way too high. Second, there's no single-inheritance trap by making say Queryable the abstract base class implementing IQueryable, just leaving 5 gaps to be implemented by developers. Finally, these extension methods allow IQueryable to grow over time without developers having to worry about it directly (although obviously new query operators will have to be recognized by the query parser written by implementers if they want to take benefit of the new operator, say for example Zip, introduced in PLINQ).

IQueryable is just one part of the picture, there's also IQueryProvider but let's skip this one for now. The take-away of this discussion is actually on the consumer side of IQueryable, the thing users are faced with when writing queries. The flexibility of IQueryable lies in its endless "fluent" chaining capability (just like e.g. System.String has lots of methods that return a System.String, allowing to build arbitrary operator chains) but if you're not going to support the lion's part of those, you're somewhat trapping developers by making them believe at compile time the whole thing will work. Say you don't have support for ordering, an IQueryable will still allows you to write things like:

var res = from product in products orderby product.Price descending select product;

while you'll throw a NotSupportedException at runtime pointing out that OrderByDescending (hidden in the fragment above) is a non-supported operator for your provider. Compile-time errors would be better in this case as pointed out in my rant about Q: Is IQueryable the Right Choice for Me?. Since it's my belief that most of the providers to be written by individuals in the near future will most likely address "basic-queryable" stores, i.e. databases of some sort not supporting the whole set of relational operators, I want to keep IQueryable as the cherry on the pie at the end of the talk, while keeping the core of the talk available for the study of query patterns and their implementation (also allowing for more live coding, which I absolutely love to do).

 

Ducks quack

As pointed out in my Q: Is IQueryable the Right Choice for Me? post, the C# 3.0 and VB 9.0 languages do not rely on interfaces being implemented in order to provider query capabilities. Any type that has the right methods available can participate in query comprehensions (read: language integrated query syntax -  I vote for the lower-case abbreviation "linq" to be used when talking about the language feature and the upper-case abbreviation "LINQ" to be used when talking about everything including providers). This is actually some form of what I'd call "statically typed duck typing": as long as there are methods with right signatures, queries will work. For example, in order for the "where" keyword to work, the compiler expects to find a method that has as its first real parameter (real because the method could be an extension method or a regular instance method) something compatible with a boolean-producing lambda. Typical samples include a Func<T, bool> or an Expression<Func<T, bool>>.

Notice this is not something new for languages like C#. Patterns like "foreach" rely on a method called GetEnumerator to be available on the type being enumerated over. It doesn't necessarily need to be an IEnumerable of some sort. Some people believe in over-typing and think this isn't "quite right" for a currently statically typed language like C#, but honestly: if an object has a GetEnumerator method returning either an IEnumerator or IEnumerable, what are the odds that object isn't enumerable? It's just like duck typing: if it walks like a duck and quacks like a duck, it ought to be a duck (or, you're right, a genetically manipulated dog - let's forget about AOP for a while :-)), it only has a stronger typed signature requirement.

 

Goals and non-goals of LINQ to MSI

Just like LINQ to AD, the goal of LINQ to MSI is to act as a sample. Although many people have mailed me already telling they absolutely love LINQ to AD and are productizing it, I have to stress the fact it's sample-level quality (which nevertheless can be very useful to extend upon). This being said, LINQ to AD and LINQ to SharePoint will get their promised updates some time in the near future (I keep a little vague on the scheduling I know, but both projects I'd categorize as "personal incubation projects" have spread their tangles into other projects I'm working on from time to time).

Where the sample distinguishes itself from LINQ to AD is in its approach:

  • We won't go down the IQueryable route.
  • We'll focus more on the structure of and cross-relationships between a query provider, entity objects, data collections and query objects.
  • There'll be less focus on implementing the query parser.

So how can MSI be queried? Well, MSIs are just little databases. There's this tool called Orca that comes with the Windows SDK that allows you to inspect an MSI's internal structure. For example, in the picture below I opened up the Windows PowerShell 2.0 CTP MSI (which just happened to be the first one in my temp folder):

image

No secrets are therein, so feel free to do the same :-). Obviously, changing an MSI is at your own risk. But why would you like to LINQify MSI? As I said, it's a sample in the first place, but the common theme in LINQ is about democratizing data access from .NET programming. The query language used by MSI is a lightweight SQL variant, but do you really want to learn another SQL dialect? Since the answer is no (allow me to answer on the reader's behalf), that's already one good reason. In addition, stores that weren't easily accessible in the past become more accessible now through unified query provider models and syntax, which is definitely a good thing. You can think of and endless number of samples with MSI, for example:

  • Create your own Orca tool.
  • Develop a setup builder tool that uses the LINQ to MSI entity model to write an MSI (this won't be supported directly though the sample implementation, but adding DML functionality shouldn't be too hard - maybe one day I'll show it to you).
  • Make an ASP.NET handler that allows users to "download this website as MSI", grabbing content dynamically and composing an MSI that will install an offline copy of the site to your local IIS (dreaming aloud, but definitely feasible for the braver readers out there).
  • Etc.

To convince you it actually works, here's the output of the following piece of LINQ code (omitted the straight-forward foreach loop):

var msi = new MyMsi(@"C:\temp\PowerShell_Setup_x86.msi");
var res = from prop in msi.Properties select new { prop.Name, prop.Value };

image

Next time, we'll start by the plumbing of MSI interop to allow querying an MSI database from managed code. You'll see classes like MsiConnection, MsiCommand and MsiDataReader appear on the surface of the bloge. Stay tuned!

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

A few days ago I got an interesting question from one of my blog readers. During my blogging battle with pattern matching, I mentioned the concept of compiling expression tree lamdas on the fly to IL-code. This is actually one of the core parts of the System.Linq.Expressions implementation when expressions are getting compiled using a call to Compile. However, in cases such as pattern matching you might want to emit your own IL-code to control the way e.g. a pattern match with multiple match clauses and types checks gets emitted into a multi-branch if-else if-else statement underneath the covers (remember only the surface of a functional pattern match needs to be functional, we're still dealing with von Neumann machines that don't have intrinsic knowledge of lambda calculus - it would be a nice thing to have some LISC or FISC instruction set though targeted at some typical tricks employed by functional languages, but we're getting off the topic now). When analyzing such an expression tree and allowing type congruential checks in the matcher, you'll be faced with end-users passing in lambdas like this:

(int age) => new { Name = "Bart", Age = age }

which obviously means we want objects that take the shape (string Name, int Age) where Name == "Bart" and selecting Age from it, passing it in to the pattern match body. Now, instead of doing a "duck type style" of object shape check you just might want to do a strict type check first and you might need to instantiate such an anonymous object deep in your code stack. If you'd write something like the below (greatly simplified sample) things won't work as expected:

using System;
using System.Reflection.Emit;

delegate void Foo();

class Skip
{
    static void Main()
    {
        var anon = new { Bar = true };

        DynamicMethod method = new DynamicMethod("MyLambda", typeof(void), new Type[] { });
        ILGenerator ilgen = method.GetILGenerator();
        ilgen.Emit(OpCodes.Ldc_I4_1);
        ilgen.Emit(OpCodes.Newobj, anon.GetType().GetConstructor(new Type[] { typeof(bool) }));
        ilgen.Emit(OpCodes.Pop);
        ilgen.Emit(OpCodes.Ret);

        Foo a = (Foo)method.CreateDelegate(typeof(Foo));
        a();
    }
}

Actually this would fail to execute with an exception like the following:

Unhandled Exception: System.MethodAccessException: <>f__AnonymousType0`1..ctor(Boolean)
   at MyLambda()
   at Skip.Main() 

The reason here is the jitter enforces visibility rules unless - and this is new in .NET Framework 3.5 - you explicitly opt-out from it using the "skipVisibilityChecks" parameter to the DynamicMethod constructor:

DynamicMethod method = new DynamicMethod("MyLambda", typeof(void), new Type[] { }, true); 

Because this is a convoluted sample, you can tell it works fine if it just returns silently :-).

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

This sounds like one of those posts that's about to draw scientifically sound conclusions. It isn't. The other day I had a discussion about various ways to express asynchronisity (or asynchronousness, whatever spell checker upsetting word you prefer) in API designs. The answer might be shocking, but it really depends and there are quite some promising evolutions around. I just want to give you a breakdown of a few possible designs, some sexier than others but that's just a matter of (good) taste. The moral of the story: asynchronous patterns (which become increasingly important in distributed systems programming) is a plural noun.

 

The dinosaur in the room: IAsyncResult

Nothing against those animals - we should appreciate the meaning giants give to life. Obviously, the asynchronous pattern realized by this giant is overly known by the average .NET developer. A fancy way of introducing it is by using a web proxy class generated by the tool of your choice (dinosaurs prefer wsdl.exe, modern age Earth citizens choose svcutil.exe) but ultimately it all boils down to what I'd call a scoped overlapped operation. It's scoped in a VB-style by means of Begin* and End* method pairs that denote the client-side view of the operation, while it's overlapped because obviously everything between Begin and End runs simultaneously with what's happening on the other side of the fence.

IAsyncResult just serves the purpose of a concert ticket: you get one when you enter, but you're not required to stay waiting for the concert to finish, you can just sneak in with the ticket to the after-party to get to hear how great it was (the result of the long-running operation). So do whatever you like in between. Obviously don't take this as my ultimate advise as the ideal way to 'attend' cultural adventures. The cool thing is you can keep waiting for one or more of those asynchronous operations to finish depending on how you want to proceed with the results. In the end, I haven't said anything more than WaitForSingleObject[Ex] and WaitForMultipleObjects[Ex] (or the Msg* equivalents in case you want to process messages during the background concert, contrary to regulations in most theatres today but highly recommended if you want to stay alert to stimuli from outer space while waiting).

 

Continuations

These beasts are more like boomerangs. Just throw them a message threaded to a boomerang and ultimately the answer comes back threaded on the boomerang. For the code-minded out there, here's a sample:

void ThrowIt(string message, Boomerang boomerang);

where the Boomerang class could be defined like

class Boomerang : Action<string> { ... }

Notice the return type of the ThrowIt operation: you don't get anything back. Well, that is, till you get the boomerang against the back of your head some time later when it returns. This just works fine, but thinking of a boomerang as the ultimate mechanism to perform asynchronous work is a little weird at first. In more theoretical terms, you're threading the future (callback, or continuation) to the current (the asynchronous call). Or think of it as the following transformation:

string result = ThrowAndWait("Hello World");
Console.WriteLine(result);

into

ThrowIt("Hello World", result => {
     Console.WriteLine(result);
});

But what's more is the subtlety of the continuation itself, because of the whole plethora of closures implemented by the language. In boomerang terms, you're not just throwing the boomerang but all bacteria on your hand (the caller site environment) travel with it and might come back in some mutated form with the result. It's a bit different from the typical big red warnings trying to protect you from side-effects, since it's nicely disguised with a little "ought to be functional, yeah" lambda-arrow. Just try to reason about this one (and assume that you do know something or do not know anything about how ThrowIt operates):

string message = "Hello World";
ThrowIt(message, result => {
     Console.WriteLine(result);
});
message = "Goodbye World";

There should be a side-effect no matter what: how on earth would a void-returning method do something useful otherwise? In fact, I bet your very first (procedural) program you ever wrote was one of your biggest lies:

static void Main() { ... }

Right, you've just declared something that heats your processor's transistors by passing in nothing and returning nothing. Obviously it get's a bit better when you return an int :-).

 

Futures

After the ancient dinosaur and the today's functional (?) programmer's wet continuation dream, let's switch gears to a more futuristic approach. No more boomerang-style rendez-vous anymore, what about just expressing we want an operation to be carried out, giving us back something somewhere in the future. Or how you go to a store to order something and get a voucher to come and get it when it's ready (or when you decide to just sit and wait because you can't live any longer without it). This is something that get's enabled by the Parallel Extensions to the .NET Framework's TPL library, so consider TPL-inspired syntax below:

Future<string> GetGreetingCard(string to);

which is a function that houses a script monk creating greeting cards. Let's assume our script monk is called String::Concat (not sure the first name is appropriate for a monk though) and submit a request:

var result = GetGreetingCard("World");
// do other shopping
Console.WriteLine(result.Value);

Actually, you get more control over it - you just pass the monk what you want to share but he doesn't get to see anything beyond that. And ultimately when you come back, there's a result that's predictable given what you gave to the monk in the first place (assuming deterministic monks, a rare sort those days). If you just want to sit and wait, you don't do any shopping between sending the request and getting the result back:

Console.WriteLine(GetGreetingCard("World").Value);

Actually, behind the scenes of the monastery there could well be laziness indeed - between receiving a request and carrying out the calligraphic job might well lie lots of secrets hidden behind thick walls. Without further elaboration one could smell the air of a thunk coming up or even SASL style of laziness.

 

Join patterns

If you've had a rendez-vous with Ada in a previous life, you'll be familiar with the concept of join patterns (or shorthand joins). Our Microsoft Research department has done quite some work on this field already, as you can read here: http://research.microsoft.com/~crusso/joins/. It's part of Polyphonic C#, which is by itself part of Cw and based on join calculus. We already got the LINQ inspiration from it, joins might be lurking around the corner too. There actually just two basic concepts to understand here:

  • Asynchronous methods: imagine a keyword async as a modifier on methods (which would imply the method is a procedure, hence the void return type becoming redundant for those type of methods). No longer do you need to create separate versions of a method with or without the Async suffix to work around return type overloading limitations imposed by the runtime. You can think of the method as a wrapper around a task scheduling call, wrapping the entire body. Whether or not this spawns a new thread is another matter.
  • Join patterns: also known as chords, is a declarative way of a WaitAll style of guard mechanism to enter a method.

Our monk would look like:

class Monk
{
     public async RequestCard(string to) { ... }
     public string GetCard() & public async RequestCard(string to) { ... }
}

Basically requests for cards queue up on the monk's GetCard operation, ready to get processed when the monk is awake. Actually this sample is convoluted and our monk suffers from some concurrency diseases (think of the cards it would hand out if there are multiple outstanding requests - any ordering guarantees without "ticketing"?) but I won't get in details for now - it's just (yet) another way of thinking about asychronous processing with today's materials in the room. If you're inspired by join patterns, check out the Polyphonic C# paper.

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

We just released a new CTP for the Parallel Extensions to the .NET Framework. Highly recommended to download immediately obviously. I plan to start a blog series on the technology pretty soon. This is a huge milestone for the Parallel Extensions work (but far from done obviously) with some major highlights like:

  • Improved notion of task hierarchies in the TPL 
  • PLINQ now runs on top of the TPL
  • Revised ordering preservation design for PLINQ (AsOrdered)
  • The new LINQ-style Zip operator
  • Coordination Data Structures (CDS) with 10 publicly available types already, including lock-free collection types

Stay tuned for more in-depth technical posts on the subject. In the meantime, get it installed and running :-).

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

More Posts