September 2005 - Posts

I'm using it for years in Word now and just discovered Visual Studio .NET 2003 supports it to: drag-and-drop while holding the ALT-key. What it does is a so-called "columnar text selection", i.e. selecting a block of text columnwise, not line-based. An example to clarify things. I was documenting a piece of code I'm writing with the following text:

/// GRAPHICAL       DATABASE                           LIMITED
/// REPRESENTATION  STRUCTURE                          QUERY RESULT
/// ==============  ==============================     =========================
/// 
/// Tree            ID     Name        Parent  Top     ID     Name        Parent
/// --------------  -----  ----------  ------  ---     -----  ----------  ------
/// Shop 1              1  Shop 1      (null)    1 -->     1  Shop 1      (null)
/// + Shop 1.1          2  Shop 1.1         1    1 -->     2  Shop 1.1         1
///   + Shop 1.1.1      3  Shop 1.1.1       2    1   +     3  Shop 1.1.1       2
///   + Shop 1.1.2      4  Shop 1.1.2       2    1   +     4  Shop 1.1.2       2
///   + Shop 1.1.3      5  Shop 1.1.3       2    1   +     5  Shop 1.1.3       2
/// + Shop 1.2          6  Shop 1.2         1    0 -->     6  Shop 1.2         1
///   + Shop 1.2.1      7  Shop 1.2.1       6    ?   -
///   + Shop 1.2.2      8  Shop 1.2.2       6    ?   -
/// Shop 2              9  Shop 2      (null)    1 -->     9  Shop 2      (null)
/// + Shop 2.1         10  Shop 2.1         9    0 -->    10  Shop 2.1         9
///   + Shop 2.1.1     11  Shop 2.1.1      10    ?   -
///   + Shop 2.1.2     12  Shop 2.1.2      10    ?   -
///   + Shop 2.1.3     13  Shop 2.1.3      10    ?   -
///   + Shop 2.1.4     14  Shop 2.1.4      10    ?   -
/// + Shop 2.2         15  Shop 2.2         9    1 -->    15  Shop 2.2         9
///   + Shop 2.2.1     16  Shop 2.2.1      15    1   +    16  Shop 2.2.1      15
///   + Shop 2.2.2     17  Shop 2.2.2      15    1   +    17  Shop 2.2.2      15
/// Shop 3             18  Shop 3      (null)    1 -->    18  Shop 3      (null)
/// + Shop 3.1         19  Shop 3.1        18    1 -->    19  Shop 3.1        18
///   + Shop 3.1.1     20  Shop 3.1.1      19    1   +    20  Shop 3.1.1      19
///   + Shop 3.1.2     21  Shop 3.1.2      19    1   +    21  Shop 3.1.2      19
/// + Shop 3.2         22  Shop 3.2        18    0 -->    22  Shop 3.2        18
///   + Shop 3.2.1     23  Shop 3.2.1      22    ?   -
///   + Shop 3.2.2     24  Shop 3.2.2      22    ?   -
///   + Shop 3.2.3     25  Shop 3.2.3      22    ?   -

Then I needed the same comment somewhere else, except for the last column (okay, maybe I should have written the comments in inverse order, so that I could add the last column after copying the existing text). Using ALT + drag mouse, you can select the last columns (indicated in bold over here):

/// GRAPHICAL       DATABASE                           LIMITED
/// REPRESENTATION  STRUCTURE                          QUERY RESULT
/// ==============  ==============================     =========================
/// 
/// Tree            ID     Name        Parent  Top     ID     Name        Parent
/// --------------  -----  ----------  ------  ---     -----  ----------  ------
/// Shop 1              1  Shop 1      (null)    1 -->     1  Shop 1      (null)
/// + Shop 1.1          2  Shop 1.1         1    1 -->     2  Shop 1.1         1
///   + Shop 1.1.1      3  Shop 1.1.1       2    1   +     3  Shop 1.1.1       2
///   + Shop 1.1.2      4  Shop 1.1.2       2    1   +     4  Shop 1.1.2       2
///   + Shop 1.1.3      5  Shop 1.1.3       2    1   +     5  Shop 1.1.3       2
/// + Shop 1.2          6  Shop 1.2         1    0 -->     6  Shop 1.2         1
///   + Shop 1.2.1      7  Shop 1.2.1       6    ?   -
///   + Shop 1.2.2      8  Shop 1.2.2       6    ?   -
/// Shop 2              9  Shop 2      (null)    1 -->     9  Shop 2      (null)
/// + Shop 2.1         10  Shop 2.1         9    0 -->    10  Shop 2.1         9
///   + Shop 2.1.1     11  Shop 2.1.1      10    ?   -
///   + Shop 2.1.2     12  Shop 2.1.2      10    ?   -
///   + Shop 2.1.3     13  Shop 2.1.3      10    ?   -
///   + Shop 2.1.4     14  Shop 2.1.4      10    ?   -
/// + Shop 2.2         15  Shop 2.2         9    1 -->    15  Shop 2.2         9
///   + Shop 2.2.1     16  Shop 2.2.1      15    1   +    16  Shop 2.2.1      15
///   + Shop 2.2.2     17  Shop 2.2.2      15    1   +    17  Shop 2.2.2      15
/// Shop 3             18  Shop 3      (null)    1 -->    18  Shop 3      (null)
/// + Shop 3.1         19  Shop 3.1        18    1 -->    19  Shop 3.1        18
///   + Shop 3.1.1     20  Shop 3.1.1      19    1   +    20  Shop 3.1.1      19
///   + Shop 3.1.2     21  Shop 3.1.2      19    1   +    21  Shop 3.1.2      19
/// + Shop 3.2         22  Shop 3.2        18    0 -->    22  Shop 3.2        18
///   + Shop 3.2.1     23  Shop 3.2.1      22    ?   -
///   + Shop 3.2.2     24  Shop 3.2.2      22    ?   -
///   + Shop 3.2.3     25  Shop 3.2.3      22    ?   -

and press delete to get rid of it. In Word, you can do the same (works - logically - best on non-proportional text fonts) e.g. to change the formatting of a column. I'm always surprised to hear that people don't know this Word trick, but as far as I can remember I got the trick from putting the "tip of the day" on in Word 6.0 (if the discussed feature was already in that version) or so a long time ago. Today was the first time I had a need for the same functionality in Visual Studio and I was so happy to see it's in there too :-).

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

I believe this must be the first acronym-only title on my blog. Don't expect to find any technical stuff in this post, but as the matter in fact, this post is a pretty important one for myself as it reflects the lion's part of my activities for the upcoming months (maybe years). As some of you might know, I've graduated from the University of Ghent in June this year (summa cum laude, see this blog post) and I'm planning to continue my studies for another two years at the same university. Because of this decision, some things will change:

  • First of all, during the week I'll stay in Ghent in a student's room, mainly to save on travelling time every day (and to reduce the pain of early mornings ;-)). I have no intentions to leave the university city for "urgent matters" (read: schedule penetrants). Only events that have been planned/announced upfront make a chance to get their way into my schedule, e.g. conferences and technical events.
  • Second, I'm allocating a fixed amount of time for technical community stuff, i.e. maintaining my blog, writing articles, doing posts. Up till now, I did these activities on an irregular basis, causing bursts of high activity followed by idle time followed by ... etc. Living a more regular life seems to be attractive to me, so I thought: "let's give it a try" :-).
  • Third, I won't accept any development projects that fall outside my scope of personal interest although I'll continue to work on some personal development projects that have started as fun and are a great experimental sandbox for me. I also intend to start working on a pretty large personal (computer-related) project that has been planned for a long time now and which is in a conceptual state for the moment being. Readers of my blog will be the first to hear about it.
  • Fourth, beside of my studies I'll spend most of my time reading and studying books in depth. This is where the acronym in this blog post's title comes from: TAOCP = The Art of Computer Programming, Donald Knuth's master piece. Since a couple of weeks, I have volumes 1 to 3 on my book shelf at home. In October, I'll start with volume 1 (reading + exercises) which I hope to finish by June next year. Donald Knuth started to write his books back in 1962, with an "RTM" in 1997 and the 18th printing in February 2005. I hope it won't take me so much time to read and understand it :o.
  • Fifth, in order to reach the goals aforementioned, I'm planning to reduce my permanent availability and reachability by processing e-mails in batch mode, once a week or so. A few months ago, I read an (in my opinion interesting) article in the newspaper about scientific research of the possible negative impact of having an e-mail (or IM) application in your near neighborhood on intelligence and concentration. I'm calling it: "running your mind in a preemptive world", ready for a context switch between your current brain activity and reading/answering e-mail or IM messages. Reading the following statement on Donald Knuth's website

    "Email is a wonderful thing for people whose role in life is to be on top of things. But not for me; my role is to be on the bottom of things."

    is an accurate reflection of what's in my mind for the moment. Building software nowadays is all about building stacks, developing things on top of existing technologies and frameworks. For me, learning another SDK or API is boring, especially when it's a technology that interests me less (thus, I'll still be focusing on certain areas for community-related work, especially CLR, C#, SQL, Indigo and WinFS). The main question however is: what makes you an expert on the field of computer science? Getting to know yet another SDK or trying to find out how things work at the very bottom line? My answer is the latter one and luckily I'm still in a position where I can dedicate time to this stuff, so I have to take advantage of my remaining two years as a student.
  • And last but not least, I'm planning to try to get more sleep at night :-). The bare minimum will be set to five hours a day, but chances are high I won't keep this last promise :-).

One last thing for now: Friday morning I'm leaving to the PDC in LA with other Belgian and EMEA community folks. Keep on reading my blog for more stuff around the PDC and much more.

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

A couple of minutes ago I had to reset the password for a bulk series of demo accounts on a web server for web design courses. Some time ago we created 99 accounts in Active Directory which have - using FTP User Isolation - access to FTP as well as FrontPage Server Extensions. The idea is pretty simple: every user has a name "web%%" (web01 - web99) and a password "COURSEweb%%" (COURSEweb01 - COURSEweb99). However, the accounts of last year did not work anymore because the password was expired. So, I had to write some app to reset the passwords in bulk. Initially I thought of using System.DirectoryServices with C# but I gave the command-line a try. With success! Here is the result:

FOR /L %%I IN (1,1,9) DO dsmod user "CN=web0%%I,OU=Course,OU=Web Users,DC=mydomain,DC=local" -pwd COURSEweb0%%I -pwdneverexpires yes

FOR /L %%I IN (10,1,99) DO dsmod user "CN=web%%I,OU=Course,OU=Web Users,DC=mydomain,DC=local" -pwd COURSEweb%%I -pwdneverexpires yes

Note: I'm using two percent signs to refer to the variable because I need to store these two commands in a .bat file. If you run it directly from the command prompt, drop the extra percent sign in front of the variable %I.

As asked by the course instructor, I set the password to "never expire" to avoid future problems when he has to teach a course again and I'm not available. But don't do this with non-locked down accounts. The accounts I'm managing in here are locked down and can only be used through FTP and FPSE, with disk quota enforced and no rights on other folders.

Challenge for readers: If you find an elegant way to reduce both lines to just one line with 0-filling at the beginning of the account name and password, let me know. A one-line solution would be much cuter :-).

Another interesting script I deliver is one to disable/enable the accounts after/before lesson sessions:

FOR /L %%I IN (1,1,9) DO dsmod user "CN=web0%%I,OU=Course,OU=Web Users,DC=mydomain,DC=local" -disabled yes

FOR /L %%I IN (10,1,99) DO dsmod user "CN=web%%I,OU=Course,OU=Web Users,DC=mydomain,DC=local" -disabled yes

I guess you can infer the code to enable accounts yourself :-).

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

Introduction

I'm running it for a couple of weeks now and I must say ... it's great. Although I'm not a fan of the most requested IE7 feature, being tabbed browsing, there's so much else - especially on the field of security - that turns me on :-). It takes a while to get used to the new look-and-feel but I must admit I like it in the end.

 

What's new in IE7?

An overview of the Internet Explorer 7 Beta 1 features can be found over here in a whitepaper. As you might expect, IE7 is based on Windows XP SP2's IE6 and works further on the improvements made in there (such as pop-up blocking, add-on management, etc). The most interesting features of IE7 are listed below:

  • A redesigned interface, including tab-based browsing.
  • Improved support for CSS and PNG (see blog post).
  • Support for RSS feeds (see blog post).
  • Direct access to a search engine of your choice (see blog post).
  • Shrink-to-fit web page printing (see blog post).
  • Better protection against malware and phishing (see blog post).
  • URLs in IE7 are now handled based on "CURI", resulting in a "normal form" for URLs (see blog post).

Please notice the following (source whitepaper):

The beta 1 release of Internet Explorer 7 includes a subset of all the new features that will be delivered in the final version of Internet Explorer 7. This release is intended to enable developers to begin to test the new browser for compatibility with their applications and Web sites. Therefore, many of the important changes in beta 1 are at a “plumbing level” of Internet Explorer and therefore will not be as evident to end users.

...the challenge of striking the right balance between security and compatibility: As security is tightened, compatibility and extensibility tend to suffer. With Internet Explorer 7, Microsoft is working hard to ensure this balance is met effectively to ensure the best possible browsing experience for end users.

 

Some personal experiences

RSS feeds

Because I'm reading blogs pretty frequently, blog readers such as SharpReader are welcome tools for me. However, the capability of a browser to detect RSS feeds and report these is a great addition. In the image below, you can see a RSS feed displayed in IE7:

 

Phishing

Today I was the phishing protection in IE7 in action for the first time over here on my machine. In the screenshots below you can see how it works. The IE team blogs about it over here. General information on anti-phishing can be found on http://www.microsoft.com/mscorp/safety/technologies/antiphishing/overview.mspx. Technical details on the phishing filter are on MSDN.

 

No Add-ons Mode

Windows has a safe mode since Windows 9x AFAIK. IE7 introduces a similar concept called "No Add-ons Mode", a special mode in which no add-ons are loaded by the browser. It can be launched from the Accessories, System Tools menu or by adding the flag -extoff to the iexplore.exe executable when launching the browser.

 

Protected Mode

Protected Mode (formerly known as "Low Rights IE") is available on Vista only. Basically, it puts Internet Explorer in a sandbox, defending against "elevation of privileges" attacks. Even when IE7 is running in the context of an Administrator account (or another highly privileged account) it can't touch system files or the registry. More information on this feature will become available in the future when Vista Beta 2 with IE7 arrives, which will include this technology. Technical details can be found in the IE7 Beta 1 Technical Overview document mentioned earlier.

 

Interesting links

Check out the following:

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

Introduction

The Indigo Windows Communication Foundation wave is around for a couple of years now, so I thought it was about time to dedicate some blog posts to this technology. In this first technical development-related post on the Indigo technology (I'll use the code-name further on instead of the new WCF name) we'll cover the basis of the technology and service orientation and will go through some simple low-level samples.

 

Service Orientation

Indigo is all about services. Not web services or Windows services but services. With the advent of the internet several decades ago, the need to have more distributed software is growing day by day. Service Orientation (abbreviated as - originally - SO) is a methodology for software design and development that puts connectivity of software components in the center of the universe, building software on top of services that interact with each other to fulfill business-related needs. An SO solution consists of a series of service that serve as building blocks for an application. Indigo is Microsoft's translation of this Service Oriented software design/development wave, enabling developers to write and build services also concentrating on the communication aspect for services and clients.

Note: You should not think of SO as a replacement for other development paradigms such as object-orientation (OO) or aspect orientation (AOP). Services is about creating facades that encapsulate the inner implementation of a service and expose the functionality of the service to the outside world. The way these services themselves are created can be OO or whatever paradigm you like to use.

SO (or SOA = Software Oriented Architecture) is based on a total of four tenets as proposed by Don Box:

  1. Explicitness of boundaries meaning that messages are sent between services across formal and explicit boundaries. Look at these boundaries as a kind of interfaces, hiding what's behind the boundary itself. Because of the explicitness, crossing a boundary is explicitly visible in code and saves you from boobytraps that are present in technologies such as DCOM, .NET Remoting, RMI, CORBA that make remote objects look as local objects.
  2. Autonomy of services makes it possible to update services independently from each other, without having to redeploy the entire solution as a whole. Loosely coupling is the keyword that makes this possible.
  3. Services share schemas and contracts. Services are not sharing types and classes but schemas and contracts instead. A schema defines data that is sent between endpoints, whileas a contract defines behavior of the service. In web services parlance, a schema can be built in XSD and a contract in WSDL. The use of schemas and contracts ensures more stability (= remain unchanged over time) than their types and classes equivalents in the distributed world.
  4. Compatibility based upon policy states that services agree on how they interact based on policies, describing the capabilities and requirements of the service (e.g. for communication with the service, encryption of data is required). Because of this, the operations (behavior, contract) is separated from contraints that should be enforced on these operations.

Mike Gilbert has a nice post on SOA over here. In the end, services can be seen as "super web services", meaning that limitations of web services are removed in the SO world. Samples include the strong binding to the HTTP protocol and the request/response model but also the removal of limitations on the field of reliability, transactions, policy expression, etc.

 

Indigo from a 10,000 foot view

Simply stated: SO is the specification, Indigo is Microsoft's implementation of SO. To kick off, we'll jump into some marketing talk on Indigo:

  • Indigo is a .NET-based technology to build service-oriented solutions. Exactly what we've been describing above.
  • Indigo provides unification of both the programming model and the runtime. In the past, several technologies existed to create distributed applications. An overview:
    • DCOM and .NET Remoting concentrate on remote objects and suffer from the "remote objects are transparent to the client" boobytrap. .NET Remoting is all about communication across application domains and is very extensible (channels, sinks, etc). The technology is fully object-oriented and works with various transport mechnanisms and protocols (e.g. binary of TCP, SOAP over HTTP, etc).
    • ASP.NET XML Web Services (.asmx) is Microsoft's first implementation of webservices on the .NET platform. As part of .NET's message, XML Web Services provide interoperability with other platforms and offers a service-oriented design. However, it lacks built-in support for new WS-* standards and can only work with HTTP.
    • Web Service Enhancements (WSE) extend the .asmx model and provide support for WS-* standards and other protocols and communication patterns than the plain vanilla HTTP response/request model. WSE is fully .NET-based and uses the power of managed code to deliver its functionality (e.g. attribute-based development). However, it lacks support for transactions, reliable messaging, etc.
    • MSMQ and System.Messaging concentrate on reliable messaging with queues as the communication mechanism. System.Messaging makes the use of MSMQ easier in the managed world. However, MSMQ is not available to services directly and is only usable in .NET-to-.NET scenarios.
    • COM+ Enterprise Services and System.EnterpriseServices are based on the notion of components that are decorated with .NET attributes to provide several services such as security and transactions. COM+ is component-based and built on the OO paradigm. However, it's a .NET-only story and lacks interoperability.
  • Indigo is scale-invariant. In more detail:
    • Scales down to devices such as printers, cameras, etc with support for WS protocols. On longer term, you can expect to see Indigo support on smart devices using the Compact Framework.
    • Scales in meaning that inter-process (or inter-appdomain) communication is damn fast (greater than or equal to .NET Remoting and/or Enterprise Services).
    • Scales up on large systems with high throughput and scalability which is mission-critical to systems with high workload and tons of consumers. Compared to existing technologies, Indigo wants to be superior.
    • Scales out by adding machines, e.g. in farms or on the routing-level (cf. WS-Routing and WS-Addressing). Information on these protocols applied to WSE 2.0 can be found on MSDN.
    • Scales away across organizations and geographies, referring to the use of interoperable protocols, reliable messaging and transmission fault tolerance and federation of security (cf. the former "TrustBridge" project and ADFS).
  • Indigo puts interoperability in the spotlight. Examples include:

Note: Also search on MSDN for more information about all of the aforementioned technologies and specifications.

 

Basic architecture of Indigo

Time for some illustrations. The picture below shows the basic layered architecture of Indigo.

Let's go through this picture from top to bottom:

  • At the uppermost layer, we have our application (a service) that's using the Indigo stack below.
  • Next, we have a series of messaging services that provide functionality to the application above. Samples include queuing, routing, eventing and discovery of services.
  • The service model provides the basic service-related functionality. This layer is all about behavior, such as error behavior, instance behavior, activation behavior, concurrency and transactions, etc.
  • The messaging layer delivers all functionality to the service model to create channels and to transport the messages between services. It also provides "side-services" such as security, reliability, policy and encoding. For transport channels such as HTTP, TCP, MSMQ, cross-process are supported.
  • Last but not least, the Indigo application is hosted in e.g. IIS, Avalon, an executable, a Windows Service or COM+.

I'll dive deeper into these in more detail further on and in upcoming posts on Indigo.

 

Some crucial concepts

Right, we know what Indigo does and we know the basic layered architecture. Time for some concepts that will be crucial for further understanding of Indigo stuff. Concentrate on the following simple sentence:

Services interact by exchanging messages.

This simple statement implies a lot of questions. Some samples: How are messages exchanged physically? Who send and who recieves messages? How does such a message look like? Are there different interaction patterns possible? Now the answers:

  • The key players in messaging are clients that initiate communication, the services that accept these messages and respond to them and possible intermediaries that route messages to their destination and inspect/manipulate the messages optionally. Notice that services themselves can also act as clients, resulting in a chain of services that are linked to each other.
  • The message structure in Indigo is based on SOAP, which is on its turn XML-based. From a macroscopic point of view, a message contains a series of headers as well as a body. Together, this forms a so-called SOAP Envelope. However, the encoding of the message and the transport protocols are configurable.
  • Channels are the pipelines over which information flows. The function of channels is twofold. First, channels are used for physical delivery of messages over various transport protocols such as HTTP, TCP, MSMQ and cross-proc, also providing reliability and/or session services if desired. Second, messages are manipulated by channels for encoding (binary, MTOM, etc) and decoration of messages to provide security features (e.g. Windows authentication).
  • Various interaction patterns exist in the world of Indigo. Starting with one-way simplex communication, over two-way duplex communication to request/reply based patterns.

Now look at a service as a black box. What should the service expose? Two seconds left for the answer :-). Right, first of all we need a description of the service because of the loosely coupling. This means it should be possible to discover information about what the service provides (behavior, operations, policy) and how to interact with it (message contracts). The discovery and description aspect of services is accomplished by WSDL, XSD, WS-Policy and WS-Metadata Exchange. Getting to know each other is one thing, working with each other is another thing. So, we also need endpoints for communication (as described by the obtained description) using SOAP.

Apart from the service's logic there should be other things inside a service as well. First, it needs to be able to provide its metadata to describe itself and the message contracts. That's the service description part of the inside story. Next, the consumer of the service needs to be able to get in through an endpoint. Therefore, we need a description of the way the service interacts with the outside world, which is called a binding. Such a binding encapsulates the transport mechanism (e.g. HTTP, TCP, named pipe, MSMQ), encoding (text, binary, MTOM) and requirements on the field of security, sessions, reliablity and transactions as well as available messaging patterns. The endpoint on the other hand is all about where the service is (an address) associated to a service contract and a binding. This brings is us the missing piece of the puzzle, being the contracts. Indigo knows three kinds of contracts:

  • Service contracts define the operations the service provides and its behavior. Compare such a contract with an interface from the OO world and you should have a pretty good understanding of such a contract. Of course, the service needs to implement the control to be useful (a class).
  • Data contracts specify data structures that are used to pass data to/from the service. The big idea of data contracts is to decouple internal class representations from the messaging format, which can be described in XSD for interoperability reasons and loose coupling.
  • Message contracts allow to manipulate the message structure and format of messages that are sent between services and consumers.

A little illustration can make things more clear I believe. Assume you have a service contract called IWeatherService that allows to report the weather using an operation that accepts a location (postal code, state, country in a data contract) and returns a weather report (temperature, air pressure in another data contract). The functionality of the service can be exposed through an endpoint consisting of an address (http://blabla/WeatherService or net.tcp://blabla:port/WeatherService or net.pipe://blabla/WeatherService or net.msmq://localhost/queue/WeatherService) and a binding mapping the address to the contract and enriching it with e.g. security. Thus, as a little rule of the Indigo thumb, remember the following:

endpoint = address + binding + contract

Services expose functionality but also have intrinsic runtime behavior. Indigo provides support for the following:

  • Concurrency - If multiple consumers use your service, how does it work with threading? Possible answers are: one single thread can access the service instance, multiple threads can access the service instance or reentrancy support meaning that threads can call into one another.
  • Error handling - Who will handle errors in the service? The developer himself, the Indigo framework or the client of the service (i.e. sending a fault to the client).
  • Instancing - People who've been working with .NET Remoting will know concepts such as singleton and singlecall. Indigo supports four mode: singleton (one instance serving all consumers), per call (used in stateless scenarios, such as classic web services), private session (each client session has its own service instance) and shared session (groups of clients can share a single session).
  • Lifetime - When using stateful sessions (e.g. a private session), there needs to be a session start en session end 'event'. The service itself can control the lifetime of the session if operations on the service are marked as session initiators and/or session terminators.
  • Metadata - This answers the question: "how can the service present itself to (possible) consumers?". By enabling metadata to be requested on demand through a request, a service is made self-describing using WSDL, WS-Policy, WS-Metadata Exchange.
  • Security - Service security is composed of various aspects, including message integrity and confidentiality, user authentication and authorization, DoS and replay (attack) detection, auditing. Indigo has support for PKI using X.509 but also supports Kerberos, plain old username/password mechanisms and more recent standards such as SAML.
  • Throttling - Used to limit the total number of service instances, concurrent threads, exchanged or accepted messages, etc. Using throttling, one can control resource consumption of the service, aiding with scalability measurement and planning.
  • Transactions - Indigo has support for transactions across services using WS-Atomic Transaction and other WS-* standards. This allows clients to start a transaction and flow it to one or more services that can then participate in that transaction. The service behavior for transactions is used to indicate whether the service will accept a transaction, requires a (new) transaction, etc.

 

Getting started

Enough theory for now, let's jump onto the Indigo wagon with a little sample. But first, some information on how to set up the stuff needed to bring our sample to a good end.

Download each of the following on either Windows XP (+ SP2) or Windows Server 2003 (+ SP1):

  1. .NET Framework Version 2.0 Redistributable Package Beta 2 (x86) - the .NET runtime on which Indigo is built
  2. Visual Studio 2005 Beta 2 - can be obtained via beta programmes (e.g. Microsoft EMEA's TheBetaExperience) or on MSDN Subscriptions
  3. "Indigo" Beta 1 MSMQ Support Package - needed for MSMQ support for queued channels (only needed for beta 1 currently)
  4. COM+ Hotfix for "Indigo" - contains WS-AtomicTransaction support for MSDTC and allows COM+ apps to be exposed using Indigo
  5. Microsoft Pre-Release Software WinFX Runtime Components Beta 1 - package with beta 1 of both Indigo (Windows Communication Foundation) and Avalon (Windows Presentation Foundation)
  6. Microsoft WinFX Software Development Kit for Microsoft Pre-Release Windows Operating System Code-Named "Longhorn", Beta 1 Web Setup - WinFX SDK with documentation and samples for Indigo and Avalon development (despite the naming of the download also supported on Windows XP and Windows Server 2003)
  7. Microsoft Visual Studio Extensions for WinFX Beta 1 - Visual Studio 2005 Beta 2 support for WinFX components and the WinFX SDK documentation

and install each of these in ascending order.

 

Your first Indigo application

As I don't want to be banned from the demo writing community I should start with the mandatory "Hello World" demo :-). The following instructions will guide you through the process of creating both an Indigo service and client:

  1. Start Visual Studio 2005 and create a new Project in C# and choose a Console Application as the template. Call it IndigoDemo.
  2. In the Solution Explorer, right-click on References and choose "Add Reference...". Select System.ServiceModel from the list and click OK.
  3. Add the following namespace import to the top of the Program.cs file:

    using System.ServiceModel;

  4. In the IndigoDemo namespace, we'll first create the service contract. We'll choose for a contract named IHello with two members, SendMeGreeting and HelloIndigo, as shown below. Notice the use of an interface and the ServiceContract and OperationContract attributes:

    [ServiceContract]
    public interface
    IHello
    {
        [
    OperationContract
    ]
        string SendMeGreetings(string
    name);
        [
    OperationContract
    ]
        string
    HelloIndigo();
    }

  5. To the same IndigoDemo namespace, add a class called HelloService that implements IHello, as shown below:

    public class HelloService : IHello
    {
        #region
    IHello Members

        public string SendMeGreetings(string
    name)
        {
            return string.Format("Hello {0}"
    , name);
        }

        public string
    HelloIndigo()
        {
           
    return "Hello Indigo"
    ;
        }

       
    #endregion
    }

  6. Now it's time to create the "service host" that will run the service. ServiceHost is the class you need to do this. The following Main method contains the Indigo hosting code for our service:

    static void Main(string[] args)
    {
         //
         //Host the service in a simple console application
         //
         using (ServiceHost<HelloService> svc = new ServiceHost<HelloService
    >())
         {
              //
              //Add endpoint for the IHello contract, with a default WS-Profile binding on the specified address
              //
             
    svc.AddEndpoint(
                  
    typeof(IHello
    ),
                  
    new WSProfileBinding
    (),
                  
    "http://localhost:1234/HelloService"
             
    );

             
    //
             
    //Launch the service
             
    //
             
    svc.Open();

             
    Console.WriteLine("Service is running. Press <ENTER> to stop."
    );
             
    Console
    .ReadLine();

             
    //
             
    //Stop the service
             
    //
             
    svc.Close();
         }
    }

    In this code, we create a (generic) instance of a ServiceHost for our HelloService. Next, an endpoint is added, which consists of an address, a binding and a service contract, being the interface IHello which was annotated with the ServiceContract tag earlier. Starting the service is as simply as calling Open, and stopping it can be done using Close.

  7. So far so good. The service is ready, now we'll jump to the client-side piece of our demo. In the Visual Studio 2005 IndigoDemo solution, add a new C# Console Application project called IndigoDemoClient.

  8. In the Solution Explorer under IndigoDemoClient, right-click on References and choose "Add Reference...". Select System.ServiceModel from the list and click OK.

  9. In order to make our client aware of the service contract of our service, we need to copy the IHello interface over here as well. We'll see a sample with discovery later on. For now, just copy the IHello interface to the IndigoDemoClient namespace:

    [ServiceContract]
    public interface
    IHello
    {
        [
    OperationContract
    ]
        string SendMeGreetings(string
    name);
        [
    OperationContract
    ]
        string
    HelloIndigo();
    }

  10. Time for the client. As you might expect, we need a proxy. How to obtain that proxy? Using the generic ChannelFactory.CreateChannel static factory method, as shown below:

    static void Main(string[] args)
    {
       
    //
       
    //Ask the user for input
       
    //
       
    Console.WriteLine("Client is running. Enter your name and press <ENTER> to talk to the service."
    );
       
    string name = Console
    .ReadLine();

        //
        //Create a proxy for the service
        //
        IHello proxy = ChannelFactory.CreateChannel<IHello
    >(
           
    new Uri("http://localhost:1234/HelloService"
    ),
           
    new WSProfileBinding
    ()
        );

        //
        //Call the service to send a greeting
       
    //
        string
    greeting = proxy.SendMeGreetings(name);

       
    //
       
    //Housekeeping
       
    //
       
    ((IChannel
    ) proxy).Close();
        ((
    IChannel
    ) proxy).Dispose();

       
    //
       
    //Output to the user
        //
       
    Console.WriteLine("The server answered: {0}"
    , greeting);
       
    Console.WriteLine("Press <ENTER> to quit."
    );
       
    Console
    .ReadLine();
    }


    This code should be pretty self-explanatory in my very opinion. As you can see, the use of generics makes creation of a proxy easier but also introduces some plumbing (see the housekeeping part). You can avoid this plumbing however by applying the following trick to IHello:

    [ServiceContract]
    public interface
    IHello : IChannel
    {
        [
    OperationContract
    ]
        string SendMeGreetings(string
    name);
        [
    OperationContract
    ]
        string
    HelloIndigo();
    }


    and changing Main like this:

    static void Main(string[] args)
    {
        //
        //Ask the user for input
        //
        Console.WriteLine("Client is running. Enter your name and press <ENTER> to talk to the service."
    );
        string name = Console
    .ReadLine();

        //
        //Placeholder for the answer
        //
        string
    greeting;

        //
        //Create a proxy for the service
        //
        using (IHello proxy = ChannelFactory.CreateChannel<IHello
    >(
            new Uri("http://localhost:1234/HelloService"
    ),
            new WSProfileBinding
    ()))
        {
            //
            //Call the service to send a greeting
            //
            greeting = proxy.SendMeGreetings(name);

            //
            //Close connection to the service
            //
            proxy.Close();
        }

        //
        //Output to the user
        //
        Console.WriteLine("The server answered: {0}"
    , greeting);
        Console.WriteLine("Press <ENTER> to quit."
    );
        Console
    .ReadLine();
    }

  11. Time to run it. Right-click the IndigoDemo solution on top of the Solution Explorer and choose Properties. In Common Properties, Startup Project select Multiple startup projects and set the action for both the IndigoDemo and IndigoDemoClient application to 'Start'. Click OK.

  12. Now hit F5 to run the application. Once the service console prints "Service is running. Press <ENTER> to stop." switch to the client console and type you name. Output similar to the following should be displayed:

    Client is running. Enter your name and press <ENTER> to talk to the service.
    Bart
    The server answered: Hello Bart
    Press <ENTER> to quit.

  13. Exit the client by pressing <ENTER> but leave the service host console window running.

In this first sample you saw how to create both a service and a client without the use of a configuration file, just by writing plain code. I chose to use the WSProfileBinding which is well-suited for HTTP (or HTTPS) services with the well-known Request-Response (or Simplex) message pattern style. One of the things we saw was how to create a proxy using the ChannelFactory class of Indigo. Now I want to take you to the next step of discovering a running service:

  1. Assuming you followed the steps above correctly, the service should still be running now. If it's not, restart the Visual Studio 2005 debugger using F5 and wait for the service to come alive.
  2. Open an instance of Internet Explorer and go to http://localhost:1234/HelloService. You should see a page "Indigo Service" that tells you "You have created an Indigo service." If you didn't realize this yet, it's time to awake now :-). The page also contains a link to the well-known WSDL discovery address on http://localhost:1234/HelloService?wsdl, just like this was the case with classic web services.
  3. Indigo provides a command-line utility called svcutil.exe which is accessible through the Visual Studio 2005 Command Prompt. Call svcutil.exe /? to see the functionality it provides:

    C:\temp>svcutil
    Microsoft (R) Service Model Metadata Tool
    [Microsoft(R) .NET Framework, Version 2.0.50215.0]
    Copyright (C) Microsoft Corporation. All rights reserved.

    svcutil.exe: Utility to generate ServiceModel code from Metadata documents, and
    vice versa.

        - USAGE -
    svcutil.exe [options] [metadataPath* | assemblyPath* | metadataUrl*]


    svcutil.exe differs from wsdl.exe on quite some fields. One notable difference is the support to work in two ways: metadata-to-assembly and assembly-to-metadata.
  4. Time to experiment with it. Execute svcutil.exe http://localhost:1234/HelloService?wsdl as mentioned on the web page from step 2. The result should look like this:

    C:\temp>svcutil.exe http://localhost:1234/HelloService?wsdl
    Microsoft (R) Service Model Metadata Tool
    [Microsoft(R) .NET Framework, Version 2.0.50215.0]
    Copyright (C) Microsoft Corporation. All rights reserved.

    Generating files...
    C:\temp\tempuri.org.Bindings.cs
    C:\temp\output.config


    Two files have been generated. Take a look at both of these using Notepad
    .
  5. In the first file, the tempuri.org.Bindings.cs file, you should find three members. First of all, there's a IHello interface that looks the same as the original one you wrote yourself in the service's code. The second interface is the IHelloChannel which derives from IHello and IProxyChannel. And last the most important class, HelloProxy. This class both derives from the IHello contract interface and the ProxyBase generic class. Notice that the address of the endpoint is not in the code, in contrast to the output of wsdl.exe in the (near) past. This variable information lives in the output.config file.
  6. Open the output.config file and notice the following sections (marked in red):

    <?xml version="1.0" encoding="utf-8"?>
    <configuration xmlns="
    http://schemas.microsoft.com/.NetConfiguration/v2.0">
        <system.serviceModel>
            <client>
                <endpoint address="
    http://localhost:1234/HelloService" bindingConfiguration="IHello"
                    bindingSectionName="customBinding" contractType="IHello">
                    <addressProperties actingAs="
    http://localhost:1234/HelloService"
                        identityData="" identityType="None" isAddressPrivate="false" />
                </endpoint>
            </client>
            <bindings>
                <customBinding>
                    <binding configurationName="IHello">
                        <contextFlow transactions="Ignore" transactionHeaderFormat="OleTx"
                            logicalThreadId="Ignore" locale="Ignore" />
                        <security algorithmSuite="Default" authenticationMode="SspiNegotiated"
                            contextMode="Session" defaultProtectionLevel="EncryptAndSign"
                            enableKeyDerivation="true" includeTimestamp="true" messageProtectionOrder="SignBeforeEncrypt"
                            securityVersion="WSSecurityXXX2005" secureConversationVersion="WSSecureConversationFeb2005"
                            trustVersion="WSTrustFeb2005" generateRequestSignatureConfirmation="false" />
                        <httpTransport manualAddressing="false" maxMessageSize="65536"
                            authenticationScheme="Anonymous" bypassProxyOnLocal="false"
                            hostnameComparisonMode="StrongWildcard" mapAddressingHeadersToHttpHeaders="false"
                            proxyAuthenticationScheme="Anonymous" realm="" transferTimeout="00:01:00"
                            useSystemWebProxy="true" />
                        <textMessageEncoding maxReadPoolSize="64" maxWritePoolSize="16"
                            messageVersion="Default" encoding="utf-8" />
                    </binding>
                </customBinding>
            </bindings>
        </system.serviceModel>
    </configuration>


    All of the marked sections should be easy to understand right now. There's an endpoint which consist of - remember? - an address, a binding and a contract. Remember that these three go hand-in-hand all the time. I won't cover the other stuff of the config file for now, but most of the generated stuff is related to the WS-Profile binding that was specified concerning security, sessions, transport protocol and encoding.
  7. Let's create a simple client now, at the command line. In the folder where the tempuri.org.Bindings.cs file was dropped, create a new file called demo.cs and add the following code to it:

    class Test
    {
        static void Main(string[] args)
        {
            HelloProxy proxy = new HelloProxy();
            string result = proxy.SendMeGreetings(args[0]);
            System.Console.WriteLine("The Hello service returned '" + result + "'");
            proxy.Close();
        }
    }

  8. Now compile the tempuri.org.Bindings.cs file first, as follows:

    C:\temp>csc -r:System.ServiceModel.dll -t:library tempuri.org.Bindings.cs
    Microsoft (R) Visual C# 2005 Compiler version 8.00.50215.44
    for Microsoft (R) Windows (R) 2005 Framework version 2.0.50215
    Copyright (C) Microsoft Corporation 2001-2005. All rights reserved.
  9. Next, build the demo.cs file, referencing both the System.ServiceModel.dll and tempuri.org.Bindings.dll assemblies:

    C:\temp>csc -r:tempuri.org.Bindings.dll,System.ServiceModel.dll demo.cs
    Microsoft (R) Visual C# 2005 Compiler version 8.00.50215.44
    for Microsoft (R) Windows (R) 2005 Framework version 2.0.50215
    Copyright (C) Microsoft Corporation 2001-2005. All rights reserved.
  10. Last but not least, rename output.config to demo.exe.config:

    ren output.config demo.exe.config
  11. Now run demo.exe with one parameter, specifying your name:

    C:\temp>demo Bart
    The Hello service returned 'Hello Bart'

Works great. So far, we've been hosting our service in a console application. However, you can host the service in IIS as well (or in other hosts). Let me show how (assuming you have IIS 5.1 or 6.0 up and running and you installed .NET v2.0 and WinFX afterwards):

  1. Stop your Visual Studio 2005 debugger if it's still running. Add a C# Type Library project to the solution and give it the name "IndigoDemoLibrary".
  2. In the Solution Explorer under IndigoDemoLibrary, right-click on References and choose "Add Reference...". Select System.ServiceModel from the list and click OK.

  3. Add the IHello interface and the HelloService class of the earlier IndigoDemo service project to the Class1.cs file. Save the file and compile the solution.

  4. Open Windows Explorer and create a folder somewhere on your system. I'll be using c:\temp\HelloService. In this new folder, create a subfolder called bin.

  5. Open the bin-folder of the IndigoDemoLibrary project, in my case C:\Documents and Settings\...\My Documents\Visual Studio 2005\Projects\IndigoDemo\IndigoDemoLibrary\bin\Debug, and copy the IndigoDemoLibrary.dll file to the bin folder (c:\temp\HelloService\bin) created in step 4 above.

  6. In the (root) folder created in step 4 above (c:\temp\HelloService), create a file called service.svc, with the following code:

    <%@Service Language="C#" class="IndigoDemoLibrary.HelloService" %>
    <%@Assembly Name="IndigoDemoLibrary" %>

  7. In the same folder, create a web.config file containing:

    <?xml version="1.0" encoding="utf-8" ?>
    <configuration xmlns="
    http://schemas.microsoft.com/.NetConfiguration/v2.0">
        <system.serviceModel>
            <services>
                <service serviceType="IndigoDemoLibrary.HelloService">
                    <endpoint address="" bindingSectionName="wsProfileBinding" contractType="IndigoDemoLibrary.IHello, IndigoDemoLibrary" />
                </service>
            </services>
        </system.serviceModel>
    </configuration>

  8. Now it's time to configure the Indigo service virtual directory in IIS. Go to Start, Run, enter "inetmgr" (without quotes) and click OK. Locate the "Default Web Site" under <computername>\Web Sites. Right-click and choose New, Virtual Directory. Click Next. In the field Alias specify "HelloService" (wtihout quotes) and click Next again. Now specify the folder you created in step 3 (c:\temp\HelloService). Click Next. On the "Access Permissions" wizard screen make sure both Read and Run scripts are enabled and click Next. Finally, click on Finish.

  9. If you have both .NET v1.x and v2.0 on the machine, right click on the HelloService virtual directory you just created and go to the Properties. On the ASP.NET tab in the first dropdownlist select version 2.0.50215.0 and click OK.

  10. Open an Internet Explorer instance and navigate to http://localhost/helloservice/service.svc. You should get to see the same page as you saw earlier with our console-hosted Indigo service.

  11. I guess you'll be able to figure out for yourself how to adapt the demo.exe.config file from the previous demo in order to use the IIS-hosted Indigo service instead. Small tip: you need to change an address on two places, somewhere in the neighborhood of the <endpoint> tag :-). In case you're not seeing clear enough, an alternative configuration file is listed below:

    <?xml version="1.0" encoding="utf-8"?>
    <configuration xmlns="http://schemas.microsoft.com/.NetConfiguration/v2.0">
        <system.serviceModel>
            <client>
                <endpoint
                    address="http://localhost/HelloService/service.svc"
                    bindingConfiguration="IHello"
                    bindingSectionName="wsProfileBinding"
                    contractType="IHello" />
            </client>
            <bindings>
                <wsProfileBinding>
                    <binding configurationName="IHello" />
                </customBinding>
            </bindings>
        </system.serviceModel>
    </configuration>

 

What's coming next?

In my next Indigo-related posts, I'll cover data and message contracts, other types of addresses and binding and much more. Stay tuned once again ... you ain't see nothing yet :-).

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

Introduction

Today I'm going to present you an overview of a tool called "ngen" in both .NET v1.x and .NET v2.0. Simply put, ngen (Native Image Generator) is a tool that comes with .NET v1.x and higher and is used to create a "native image" for a given assembly. As you probably know, the CLR executes IL-code which cannot be understood by a processor. Instead, it needs to be translated into native code which can run on the computer's physical processor(s). The CLR execution engine contains a so-called JIT (just in time) compiler to accomplish this IL-to-native-code translation at runtime. However, this compiled (native) code isn't stored, it's just kept in memory during the containing process' lifetime. When the application is restarted, the JIT compiler comes into play again to compile the same code to native code again. The ngen tool lets you create a native image and store it physically on disk in a "native image cache" (aka NGen cache). Let's give a simple usage sample:

  1. Open up the Visual Studio .NET 2003 Command Prompt.
  2. Open Notepad and create a simple "Hello NGen" (I guess I'll never win a prize for originality :-)) like this:

    class Test
    {
       public static void Main()
       {
          System.Console.WriteLine("Hello NGen");
       }
    }

  3. Compile the application using csc.exe and run it. Nothing exciting yet:

    C:\temp>csc test.cs
    Microsoft (R) Visual C# .NET Compiler version 7.10.6310.4
    for Microsoft (R) .NET Framework version 1.1.4322
    Copyright (C) Microsoft Corporation 2001-2002. All rights reserved.

    C:\temp>test
    Hello NGen

  4. Open another Visual Studio .NET 2003 Command Prompt window and go to %windir%\assembly and show the contents of the directory:

    C:\WINDOWS\assembly>dir
     Volume in drive C has no label.
     Volume Serial Number is DC7C-7836

     Directory of C:\WINDOWS\assembly

    08/31/2005  12:52 AM    <DIR>          GAC
    08/31/2005  12:57 AM    <DIR>          GAC_32
    08/31/2005  12:46 PM    <DIR>          GAC_MSIL
    08/31/2005  12:11 AM    <DIR>          NativeImages1_v1.1.4322
    08/31/2005  02:08 PM    <DIR>          NativeImages_v2.0.50215_32
    08/31/2005  01:14 PM    <DIR>          temp
    08/31/2005  01:15 PM    <DIR>          tmp
                   0 File(s)              0 bytes
                   7 Dir(s)  119,380,508,672 bytes free

  5. The NativeImages1_v1.1.4322 is where native images live. Cd into this directory and check that no "test" subdirectory exists over there.
  6. Switch back to the first command prompt and make a native image of the test.exe file using ngen and run it again, as follows:

    C:\temp>ngen test.exe
    Microsoft (R) CLR Native Image Generator - Version 1.1.4322.573
    Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.
    test, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null

    C:\temp>test
    Hello NGen
  7. Next, open the test.exe file using ildasm.exe. You should see the manifest as well as the IL code definition of the class Test.
  8. Switch to the second command prompt, the NGen cache. Now, there should be a directory called test:

    C:\WINDOWS\assembly\NativeImages1_v1.1.4322>dir test
     Volume in drive C is W2K3ENT
     Volume Serial Number is 1C71-360F

     Directory of C:\WINDOWS\assembly\NativeImages1_v1.1.4322\test

    08/31/2005  04:00 PM    <DIR>          .
    08/31/2005  04:00 PM    <DIR>          ..
    08/31/2005  04:00 PM    <DIR>          0.0.0.0___2b518c8d
                   0 File(s)              0 bytes
                   3 Dir(s)     619,991,040 bytes free

  9. Cd into that directory and display the contents:

    C:\WINDOWS\assembly\NativeImages1_v1.1.4322\test\0.0.0.0___2b518c8d>dir
     Volume in drive C is W2K3ENT
     Volume Serial Number is 1C71-360F

     Directory of C:\WINDOWS\assembly\NativeImages1_v1.1.4322\test\0.0.0.0___2b518c8d

    08/31/2005  04:00 PM    <DIR>          .
    08/31/2005  04:00 PM    <DIR>          ..
    08/31/2005  04:00 PM             5,120 test.exe
    08/31/2005  04:00 PM                90 __AssemblyInfo__.ini
                   2 File(s)          5,210 bytes
                   2 Dir(s)     619,991,040 bytes free

  10. You'll notice the test.exe file being larger than the original one (the original one should be 3,072 bytes in size). Again, run ildasm.exe on the test.exe file. Now you won't find any IL, there's just a manifest left.

<GeeksWantMore AlternativeMethodNumber="1">

Geeks can run ildasm /Adv (which is a hidden option in v1.x) to take a look at the COR Header (View, COR header). The original test.exe file should have the following CLR header:

 CLR Header:
 72       Header Size
 2        Major Runtime Version
 0        Minor Runtime Version
 1        Flags
 6000001  Entrypoint Token
 207c     [200     ] address [size] of Metadata Directory:       
 0        [0       ] address [size] of Resources Directory:      
 0        [0       ] address [size] of Strong Name Signature:    
 0        [0       ] address [size] of CodeManager Table:        
 0        [0       ] address [size] of VTableFixups Directory:   
 0        [0       ] address [size] of Export Address Table:     
 0        [0       ] address [size] of Precompile Header:        

whileas the ngen-ed one looks like this:

 CLR Header:
 72       Header Size
 2        Major Runtime Version
 0        Minor Runtime Version
 6        Flags
 0        Entrypoint Token
 2200     [398     ] address [size] of Metadata Directory:       
 0        [0       ] address [size] of Resources Directory:      
 0        [0       ] address [size] of Strong Name Signature:    
 0        [0       ] address [size] of CodeManager Table:        
 0        [0       ] address [size] of VTableFixups Directory:   
 0        [0       ] address [size] of Export Address Table:     
 2050     [40      ] address [size] of Precompile Header:        

Basically, the runtime will check whether a managed code file has an associated native image. If that's the case, the native image is used instead of the file containing the (non-JITted) IL code.

</GeeksWantMore>

<GeeksWantMore AlternativeMethodNumber="2">

If you want to see more evidence of the native image being used instead of the original (non-pre-jitted) one, you can use "Fusion", otherwise known as "Assembly Binding Log Viewer". Open the Registry Editor (regedit.exe) and locate HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Fusion. In there, add a REG_DWORD value with name "ForceLog" (without quotes) and data 0x00000001 (1). Go to the first command prompt window and execute the test.exe file. Then, start fuslogvw.exe:

C:\temp>test
Hello NGen

C:\temp>fuslogvw

You should see something like this:

Select the test.exe log entry and click "View Log". At the bottom of the Assembly Binder Log Entry you should find the following:

LOG: Not processing DEVPATH because this is a pre-jit assembly bind.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Post-policy reference: test, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
LOG: Found assembly by looking in the cache.

After playing this trick, disable the ForceLog again in the registry by removing the value you added previously.

</GeeksWantMore>

 

Shortcomings in v1.x

In the introduction we took a look at the ngen.exe tool of the .NET Framework v1.x. Although the principle is great (and it's being used by dotnetfx.exe during installation of the BCL assemblies), it has some serious drawbacks. An overview:

  • Native images cannot be shared across application domains, which renders ngen.exe useless for ASP.NET and other scenarios where more than one application domain is involved.
  • Managed assemblies have dependencies (at least mscorlib.dll is required). When an assembly is ngen-ed, stuctural information about those dependencies is introduced in the native image. A change of such a dependency will invalidate the native image which was generated by ngen, falling back on JIT compilation instead. Imagine what would happen if System.dll changes (e.g. by applying a service pack).
  • Ngen is invoked on a single assembly, not on an entire application. It's up to the developer to make sure dependencies are ngen-ed as well.
  • The ngen.exe tool doesn't use the standard assembly probing rules to locate the specified assembly. Instead, it just looks in the current folder.
  • NGen can only run synchronously. You have to wait till the compilation to native code has completed.
  • There is no support for a thing called "hardbinding". Although only handy in somewhat rare situations, hardbinding can be useful.

    Hardbinding eliminates additional costs associated with maintaining "call stubs" which are introduced in code to call methods etc in another assembly. Because the target is not known at compile-time, the virtual address at runtime is unknown. Instead, a little stub function is created which is responsible to calculate that virtual address of the call target. Once the address is found, the call is "backpatched" in order to bypass the stub and call the target directly. However, because of this a code page which was read-only in the past, now needs to be changed to writable (to apply the backpatch) and thus it cannot be shared anymore. Reduction of sharable pages can hurt performance.

 

Whidbey's new NGen technology

Overview of ngen.exe

NGen 2.0 tries to solve a series of the aforementioned shortcomings. Start by opening a Visual Studio 2005 Command Prompt and launch the new ngen.exe tool. This is the output:

C:\>ngen
Microsoft (R) CLR Native Image Generator - Version 2.0.50215.44
Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.

Usage: ngen <action> [args] [/nologo] [/silent] [/verbose]
       ngen /? or /help

    /nologo    - Prevents displaying of logo
    /silent    - Prevents displaying of success messages
    /verbose   - Displays verbose output for debugging

Actions:
    ngen install <assembly name> [scenarios] [config] [/queue[:[1|2|3]]
        Generate native images for an assembly and its dependencies
        and install them in the Native Images Cache

NGen now works on applications, not on stand-alone assemblies.

        If /queue is specified compilation job is queued up.  If a priority
        is not specified, the default priority used is 3.

Can run asynchronously by using a queue. More info follows later.

    ngen uninstall <assembly name> [scenarios] [config]
        Delete the native images of an assembly and its dependencies from
        the Native Images Cache.
    ngen update [/queue]
        Update native images that have become invalid
        If /queue is specified compilation jobs are queued up.
    ngen display [assembly name]
        Display the ngen state
    ngen executeQueuedItems [1|2|3]
        Executes queued compilation jobs.
        If priority is not specified all queued compilation jobs are done.
        If priority is specified compilation jobs with greater or equal.
        priority than the specified are done.
    ngen queue [pause|continue|status]
        Allows the user to pause and continue the NGen Service Queue, and to
        query its status.

Scenarios:
    /Debug          - Generate images that can be used under a debugger
    /Profile        - Generate images that can be used under a profiler
    /NoDependencies - Generate the minimal number of native images
                      required by this scenario

Config:
    /ExeConfig:<path to exe> - Use the configuration of the specified
                 executable assembly
    /AppBase:<path to appbase directory> - Use the specified directory as
                 the appbase

I've annotated the output above with red text, pointing to some of the shortcomings addressed in the new version. First of all, dependencies are also ngen-ed automatically because the tool detects all dependencies automatically. An easy sample will show this:

  1. Open a Visual Studio 2005 Command Prompt.
  2. Create a new file testhelper.cs using Notepad:

    public class TestHelper
    {
       public string GetMessage()
       {
          return "Test";
       }
    }
  3. Create a new file test.cs using Notepad:

    class Test
    {
       public static void Main()
       {
          System.Console.WriteLine(new TestHelper().GetMessage());
       }
    }
  4. Compile both files and run:

    C:\temp>csc -t:library testhelper.cs
    Microsoft (R) Visual C# 2005 Compiler version 8.00.50215.44
    for Microsoft (R) Windows (R) 2005 Framework version 2.0.50215
    Copyright (C) Microsoft Corporation 2001-2005. All rights reserved.

    C:\temp>csc -r:testhelper.dll test.cs
    Microsoft (R) Visual C# 2005 Compiler version 8.00.50215.44
    for Microsoft (R) Windows (R) 2005 Framework version 2.0.50215
    Copyright (C) Microsoft Corporation 2001-2005. All rights reserved.

    C:\temp>test
    Test

  5. Open %windir%\assembly\NativeImages_v2.0.50215_32 in another Visual Studio 2005 Command Prompt window. You should find no folder called test or testhelper.
  6. Switch back to the first command prompt window and execute ngen as follows (new syntax!):

    C:\temp>ngen install test.exe
    Microsoft (R) CLR Native Image Generator - Version 2.0.50215.44
    Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.
    Installing assembly C:\temp\test.exe
    Compiling 2 assemblies:
        Compiling assembly C:\temp\test.exe ...
    test, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
        Compiling assembly testhelper, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null ...
    testhelper, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null

  7. Now native images for both test and testhelper should be present in the assembly\NativeImages_v2.0.50215_32 folder. The new filename of native images contains .ni. in the middle between the original filename and the extension. So, the following dir command shows us all of the native images for test* assemblies:

    C:\WINDOWS\assembly\NativeImages_v2.0.50215_32>dir test*.ni.* /s
     Volume in drive C has no label.
     Volume Serial Number is DC7C-7836

     Directory of C:\WINDOWS\assembly\NativeImages_v2.0.50215_32\test\8714e29e626e8e3e8323436ee93dd387

    08/31/2005  05:54 PM            12,800 test.ni.exe
                   1 File(s)         12,800 bytes

     Directory of C:\WINDOWS\assembly\NativeImages_v2.0.50215_32\testhelper\b6c46520e8eb8c37b273b092ff53c2fa

    08/31/2005  05:54 PM            12,288 testhelper.ni.dll
                   1 File(s)         12,288 bytes

         Total Files Listed:
                   2 File(s)         25,088 bytes
                   0 Dir(s)  119,786,254,336 bytes free

  8. Again you can use ildasm and/or fuslogvw to find out more:
    • ildasm.exe should only show the manifest in the native image files test.ni.exe and testhelper.ni.dll (note: the /Adv flag for ildasm.exe has been removed and is on by default; use View, Headers to view the COR Header)
    • fuslogvw.exe now has a Settings button to eliminate the need to go to the registry to change the logging level (select "Log all binds to disk") and also has a "Log Categories" section where you can choose "Native Images" from

Now, how does ngen.exe determine the dependencies of a given assembly? The answer is the IL header which contains all static "imports" in the assembly:

// Metadata version: v2.0.50215
.assembly extern mscorlib
{
  .publickeytoken = (B7 7A 5C 56 19 34 E0 89 )                         // .z\V.4..
  .ver 2:0:0:0
}
.assembly extern testhelper
{
  .ver 0:0:0:0
}

Now, what happens if we want to uninstall the native images for our test.exe application. Let's run it to get a clue about this (warning: make sure to cd out of the NativeImages_v2.0.50215_32\test* directories if these are still open in some command prompt window):

C:\temp>ngen uninstall test.exe
Microsoft (R) CLR Native Image Generator - Version 2.0.50215.44
Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.
Uninstalling assembly C:\temp\test.exe

and

C:\WINDOWS\assembly\NativeImages_v2.0.50215_32>dir test*
 Volume in drive C has no label.
 Volume Serial Number is DC7C-7836

 Directory of C:\WINDOWS\assembly\NativeImages_v2.0.50215_32

File Not Found

As you can see, also the testhelper.dll native image has disappeared. There has to be some kind of dependency tracking, right? Let's do another test:

  1. In the folder where testhelper.dll resides, create a new file called test2.cs with the following code in it:

    class Test2
    {
       public static void Main()
       {
          System.Console.WriteLine(new TestHelper().GetMessage());
       }
    }

  2. Compile the test2.cs file:

    C:\temp>csc -r:testhelper.dll test2.cs
    Microsoft (R) Visual C# 2005 Compiler version 8.00.50215.44
    for Microsoft (R) Windows (R) 2005 Framework version 2.0.50215
    Copyright (C) Microsoft Corporation 2001-2005. All rights reserved.

  3. Now send both apps to the NGen cache:

    C:\temp>ngen install test.exe
    Microsoft (R) CLR Native Image Generator - Version 2.0.50215.44
    Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.
    Installing assembly C:\temp\test.exe
    Compiling 2 assemblies:
        Compiling assembly C:\temp\test.exe ...
    test, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
        Compiling assembly testhelper, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null ...
    testhelper, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null

    C:\temp>ngen install test2.exe
    Microsoft (R) CLR Native Image Generator - Version 2.0.50215.44
    Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.
    Installing assembly C:\temp\test2.exe
    Compiling 1 assembly:
        Compiling assembly C:\temp\test2.exe ...
    test2, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null

  4. In the other command prompt window, check the existence of the expected folders:

    C:\WINDOWS\assembly\NativeImages_v2.0.50215_32>dir test*
     Volume in drive C has no label.
     Volume Serial Number is DC7C-7836

     Directory of C:\WINDOWS\assembly\NativeImages_v2.0.50215_32

    08/31/2005  06:12 PM    <DIR>          test
    08/31/2005  06:12 PM    <DIR>          test2
    08/31/2005  06:12 PM    <DIR>          testhelper
                   0 File(s)              0 bytes
                   3 Dir(s)  119,785,574,400 bytes free

  5. Now uninstall test.exe using ngen:

    C:\temp>ngen uninstall test.exe
    Microsoft (R) CLR Native Image Generator - Version 2.0.50215.44
    Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.
    Uninstalling assembly C:\temp\test.exe

  6. Check the NGen cache again:

    C:\WINDOWS\assembly\NativeImages_v2.0.50215_32>dir test*
     Volume in drive C has no label.
     Volume Serial Number is DC7C-7836

     Directory of C:\WINDOWS\assembly\NativeImages_v2.0.50215_32

    08/31/2005  06:12 PM    <DIR>          test2
    08/31/2005  06:12 PM    <DIR>          testhelper
                   0 File(s)              0 bytes
                   2 Dir(s)  119,785,574,400 bytes free

  7. Cool, testhelper is still alive and kicking out there. Dependency tracking does its job.

So, now the question you'll be asking yourself probably: where does the CLR keep track of these dependencies? The answer is the registry, more specifically in the following key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Fusion\NativeImagesIndex\v2.0.50215_32. In the Registry Editor, search for "test" starting from the specified key and you should find both the test2 and testhelper entries. For example, I have a key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Fusion\NativeImagesIndex\v2.0.50215_32\IL\2988d581\392c9598\f1over here for the testhelper assembly. The subkey InvertDependencies contains links to the assemblies depending on the current one. An overview of the local situation:

  • IL\2988d581\392c9598\f1
    • DisplayName: testhelper,0.0.0.0,,
    • InvertDependencies: 1793d581\fe08a65\d2, 5b607b80\66bc3064\a1
  • IL\6897b81\795befb3\f8
    • DisplayName: test2,0.0.0.0,,
    • InvertDependencies: 5b607b80\66bc3064\a1
  • NI\1793d581\fe08a65\d2
    • DisplayName: testhelper,0.0.0.0,,
    • MVID: 6B DA A9 0D 8E 31 A3 38 96 05 C7 87 17 C8 65 98
  • NI\5b607b80\66bc3064\a1
    • DisplayName: test2,0.0.0.0,,
    • MVID: 2C 25 C4 D5 6F 30 FB 3F B3 CA AC 5F 2D 46 5B 52

The NI keys contain information about the native images, i.e. testhelper and test2. The MVID points to the subdirectory in the NativeImages_v2.0.50215_32 folder where the native image resides. E.g. testhelper lives on my system in C:\WINDOWS\assembly\NativeImages_v2.0.50215_32\testhelper\6bdaa90d8e31a3389605c78717c86598. In the IL part you'll find the InvertDependencies list for each assembly in the NGen cache. As you can see, testhelper is needed for testhelper itself (the native image) and test2. This means that a change to testhelper will cause an ngen to occur to testhelper and test2, because these native images would be invalidated by changing testhelper.

Note: Do not confuse these mechanisms with the old COM registration mechanism. COM suffers from the infamous DLL Hell, whileas the .NET Framework does not suffer from this problem (cf. versioning). Furthermore, the use of the registry for "registration" and "dependency tracking" is a ngen-only thing in the world of .NET, assemblies can be xcopy deployed. Also notice that the assemblies act as a kind of stubs to the native images when such a native image exists. You can't delete the testhelper.dll file in order to run test.exe or test2.exe although it resides in the NGen cache. If you'd like to do this, you'll need to register the assembly in the GAC (thus requiring strong naming). Thus, ngen.exe is not a .NET look-alike for regsvr32.exe.

So far, we've seen the ngen install and ngen uninstall actions. It's worth to mention that a fully-qualified assembly name can be used to perform the install action. Notice that such a fully-qualified name in .NET v2.0 has now 5 components: name (test), version (0.0.0.0), culture (neutral), public key token (...) and processor architecture (MSIL). Two other useful features (flags at the command line) are:

  • Scenarios: used to generate native images that can be used by a debugger (/debug) or a profiler (/profile) or to generate a minimum number of native images by not ngen-ing dependencies (/nodependencies).
  • Config: /ExeConfig to point to a configuration file (.exe.config) that contains additional information used by ngen and /AppBase to override assembly probing settings by specifying an "appbase directory" to search for assemblies in.

Another useful action is the ngen display action. An example is shown below:

C:\temp>ngen display test.exe
Microsoft (R) CLR Native Image Generator - Version 2.0.50215.44
Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.

NGEN Roots:

C:\temp\test.exe

NGEN Roots that depend on "C:\temp\test.exe":

C:\temp\test.exe

Native Images:

test, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null

Using verbose output (add /verbose) you get a lot more information however:

Microsoft (R) CLR Native Image Generator - Version 2.0.50215.44
Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.

NGEN Roots:

C:\temp\test.exe
    ScenarioDefault  --no debug or profile scenario present
        C:\temp\test.exe
            DisplayName = test, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
            Native image = {CC9FE28A-004D-38EB-82F9-3D97AB21A051}
            Hard Dependencies:
                mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
            Soft Dependencies:
                testhelper, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null --what we were looking for; "forward dependency"
        mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
            DisplayName = mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
            Native image = {DAAB7D0D-ABB0-394D-82EF-037B571B1032}
            Hard Dependencies:
            Soft Dependencies:
        testhelper, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
            DisplayName = testhelper, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
            Native image = {346E10F6-8E71-3605-91A2-C732FC2D5866} --the location of testhelper can be derived from the GUID: F6106E34718EA29166582DFC32C7 by a bytewise revert of each "GUID group"
            Hard Dependencies:
                mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
            Soft Dependencies:

--Note: In the end, the information shown above forms a tree of dependencies. Try doing the same on an application that uses e.g. System.Data.DataSet.

NGEN Roots that depend on "C:\temp\test.exe":

C:\temp\test.exe
    ScenarioDefault
        C:\temp\test.exe
            DisplayName = test, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
            Native image = {CC9FE28A-004D-38EB-82F9-3D97AB21A051}
            Hard Dependencies:
                mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
            Soft Dependencies:
                testhelper, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
        mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
            DisplayName = mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
            Native image = {DAAB7D0D-ABB0-394D-82EF-037B571B1032}
            Hard Dependencies:
            Soft Dependencies:
        testhelper, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
            DisplayName = testhelper, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
            Native image = {346E10F6-8E71-3605-91A2-C732FC2D5866} --the location of testhelper can be derived from the GUID: F6106E34718EA29166582DFC32C7 by a bytewise revert of each "GUID group"
            Hard Dependencies:
                mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
            Soft Dependencies:

Native Images:

test, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
 Source MVID: {C456F353-332D-401F-B9FC-93495C519EB1}
 Source HASH: f9149373069fb3e81111bb24db59018bc2d48d98
 NGen GUID sign: {CC9FE28A-004D-38EB-82F9-3D97AB21A051}
 OS:  WinNT
 Processor: x86(Pentium 4) (features: 00008001)
 Runtime: 2.0.50215.44
 mscorwks.dll: TimeStamp=42578752, CheckSum=0053D2C1
 Flags:  
 Scenarios:  <no debug info> <no debugger> <no profiler> <no instrumentation> --try to add other scenarios as explained earlier
 Granted set: <PermissionSet class="System.Security.PermissionSet"
version="1"
Unrestricted="true"/>

 File:  C:\WINDOWS\assembly\NativeImages_v2.0.50215_32\test\8ae29fcc4d00eb3882f93d97ab21a051\test.ni.exe --the location of the native image
 Dependencies:
  mscorlib, Version=2.0.0.0, PublicKeyToken=b77a5c561934e089:
   Guid:{7125636A-C067-4930-9DA0-DCD6DC6A10A9}
   Sign:66c33e7153140d9397fd66ecb5d9c4b19f00e8e3
   Hardbound Guid:{DAAB7D0D-ABB0-394D-82EF-037B571B1032}
  testhelper, Version=0.0.0.0, PublicKeyToken=null:
   Guid:{58EDC827-7258-43EF-BD02-B694FA4C806C}
   Sign:0cd07a972e90d8faf86ec768d0705a17211306d3

The NGen service

Time to move on to another piece of the renewed NGen technology, being the ".NET Runtime Optimization Service". What's in a name? It's a service and it's used to optimize the .NET Runtime. So, what about looking at the SCM MMC using services.msc? The first service in the list that you should see is the following:

  • Service Name: clr_optimization_v2.0.50215_32
  • Display Name: .NET Runtime Optimization Service v2.0.50215_X86
  • Description: Provides support for optimizing managed assemblies using NGEN technology.
  • Path to executable: C:\WINDOWS\Microsoft.NET\Framework\v2.0.50215\mscorsvw.exe

People who are somewhat familiar with services will notice that the recovery settings are rather "exceptional": on every failure, the service is restarted and there's a pretty agressive recovery policy. By default, the service will have a Manual startup type, but the service itself changes this when needed (see further). Now, what's this service all about?

Imagine an application you want to ngen when it's installed but you don't want to wait for the entire ngen compilation to complete at setup time. In other words, some asynchronous triggering of the ngen compilation mechanism for some application. This is the scenario for which this service was made: to perform IL-to-native code compilations in the background when the system is idle or when ngen compilation is enforced through ngen.exe.

Let's go through a end-to-end scenario using the NGen service:

  1. Make sure all test.exe, test2.exe and testhelper.dll native images are deleted from the system by running ngen uninstall on each of these assemblies. When doing so, make sure no other command prompt is active in the NGen cache folders for these assemblies' native images.
  2. First, query the service status. In my case this is the output:

    C:\temp>sc queryex clr_optimization_v2.0.50215_32

    SERVICE_NAME: clr_optimization_v2.0.50215_32
            TYPE               : 10  WIN32_OWN_PROCESS
            STATE              : 1  STOPPED
                                    (NOT_STOPPABLE,NOT_PAUSABLE,IGNORES_SHUTDOWN)
            WIN32_EXIT_CODE    : 0  (0x0)
            SERVICE_EXIT_CODE  : 0  (0x0)
            CHECKPOINT         : 0x0
            WAIT_HINT          : 0x0
            PID                : 0
            FLAGS              :


    If the service is running, it has work to do. In that case, execute the following command to "purge" the queue of outstanding work (note this can take a while and will cause high CPU usage):

    C:\temp>ngen executeQueuedItems 3
    Microsoft (R) CLR Native Image Generator - Version 2.0.50215.44
    Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.
    (...)

    If there was work to do and the service is still running, reboot the machine (just for sake of the demo). Next time the service should be stopped and have a startup type of Manual.
  3. Now we'll ngen the test.exe file, not directly but deferred. In order to do this, we need to execute the ngen install action but with the /queue flag:

    C:\temp>ngen install /queue test.exe
    Microsoft (R) CLR Native Image Generator - Version 2.0.50215.44
    Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.
    Installing assembly C:\temp\test.exe

    Compare this with earlier executions of ngen install without the /queue flag. The assembly is not being compiled directly right now.
  4. Go to the C:\WINDOWS\assembly\NativeImages_v2.0.50215_32 folder in the another command prompt window. You should find no subfolder called "test" in there. Native image compilation has not been done yet...

    C:\WINDOWS\assembly\NativeImages_v2.0.50215_32>dir test
     Volume in drive C has no label.
     Volume Serial Number is DC7C-7836

     Directory of C:\WINDOWS\assembly\NativeImages_v2.0.50215_32

    File Not Found

  5. Now run the ngen display action for the test.exe assembly:

    C:\temp>ngen display test.exe
    Microsoft (R) CLR Native Image Generator - Version 2.0.50215.44
    Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.

    NGEN Roots:

    C:\temp\test.exe (StatusPending) (Pri 3)

    Native Images:

    This output indicates that only the root of test.exe is known to the system and the status of its compilation is pending with priority 3 (see further).
  6. Query the service status again. It's now alive (and kicking?):

    C:\temp>sc queryex clr_optimization_v2.0.50215_32

    SERVICE_NAME: clr_optimization_v2.0.50215_32
            TYPE               : 10  WIN32_OWN_PROCESS
            STATE              : 4  RUNNING
                                    (STOPPABLE,PAUSABLE,IGNORES_SHUTDOWN)
            WIN32_EXIT_CODE    : 0  (0x0)
            SERVICE_EXIT_CODE  : 0  (0x0)
            CHECKPOINT         : 0x0
            WAIT_HINT          : 0x0
            PID                : 3040
            FLAGS              :

  7. When you investigate the task list, you'll find one instance of mscorsvw.exe, as shown below:

    C:\temp>tasklist /FI "IMAGENAME eq mscorsvw.exe"

    Image Name                   PID Session Name     Session#    Mem Usage
    ========================= ====== ================ ======== ============
    mscorsvw.exe                3040 Console                 0      3,064 K

    This is the service which is running. You can also check the state of the service as follows, using ngen.exe:

    C:\temp>ngen queue status
    Microsoft (R) CLR Native Image Generator - Version 2.0.50215.44
    Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.

    Service name is: clr_optimization_v2.0.50215_32
    The .NET Runtime Optimization Service is running.

  8. In step 5, I've shown that the test.exe compilation is pending and has a priority of 3. This means the service is waiting till the computer is idle (which is measured based on the time of the last user input and the state of a screensaver, but this "idleness detection mechanism" is still to be finetuned towards RTM of .NET v2.0) in order to start working on priority 3 jobs that are in the queue. However, you can force this work to be done immediately, using the following command:

    C:\temp>ngen executeQueuedItems 3
    Microsoft (R) CLR Native Image Generator - Version 2.0.50215.44
    Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.
    Compiling 2 assemblies:
        Compiling assembly C:\temp\test.exe ...
    test, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null
        Compiling assembly testhelper, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null ...
    testhelper, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null


    Note: While this command is running (you need to be quick to see), another instance of mscorsvw.exe will be created (watch your Task Manager task list, sorted by Image Name).
  9. When you inspect the NGen cache now, you should find the following entries:

    C:\WINDOWS\assembly\NativeImages_v2.0.50215_32>dir test*
     Volume in drive C has no label.
     Volume Serial Number is DC7C-7836

     Directory of C:\WINDOWS\assembly\NativeImages_v2.0.50215_32

    09/01/2005  01:24 AM    <DIR>          test
    09/01/2005  01:24 AM    <DIR>          testhelper
                   0 File(s)              0 bytes
                   2 Dir(s)  119,775,739,904 bytes free

  10. In order to visualize the difference between priority 1, 2 and 3 perform now the following:

    C:\temp>ngen install test2.exe /queue:1
    Microsoft (R) CLR Native Image Generator - Version 2.0.50215.44
    Copyright (C) Microsoft Corporation 1998-2002. All rights reserved.
    Installing assembly C:\temp\test2.exe

  11. Go to the other command prompt window. Because the priority was set to 1, the service is performing the compilation immediately, as you can see by showing the test* contents of the NGen cache:

    C:\WINDOWS\assembly\NativeImages_v2.0.50215_32>dir test*
     Volume in drive C has no label.
     Volume Serial Number is DC7C-7836

     Directory of C:\WINDOWS\assembly\NativeImages_v2.0.50215_32

    09/01/2005  01:24 AM    <DIR>          test
    09/01/2005  01:29 AM    <DIR>          test2
    09/01/2005  01:24 AM    <DIR>          testhelper
                   0 File(s)              0 bytes
                   3 Dir(s)  119,775,760,384 bytes free

I won't cover the other ngen.exe capabilities (such as the update action) over here for now. However, based on what I've shown above, you should have a good clue about the power the renewed ngen technollogy is offering in .NET v2.0. A classic scenario is queuing up precompilation work for a set of assemblies associated with your application at installation time. You'll queue compilation of the most critical files using priority 1 or 2 whileas you'll queue up compilation for the other executables (e.g. less used tools) using the default priority of 3 (compile during idle time). At installation time, you'll benefit from both queue-ups because the installer does not need to wait for the ngen tasks to complete (the service does the work in the background for you). The general skeleton looks like this:

ngen queue pause --keep the NGen image cache stable during setup (recommended)
ngen install <...> /queue --default priority = 3; recommended for less-used apps
ngen install <...> /queue:1 --enforce high priority compilation
ngen queue continue --okay, the NGen service can continue its mission

 

NGen: use it or leave it?

The official answer: it depends. The correct answer: it depends. You should measure the performance of both the JITted application and the NGen-ed application and compare the results in order to get a clue about the performance gains for your application. Purely theoritically, the following statements hold:

  • Precompiled applications require less memory because the JIT compiler doesn't need to be loaded into memory.
  • Precompiled applications can benefit from sharing memory pages for shared code across applications (i.e. DLLs). This also reduces memory consumption.
  • (Warm) startups of precompiled applications are faster than their non-JITted equivalents.

Furthermore, in v2.0 NGen has support for sharing of native images across application domains and all of the reflection stuff on native images. For a discussion on "hardbinding", I want to refer to the MSDN Article mentioned below in the "Useful links" section.

 

Useful links

 

Conclusion

Visual Studio 2005 is great, but don't forget about the .NET Framework SDK tools in .NET v2.0. There are just so many improvements in there (in the end, these tools are the core of the .NET Framework), it would be a pitty for you to miss any of these improvements. I'll try to point out things like this on a regular basis on my blog.

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

More Posts « Previous page