Server-side Office Development with SharePoint

I’ve talked to many developers who have recently started doing some SharePoint programming and have gotten hooked. In fact, they like SharePoint development so much, especially the event receivers, that these developers are starting to think about using SharePoint as a central clearinghouse for document processing.

SharePoint event receivers allow developers to programmatically interact and alter data before or after the data is saved into, updated in, or removed from SharePoint. Event receivers can work on list items as well as documents, including Office documents.

A world of possibilities

Combined with SharePoint’s metadata storage and workflow capabilities, you can imaging some pretty compelling applications built using event receivers. For example, you could create some code to run as a document is saved into SharePoint. If the document is in a draft state according to its metadata, the code would alter the document to include a “Draft” watermark right in the contents of the document. Once the metadata changes to a published status, the code could remove the watermark and also save a PDF copy of the document.

What not to do

A common implementation strategy is to use a SharePoint event receiver or workflow to invoke the Word object model using  the office primary interop assemblies (OPIA). Then using the OPIA add or remove the watermark or otherwise after the contents of the document according to the metadata associated with the document.

This is a bad architecture.

For details on why this does not work, please see my post: “AnyCPU, x86, x64 – What’s the Difference?” To summarize that post, when SharePoint is installed on a 64-bit operating system, the various SharePoint processes are running in 64-bit mode, therefore your assemblies will be running in 64-bit mode. A 64-bit process can not load a 32-bit DLL. Office is a 32-bit application and so are the primary interop assemblies most .Net developers use to automate Office. So you will receive a BadImageFormatException when trying to invoke any of the Office applications from your SharePoint code when SharePoint is running on an x64 operating system.

Beyond the issue of x86 vs. x64 interoperability, there are other reasons to avoid this software design. First, to save money many developers will implement the software in a way that requires installing Office on the SharePoint web front end servers. That’s a very bad idea, period. Security, threading, and DCOM issues will make the software a nightmare to maintain.

Even if the approach functions on the developer’s x86 development SharePoint machine with a single user and the test documents are 20 KB, what will happen in a production environment where there are thousands of users and 10 MB documents? What happens when 20 documents per minute are being checked in and need to be processed?

A better idea

Don’t get me wrong. I am not saying that you should not try to write an application that does document processing using the SharePoint and Office technology stacks. Document processing applications can serve to increase the data quality of documents and increase worker productivity by reducing the labor involved in document maintenance. Just don’t use the approach outlined earlier. Use the shared a services approach instead.

Document processing as a shared services

The concept of shared services was introduced in SharePoint Portal Server 2003 (SPS) and greatly expanded upon in Microsoft Office SharePoint Server 2007 (MOSS). In a nutshell, the bulk of SharePoint is content driven. Shared services provide data storage and processing that are available globally to the SharePoint farm and are not tied to a particular SharePoint content repository.

Content indexing is a good example. The content indexer is a process that runs outside the scope of any SharePoint web application. The data the indexer stores is not user content and so does not go into any SharePoint content database. The data the indexer collects and processing the indexer performs is available to all SharePoint web applications in the farm, and potentially to other farms.

I am advocating that the document processing operations described earlier should be treated the same way. Sure the documents themselves are user content and reside in a SharePoint content database, but the process of adding a watermark to or removing a watermark from a Word document has nothing to do with a SharePoint web application and thus should not be performed in an event receiver, or from anywhere within the w3wp.exe process.

How do you Easily create a SharePoint Shared Service?

There are a couple of constraints our solution needs to adhere to in order to function properly. The solution needs to be able to:

• detect document changes
• process document changes
• process long-running operations
• support both x86 and x64 SharePoint installations
• support both XML (.docx) and legacy file formats (.doc)

Detecting document changes is fairly easy. Use a list item event receiver to catch the changes.

Processing the document changes, especially changes to large documents can be more involved. If you throw in the requirement to support legacy Office document formats and x64 installations, there seems to be only one way to do it: create a Windows Service (can’t use a timer job) that is compiled with the x86 option. See my post on the differences between AnyCPU, x86, and x64 compiler options for details on why we need to be explicit about the target platform. This service can reside on one of the SharePoint application servers in the farm, on on a server that is outside of the farm. How?

Create a web service interface that accepts document processing requests and triggers the processing service to do its work. The web service doesn’t actually do any work, it simply accepts documents as input and drops them into a folder on the file system that the processing service is watching. The web service must be compiled with the AnyCPU option so that it will run properly on IIS on both x86 and x64 systems. Because the document acceptor web service and the document processing Windows service are communicating out of process, the web service can run as an x64 process for compatibility with x64 IIS, and the Windows service can run as an x86 process for compatibility with Office.

That reminds me, Office will need to be installed on the server that is running the document processing service. While it is easy to work with the XML Office file formats without Office, working with the legacy file formats either requires that you have Office installed or that you use third party middleware.

Putting everything together

What does the final solution look like and how does it operate? The list item event receiver detects a change to a document that needs processing. The event receiver invokes the document acceptor web service running on a SharePoint application server or on an external server. The document acceptor service receives the file to be processed and save the file to a file system folder. The document processor service sees the saved file on the file system and processes the file. Once processing is complete, the document processing service saves the file back to the SharePoint document library.

For a more in-depth discussion of using Shared Service architectures in a SharePoint environment, join me at the Best Practices Conference. I will be covering this and related topics in my presentation: Using Service Oriented Methodologies to Create SharePoint Products.

Technorati Tags: x64,SharePoint,WSS,MOSS,Office,Best Practices Conference

Run STSADM commands remotely

Are you tired of having to bring up a Remote Desktop to your SharePoint server just to run an stsadm command? I sure was. Here’s a very simple way to remotely execute stsadm commands against any SharePoint server to which you have a LAN connection. Best of all, unlike some other methods out there, this method does not require deploying anything to the SharePoint server. However, this method is primarily for LAN scenarios, and it is very unlikely to work over the Internet.

What you’ll need

PsExec.exe by Sysinternals (now part of Microsoft) is a great utility that allows you to execute any command line against any Windows computer to which you have a LAN connection and proper credentials without installing any software on the remote server. PsExec sits on the workstation computer that you are using. Since stsadm is a command line tool and PsExec is designed to execute command lines, we can combine stsadm with PsExec to create a very nifty way to work on SharePoint servers remotely using only a command prompt.

PsExec uses Windows RPC ports to execute the commands, so if you will need to have the RPC ports open between your workstation and the SharePoint server that you want to manage. This is why it is unlikely to work over the Internet. This article has a nice primer on Windows TCP ports and references to other resources.

How to use PsExec with Stsadm

Here’s what you do. Remember that you will do all of these steps from your desktop computer, not the SharePoint server.

1. Download PsExec from Microsoft’s web site to your computer, not to the SharePoint server.
2. Open a command prompt (in administrative mode if you are using Vista with UAC enabled), and change to the directory where you have downloaded and extracted the PsExec tool.
3. Execute stsadm.exe through the PsExec command. Use the following format:
   psexec \\[SharePoint_Server_NetBIOS_Name] –u [username] –p [password] “C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN\stsadm.exe” –o [stsadm_parameters]
   
   Note: You need “\\” in front of the remote SharePoint computer’s name.

Notice that all files names and paths should be absolute paths and in the context of the remote SharePoint on which you are executing the stsadm command.

A few sample commands

Here’s a sample that will enumerate all solutions:

psexec \\wss01 –u mydomain\administrator –p password “C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN\stsadm.exe” –o enumsolutions

Here’s a sample that will do a SharePoint backup to the “c:\backup folder” directory on the SharePoint server. Notice that parameters that contain spaces need to be enclosed in quotes:

psexec \\wss01 –u mydomain\administrator –p password “C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN\stsadm.exe -o backup –directory “c:\backup folder” -backupmethod full

Here’s a sample that will copy a WSP solution to a remote SharePoint server, add the solution to the solution store, and deploy the solution to the farm

REM Set an environment variable to point to stsadm. This is just for convenience

set s=“C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN\stsadm.exe”

REM Copy the WSP solution file to the remote SharePoint server through the admin$ administrative share. This share points to the Windows directory. You must have administrative rights on the server to access this share. The “net use” command will establish credentials on the remote server share.

net use \\wss01\admin$ /USER:mydomain\administrator password

copy C:\Deploy\MySolution.WSP \\wss01\admin$\temp\MySolution.WSP

REM Add the WSP solution to the solution store

psexec \\wss01 –u mydomain\administrator –p password “%s%” –o addsolution –filename c:\windows\temp\MySolution.WSP

REM Create a timer job to deploy the solution to all content web applications in the SharePoint farm

psexec \\wss01 –u mydomain\administrator –p password “%s%” –o deploysolution -name MySolution.WSP –allcontenturls –allowgacdeployment –immediate

REM Start the deployment timer job

psexec \\wss01 –u mydomain\administrator –p password “%s%” –o execadmsvcjobs

Why not PowerShell

You’ve probably noticed that I’ve been using all batch command statements in this article. The natural question to ask is: “Why not use PowerShell" instead?” After all, PowerShell is certainly far more powerful than the dated DOS commands that are available through batch files. While this is true, the fact is that PowerShell is not installed on a fairly large portion of SharePoint servers because PowerShell was not released when Windows Server 2003 came out, which still forms a very large percentage of the base operating system used for SharePoint deployments. Because one of the goals of this article was to come up with a remote SharePoint management method that could be used with all LAN SharePoint servers with no additional server software install, PowerShell was out.

Technorati Tags: WSS,SharePoint,MOSS

RunAs in C#

I do a lot of server programming. I frequently need to be able to run some code as a different user than the currently logged-in user. Unfortunately, the .Net Framework does not make this particularly easy. Sure, there is the WindowsIdentity class and its Impersonate method. But how do you create an instance of a WindowsIdentity object? You need to pass the constructor a logon token. How do you get a logon token? Well, .Net is rather silent on that point.

PInvoke is your friend

The Win32 API does have a LogonUser function in the advapi32.dll DLL. Thanks to the wonderful work of PInvoke (now part of RedGate), you can easily get the signature of the LogonUser function translated into .Net. It looks like this:

[DllImport("advapi32.dll", SetLastError = true)]  
public static extern bool LogonUser(string lpszUserName, string lpszDomain, string lpszPassword, int dwLogonType, int dwLogonProvider, out IntPtr phToken);

I found a nice code snippet by Viji RAJKUMAR that showed how to use the LogonUser function, handle the Win32 error codes, and handle the .Net impersonation contexts. But do you really want deal with all of that whenever you have to run some code as someone else? I sure don’t.

Inspiration from SharePoint’s RunWithElevatedPrivileges

One thing that the SharePoint API has is a very handy method on the SPSecurity class called RunWithElevatedPrivileges. RunWithElevatedPrivileges takes a function pointer (called a delegate in .Net) and executes the code in the function in the context of a pre-defined user identity (the identity of the IIS application pool in which the code is running). I really liked the ease of invocation of RunWithElevatedPrivileges method, but I needed to have a method that allowed me to specify the user name and password of any arbitrary user account.

The RunAs method is born

I used Viji’s code snippet as a starting point, and enhanced it to accept a delegate function. The result is that I can easily call RunAs, passing in a username and password, and a block of code I want executed under that user’s identity. Calling RunAs looks like this:

   1:  BlackBlade.Utilities.SecurityUtilities.RunAs(delegate()

   2:      {

   3:          File.Move(“local_file”, “network_file_location”);

   4:      },

   5:      “a_username”,

   6:      “some_password”);

Line 3 in the above code snippet is the code that will be executed with the credentials supplied to the RunAs method. You can have as many lines of code between lines 2 and 4. And those lines of code can interact with variables outside the code block. The RunAs method takes care of the details of interoperating with the Win32 API and handling the impersonation and reversion of .Net security contexts.

RunAs implemented

Here’s the full implementation of the RunAs method. Less than 75 lines of code. You can compile this as an assembly and reference it form other projects. It’s very handy.

   1:  using System;

   2:  using System.Collections.Generic;

   3:  using System.Text;

   4:   

   5:  using System.Security.Principal;

   6:  using System.Runtime.InteropServices;

   7:   

   8:  namespace BlackBlade.Utilities

   9:  {

  10:      public class SecurityUtilities

  11:      {

  12:          [DllImport("advapi32.dll", SetLastError = true)]

  13:          private static extern bool LogonUser(string lpszUserName, string lpszDomain, string lpszPassword, int dwLogonType, int dwLogonProvider, out IntPtr phToken);

  14:          public delegate void RunAsDelegate();

  15:   

  16:          public static void RunAs(RunAsDelegate MethodToRunAs, string Username, string Password)

  17:          {

  18:              string userName;

  19:              string domain;

  20:   

  21:              if (Username.IndexOf('\\') > 0)

  22:              {

  23:                  //a domain name was supplied

  24:                  string[] usernameArray = Username.Split('\\');

  25:                  userName = usernameArray[1];

  26:                  domain = usernameArray[0];

  27:              }

  28:              else

  29:              {

  30:                  //there was no domain name supplied

  31:                  userName = Username;

  32:                  domain = ".";

  33:              }

  34:   

  35:              RunAs(MethodToRunAs, userName, Password, domain);

  36:          }

  37:   

  38:          public static void RunAs(RunAsDelegate MethodToRunAs, string Username, string Password, string Domain)

  39:          {

  40:              IntPtr imp_token;

  41:              WindowsIdentity wid_admin = null;

  42:              WindowsImpersonationContext wic = null;

  43:   

  44:              try

  45:              {

  46:                  if (LogonUser(Username, string.IsNullOrEmpty(Domain) ? "." : Domain, Password, 9, 0, out imp_token))

  47:                  {

  48:                      //the impersonation suceeded

  49:                      wid_admin = new WindowsIdentity(imp_token);

  50:                      wic = wid_admin.Impersonate();

  51:   

  52:                      //run the delegate method

  53:                      MethodToRunAs();

  54:                  }

  55:                  else

  56:                      throw new Exception(string.Format("Could not impersonate user {0} in domain {1} with the specified password.", Username, Domain));

  57:              }

  58:              catch (Exception se)

  59:              {

  60:                  int ret = Marshal.GetLastWin32Error();

  61:                  if (wic != null)

  62:                      wic.Undo();

  63:   

  64:                  throw new Exception("Error code: " + ret.ToString(), se);

  65:              }

  66:              finally

  67:              {

  68:                  //revert to self

  69:                  if (wic != null)

  70:                      wic.Undo();

  71:              }

  72:          }

  73:      }

  74:  }

Line 53 in the previous code block invokes the caller-supplied code that should be run unsung the new impersonation context.

Technorati Tags: .Net,C#,Impersonation,LogonUser

Windows 7 RTM now available for MSDN Subscribers

The Windows 7 RTM along with support and technical content is now available for download to MSDN subscribers.

• MSDN
• Windows 7 Downloads

Technorati Tags: Windows 7,MSDN

Missing History in TFS

A few months ago our Team Foundation Server (TFS) box failed and we lost some data. I thought our new TFS box was headed for the same end the other day when I pulled up a history list for a Visual Studio project and saw that there were not nearly as many entries in the history list as there should have been.

A few minutes and a few valiums later, I figured out why there were so few entries in the history list. It turns out there are two ways to get the TFS history from Visual Studio.

View History from Solution Explorer

The first way is to select the View History option from the context menu of a file in in the Visual Studio Solution Explorer (see screenshot below). This way of viewing TFS history only pulls up the history items for the specifically selected item (the selected project file in this case), and not a rolled-up history for its child items (the files within the project). The screenshot below and the right shows the history list for just the selected project.

   

View History from Source Control Explorer

The other way to view TFS from Visual Studio is to do so from an item’s context menu in the Source Control Explorer in Visual Studio. From the screenshot below and to the left, there is no visual distinction between the View History menu item in the Source Control Explorer and the one in the Solution Explorer. However, look at the change-sets returned by the View History menu item from the Source Control Explorer in the screenshot below and to the right. There are three change-sets that for the project that were not present in the history view returned by the Solution Explorer. These additional change-sets represent a rolled-up view of changes that were made to files within the project (code changes), but that did not include changes to the project file itself (like adding a file to or removing a file from the project). The history view from the Source Control Explorer shows a complete roll-up of all changes made to the selected item as well as child items.

   

Summary

The moral of this story is:

• If you want to see TFS change history for just the selected item, use the View History command from the items context menu in the Visual Studio Solution Explorer.
• If you want to see a rolled-up TFS change history for the item and any child items, use the use the View History command from the items context menu in the Visual Studio Source Control Explorer.

Say “No” to Java Trial-ware

The latest update to the Sun Microsystems’ (Sun) Java Runtime Environment (JRE) for Windows comes with a trial-ware Internet backup service called “Carbonite Online Backup”. Unsuspecting users who install the Java update will also be installing potentially unwanted software that could cost them $5 per month.

I’m a seasoned geek. I’ve grown to expect such antics from less reputable Internet companies. No names need be mentioned. But I’ve generally considered Sun a trust-worthy software provider, especially in terms of its JRE for Windows. This is a low move, trying to sneak trial-ware into a software distribution channel that in the past has had a high degree of integrity. Sun: please remove the trial-ware from future JRE updates.

AnyCPU, x86, x64 – What’s the Difference?

A colleague of mine recently posed an interesting question: Why would you ever want to specifically target an assembly compilation to x86 or x64? Wouldn’t you always want to use the AnyCPU option?

What are the options?

Here’s a bit of background on what each option does:

• AnyCPU – The assembly is compiled in a CPU-independent manner. If you are compiling an EXE, the EXE will run as an x64 processes when loaded by an x64 version of the .Net Framework on an x64 operating system. Otherwise the EXE will run as an x86 process. When you’re compiling a DLL, the DLL will load as an x64 DLL when loaded from an x64 process. Otherwise, the DLL will load as an x86 DLL.
• x86 – The assembly will always load as an x86 assembly, regardless of the operating system or .Net Framework version. The assembly will not load from an x64 process.
• x64 – The assembly will always load as an x64 assembly, regardless of the operating system or .Net Framework version. The assembly will not load on an x86 operating system or from an x86 process.

Well, it looks like the AnyCPU option is definitely the way to go. You get the most flexibility with the least amount of headaches due to platform and process differences. In most cases (99 out of 100) that is true.

Subtleties of assembly loading

However, there is one subtlety to how assemblies load into processes. A process can only load assemblies that are of the same architecture at runtime as the process itself. An x64 process can only load x64 assemblies, and x86 processes can only load x86 assemblies. This restriction still applies to assemblies compiled with the AnyCPU option. Even though assemblies compiled with the AnyCPU option are processor architecture-independent at compile-time, the assemblies do get tied to a particular processor architecture when they are loaded for execution. If a process tries to load an assembly with an incompatible architecture, a BadImageFormatException will be thrown.

Why does that matter? Wouldn’t all of this be irrelevant if all assemblies were compiled with the AnyCPU option? Yes. Except, that not all code your assembly needs to load or that will be loading your assembly is processor architecture-independent. In particular, this is most apparent when you need to interoperate with un-managed code, such as COM.

So what? I only use managed code

Many .Net developers believe that all of .Net is a brand new, shiny framework that frees them from the headaches or working with legacy runtimes and unmanaged binaries. This is not true. In fact, a large portion of the .Net Framework simply wraps existing un-managed operating system components, such as the EnterpriseServices and DirectoryServices namespaces. Here’s a screenshot of a COM+ component EnterpriseServices uses to extend transaction support into the .Net Framework:

There are also many components and products that are based on un-managed code and only wraps this code for interoperability with .Net assemblies. Many Microsoft applications fall into this category as well, such as Microsoft Office, even if you are using the Office primary interop assemblies.

When do I need to specify x86 or x64 target for my assembly?

Given what we now know about how assemblies are loaded, we can now see that when we need to interoperate with un-managed code, even code that has .Net wrappers, we may need to target our assemblies, particularly when our process or assembly is likely to be loaded as a 64-bit (x64) architecture but the unmanaged code is of 32-bit (x86) architecture.

For example, say we need to automate a Microsoft Office application and our assembly is running on an x64 operating system. We need to have a 32-bit process, since Office is strictly a 32-bit application (that is until Office 2010 is released), but because our process is running on an x64 operating system, the process will be loaded in x64 mode. In this case, we want to specify that our assembly should always be targeted to x86 architecture, even when running on an x64 operating system.