AIIM ‘Sharepoint Meets ECM’ sessions reviewed

Greg Clark of C3 associates has been attending the ‘AIIM Sharepoint Meets ECM’ sessions in Chicago and has written a couple of excellent blogs about Sharepoint as an ECM platform. The blog covers different opinions on the richness of MOSS as an ECM system. I recommend reading them if you are into ECM.

Liveblogging AIIM SharePoint Meets ECM Session (1 of 5)

SharePoint Meets ECM: Doculabs Summary (2 of 5)

SharePoint Meets ECM: Document Imaging Breakout Sessions (3 of 5)

SharePoint Meets ECM: SharePoint as an ECM Platform (4 of 5)

SharePoint Meets ECM: Doculabs on the Positioning of SharePoint and Traditional ECM Tools (5 of 5)

External storage API for WSS, opening the door for CAS and HSM storage?

In my ECM article I mentioned that WSS didn’t provide support external storage for files and documents. However, Microsoft has recently released new functionality in an API trough a hotfix (which is probably why I didn’t notice it before. Who puts new functionality in a hotfix?) . I didn’t have any time yet to investigate further in this API, but sounding from the description it looks promising when there is an requirement for CAS & HSM storage of Sharepoint documents.

“An external storage API is available for Microsoft Windows Sharepoint Services 3.0. The external storage API lets you store documents or files on an external storage device other than Microsoft SQL Server. This API also lets you upgrade existing Windows Sharepoint Services 3.0 sites to point to an external storage device.”

It could be a very powerful feature in the records center in combination with the DoD-5015.2 that should be released fall 2007. Too bad that  there isn’t any documentation available yet.

You can find information on the hotfix here:
http://support.microsoft.com/kb/938499/en-us

Getting started with WCM (Part 1) – WCM Introduction

Before you you can start building an WCM Internet presence site. You need to know WCM. Its capabilities and limitations. If you are new MOSS WCM i would recommend reading every part of MOSS WCM you can find on the Internet. For now, I will post a couple of good resources here which should get you started immediately. I highly recommend starting with the videos to get the general idea.

Understand WCM core concepts

Get right to the action with 2007 Office System Video: Enterprise Content Management. Follow up with Customizing and Branding Web Content Management-Enabled Sharepoint Sites (Part 1 of 3): Understanding Web Content Management and the Default Features series.

Read the Microsoft’s excellent resources Managing Web Content and the conceptual WCM and branding – How do I…. If you get the overall idea you can start reading the planning documentation with Plan Web site structure and publishing (Office Sharepoint Server). 

The latter includes a reference to some good whitepapers: ‘Building Multilingual Solutions by Using Sharepoint Products and Technologies’  (also covers localization of custom webparts), ‘Information architecture in Office Sharepoint Server‘ and the excellent Web Publishing and Planning Guide with Microsoft Office Sharepoint Server book

Know what consequences an moss presence site has for security environment and what to do about it here: How to lockdown an Internet facing MOSS-based web site and Planning, Designing & Securing an Extranet and Internet Facing WSSv3 and Sharepoint Server 2007 Environments

You are probably going to use content deployment, and you might have some troubles with it. There is an excellent blog post that referers to situations regarding content deployment check it out here.

While you are at it, check out the hardware requirements for an WCM configuration at Estimate performance and capacity requirements for Internet environments (Office Sharepoint Server)

Master Sharepoint designer

Sharepoint designer is your sidekick in MOSS WCM world.  So make sure SPD becomes second nature.

WS313 – Microsoft Office SharePoint Designer 2007: Building No-Code Sharepoint Solutions (WS313_Mauceri.wmv)
Also get the 40 application templates under the hood for some general Sharepoint Designer patterns and practices.

Get in-depth

As you build an WCM presence site you might run into several requirements that WCM doesn’t cover out of the box. It is key to know how to extend MOSS WCM. MOSS uses the ASP.NET 2.0 provider model extensively. The two most frequent custom implemented providers are the SiteMapProvider and the MemberShipProvider.

Learn from others’ experiences

There are a couple of good tutorials on how other companies incorporated WCM.  See the How we did it.. – SPS Blog series. This blog references to real world implementations which covered implications like Search Engine Rankings & SEO, reducing page size , hybrid authentication providers (#Authentication) and ‘friendly URLs‘.

More resources

Basically your one stop shop for all other WCM resources, purposely put it as last so it doesn’t confuse with it sheer amount of information.
http://www.andrewconnell.com/blog/articles/MossWcmResources.aspx

Microsoft Enterprise Content Management Team blog (Includes WCM)
http://blogs.msdn.com/ecm/

Microsoft Sharepoint Blog
http://blogs.msdn.com/sharepoint/

Upcoming WCM series

Recently, I’ve been contracted for building and configuring an presence site for an Dutch company with multiple offices across Europe . I will dedicate some of my upcoming posts to the process that involves creating an multinational site.

If you are into WCM, watch this space.

The role of MOSS in an large scale ECM environment (Part 2)

In this series

- Part one
- Part two

Introduction

In the previous post I gave an introduction what it takes to be an ECM enterprise. At this point any Sharepoint professional can already detect what functional requirements lack in the current MOSS implementation. However, be aware that only a relatively small portion of companies will actually need all the requirements of an such an ECM enterprise. So you want to review whether the standard MOSS ECM features provide enough functionality for your company. In this post I will dig deeper on the matter, and where to draw the line in the gray area of MOSS only or extending MOSS with third party ECM tooling. In many situations it is  unclear for both parties. The ECM party knows what it means to be an ‘true’ ECM but doesn’t seem have enough Sharepoint knowledge or experience to make a comprehensive decision. On the other hand, the overall Sharepoint party is new to ECM and has none to limited ECM field experience.

Its not really a matter whether the full ECM functionality is possible in Sharepoint since the foundation of MOSS is excellent and every ECM requirement can be custom build on top of MOSS. But more whether it is out-of the-box available, and ready for the real world.  For instance, if you want to store millions of files you can easily develop logic that distribute the files over different document libraries and folders. Single instance archiving and CAS can be achieved custom policy development. So you can basically bend Sharepoint enough to be full ECM compliant. However, custom development is costly and it is your responsibility to maintain the code throughout your enterprise existence. You also want to incorporate more tooling and content services to manage your content on an overall basis. You also need to keep in mind to keep the code working with every iteration of MOSS.

Is MOSS out-of-the-box ECM enterprise ready?

As an independent Sharepoint consultant I feel to obliged to advise clients on whether to use Sharepoint standalone or consider augmenting Sharepoint with third party ECM tooling. So is the current version of MOSS out-of-the-box ECM enterprise ready? Well, it really depends on your company requirements. As it comes to the features, MOSS has a wide range of ECM capabilities that would cover most companies’ needs. However, its not as specialized or out-of-the-box ready as the older ECM players of this world like OpenText, Interwoven or EMC2. MOSS has very good and tight office integration and user experience for creation of documents and supersedes other players in this area. But lacks the out-of-the box ECM features like document imaging, single instance archiving and hierarchical storage management that other vendors provide.

MOSS is very good for the ‘active’ part of document management like ad-hoc document creation, collaboration, basic approval workflows and sorts. But for more complex services one might rely on other vendor’s products or custom development. As for record management MOSS is certainly going in the right direction, but at this iteration of MOSS there is no information on the amount of data the record center can handle. Also the current out-of-the-box functionality is rather limited. However, the 5015.2-STD add-on pack which will be released later this year (Fall 2007) will increase the amount of functionality, and will cover probably enough of the record management requirements for most companies. There is however no detailed information available on what this record management pack will include, or whether it supports CAS or HSM.

Common ECM scenarios when one might consider the use third party add-ons or custom development

I’ve created an basic ECM requirement list that can be used to identify whether one might want to extend MOSS with third party tooling. Keep in mind that this is just a very limited set of requirements. At the moment I am building an extensive requirement list that can act as a guide when making a choice.

ECM requirement

Need for a single instance archiving

Need for hierarchical storage management or content addressed storage

Need for complex workflows, that need to be globally monitored

Need for high volume batch import & export of large quantities of files

Need for multiple classifications of files

Need for meta-data inheritance

Need for digital imaging

Need for extensive reporting over content

Need for rich content services like case management

Need for extremely large storage (Dozens of TBS)

The role of MOSS in an ECM enterprise

Depending on the amount of content and the level of ECM complexity your company requires, one could go for an MOSS only solution or augment MOSS. If the content management complexity is fairly limited, no rich content services required and the amount of content doesn’t fall in the range of HSM you could configure MOSS as your central record system. I do recommend to configure a very hard and explicit line between what is active and what is passive and can be declared as an record. Without such a explicit architecture you are risking fragmentation of your information.

Personally, I think that an augmentation or an co-existence of both ECM systems would create the ideal solution for a large full fledged ECM enterprise. This way one can use MOSS for the active in-flight part of document management such as ad-hoc creation and management of content and basic workflow. Use sites for showing context related files from external repositories. Use the CAS and HSM, DIS, case management functionality provided by third party tooling. When information is solid enough to be of critical value and can be defined as a record, it can be transparently transferred from MOSS to a more advanced ECM system. Also from my point of view, Sharepoint should not be bothered with the high speed import of large quantities of files. After classification only the files that need to be processed by humans should be promoted to the Sharepoint environment to be augmented. The others can go to the advanced ECM system directly.

However the tooling nowadays  provided by the leading ECM parties create an overlap of functionality which you might not want to pay for. The whitepaper of Doculabs describes more on this issue. Personally, I think its a matter of time before ECM vendors start building software solely for bridging the gap without the functional overlap. Microsoft surely provides enough extension points and documentation in its MOSS & WSS foundation to enable this.

What others say

Gartner has written an couple of good documents on MOSS regarding to its ECM capabilities and its position in the ECM world. Unfortunately, since these require you to be a member of Gartner, I can’t go in detail. Make sure to get them if you are.

Doculabs provides an excellent (free) whitepaper regarding the coexistence of MOSS in an ECM environment.

AIIM has an (free) seminar dedicated solely on how to efficiently let MOSS2007 and core ECM systems coexist in the same environment. If you live in the US I would recommend visiting. You can find the link below. I would highly recommend reading this. 

There are more resources available in the section ‘Recommended Reading’.

What’s next

My upcoming project will be a WCM one, so I probably won’t be blogging too much these two months about this topic. However, I am scheduled for another ECM project which involves integrating MOSS with an external ECM system later this year.

Recommended reading

Gartner:
 - Microsoft’s 2007 Sharepoint Products and Technologies in Action (June 2007) - (Excellent)
 - Q&A: Microsoft’s Content Management Software and Strategy (September 2006) 

Doculabs:
 - Analyst Report: Doculabs: The Coexistence of Sharepoint and Advanced ECM platforms ? What You Need to Know  (See the solution information bar)
 - Microsoft Sharepoint 2007 and Your Existing ECM Solution: Which Should Be the System of Record?

AIIM:
 - Free seminar by AIIM  ‘Sharepoint meets ECM’ 

Misc:
 - Office 2007, Records Management, and ECM

Blogs:
 - Sharepoint meets ECM  (AIIM)
 - Better ECM
 - So is MOSS an ECM tool or not? (C3 Associates)
 - EMC Industry Watch

Back from France! And back for business!

Well, I’m back from my holiday in beautiful France. Now plowing trough all the posts in my RSS reader. Some interesting things you probably already know, nevertheless I’ll blog them here. These nifty updates make my social life disappear once again.

The new Moss2007 SDK 2.1 & WSS 3.0 SDK 2.1

You can download the MOSS SDK here and the WSS SDK here at microsoft.com.

Here’s an excerpt of some of the new tools (from the link below):

Business data catalog samples and utilities

• Microsoft business data catalog definition Editor
  (No more XML writing for BDC) And free too. More on this tool here (SPS Teamblog)
• WSHelloWorld web service
• Excel services user defined function sample
• WSOrders custom proxy sample
• SAP sample

Document management and content processing samples

• Comment scrub document converter
• Term replacement document inspector

Search samples

• Sample protocol handler
• Custom content source

For a full description on the contents of this SDK will refer to this nice post.  (SPS Teamblog)

Visual Studio Extensions for Sharepoint development CTP

Glad to see that the SPS team is still building more tools to aid Sharepoint development. The Sharepoint designer team released an new version of the VSeWSS kit that extends visual studio to help the developer create Sharepoint solutions. Personally, I really like the WSP view. It nicely abstracts all the files needed for an WSP solution and shows the developer the Sharepoint extension points only (like libraries, webparts etc). One could consider it as an functional view on your Sharepoint solution. Although I miss the functionality to add new functional components in the WSP tree, it is still pretty interesting. I’m also wondering why the team doesn’t use the Guidance Automation Toolkit for building such a factory tool.

You can find more on the VSeWSS here.

Looking for Sharepoint experts!

I’m looking for more Sharepoint developers, consultants, analysts, to exchange information, implementation experiences and projects with! If you are one, you know one, you might know one but you are not entirely sure. Refer him/her to me. That would make me very happy.

LinkedIn: http://www.linkedin.com/in/ebosch

The role of MOSS in an large scale ECM environment (Part 1)

In this series

- Part one
- Part two

Introduction

One of my clients asked me whether MOSS can be considered a ‘true’ ECM application. That certainly raised an eyebrow. Well, sure it supports ECM functionality like record management, document management IRM and sorts but does that make MOSS an enterprise ready ‘true’ ECM system?

The couple of weeks after I have been discussing the matter with ECM and fellow Sharepoint consultants. The whole large scale ECM world was new for me, but it is quite an interesting one. Now, I am not an ECM wizard but I wish to share my experiences nevertheless.

In this series of posts I will dig deeper into the matter and provide a insight in how such an enterprise works. Furthermore, I will discuss the capabilities and the role of MOSS in such an organization. What you should and what you should not do.

Defining the scope

So the first question one might ask is, what defines an ECM system? In what context are we going examine Sharepoint’s ECM abilities? For answering this question I will be referring to Wikipedia’s definition (AIIM)

“Enterprise Content Management is the technologies used to Capture, Manage, Store, Preserve, and Deliver content and documents related to organizational processes. ECM tools and strategies allow the management of an organization’s unstructured information, wherever that information exists.”

To understand the span of this abstract explanation one needs to have at least limit knowledge on what it takes to be an ECM enterprise. So let’s take a look at your typical ECM enterprise.

Anatomy of an ECM enterprise in a nutshell

Imagine, that this nifty diagram is our typical ECM enterprise called ‘claims-R-us’. Everyday large amounts of claims forms are send trough regular paper mail and need to be extracted, managed, audited, augmented, stored and destroyed in our enterprise. As you can see, our role model enterprise exists out of a few functional areas:

Information acquisition (process services)

At some point information needs to be imported in the system trough channels. Each channel represent an information flow. This can be of all sorts like paper (regular forms), recorded sound (phone) or information from third parties.

Some channels require conversion of the original media before it can be stored digitally. Think of paper documents that need to be scanned to a certain format. Often law regulations enforces that information that enters the system and can be identified as a record, needs to be stored as-is thus in its original format.

Its not unusual for a large scale enterprise to scan up to half a million forms a day by using multiple scanners. The scanning process is the responsibility of a DIS (Digital Imaging System) .The information from these channels are often augmented with information that enables the process to identify their further journey trough the enterprise.

Information management (content services)

The information that enters the system needs to be classified. Classification happens often from an business point-of-view. So when the scanned document is an ‘car damage form’ it will be classified a such. Often there are multiple classifications possible per document. There will be made a record to keep track of the document instance and the classification data. Usually at this point an separation of document flow will occur. The system will identify which classifications of information can be files directly in the archive (like receipts) or whether it needs further processing (for instance analyze, addition and mutation of claim forms by humans)

Records management needs to be applied that enforces that law regulations such as DoD 5015.2-STD are met. Transformations may be required to make sure that the document is un-modifiable. When necessary IRM is applied. Information Workers work in processes to create new documents or edit existing ones.

This information management functional area is the central information system of your enterprise. All the common document management functionality such as editing, checking in and out, information worker flows take place here.

Storage management (repository services)

Information that is significant for your business needs to be stored somewhere safe. Excuses me as I go a little in depth here since this storage is such an important part of the process.

For this high volume storage of documents, enterprises often implement hierarchical storage management (HSM) which is a data storage technique which automatically moves data between high-cost and low-cost storage media. HSM systems exist because high-speed storage devices, such as hard disk drive arrays, are more expensive (per byte stored) than slower devices, such as optical discs and magnetic tape drives. (Wiki). The surpassing of storage medium is mostly based on activity. For instance, when there is lots of file activity the file remains on the high-speed storage device, with lesser activity it gets demoted to the slower disks.

Often these systems implement an technique called “Content Addressed Storage” (CAS). Which can be seen as a wardrobe at your local club. You give your coat, get a number in return, later on you supply the number and you will get your coat back. A CAS system does basically the same thing with files. It takes care itself on where to store the file, how it works internally, is not for you to worry. You can retrieve it any time. EMC Centera is an example of such a CAS system.

Coming up in the next series

So now we described an typical enterprise we can discuss the matter further in our next post. Where we will describe the possible roles of Sharepoint within this organisation.

Recommended reading

So, is MOSS an ECM Tool or Not?
http://www.c3associates.com/2007/04/25/so-is-moss-an-ecm-tool-or-not/

Update (17th-August): Edited some links and made introduction.

BDC: Writing programmatically to the BDC using GenericInvoker

Last couple of days I was ranting about my upcoming BDC Toolkit project, I didn’t fully test the writing theory yet.  There was quite a lack of documentation and I was worried for a while that I had to come back to my idea but luckily I don’t have to.

So you can actually write to your BDC sources using the object-model and the GenericInvoker method instance. It ain’t that hard, but it might be a bit getting used to.

For this sample we will need a table to write to (imagine that this rather poor table as your enormous enterprise datamodel). For simplicity sake I am writing directly to a table, if you have read my previous post you will know that you should use a stored procedure as an indirection layer for your datamodel.

The following script has been used for creating the table.

USE [MYDATABASE]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[Test](
    [MyValue] [int] NULL
) ON [PRIMARY]

First off, you need your regular XML application definition file. I clipped it for readability purposes.

<Method Name="AddRecordMethod">
    <Properties>
        <Property Name="RdbCommandText" Type="System.String">
            INSERT INTO Test (MyValue) VALUES (@MyValue)
        </Property>
        <Property Name="RdbCommandType"
                  Type="System.Data.CommandType">Text</Property>
    </Properties>
    <Parameters>
        <Parameter Direction="In" Name="@MyValue">
            <TypeDescriptor TypeName="System.Int32"
                 IdentifierName="MyValue"  Name="MyValue"/>
        </Parameter>
        <Parameter Direction="Return" Name="MyReturnValue">
            <TypeDescriptor TypeName="System.Int32" Name="MyReturnValue"/>
        </Parameter>
    </Parameters>
    <MethodInstances>
        <MethodInstance Name="AddRecord" Type="GenericInvoker"
                ReturnParameterName="MyReturnValue" />
    </MethodInstances>
</Method>

As you can see in the XML definition above, I’m using an generic invoker as my method instance. The return value serves no purpose here but is required by the MethodInstance tag.

The code below is used for executing this method on the Business Data Catalog.

SqlSessionProvider.Instance().
   SetSharedResourceProviderToUse(yourSSPName);

NamedLobSystemInstanceDictionary sysInstances = ApplicationRegistry.
  GetLobSystemInstances();

LobSystemInstance writeInstance = sysInstances["yourInstanceName"];

NamedEntityDictionary entities = writeInstance.GetEntities();
Entity testEntity = entities["yourEntityName"];

NamedMethodInstanceDictionary instances = testEntity.
GetMethodInstances();

//Gets the method definition for our AddRecord invoker
Method mymethod = instances["AddRecord"].GetMethod();

//Creates a set of default paramers (which we are going to overrule)
object[] param = mymethod.
     CreateDefaultParameterInstances(instances["AddRecord"]);

param[0] = 31; //This is the MyValue parameter
                         //we defined earlier

//Finally excute the method
testEntity.Execute(instances["AddRecord"], writeInstance, ref param);

So, now you can use the BDC as your unified access for reading and writing to your external system.

BDC: Introducing the BDCToolkit

I’ve created an BDC Toolkit stub project on CodePlex that should make life somewhat easier for developers that are struggling with the BDC runtime model. This can generate an typed layer to your BDC.  

Here is an excerpt from the project page:

So, how does it work?

The console application generates strongly typed code from your ‘Business Data Catalog Application Definition’ that describes the external system and generates one of the following artifacts:

Typed Data-Access-Layer
A strongly typed set of C# classes that provide an indirection layer for programming against the BDC. These classes map directly against the methods defined in the BDC application definition file. This way the developers don’t need to worry about the BDC object model, and enables them to focus more on the overall functionality of the integration process. The classes are two way, so reading and writing using the BDC is covered.

Web service Layer
A strongly typed web service that enables users to access their BDC application definition methods using a web service. It uses the same DAL as above but provides a thin web service layer so that users can easily integrate their BDC application definition in products like InfoPath.

So what are you waiting for? Check it out: BDC Toolkit. I will be committing files soon!