What seems like a lifetime ago I found myself as the requirements holder and manager for a project that was to bring a modern (for 2002) document management system to a small security company who some may have heard of, called NATO. Several years later we end up implementing a solution based on Microsoft SharePoint and little did I know at the time, but this project was set to transform my life as I evolved from British Army Officer to SharePoint consultant, developer and evangelist.

I’ll spare you the details of this metamorphic journey but at the outset I knew precious little about document management. Surely a document is a document is a document, right? We’ll that’s true if you view a document as simply a medium to convey information, because they all share that purpose in common.

However, over time it becomes clear that are different types of document. And here I am not talking about the obvious distinctions between emails formal correspondence, a PowerPoint presentation or a CAD drawing. No, I am talking about something even more basic than that; documents can be classified according to their purpose. My epiphany moment was the realisation that documents, in every organisation, and in every context can be classified into 3 main groups – or buckets as I chose to call them – each with a different lifecycle.

Controlled Documents

These are all the documents that are used to govern business processes. They might be externally produced, like government legislation or industry regulations as issued by governing bodies or they might be internally developed by an organisation, these days with the help of ChatGPT or Copilot, no doubt!

We are talking about such things as Policies, Procedures, Directives, Guidelines, SOPs, Work Instructions, Form Templates and even Corporate Strategy documents and Mission Statement. These are reference products that should be used by the staff to govern how they execute business processes.

There is only one logical instance of each of these documents, in the sense that there will only be one Procedure document that describes how to go about procuring paperclips, for example. There may be several physical or electronic instances of said documents, whether they be digital copies accessed from your corporate Intranet or hard copy printouts stored on bookshelves, but logically we only have (or should only have) a single Procedure to be followed when we feel compelled to buy paperclips.

This class of documents are most frequently referred to collectively as Controlled Documents which implies both that they are to be used to govern (control) business processes and that they need to be tightly controlled so that the staff are supported with Consistency (everyone is using the same information to execute the process) and Currency (everyone is using the latest approved version of that information).

Controlled Documents have a lifecycle where they will need to be reviewed, updated and formally approved as the latest authoritative product that the staff should use to govern their work. The review and update process may be triggered by several factors such as:

• Time: You organisation might have a directive that all Controlled Documents need to be reviewed periodically, say annually or biennially, regardless of whether an update results from that process.
• Law: Changes in the law may require an organisation to review some of its Controlled Documents.
• Environment: Changes in the environment might include societal changes at a glacial pace (such as with workplace equality) or be triggered by cataclysmic events (think of updated security procedures at airports in the wake of 9/11)
• Organisational: Such things as changes in leadership or business expansion into new markets or just when someone is charge says, “Its time be reviewed this…”.

The key point is that change is both normal and necessary.

It’s a deceptively simple process:

• The trigger causes a review.
• Subject matter experts work collaboratively to make any necessary updates.
• These changes get approved (signed off) by someone in authority.
• The approved new version is published to replace the earlier version in such a way that Currency and Consistency are preserved.

Whilst this process is seemly simple, in practice it can be difficult to manage and many an auditor has eked a living being able to pick holes in an organisation’s lack of an effective governance process in the management of their Controlled Documents.

And in case you think this stuff doesn’t really matter – think again! Failures to get this right can be cost organisations millions in fines or even result in an organisational implosion as happened to Author Anderson LLP after the Enron scandal!

In later blog posts I will describe how we can build a well-governed process for the lifecycle or Controlled Documents stored in SharePoint, but this article is focused on providing the foundational concepts on which to start.

Transaction Documents (Records)

Transaction Documents are those documents which an organisation consumes and generates as a result of executing business processes.

Some of these documents will be produced externally and received into the organisation, whilst others will be generated internally by the staff as a product of their work, but collectively, they provide a record of a transaction and hence why this class of documents are more usually and simply, referred to as Records!

This lifecycle for Records is very different from that of Controlled Documents but again, conceptually it is quite simple:

• An instance of a business process is triggered.
• Documents (which when completed will become part of the record) are generated by the organisation in order to achieve an intended business outcome.
• These documents may need to be sent to other parties so that they can perform their part in the process. These other parties might be:
  • Specific named people.
  • Other departments within the organisation.
  • Government agencies, partners, suppliers, and other external organisations.
• This may result in other documents, generated by these parties.
• And there may be much toing and froing of documents between stakeholders involved in the process until an outcome has been achieved and the process instance has been concluded such that no further work is needed.
• At which time all documents (records) relating to that business process are no longer relevant to current operations but must be retained somewhere as an archive.
• At some point in the future, these records may be deemed truly obsoleted and can be disposed of, although it should be noted that some records may need to be retained indefinitely.

Many organisations, especially large businesses who work in highly regulated industries, invest in an Electronic Document and Records Management System (EDRMS) to help them manage the lifecycle of their records in expectation that it provide them with the tools necessary to ensure compliance, avoid fines and keep executives out of jail!

These products tend to be expensive and require staff dedicated to running and managing them but they worth that investment if an organisation must produce their records for any manner of reasons including auditing, litigation and proof of compliance.

Most professional Records Managers would (still) not consider SharePoint to be complete EDRMS. It’s a highly capable Document Management System (DMS) for sure, but historically it has lacked the tools to ensure the effective preservation of records, and governance over the retention and disposal process.

Extensions to M365 environment, specifically Microsoft Purview, have introduced significant advances in information governance, security, risk and compliance and, like most high-end EDRMS solutions work at the product meta-level. That is to say they seek to provide management and compliance tools that work across multiple technologies and repositories of information, of which documents stored in SharePoint be only one. Other might include MS Team (chat and online meetings), Exchange for email correspondence and products for 3rd party vendors (Dropbox, Google Docs, Slack, Atlassian and dozens more).

Purview still doesn’t (currently) meet the expectations of most professional Records Managers, primarily because it doesn’t allow you to import a full set of standards which can then be used as a means to prove compliance. Records management is an activity driven by standards. But this post is intended to be foundational and so I won’t dwell on records management any more here, but will come back to it later, I’m sure.

Transient Documents

Anything that isn’t Controlled Document or a Record is by my definition a Transient Document. Transient Documents oil the wheels of business processes but do not have any long-term value to the organisation and do not materially add to the record.

Typically, this will include 90% of email and chat messages, although it is essential to understand that what is import is the information and not the medium through which that information is communicated. If a director sends an email to cancel a procurement order, that email is clearly important to the record and so must be treated as a Transaction Document. It is not Transient as it holds critical information about how the organisation decided to proceed (or in this case not proceed) with a business transaction.

Once they have served their purpose, Transient Documents can be disposed of. They have no value to the organisation and simply add noise to the pool of information if left lying around making it more difficult to find current, relevant information. The main challenge here is that most users tend to hoard information “just in case”, when there really is no justification to do so.

Scenario

Let’s walk through a simple scenario to make the distinction between these classes of documents crystal clear.

My boss walks over to the stationary cupboard and picks up a pack of paperclips and notices that we are running low. She send me an email tasking me to get more – “we can’t possible run out of this most essential of office supplies!”. Her email is a Transient Document, but I won’t dispose of it just yet in case some asks “Who told you to buy more paper clips?”.

Eager to please, but unfamiliar with the process, I strike up a Teams chat with Jenny in Procurement. “Hey Jenny, we need more paper clips, what do I do?”.

Jenny replies with a link to Stationary Procurement Procedure and to the Requisition Form that I need to complete to start the process. Our chat session is a Transient Document, but the procedure and the requisition form are Controlled Documents.

I digest the procedure and discover that I need a budget code and to complete the requisition form and send it to Jenny (she could have just told me that!). I look up the budget code, as the procedure was useful enough to tell me how to do that. I then download the form and fill it out to indicate that we need a gross pack of paperclips.

At this point my instance of the form becomes a Transaction Document destined to become part of the procurement record for this process instance. Note that source requisition form I used is more accurately described as a Form Template and remains a Controlled Document, it is my completed instance of the form that is the record.

I send the form to Jenny who raises a Purchase Order (Transaction Document) with Clips R Us, our preferred stationary supplier, who dutifully send back an invoice and shipping reference number (Transaction Document). Jenny forwards the PO and invoice to Colin in Accounts, who issues an Accounts Payable payment order (Transaction Document).

The paperclips arrive and Jenny and Jenny signs the receipt and scans a copy (Transaction Document) which she sends Colin to let him know to that he can now pay their invoice. Jenny then sends me an email to come pick up the paperclips from her desk (Transient Document).

Finally, I send a chat message to the boss to tell her she can sleep easy, paperclip stocks are back to normal (Transient Document) and then prune my email by deleting an messages from my inbox that don’t need to be retained.

In summary:

• The Stationary Procurement Procedure and to the Requisition Form Template are Controlled Documents which govern the process.
• The PO, Invoice, AP order and Receipt are Transaction Documents which collectively, constitute the record of this transaction.
• Email and chat messages between the boss and myself, myself and Jenny and between Jenny and Colin are Transient Documents and can be deleted.

In this scenario, we need to be careful with the emails, because it is used as the bearer mechanism for the Transaction Documents and so preserve the record we might chose to retain the email, complete with attachments, or download the attachments and store them somewhere useful, where they can be managed as a record.

In Conclusion

I hope I have promoted the case that not all documents have the same purpose and so how we need to manage the lifecycle of these documents is different.

At the simplest conceptual level there are 3 classes of documents that every organisation (more complex than a sole trader) will have. These are:

• Controlled Documents: This documents, such as policies, procedures and form templates which govern business processes.
• Records: This documents which are generated as a result of executing business processes.
• Transient Documents: Those documents which oil the wheels of business processes.

The effectiveness of Controlled Documents relies on a well governed system to ensure that the staff can rely on the guidance provided by these documents and that the information is current (the most up-to-date information) and consistent (the same information that everyone else is using).

Records need to be subject to good governance processes also, to ensure that documents we create or receive into the organisation are appropriately used, retained and disposed in compliance with organisational and regulatory standards.

Any documents that are neither Controlled Documents nor Records should be disposed of when they have served their purpose, as a matter of good housekeeping. Any document that has long term value and needs to be retained in not, by definition, a Transient Document.