Introduction

This is the 5^th part of a series of blog posts about how I am seeking to build a new UI to access SharePoint using the D3 data visualisation library and SharePoint client-side web parts, built using the SPFx. So far in the series, we have:

• Part 1: Getting started with SPFx and D3
• Part 2: SPFx and D3 Trees
• Part 3: Making D3 Trees SharePointy in SPFx Web Parts
• Part 4: Controlling D3 Trees via SPFx Web Part Properties

We’ve managed to get a collapsible/expandable D3 tree up and rending inside a SharePoint client web part built using SPFx. In Part 3, we managed to make it look a bit less like a D3 demo and a bit more like a SharePoint navigation tool and in part 4, I demonstrated how we could externalise chart properties as web part properties so that users can easily control how the tree is rendered. And this work has resulted in something that looks like what is shown in the screenshot below:

Nodes that represent folder can be expanded and collapsed at will and leaf node, represent the documents, that might be contained within these folders. The only problem is, this data is fake. It comes from a data object that is statically define inside the web part as shown below:

private treeDataSet =
{
  "name": "Documents",  
  "children": [
    {
      "name": "Folder A",       
      "children": [
        {"name": "Doc 1"},
        {"name": "Doc 2"}]
    },
    {
      "name": "Folder B",       
      "children": [
        {"name": "Doc 3"},
        {"name": "Doc 4"},
        {"name": "Doc 5"},
        {"name": "Doc 6"},
        {"name": "Sub Folder B-1",          
          "children": [
            {"name": "Doc 7"},
            {"name": "Doc 8"},
            {"name": "Doc 9"},
            {"name": "Doc 10"}
          ]
        }
      ]
    }
  ]
};

The challenge before us is this; we need to be able to hook the web part up with an object in SharePoint in real time that can return us data that can be manipulated into a compatible format so that can be fed into the D3 Tree which can then be rendered inside the client web part.  That then is what I shall be covering in this post.

If you’ve not being following this series of articles then I suggest you start at the beginning as I think you might struggle to follow some elements of what is coming next without that context.

SharePoint Data Sources

But what to use as a SharePoint data source? There are potentially lots of SharePoint objects that we could use, including:

• Data in a text file that contains JSON or elements in CSV format which is stored in a SharePoint library somewhere.
• The folder structure of SharePoint Lists and Libraries.
• List views using group-by columns to provide us with the hierarchy we need for a tree.
• The metadata driven navigation settings of a document library. This has always been a great concept, poorly implemented IMHO and we could potentially use this approach to deliver upon the vision – how cool would that be?
• The Content Types in a list or library, possibly combined with the folder structure.
• The SharePoint User Profile Service (UPS) might be used to construct an organisation chart for example.
• A Term Set in the Managed Metadata Service (MMS) – I like this option as we can potentially use it to define a tree that cuts across site collection boundaries.
• And why limit it to SharePoint? We potentially have the whole Office 365 Graph at our disposal so could reach into Teams, Planner and Delve and elsewhere even outside of O365.
• And how about using the site collection structure to develop an interactive site map tool?
• Or we might be able to hook it up to search and build a tree using search refiners.

When you start to think about it, the possibilities are endless!

But we’ve got to start somewhere and so I thought I’d begin with a conceptually simple scenario and build the data from the folder structure of a SharePoint document library.

Data Access Strategies

Mmm, before we get building, we’d better think about how we are going to extract the data we need to generate the tree.

We could just run a single request when the web part is loaded to get back all the data items in the library, construct the JSON and feed it to the tree. I think that for many of the potential use case scenarios listed above, this will be just fine. And, I think that it will work just fine for the scenario I am starting with here, so long as the library contains a manageable number of documents. However, I envisage problems with this approach when the library contains several thousand documents as it will likely result in some appreciable delay while the data is first retrieved, loaded and the tree is constructed.

To maybe side-step this, we might consider caching the data. We could either save the tree data in a file that gets stored in some SharePoint library such as Site Assets or we could cache the data in a property of the web part itself. D3 is more than capable of rendering a tree with thousands of nodes in a timely way and loading data from the cache will of course be much faster as we wouldn’t have to wait for a query response from SharePoint, which is where I envisage any bottleneck might be.

Caching might help but what do we do when the cache needs updating i.e. someone adds or deletes a document? We’d have to invalidate the cache and then rebuild it again. On a busy library with documents being added, updated or deleted all the time we might not get much benefit from the cache and end up continually rebuilding it and as often as not and the user would still have to wait to load the web part with fresh data.

How about this then? Instead of making the user wait for data to be refreshed, why not asynchronously update a tree data file every time the library changes? In an on-prem SharePoint deployment we could do this I think by using Event Receivers. Every time a document is added, updated or deleted we could update the data file that feeds the web part with tree data. In SharePoint On-line though, that’s not so easy. We’d have to build a Remote Event Receiver (RER) solution or use Web Hooks or Azure Functions or write a Flow.

We might think to improve things by generating the refreshed data file periodically, instead of on demand. It might be perfectly acceptable for there to be a time delay before the data is refreshed. On-prem, we could write a custom timer job for this but that’s not possible for the SharePoint Online so we’d have to find some other way.

All of these options might work but they rely on other services to do their job and I don’t know about you, but I would prefer a solution with as few moving parts as possible.

Alternatively, we might also consider a pull-data-as-you-go strategy. That is to say, only pull the contents of the folder when the user expands the folder node. This might work but I’m a bit concerned that the slinky, sexy, gliding UI that D3 gives us will be hampered by waiting for the data to arrive asynchronously. Still we might get around this if we could fill the data far enough ahead. So, when the user clicks to expand a folder then we don’t just grab the immediate child content but all the grandchildren content as well. Initially we load data to Level 2 but when the user clicks on a Level 1 node, we retrieve that branch data down to Level 3, staying just that step ahead of the user. This is obviously a more technically challenging exercise as well as we’d have to find a way of merging new content into the tree as it arrives.

Another problem I foresee with the load-as-you-go approach is that we might want to build some quick search capability and this will be much easier to do if we have all the data objects are at our immediate disposal. It might be challenging and confusing for end users if we build a quick search capability that only scanned over the sub-set of documents that have thus far been loaded.

At this stage I have no definitive answers to these dilemmas as I don’t yet know the impact that capacity will have on performance as that remains an unknown at the moment. With that in mind, I’m going to start out simple and work with a relatively modest source library of about 300 documents and load all the data in one go. Then I’ll ramp up the number of documents in the library to see at what point it becomes unusable and then we will be better positioned to consider alternate strategies if required. If we discover that we can handle say 2000 documents in a library in a single query/tree rendering transaction, then I think I’ll settle for that because I think anything more than that will likely be too cumbersome to work with in the UI in any case.

I hope that sound like a fair enough plan.

The PnP JS Library

As we are now going to be working with SharePoint data, I strongly recommend that you make life easier for yourself by using the PnP JS library. This marvel of software engineering abstracts the complexities of most REST API calls that you will need to make. Sometimes you might have to resort to native REST but for the most part this library hides that detail, so you don’t need to worry about it.

As an aside, the self-imposed 2000 document limit for our UI, is not just an arbitrary figure. It happens to be the maximum number of item that the PnP JS library will pull back from a query in a single request, so it seemed an appropriate and fitting constraint.

Head on over to the Getting Started page and follow the guidance there to using npm to install the packages you will need in your project.

Be aware that if you use the PnP JS Library then your solution won’t natively work with Internet Explorer. To work round that limitation, you need to install the polyfills package for IE 11. Yes, this will just give you support for IE 11 but I think that that’s acceptable these days. In my view, providing a solution that is incompatible with IE11 is still currently not acceptable however, so I recommend you install the polyfills, import the package in your web part file and be done with it.

Hooking up to a Document Library

We have to find a way to let the user select which document library to be used for the visualisation. Now we could just take the easy way and ask the user to provide the name of a document library in the current site, in a textbox control in the property pane of the web part but that’s a bit crude. Instead, what I’m going to do here is to provide a dropdown control that will list all the document libraries in the site and let the user select one from the list.

Adding a PropertyPanelDropdown field control

The first thing we need to do is to add a field control to the web part property pane that will contain the list of libraries. As with all web part properties that we wish to expose for configuration by users, we need to declare a suitable field control in the getPropertyPaneConfiguration method and hook that up to suitable data property in the web part, as shown below:

protected getPropertyPaneConfiguration(): IPropertyPaneConfiguration {
  return {
    pages: [
      {
        header: {
          description: ""
        },
        groups: [
          {
            groupName: "LIBRARY",
            groupFields:[
              PropertyPaneDropdown('listId', {
                label: "Select a list",
                options: this.libraryOptions,                 
              })                                                 
            ]             
          }         
        ]
      }
    ]
  }
}

If you are not yet comfortable working with web part properties, might I suggest you check out my article on the subject: Adventures with the SharePoint Framework (SPFx): Part 6 – Web Part Properties.

I have declared a PropertyPaneDropdown control and linked it to the listId property of the web part interface as shown below:

export interface ID3TestWebPartProps {  
  listId: string;  
}

Whatever value is selected in the drop down will be returned and persisted in the listId property of the web part.

Library Options

The list of options available to the user (the list of document libraries for which they can select) must be loaded into the field control and those options are what gets returned by the libraryOptions property of the web part which is an array of IPropertyPanelDropdownOptions as show below:

private libraryOptions: IPropertyPaneDropdownOption[];

But this is just a property and we need to supply an actual array of values .

That magic happens within the onPropertyPaneConfigurationStart method which gets called prior to rendering the property pane and so provides us with the opportunity to initialise controls.

protected onPropertyPaneConfigurationStart(): void{

  if (!this.libraryOptions){   
    DataUtilities.getLibraryOptions().then((libOptions) => {
      this.libraryOptions = libOptions;
      this.context.propertyPane.refresh();       
      
    });   
  }
}

First, we check to see if the libraryOptions is null or empty as we only need to load the libraries once and it could be that the user has configured the web part before and so the list of library options has already been loaded.

However, assuming that this is the first time the user is accessing the web part property pane, the libraryOptions array will be null and then assume that there is a need to enumerate the options that represent the libraries within the current site.

I have created a DataUtilities class in a separate DataUtilities.ts file in the project. Like the D3Utilities class I developed in Part 2 of this series it is defined as a support package with static methods as shown below:

import { sp, Web, List, View, Views, Files, File, Item, Folder } from "@pnp/sp";
import { WebPartContext } from "@microsoft/sp-webpart-base";
import { IPropertyPaneDropdownOption } from '@microsoft/sp-webpart-base';

export abstract class DataUtilities {

    public static async getLibraries() : Promise<List[]> {
        return await sp.web.lists.filter("(BaseTemplate eq 101) and (Hidden eq false)").select("Title", "ID").get();            
    }

    public static async getLibraryOptions(): Promise<IPropertyPaneDropdownOption[]> {
        const libs = await DataUtilities.getLibraries();

        let libraryOptions: Array<IPropertyPaneDropdownOption> = new Array<IPropertyPaneDropdownOption>();

        libs.forEach((lib: any) => {           
            libraryOptions.push({ key: lib.Id, text: lib.Title });
        });

        return libraryOptions;
    }
}

The getLibraryOptions method returns a Promise of an array of IPropertyDropdownOptions items, which is what then gets assigned to the libraryOptions property of the web part and ultimately what provides the dropdown options in the PropertyPaneDropdown field control.

The getLibraryOptions method makes a call to another static method defined in the DataUtilities class, namely getLibraries. This method uses the PnP JS library to query the current web site for lists that are not hidden and have a BaseTemplate property of being equal to 101 (which signifies that the list is a document library). This is a wrapper around the REST API and just makes life so much easier.

Note how the query is selecting just the Title and ID properties of the list because that’s the only information we need to create the options to allow users to select libraries. The use of the filter and select elements in the query chain mean that the result returned from the SharePoint query will be as lean as it can possibly be.

When we get the results back from the call to getLibraries we then create a result array of IPropertyPaneDropdownOption and loop round the libraries pushing a new option item into the array and when that’s done, we return the array of options back to the onPropertyPaneConfigurationStart method of the web part and assign it to the libraryOptions property.

Note the need to call propertyPane.refresh method because unless you do, the list of libraries won’t appear.

Gulp serve and see it in action in the work bench.

Remember that this is real SharePoint data we are working with now so you’ll need to use the remote workbench of a test site somewhere, as the local workbench has no SharePoint context, unless you fake one which is too much like hard work for my liking.

This means calling gulp serve with the –nobrowser flag and then opening a suitable workbench page on a target test site somewhere. As SharePoint provides no UI to access the workbench, you’ll have to assemble the URL yourself as indicated below:

    {your test site URL}/_layouts/15/workbench.aspx

Be sure to chose a site that contains a suitable document library, one loaded with a reasonable set of folders and documents.

Because we have linked the PropertyPaneDropdown field control with the listId property of the web part, SPFx takes care or persisting the selected value, which is the ID of the library in this case and will restore the selected item if a matching option item is found in the field control.

It all begins to make sense!

Fetching Items

We’ve now got a reliable, elegant and persisted way to select a document library but we still don’t have any documents and folders returned as data from the library. For that we need to develop another static method in our DataUtilities class called getAllItems.

Getting items from a list or library is actually very simple with the PnP JS library, in fact it can all be done in a single line of code such as this:

public static async GetAllItems(listId: string): Promise<any>{   

    return await sp.web.lists.getById(listId).items.get();

}

There are a couple of problems here though.

First of all, the call to items.get only returns the first 100 items in the library. I can understand why this might be the case; it’s a sort of protection mechanism to make sure that the query doesn’t choke the UI.

Second, this returns us the list items, but in this scenario, we really need to be able to access the underlying files and folders for which the SharePoint list item is really just a wrapper.

We can address both of these issues by changing the method to the following:

public static async GetAllItems(listId: string): Promise<any>{   
   
    return await sp.web.lists.getById(listId).items.expand("File", "Folder").getAll();               

}

The call to expand gets us File and Folder child objects and the call to getAll at the end of the chain will return us all items in the library, up to the maximum of 2000 items as mentioned above.

I am aware that this will return everything is a flat array and have no hierarchical structure to it but that’s ok for now, we’ll come back to that in a minute.

We might also be able to refine and reduce the number of properties returned by the query by applying a select statement, just like I did for retrieving the list of libraries in the site but as I’m not sure exactly what properties I will need and so I shall save that fine-tuning exercise for a later date.

Tree Nodes

Now we have a way to get at all the list items, files and folders we need, although to build the tree data structure we really only need 2 elements:

• Text: Some text to display to identify the node, which for our document library scenario will be the name of the folder or the file name of the document.
• Children: A collection of child nodes representing the SharePoint objects with a  folders.

However, to build something more useful we are going to need more data than this and to make things nice and object-orientated what I’m going to do is create a new class that is used to define a node in the tree to represent each SharePoint object.

This TreeNode class is defined in its own file in the project called TreeNode.ts and is shown below:

import { NodeType } from './enums';
import { DataUtilities } from './DataUtilities';
import { ChartConfig } from './ChartConfig';

export class TreeNode {
   
    public name: string;   
    public nodeType: NodeType;
    public imageUrl: string;
    public imageUrlExpanded: string;
    public serverRelativeUrl: string;
    public parentFolderUrl: string;   
    public nodeObject: any;
    public itemCount: number = 0;
    public ind: number = 0;
    public children: Array<TreeNode>;

    constructor(item: any, config: ChartConfig){
                     
        if (item.File){
            this.nodeObject = item.File;
            this.nodeType = NodeType.File;
            this.name = this.nodeObject.Name;
            this.imageUrl = DataUtilities.getFileIconUrl(item.File.Name, config.iconSize);
        }

        else if (item.Folder){
            this.nodeObject = item.Folder;
            this.name = this.nodeObject.Name;
            this.itemCount = this.nodeObject.ItemCount;          
            if (this.nodeObject.ProgID == "Sharepoint.DocumentSet"){    
                this.nodeType = NodeType.DocumentSet; 
                this.imageUrl =  DataUtilities.getDocSetIconUrl(config.iconSize);
                this.imageUrlExpanded = DataUtilities.getDocSetIconUrl(config.iconSize);
            }  

            else {
                this.nodeType = NodeType.Folder;               
                this.imageUrl = DataUtilities.getFolderClosedIconUrl(config.iconSize);
                this.imageUrlExpanded =  DataUtilities.getFolderOpenIconUrl(config.iconSize);
            }           
        }

        else if (item.RootFolder){  //This will only be the case when the list is passed in as the item parameter
            this.name = item.Title
            this.nodeObject = item.RootFolder;
            this.nodeType = NodeType.RootFolder;
            this.itemCount = this.nodeObject.ItemCount;
            this.imageUrl =  DataUtilities.getLibraryIconUrl(config.iconSize);
        }
      
        this.serverRelativeUrl = this.nodeObject.ServerRelativeUrl;
        this.parentFolderUrl = (item.RootFolder) ? null : DataUtilities.getParentFolderUrl(this.serverRelativeUrl);
    }
                
    public appendChild(childNode: TreeNode){
        if (!this.children){
            this.children = new Array<TreeNode>();
        }
        this.children.push(childNode);
    }

}

This is a fairly simple data structure consisting of a few key properties as follow:

• name: The name of the node as it will be shown in the tree. This will essentially be the file or folder name.
• nodeType: This is a custom enumeration type (declared in the enums.ts file) that is used to differentiate between the different types of SharePoint object that might be represented by a node and can set to one of the following values:
  • File
  • Folder
  • DocumentSet
  • RootFolder
• imageUrl: The URL to a graphic used to represent the SharePoint object.
• imageUrlExpanded: The URL to a graphic file to represent a folder when it is expanded.
• serverRelativeUrl: The server relative URL of the SharePoint object being represented by the node.
• parentFolderUrl: The server relative URL of the parent folder in which the SharePoint object is contained.
• nodeObject: The actual instance of the object SharePoint object being represented by the node.
• itemCount: The immediate number of child items contained within a folder (or document set). This will be 0 for files and empty folders.
• children: Any array of TreeNode objects that will represent the files and folders within a folder.

I’m not sure whether we’ll actually need to store the SharePoint object in the tree nodes and, as mentioned above, at the moment I am probably bringing back from SharePoint much more data than I will actually be using. But these potential optimisations I shall leave for another day.

The class constructor requires 2 parameters as follow:

• item: This is the SharePoint object to be represented by the tree node and will either be a:
  • List: The root node in the tree will created from the SharePoint library itself, or more specifically the root folder in the library
  • List Item: The list item used to create other nodes in the tree will either have a File or a Folder child object depending on whether the list item is the bearer of a document or a folder/document set, respectively.
• config: This is an object instance of a ChartConfig class and is used to pass around various parameters that govern how the tree and nodes within it are to be rendered in the chart area.

Some OOP purists will be turning in their graves at this point because what I could (and probably should) have done is to make the TreeNode class abstract and then derived sub-classes for folders, files and document sets. Instead of which, I am using a single class to represent the different types of SharePoint object and potentially carrying around excess and unnecessary data properties. For example, the itemCount property for a file will always be zero and we don’t need to worry about the imageUrlExpanded property for files either. Again, these are optimisations that we might consider down the line as part of a refactoring exercise but for now, I think the current approach will be good enough.

We currently get the data back from SharePoint as a flat data set of course and so we’ll have to construct the hierarchy out of the data somehow and to support this I also defined a single method, called appendChild, that allows child tree nodes to be added to the children array of a node. As it turned out I didn’t need this method but I’ve left it in here for now, just to explain how I initially went about this task of constructing a hierarchy from flatness.

In the class constructor, I set various properties according to the type of object passed in as the item parameter. I determine the type of SharePoint we are dealing with by looking for child object properties of interest. For a SharePoint list item that represents a document, the File property will exist and have a value but the Folder property won’t exist. So, all we have to do is to check for the existence of these objects to know what sort of SharePoint object we are dealing with and then configure the tree node accordingly.

If a SharePoint List (or a rather a document library) is passed in as the item parameter it will have neither a folder nor a file but we can send it through with the Root Folder of the library and check for that.

A Document Set is really just a special kind of folder but notice how we can check the ProgID property to see whether we need to provision the tree node to represent either a Document Set or a folder. A standard folder will have null assigned to its ProgID property, whereas for a Document Set this value will be set to “Sharepoint.DocumentSet” (note the atypical text-case, its Sharepoint rather than the more usual SharePoint).

I think the rest of the code in the constructor is fairly straight forward. I basically set the URL of an appropriate graphic to be used for the object by calling out to helper methods defined in the DataUtilities class. I won’t go into the details of these methods here but essentially the getFileIconUrl method extracts the file extension from the file name and uses that to determine which of the graphics files in the standard SharePoint images folder to point to.

Actually, SharePoint provides us with the opportunity to exploit file type graphics for most common file types at 3 different sizes, namely:

• 16 x 16 pixels
• 32 x 32 pixels
• 256 x 256 pixels

Choosing which size of graphic to use is then the purpose of the iconSize property of the ChartConfig class and you can see that this property value is passed into the utility methods that return the graphics. So basically, I’m doing some thinking ahead here and planning for the possibility of the tree being able to use node graphics of different sizes (must try and stop doing that thinking ahead stuff).

I’m not going to go into the details of the ChartConfig class in this post, I’ll save it for the next one, because here I want to focus on how to get a collection of tree nodes that can be constructed from SharePoint objects, specifically files, folders/document sets and the library itself.

getAllItems Revisited

The getAllItems method that I showed earlier, returns an array of SharePoint objects. But actually, what we need is an array of TreeNode objects.  To achieve that, we then need to update the getAllItems method so that it now looks like the following:

public static async getAllItems(listId: string, config : ChartConfig): Promise<TreeNode[]>{   

    let items = await sp.web.lists.getById(listId).items.expand("File", "Folder").getAll();                          

    let treeNodes: Array<TreeNode> = new Array<TreeNode>();     

        items.forEach(item => {
                       
            let treeNode : TreeNode = new TreeNode(item, config);            

            treeNodes.push(treeNode);

        });       
   
    return treeNodes;
}

What’s happening here is that I first create an empty array to store TreeNode objects and then loop round the collection of items returned by the query and then for each I am creating a TreeNode and pushing it onto the array, before finally returning the array.

Building a Mountain out of a Pancake

The call to getAllItems still returns us a flat list of objects, all that’s different is that we are returning TreeNodes constructed using SharePoint objects and not the SharePoint objects themselves. Somehow, we’ve got to build a hierarchical structure out of this flatness and initially the way I did this was to write my own utility function that recursively parsed the collection of TreeNodes using the serverRelativeUrl and parentFolderUrl properties.

As most developers know, writing a recursive function often requires the mental dexterity of chess grand master combined with saintly patience, although, with success comes a hugely gratifying smugness of knowing how clever you can be when you put your mind to it!

So that’s how I justified the time (more of it than I’d care to admit to) I spent on developing the stackNodes method.

public static stackNodes(rootNode: TreeNode, nodes: TreeNode[]) : TreeNode[]{     

    if ((nodes) && (nodes.length > 0)){

        let remainingNodes : Array<TreeNode> = new Array<TreeNode>();          

        let nodeIndex : number = 0;

        nodes.forEach(node => {                              

            if (node.parentFolderUrl == rootNode.serverRelativeUrl){
                node.ind = nodeIndex;
                nodeIndex++;
                rootNode.appendChild(node);
            }

            else{
                remainingNodes.push(node);
            }                

        });
      
        if ((remainingNodes.length > 0) && (rootNode.children)){

            rootNode.children.forEach((childnode : TreeNode) => {

                this.stackNodes(childnode, remainingNodes);

            });

        }
    }

    else {
        return null;
    }

}

The stackNodes method is called with 2 parameters. The rootNode parameter is the TreeNode for which we will search the array of TreeNode objects passed in via the nodes parameter. We ascertain whether a specific node is a direct child of the rootNode by comparing the serverRelativeUrl value of the rootNode with the parentFolderUrl property of the node. Where they are equal, we know we have found an immediate child of the rootNode and so can append that node to the children array of the rootNode by calling the appendChild method of the TreeNode class.

If the node is not a match then it is added to an array of nodes, called remainingNodes which are those yet to find their home in the tree hierarchy.

While there are still items in the remainingNodes array we make a recursive call to stackNodes until such time as we have nothing left in remainingNodes which tells us that every node has found its proper place.

Smugness abound, and this what the render method of the web part now looks like:

public render(): void {

  if (this.selectedListId)
  {

    this.resetControls();
    this.domElement.innerHTML = `
`;

    DataUtilities.getRootNode(this.selectedListId, this.chartConfig).then((rootNode: TreeNode) => {       
      DataUtilities.getAllItems(this.selectedListId, this.chartConfig).then((nodes: TreeNode[]) => {

        DataUtilities.stackNodes(rootNode, nodes);                  
        
        this.chartConfig.root = d3.hierarchy(rootNode);             

        D3Utilities.initialiseNodes(this.chartConfig.root, 0, 0 );           

        D3Utilities.renderChart(this.chartConfig.root, this.chartConfig);           

      })

    })

  }

}

After the call to getAllItems which gives us the flat list of TreeNodes, we then call stackNodes which brings hierarchical structure into the nodes. After that we call the d3.hierarchy method as we have done before and similarly make subsequent calls to initialise the nodes and then render the tree. And it works!

Then my smugness over stackNodes evaporated as I discovered the d3.stratify method.

Stratification

If there were ever a justified reason to shout RTFM it would be about now because the time, I spent on writing stackNodes was pretty much wasted. It turns out that the d3.hierarchy method has a sister called d3.stratify. Both of these methods do basically the same thing in that they augment the tree nodes with coordinate data (x and y values), provide additional functions that will return the descendants and parent of a node and return the root node of the data structure. The difference is that when calling d3.hierarchy the hierarchical structure of the data is assumed to be in place. That’s what the call to stackNodes did for us. It created that hierarchal structure.

However, it turns out that I can call d3.stratify instead of d3.hierarchy. Stratify provides us with the means to sort out the hierarchical structure of the tree internally without the need of external utility methods. Using stratify instead of hierarchy, the render method now looks like this:

public render(): void {

  if (this.selectedListId)

  {
    this.resetControls();
    this.domElement.innerHTML = `
`;

    DataUtilities.getRootNode(this.selectedListId, this.chartConfig).then((rootNode: TreeNode) => {       
      DataUtilities.getAllItems(this.selectedListId, this.chartConfig).then((nodes: TreeNode[]) => {          

          nodes.push(rootNode);
         
          this.chartConfig.root = d3.stratify()
                                    .id(function (d : any) { return d.serverRelativeUrl;} )
                                    .parentId(function (d: any) { return d.parentFolderUrl;})(nodes);

          D3Utilities.initialiseNodes(this.chartConfig.root, 0, 0 );           

          D3Utilities.renderChart(this.chartConfig.root, this.chartConfig);         

      })
    })
  }
}

In the above code you can see that d3.stratify allows us to specify an id and parentId property using a function that returns a property of the node. In this case we are using the serverRelativeUrl property as the id and the parentFolderUrl as the parentId of the parent folder.

The only other significant difference is that the call to getAllItems does not include the rootNode itself. When we called stackNodes we coded round this because the method signature included the rootNode as a parameter. However, this doesn’t apply to d3.stratify which needs the complete set of nodes including the root node and so we need to push the rootNode into the array of nodes to be processed before the call to stratify is made.

As d3.stratify is likely to be more robust and better performing than my original code it means we can dispense with the stackNodes altogether. Not only that but we no longer need the appendChild method in the TreeNode class and can get rid of that as well!

The lesson here is to do some research before cutting code because someone has likely done the grunt work already and RTFM!

The Reveal

And so, we come to the reveal. We can now build a tree based on folder structure of a SharePoint library as depicted below:

We can expand and collapse folders at will and without post-backs and marvel how everything slides and glides into place. How cool is that?

In Summary

In this post I showed you how to access data from SharePoint using the PnP JS library and then how to transform folder and document objects returned by querying a SharePoint document library into a collection of tree nodes.

I then explained how to reconstitute a hierarchical structure from the flat data result set using the ServerRelativeUrl property which is common to both files and folders and which is guaranteed to be unique (SharePoint won’t let you have 2 objects with the same URL).

First, I built my own utility function to process the tree nodes into the required hierarchy but then I discovered that the d3.stratify method would do that for us.

More to come?

We’re not done yet of course. At the moment we’ve only got a way of visualising and navigating the tree but we don’t yet do anything useful like being able to show users a document preview, or metadata properties or provide a means to actually open documents. Pretty as this thing is, its still functionally pretty useless!

However, before we get onto that, I should point out that I have updated what goes on inside the renderChart method from what we built before. In the screenshot you might have noticed how I’m now rendering file graphics based on the file extension and I’m also now showing the number of child items within a folder as a number rendered in superscript on the top right of folder icon. In the next post, I’ll cover how I do that and also how we can add more properties to the web part to govern how it renders and behaves.