SharePoint 2010 : Custom BCS connector for Search with Security Trimming, Batching, Incremental Crawling

This guide is designed to highlight the solution for enabling security trimming on search results in SharePoint 2010 for external database. It also focuses on large datasets and file system crawling. It provides an overview of the requirement, a solution for the same, and concludes with implementation of the solution suggested. In order to have item level security implemented and search result trimming based on security, we need to have the security descriptor generated for each record and present in the table for each row. BCS is used to pull data and index in SharePoint for implementing search driven applications, we need constant updation of data from the external system. In such systems we need incremental update to work to ensure CRUD operations. If you have documents stored in your External Systems then it is possible using the Business Connectivity Services (BCS) give your end users the ability to access and crawl these artifacts with ease. The Stream Accessor method allows you to pull System.Byte[] data from your External System and make it available from within SharePoint.

http://social.technet.microsoft.com/wiki/contents/articles/22945.sharepoint-2010-custom-bcs-connector-for-search-with-security-trimming-batching-incremental-crawling.aspx

How to programmatically update a content source in SharePoint 2010 enterprise / fast search

Following are the assembly references required to execute the code

using Microsoft.Office.Server.Search.Administration;
using Microsoft.SharePoint.Administration;
using Microsoft.SharePoint.BusinessData.Administration;
using Microsoft.SharePoint.BusinessData.Parser;
using Microsoft.SharePoint.BusinessData.SharedService;
using Microsoft.SharePoint.BusinessData.Infrastructure;
using Microsoft.BusinessData.Infrastructure;
using Microsoft.BusinessData.MetadataModel;
using Microsoft.Office.Server.Search;
using Microsoft.SharePoint.Search.Extended.Administration;
 

Here is the function that take two parameters as first is your central application url and other parameter to define if you want to configure fast search or enterprise search. You can use the code individually also in case you dont want to configure both.

public void UpdateContentSource(string WebAppUrl, bool IsFast)

        {

            try

            {

                string strContentSourceName = "<CustomContentSourceName>";

                string strLogSystemName = "<BDCLogSystemName>";

                //Content Source Update Started

                //execute this code if you are updaameting a fast search content source. else user for enterprise content source

                if (IsFast)

                {

                    foreach (SearchServiceApplication FASTServiceApp in SearchService.Service.SearchApplications)

                    {

                        SearchServiceApplicationType at = FASTServiceApp.SearchApplicationType;

                        if (FASTServiceApp.SearchApplicationType == SearchServiceApplicationType.ExtendedConnector)

                        {

                            Content FastContent = new Content(FASTServiceApp);

                            bool CheckExist = false;

                            ContentSourceCollection FastCSColl = FastContent.ContentSources;

                            foreach (ContentSource cs in FastCSColl)

                            {

                                if (cs.Name.Equals(strContentSourceName))

                                {

                                    CheckExist = true;

                                    Console.WriteLine(strContentSourceName + " FAST Search Content Source already exist");

                                }

                            }

                            BusinessDataContentSource CustomConSource = null;

                            if (CheckExist)

                            {

                                CustomConSource = (BusinessDataContentSource)FastCSColl[strContentSourceName];

                                if (CustomConSource.StartAddresses.Count > 0)

                                {

                                    CustomConSource.StartAddresses.Remove(BusinessDataContentSource.ConstructStartAddress("Default", new Guid("00000000-0000-0000-0000-000000000000"), strLogSystemName, strLogSystemName));

                                }

                                CustomConSource.Delete();

                                FastCSColl.Update();

                                Console.WriteLine(strContentSourceName + " FAST Search Content Source Deleted");

                            }

                            CustomConSource = (BusinessDataContentSource)FastCSColl.Create(typeof(BusinessDataContentSource), strContentSourceName);

                            CustomConSource.StartAddresses.Add(BusinessDataContentSource.ConstructStartAddress("Default", new Guid("00000000-0000-0000-0000-000000000000"), strLogSystemName, strLogSystemName));

                            CustomConSource.CrawlPriority = CrawlPriority.Normal;

                            FastCSColl.Update();

                            Console.WriteLine("External Source FAST Content Source Created");

                        }

                    }

                }

                else

                {

                    using (SPSite site = new SPSite(WebAppUrl))

                    {

                        foreach (SearchServiceApplication objServiceApp in SearchService.Service.SearchApplications)

                        {

                            if (objServiceApp.SearchApplicationType == SearchServiceApplicationType.Regular)

                            {

                                SearchServiceApplicationProxy proxy = (SearchServiceApplicationProxy)SearchServiceApplicationProxy.GetProxy(SPServiceContext.GetContext(site));

                                Guid appId = proxy.GetSearchServiceApplicationInfo().SearchServiceApplicationId;

                                SearchServiceApplication SearchApp = objServiceApp;         //SearchService.Service.SearchApplications.GetValue<SearchServiceApplication>(appId);

                                Content content = new Content(SearchApp);

                                ContentSourceCollection csColl = content.ContentSources;

                                bool CheckExist = false;

                                foreach (ContentSource cs in csColl)

                                {

                                    if (cs.Name.Equals(strContentSourceName))

                                    {

                                        CheckExist = true;

                                        Console.WriteLine(strContentSourceName + " is already exist");

                                    }

                                }

                                BusinessDataContentSource CustomConSource = null;

                                if (CheckExist)

                                {

                                    CustomConSource = (BusinessDataContentSource)csColl[strContentSourceName];

                                    if (CustomConSource.StartAddresses.Count > 0)

                                    {

                                        CustomConSource.StartAddresses.Remove(BusinessDataContentSource.ConstructStartAddress("Default", new Guid("00000000-0000-0000-0000-000000000000"), strLogSystemName, strLogSystemName));

                                    }

                                    CustomConSource.Delete();

                                    csColl.Update();

                                    Console.WriteLine(strContentSourceName + " Deleted");

                                }

                                CustomConSource = (BusinessDataContentSource)csColl.Create(typeof(BusinessDataContentSource), strContentSourceName);

                                CustomConSource.StartAddresses.Add(BusinessDataContentSource.ConstructStartAddress("Default", new Guid("00000000-0000-0000-0000-000000000000"), strLogSystemName, strLogSystemName));

                                CustomConSource.CrawlPriority = CrawlPriority.Normal;

                                csColl.Update();

                                Console.WriteLine("External Source Content Source Created");

                            }

                        }

                    }

                }

            }

            catch (Exception ex)

            {

                Console.WriteLine("Error:" + ex.Message);

            }

            Console.WriteLine("Content Source Update Ended");

        }

How to update BDC / BCS custom connector DLL programmatically in SharePoint 2010

using System.Management.Automation.Runspaces;

using System.Management.Automation;

public static void UpdateBDCDLL(string webappURL, string strModelName)

        {

            try

            {

// Mapping Assembly of BDC

                string DllPath = @"@"<DLL PHYSICAL PATH>";

                //create powershell runspace

                RunspaceConfiguration config = RunspaceConfiguration.Create();

                PSSnapInException OExSnapIn = null;

                PSSnapInInfo pssnap = config.AddPSSnapIn("Microsoft.SharePoint.PowerShell", out OExSnapIn);

                Runspace cmdlet = RunspaceFactory.CreateRunspace(config);                cmdlet.Open();

                RunspaceInvoke scriptInvoker = new RunspaceInvoke(cmdlet);

// set powershell execution policy to unrestricted

                scriptInvoker.Invoke("Set-ExecutionPolicy Unrestricted");  

// create a pipeline and load it with command object

                  Pipeline pipeline = cmdlet.CreatePipeline();

              StringBuilder sbPSCommand = new StringBuilder();

                sbPSCommand.AppendLine("$assemblyPath = '" + DllPath + "'");

                sbPSCommand.AppendLine("$lobSystem = '" + strModelName + "'");

                sbPSCommand.AppendLine("$url='" + webappURL + "'");

                sbPSCommand.AppendLine("$serviceContext = Get-SPServiceContext $url");

                sbPSCommand.AppendLine("$lobSystem = Get-SPBusinessDataCatalogMetadataObject -ServiceContext $serviceContext -BdcObjectType lobsystem -Name $lobSystem");

                sbPSCommand.AppendLine("Import-SPBusinessDataCatalogDotNetAssembly -LobSystem $lobSystem -Path $assemblyPath -Confirm:$false");

                Pipeline pipeline2 = cmdlet.CreatePipeline();

                pipeline2.Commands.AddScript(sbPSCommand.ToString());

                pipeline2.Invoke();

            }

            catch (Exception ex)

            {

                //exception handling goes here

            }

        }

Parameter count mismatch

If you are working with BCS to pull data for search service application or FAST Search then you might have encountered this error when ever you try to update a dll where you have added or removed parameters for either paging or some custom logic.

"Parameter Count Mismatch"

Here are the steps that you should use to resolve this issue :

1. Upload your dll in GAC
2. Run following commands in powershell



$url = "http://gdc-encore"
$assemblyPath = "Assembly path"
$lobSystem = "Lob system name"
Write-Host "Adding assembly to LOBSystem"
$serviceContext = Get-SPServiceContext $url
$lobSystem = Get-SPBusinessDataCatalogMetadataObject -ServiceContext $serviceContext -BdcObjectType lobsystem -Name $lobSystem
Import-SPBusinessDataCatalogDotNetAssembly -LobSystem $lobSystem -Path $assemblyPath



3. Reset search service from the services named "SharePoint Server Search 14" which is running mssdmn behind the scenes and caches the dll which causes the mismatch in assembly reference.

Disable - "Did you mean" feature for some words - Spell Checking in FAST Search - SharePoint 2010

In FAST Search Center, there is a feature - Did you mean... If user search for some word and there is some mistake in spelling then "Did you mean" feature show some suggestions.

FAST Search with SharePoint 2010 provides spell checking capability. So when a user types a wrong spelling for a word it suggest a correct word with Did you mean functionality. Suppose if we are searching with a "Shareepoint" it suggests that Did you mean sharepoint.

But if we do not want to offer some words as suggestion, means, words that are never offered as suggestions to searchers, then there is a feature - Add Spell Checking Exception, using that we can achieve this.

These are the following steps to do this:
1. Go to the Central Administration -> Manage Service Application -> FAST Query SSA.

2. Go to FAST Search Administration, Under Spell Checking you will find Spell Checking Management link.

3. Now in this screen we can add the words for which we dont want to recieve the suggestions. To add a exception click on the "Add Spell Checking Exception" as shown:

4. There is a timer job with the name "FAST Search Server 2010 for SharePoint Dictionary Compilation Job" which compiles these suggestion. We can run this job manually or wait for this job to run.

Once this job is completed user will not receive the suggestion for the word shareepoint.

SharePoint 2010 FAST Search Server - Erase/Reset the content index and Clear the content collection

In SharePoint 2010 FAST Search, if you have crawled your site and after that you have deleted some documents. Again, you have full crawled your site, but the document you have deleted still show while search. Because the content index and content collection is not reset for these documents.

In SharePoint Server 2010 to reset the Content index, we need to follow following steps:

1. Open Central Administration and in the Application Management section, click Manage service applications.
2. On the Service Applications page, in the list of service applications, click your FAST Search Content SSA.

3. On the Search Administration page, under Crawling, click Index Reset.

4. On the Search Service Application: Index Reset page, click Reset Now. A confirmation dialog box appears; click OK to confirm the content index reset. The Search Service Application: Search Administration page opens and the System Status is displayed.

Before you start any new crawls, you must manually clear the content from the content collection your FAST Search Content Search Service Application was feeding to. The default collection name is sp.
Note - that clearing the content collection is irreversible.

Manually Clear the content collection:

1. On the Start menu, click All Programs, Click Microsoft FAST Search Server 2010 for SharePoint.

2. Click the Microsoft FAST Search Server 2010 for SharePoint shell.
3. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:
Clear-FASTSearchContentCollection -Name [collection name]
e.g.:
       Clear-FASTSearchContentCollection -Name sp

Wait for the command to finish. After successful completion, you content collection will be empty.

And after that, if you do Full Crawl, the the deleted documents will not be available.

Thanks.