Integrating a custom processor in FAST pipeline – part 1 of 2

FAST for SharePoint 2010 allows us to integrate our own custom processor into the search engine’s processing pipeline. While attempting to modify the default processing pipeline may sound like an hazardous task, its actually not that bad. In part 1 of the series we will set the stage for our custom processor by creating a BCS connector, set the managed properties and add a refiner to the refinement panel.

For the purpose of this article, we’ll use BCS as a connector for a database with the following columns:

There is nothing special about the ProductID, ProductName or ProductDescription columns. These are just your regular run of the mill nvarchar and int columns. ProductTags on the other hand is a bit different.

ProductTags is going to represent a set of tags, separated by the | (pipeline) character. Once crawled, this property will serve as a refinement property. The problem with this column is that we need to separate the tags using the pipeline character so each tag will be counted as an individual tag. If our search result return 3 products with the tag Home Styling, the user expects that the refinement panel will show Home Styling (3).

So how do we get this result? how do we separate the tags? The short answer would be integrating a custom pipeline processor that will receive the tags as input, and replace the pipeline separator character with a character FAST recognize as a separator – the char u2029.

Before we go on with creating the custom processor make sure you have some products in your data base. Pay special attention to the ProductTags column and make sure you add data to it with the following format: tag|tag|tag etc.

To get the data from the SQL database to our SharePoint we are going to use a BCS connector built in SharePoint Designer 2010. Using SharePoint Designer 2010 to create BCS connections is a fast and simple method to create BCS connections and its great to use when the data you wish to represent is simple, as in our case.

Creating the BCS connection

To keep this post focused on the subject of item processing, follow the great walkthrough by Ingo Karstein – Create a simple BCS connection with SharePoint Designer 2010

Once you have the BCS created we can go ahead and create our crawling content source.

Creating a crawl content source

Fire up SharePoint 2010 central admin and choose Manage service applications. Choose your FAST content SSA and click Content Sources on the left side menu. We are now in the Manage Content Sources page.

Click on the New Content Source button on the upper horizontal bar to add a new content source to crawl. The Add Content Source page shows up. Name the new content source to your liking. At the Content Source Type area, make sure you select Line of Business Data and select the name of the BCS connector you created earlier.

In my system the form looks as follows:

Once all the fields are filled, scroll to the bottom of the page, check the Start full crawl of this content source box and click the OK button. Wait for the crawler to finish its thing and check the crawl log by clicking on Crawl Log on the left menu.If your content source shows successes or warnings you are clear to move on to the next step.

Mapping managed properties

If we try to perform a search right now we will see that the title of the result items is wrong. For example, if we search for the term Xbox, which i have in my database as a product name, we will get the following result:

Not the most informative title is it… In order to show the right title, we have to set the managed property Title to include data from the crawled property that represent ProductName from our database. This process is called mapping. We map a managed property to include data from crawled property.

Fire up the ol’ central admin and click on Manage service applications. Click on your FAST Query SSA application and then click on FAST Search Administration on the top part of the left menu. Under Property Management, click on Managed properties:

In the Managed properties page, search for the managed property Title. Once found, click on it and choose Edit Managed Property. At the Mappings to Crawled Properties section of the page click on the Add Mapping button. Change the category to Business Data, find the ProductName crawled property, mark it and click on Add. The name of the crawled property might differ a bit in your setup with an additional prefix such as read listelement or something similar.

Once added click on the OK button. Again under the Mappings to Crawled Properties section highlight the newly added property and using the Move Up button move it all the way to the top. That will make sure that FAST will use the data from this property.

Finally, click on the OK button and perform a full crawl again on our content source. If we perform the same search for Xbox now, the result will be as follows:

Now that we got this issue out of the way, let’s add the ProductTags property to the refinement panel.

Adding a refinement property

Before we can add our tags property to the refinement panel we first have to create, well a property for it. Once again, fire up the central admin, click on Manage service applications, then click on your FAST Query SSA application and finally click on FAST Search Administration on the top part of the left menu.

Under Property Management, click on Managed properties and then click on Add Managed Property. Name the new property as you wish. In my example i called it ProductTags.

Next, under the Mappings to Crawled Properties section make  sure the first radio button is selected and click on Add Mapping. Once again, change the category to Business Data and look for an entity with the name ProductTags. Add it and click OK.

Under the Refiner Property section check both Refiner property and Deep Refiner.

Under the Query Property section check the Query property check box and click OK.

For FAST to recognize our new property we must perform a crawl again, so go ahead and perform another full crawl on our content source. Once done get back to the search result page and under Site Settings click on Edit Page.

Click on Edit Web Part for the Refinement Panel web part and expend the Refinement section. Copy the xml from the Filter Category Definition property to your favorite text editor and add the following snippet above the closing FilterCategories tag (the last line):

<Category Title=Product Tags” Description=Tags for product” Type=Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator” MetadataThreshold=1
NumberOfFiltersToDisplay=4” MaxNumberOfFilters=20” ShowMoreLink=True” MappedProperty=ProductTags” MoreLinkText=show more” LessLinkText=show fewer” ShowCounts=Count />

Copy the XML back to the Filter Category Definition property and make sure you uncheck the Use Default Configuration check box!

Once done click OK and Save & Close to finish editing the page. The result of the change is as follows:

Well we are half way trough.. we have the tags showing in the refinement panel but they aren’t separated. All the tags are shown in the same line. As stated above, the reason for this is that FAST doesn’t recognize the pipeline char as a separator and as such treat the text as one big line.

To solve this issue we will need to build a custom processor to replace the pipeline char with FAST’s separator char. We will build this processor in part 2 of the series, so take a short rest (you deserve it after all these settings) and check back soon for part 2 🙂

Working with FAST Search For SharePoint 2010 Document Promotions Programmatically

As much as I love working with the FAST for SharePoint 2010 search engine, finding documentation for it online is quite a challenging task. This is why I’m going to try and document as much as possible about the little ‘gems’ I find while working with FAST.

Using FAST for SharePoint you can define documents to be promoted within the search results list, indicating that they are highly relevant results for a particular keyword or its synonyms. The promoted documents will appear at the top of the search result list.

While this feature works great when using OOTB Core Search Results web part, it fails miserably when using an extended Core Search Results web part (a web part that inherit from the base Core Search Results web part and extend its functionally, such as giving the developer full control over the FQL (Fast Query Language) query sent to the server).

Today I had a client who uses such web part and thus has to get the promoted documents urls programmatically so he can append an xrank property to the FQL query. This was much trickier task than expected. To get the promotions you first have to get the Search Setting group for your site. This can be done using the following code:

var ssaProxy = (SearchServiceApplicationProxy)SearchServiceApplicationProxy.GetProxy(SPServiceContext.Current);
if (ssaProxy.FASTAdminProxy != null)
    var fastProxy = ssaProxy.FASTAdminProxy;
    KeywordContext keywordContext = fastProxy.KeywordContext;
    var searchGroup = (from x in keywordContext.SearchSettingGroups where x.Name==SPContext.Current.Site.ID.ToString() select x).First();  

Once we have the search group, we need to get the keyword itself:

var getKeyWord = from x in searchGroup.Keywords where x.Term == searchKeyword select x;

Where searchKeyword is the term that was searched.

Keyword keyword = getKeyWord.ElementAt(0);

Now all we have to do is go through the collection of promoted items for that keyword, and look for a PromotedItem of type PromotedDocument:

foreach (var promo in keyword.Promotions)
foreach (var item in promo.PromotedItems)
if (item isPromotedDocument)
// your logic here

To get the URL for the promoted document use the following code:

(item asPromotedDocument).DocumentId

Once you have the url you can add it to your FQL query like

xrank(path:equals(“<DocumentId>”), boost=100000, boostall=yes)

I hope this article saves you a healthy dose of slamming your head against the wall while trying to figure out where the promoted documents urls are hiding!

Fast Tips for FAST for SharePoint #2

When trying to query for a document that has a question mark (?) in its name or address as part of a FQL query, you may notice weird behavior from FAST.

For example, let’s send a query that look for a document with the URL address of http://localhost/blogs.aspx?blogid=5:

string fqlQuery=@”and(path:(string(“”http://localhost/blogs.aspx?blogid=5″&#8221;, Mode=””PHRASE””)), filter(sitename:equals(“”Local SharePoint””)))”;

Sending the query as is, will cause FAST to automatically evaluate the question mark as a wildcard character that will match any single character. Since this is not our desired result, we need to tell FAST not to use wildcards in our specific string. This can be done easily using the wildcard parameter of string. Changing the query with this parameter will look as follows and will make sure you get the desired result:

string fqlQuery = @”and(path:(string(“”http://localhost/blogs.aspx?blogid=5″&#8221;, Mode=””PHRASE””, wildcard=off)), filter(sitename:equals(“”Local SharePoint””)))”;

Fast Tips for FAST for SharePoint #1

In the next couple of short articles I’m going to give some fast tips (pun intended) for developing with the FAST search engine for SharePoint 2010.

My first tip deals with setting Managed Metadata Properties for FAST without the use of PowerShell.

1) Start by going to your central admin->Manage service applications.

2) Click on your FAST Query SSA service:

3) Click on FAST search administration.

4) Click on Managed Properties.

5) Play with the properties as you wish 🙂