There’s a WCI App For That 5: SearchFixer

We’ve discussed a tiny bit about Knowledge Directory cards and how the WCI Search Update plays into the crawler ecosystem, and seen that it’s possible to directly query the WebCenter Search Service, so how ’bout a quick real-world application example, expanding both of those concepts?

Here’s the scenario:  I had a client that was showing discrepancies between “Browse” and “Edit” modes in the ALUI Knowledge Directory, and in Snapshot Queries.  I suppose I owe you all a more detailed explanation of these topics – which I’ll put up in a couple of days – but for the purposes of this article, suffice it to say that the “Search Index” and “Database” were mis-matched, and the WCI search index didn’t match the database.  Worse, the regular method of repairing this discrepancy (using the Search Update job after scheduling a Search Repair) wasn’t working.

So, to fix this issue, I developed another quick and dirty application that enumerated all folders in the Knowledge Directory, doing a search for cards within the folder, then querying the database.  The application would then compare the results, and if they were different, would allow the admin to “fix” the problem by deleting all cards from the Search Index for that folder.  When the Search Repair job next ran, it would re-create these entities without all the extraneous records in there.

Like this post, I’m not particularly proud of the code as a well-architected solution, but it works and I’d be happy to help you out if you want to get in touch.  Some of the relevant code is after the break.


        private void recurseFolder(IPTFolder folder, int level)
        {
            if (!running)
                return;
            Application.DoEvents();

            try
            {
                int folderID;
                string folderName;
                IPTFolder childFolder;
                string statusString;
                List<int> dbObjIDs;
                List<int> searchObjIDs;

                int cardsUnapproved = folder.QueryCardsCount(true);
                int cardsApproved = folder.QueryCardsCount(false);

                IPTQueryResult res = folder.QuerySubfolders(PT_PROPIDS.PT_PROPID_OBJECTID + PT_PROPIDS.PT_PROPID_NAME, 0, PT_PROPIDS.PT_PROPID_NAME, 0, 50, null);
                for (int i = 0; i < res.RowCount(); i++)
                {
                    folderID = res.ItemAsInt(i, PT_PROPIDS.PT_PROPID_OBJECTID);
                    folderName = res.ItemAsString(i, PT_PROPIDS.PT_PROPID_NAME);
                    childFolder = session.GetCatalog().OpenFolder(folderID, false);
                    statusString = "";
                    dbObjIDs = new List<int>();
                    searchObjIDs = new List<int>();
                  
                    // add tab-separated folder name based on depth
                    for (int y = 0; y < level; y++)
                        statusString += "\t";
                    statusString += childFolder.GetName();
                    for (int y = 0; y < FOLDER_LEVELS-level; y++)
                        statusString += "\t";

                    // add number of cards (from DB)
                    statusString += "," + childFolder.QueryCardsCount(true);
                    statusString += "," + childFolder.QueryCardsCount(false);
                    IPTQueryResult dbquery = childFolder.SimpleQueryCards(PT_PROPIDS.PT_PROPID_OBJECTID, null);
                    for (int k = 0; k < dbquery.RowCount(); k++)
                    {
                        dbObjIDs.Add(dbquery.ItemAsInt(k,PT_PROPIDS.PT_PROPID_OBJECTID));
                    }

                    // add number of cards (from Search)
                    IPTSearchRequest request = session.GetSearchRequest();  //create search request
                    request.SetSettings(PT_SEARCH_SETTING.PT_SEARCHSETTING_APPS, PT_SEARCH_APPS.PT_SEARCH_APPS_PORTAL);  //turn off requests for collab/content apps
                    request.SetSettings(PT_SEARCH_SETTING.PT_SEARCHSETTING_OBJTYPES, new int[] {
            PT_CLASSIDS.PT_CATALOGCARD_ID});  // Restrict the search to cards
                    request.SetSettings(PT_SEARCH_SETTING.PT_SEARCHSETTING_RET_PROPS, new int[]
                  {
                   PT_PROPIDS.PT_PROPID_OBJECTID,
                   PT_PROPIDS.PT_PROPID_CLASSID
                        }); // make sure the appropriate fields are returned.
                    request.SetSettings(PT_SEARCH_SETTING.PT_SEARCHSETTING_DDFOLDERS, new int[] { folderID }); // restrict to folder
                    request.SetSettings(PT_SEARCH_SETTING.PT_SEARCHSETTING_INCLUDE_SUBFOLDERS, false); // don't include subfolders
                    request.SetSettings(PT_SEARCH_SETTING.PT_SEARCHSETTING_MAXRESULTS, MAX_ITEMS); // max items to return
                    IPTSearchQuery query = request.CreateBasicQuery("*", "PT" + PT_INTRINSICS.PT_PROPERTY_OBJECTNAME);
                    IPTSearchResponse results = request.Search(query);
                    statusString += "," + results.GetResultsReturned();

                    //enumerate the results
for (int z = 0; z < results.GetResultsReturned(); z++)
                    {
                        searchObjIDs.Add(results.GetFieldsAsInt(z,PT_INTRINSICS.PT_PROPERTY_OBJECTID));
                        // handle progress bar
                        totalCardsFound++;
                        try
                        {
                            prgStatus.Value = totalCardsFound;
                            if (totalCardsFound % 10 == 0)
                                Application.DoEvents();
                        }
                        catch (Exception)
                        {
                            // ignore the progress update if there are too many cards found in case there's a discrepancy
                        }
                    }

                    dbObjIDs.Sort();
                    searchObjIDs.Sort();

                    if (!CheckEqual(dbObjIDs, searchObjIDs))
                    {
                        handleUnequal(childFolder.GetPath(), folderID, dbObjIDs, searchObjIDs);
                    }

                    if (chkIncludeOIDs.Checked)
                        statusString += "," + listString(dbObjIDs) + "," + listString(searchObjIDs);
                    addStatus(statusString);
                    recurseFolder(childFolder, level + 1);
                }
            }
            catch (Exception ex)
            {
                addStatus("Exception in recurseFolder: " + ex.Message);
            }
        }

        private void btnFix_Click(object sender, EventArgs e)
        {
            if (btnGo.Text == "GO")
            {
                btnFix.Text = "FIX";
                running = true;
                prgStatus.Value = 0;
                prgStatus.Maximum = searchFoldersToPurge.Count;
                for (int x = 0; x < searchFoldersToPurge.Count; x++)
                {
                    prgStatus.Value = x;
                    Application.DoEvents();
                    if (running)
                    {
                        int folderID = searchFoldersToPurge[x];
                        purgeSearchForFolder(searchFoldersToPurge[x]);
                    }
                }
                running = false;
                btnFix.Text = "FIX";
            }
            else
            {
                btnFix.Text = "STOP";
                running = false;
            }
        }

        private void purgeSearchForFolder(int folderID)
        {
            string messageSent = SEARCH_PURGE_START + folderID + SEARCH_PURGE_END;
            TcpClient tcpClient=null;
            NetworkStream networkStream=null;
            try
            {
                addResult("---------------");
                addResult("Purging search folder " + folderID + " with TCP Message: ");
                addResult(messageSent);
                Application.DoEvents();

// connect to the search server port directly and purge the folder's cards
                tcpClient = new TcpClient(txtSearchServer.Text, int.Parse(txtSearchPort.Text));
                Byte[] bytesSent = Encoding.ASCII.GetBytes(messageSent + "\r\n");
                Byte[] bytesReceived = new Byte[256];
                networkStream = tcpClient.GetStream();
                networkStream.ReadTimeout = 1000;
                networkStream.Write(bytesSent, 0, bytesSent.Length);
                String responseData = String.Empty;
                int bytes = 0;
                do
                {
                    bytes = networkStream.Read(bytesReceived, 0, bytesReceived.Length);
                    responseData = responseData + Encoding.ASCII.GetString(bytesReceived, 0, bytes);
                }
                while (bytes > 0);

                addResult("RESPONSE: " + responseData);
                Application.DoEvents();
            }
            catch (Exception ex)
            {
                addResult("EXCEPTION UPDATING SEARCH [" + folderID + "]: " + ex.Message);
            }
            finally
            {
                if (null == networkStream)
                    networkStream.Close();
                if (null == tcpClient)
                    tcpClient.Close();
            }

        }

Tags: , ,

2 Responses to “There’s a WCI App For That 5: SearchFixer”

  1. Geoff Garcia says:

    Hey Matt,
    Wouldn’t rebuilding the search index solve the issue also?
    Great solution though, rebuilding can be a bear.

  2. Matt Chiste says:

    The problem with this client was that, since there were items indexed in the search server but no entries in the database, a search REPAIR and a search UPDATE did nothing to remove the extraneous entries. And, because there were tens of thousands of cards in the Knowledge Directory, it would have taken about 24-48 hours to purge the index and do a complete rebuild. So this solution allowed us to identify only the folders that had a mismatch and purge them individually, so that a search update would recreate the cards.

Leave a Reply