At our client site, we noticed that Analytics was logging gigs worth of data to the weblogic.out file – a catch-all file for logging reported via the standard out pipe. Note that IIS doesn’t record logs on this stream, so this is a Weblogic-only problem. The line recorded over and over (one for each hit):

OPENUSAGE(INFO ): ==> Event sent to: //UsageTracking@openusage/ wci-analyticscollector| port=31314/ BYTEARR

To prevent these events from being logged, he suggests updating %PT_HOME%/settings/configuration.xml to reduce logging and turn off console logging:

<component name="analytics-communication:config" type="http://www.plumtree.com/config/component/types/analytics-communication">
<setting name="logging:level">
<value xsi:type="xsd:string">WARN</value>
</setting>
<setting name="console-logging:enabled">
<value xsi:type="xsd:boolean">false</value>
</setting>
</clients>
</component>

In my own insignificant way, I’ll add a little bit to this tip: These settings, like most of the configuration settings in configuration.xml, can (and should, if possible) be updated in the Oracle Configuration Manager:

The Analytics Context could not be created.  This is typically due to a configuration problem.  Review the Analytics UI log for more information.

While there are many causes for this and many fixes such as re-scripting the security database, sometimes the simplest solution is overlooked: startup order.

When Analytics needs to be (re)started, the services need to be restarted in the proper order:

1. WSAPI Server.  The API Server provides SOAP access to the portal objects, such as users.
2. LDAP Directory Service.  The LDAP Directory Service connects to the API Server to surface Plumtree users and groups via LDAP.
3. Analytics UI.  This is the service that ultimately provides all the fancy reporting, and can’t work without the other two already set up, since it needs to check credentials (which introduces its own set of problems):

As a side note, the Analytics Collector doesn’t require the API or LDAP service.  It simply accepts inbound events such as searches and logins via UDP from the portal and records themto the database.  It’s a good thing that the services are separate, since even if the UI isn’t working, in most cases you can be reasonably confident events are still be recorded and not lost forever.

Every now and then, products in the WebCenter Interaction stack have a bug. And occasionally, Oracle releases a fix for said bug, and things are right with the universe again. But, once in a blue moon, that patch disappears when then the next version of the product is introduced. Such is the case with the “WebCenter Analytics Documents Report Export May Return Different Report [ID 783591.1]” issue.  The patch addresses this problem:

If you choose to “Export User Detail” for the “Other Metrics” section, “Documents” tab report, the results from a different report will be exported.

Or, more succinctly: if you export the User Detail report in Analytics for Collab Documents, it will not actually include document details.  The feature worked in the Aqualogic days, so what’s the deal now?  Well, the story is that you used to be able to export that report properly, then it was broken in ALI Analytics 2.5.  Oracle released a hotfix in 2.5 to repair the issue last year, and their release notes for the patch say to install patch 8198674, or upgrade to Analytics 10.3.

Problem is: the patch that worked for Analytics 2.5 isn’t applicable to 10.3, and 10.3 doesn’t include the patch.

Solution?  I’ll spare you the details of this pretty complicated trick, but at a high level, you need to:

1. Download this fixed version of BaseCollabServerDataProvider.class, add it to analytics-webui.jar, and put that back into analytics.war.
2. Add this line to the ptanalytics\10.3.0\settings\config\wrapper.conf file:
   

   wrapper.java.additional.21=-Dcom.bea.analytics.CollabInfoAtDocumentLevel=true

3. Reinstall the service (see the patch release notes)

As shown in this source code diff, this class just defines the original “dimensions” in Analytics that include the document details:

This will cause your “export user details” report to change from this:

… to this:

Release notes for the patch after the break…

In this Document
Symptoms
Cause
Solution
References

Applies to:

Oracle WebCenter Analytics – Version: 2.5 to 10.3
This problem can occur on any platform.

Symptoms

If you choose to “Export User Detail” for the “Other Metrics” section, “Documents” tab report, the results from a different report will be exported.

Cause

An unpublished bug is the root cause of this problem.

Solution

Download and install Patch 8198674 or upgrade to Analytics 10.3.0.1.

References

Readme for AquaLogic Interaction Analytics 2.5 MP1 Patch 343287, February 13, 2009
Copyright © 2009 Oracle, Inc. All Rights Reserved.

Contents

Issues Resolved in the Patch
Installing the Patch
Uninstalling the Patch

Issues Resolved in the Patch

This patch provides an update to Analytics 2.5 MP1 to fix bugs 7826985 and 8198674

Installing the Patch

This Patch is based on Analytics 2.5 MP1 and involves an update to the AL Analytics service.

1. Stop the AL Analytics service

2. Rename or move the existing analytics.war file stored in:

<bea root>\alui\ptanalytics\2.5\webapp

3. Copy the new analytics.war file contained in this fix to:

<bea root>\alui\ptanalytics\2.5\webapp

4. To enable document level detail for exported Document Views report, add this line to wrapper.conf file:

wrapper.java.additional.21=-Dcom.bea.analytics.CollabInfoAtDocumentLevel=true

To disable document level detail for exported Document Views report, comment previous line

5. After changing wrapper.conf file, run the following commands from <bea root>\alui\ptanalytics\2.5\bin

analyticsd.bat remove

analyticsd.bat install

6. Start the AL Analytics service

Uninstalling the Patch

To uninstall the Patch, replace the original analytics.war file saved in step 2, comment line added in step 4, and run commands detailed in step 5 above.

WebCenter Analytics now requires you to have the binaries for Hibernate 3.0.5 (for DB access) and Cewolf 0.10.3 (for charting functionality).  Unfortunately, Cewolf 0.10.3 is no longer available for download, and the installer throws warnings about the checksums not matching if you use a later version.  In fact, I’ve had difficulties with getting Analytics to work with the newer version after disregarding the error, and it’s best to just use the older versions that Analytics wants if you ever plan on contacting Oracle Support anyway.

The older binaries themselves aren’t too hard to find, but if you’d like to just grab them here, they’re all yours:

• Hibernate 3.0.5
• Cewolf 0.10.3

Again, use caution with any files other than these; otherwise you may end up with ugliness like this:

The reality, though, is that these types integrations take time – and sometimes, lots of it.  As evidence of this, look no further than Aqualogic Analytics.  When BEA acquired Plumtree, one of the gaping holes in WLP was Analytics, or usage reporting of the product.  Plumtree Analytics was becoming a solid product, but it was very tightly integrated into the Plumtree portal.  So the decision was made to try and abstract some of the major pieces out, with the thinking that these abstractions could be useful elsewhere.  For example, the ill-fated Security Services (once used by the also ill-fated PEP line and now just built into Analytics) and the existing Directory Services came of this integration attempt.  The idea was that by abstracting out security and user management:

1. These services would be available to other applications that were developed down the line
2. Analytics would be more compatible with WebLogic Portal, which also had an LDAP repository to access user and group information

While I think that if more time had been available for the integration to become more seamless, the problem is that no one won in this attempt because it was aborted too early.  I have no idea whether WLP still supports Analytics integration, the old Security Services are now just built into the product as a phenomenally complicated set of DB tables that make little sense, and Directory Services are a dramatically inefficient way to access user and group information.

Case in point – I’ve had a couple of clients report Analytics timeouts for some users, but other users were seeing the proper report:

How does this relate to the whole Plumtree/BEA/Oracle integration saga? The old Plumtree Analytics application used a SQL query called something like QUERY_USER_FLATTENED_GROUPS.  Basically, this was a Plumtree Portal-specific SQL query that, given a user ID, would spit out all nested groups that that user was a member of.  So if a user was a member group A, and Group A was a member of Group B, the query would return both Group A and B.

The ALUI version of Analytics, though, utilized Directory Services so that group membership didn’t need to come from PT-specific SQL queries.  It could come from generic LDAP queries.  The problem is, LDAP doesn’t have a mechanism like QUERY_USER_FLATTENED_GROUPS, so for any given user, Analytics needs to query LDAP for the groups a user is in.  And then, Analytics needs to check to see which groups THOSE groups are a member of.  And so on, and so on.  You’d be surprised – through inheritance, you may be a member of thousands of groups, and rather than a single SQL query, you’re now dealing with tens of thousands of LDAP calls.

Bottom line:  Integrations and product convergence can work, but they’re phenomenally complicated, because every piece of abstraction added can cause unanticipated side effects.  Which is probably why Oracle is taking this whole process pretty slowly.

Full Bug Report after the break for your convenience.

*** 04/15/10 03:33 pm ***
  BUG TYPE CHOSEN
  ===============
  Code
�
  Product: Oracle WebCenter Analytics (5388)
  ==========================================
  Problem Description
  ===================
  The WebCenter Analytics Console community in the WebCenter Interaction
(WCI)
  portal loads slowly or experiences a timeout when the user accessing the
  product is a member of many portal groups, especially if membership is
  gained through nested groups-in-groups. For example, a user with less than
  200 group memberships loads Analytics in 7 seconds while a user with
  more than 1000 direct group memberships loads Analytics in 84 seconds and
  a user with more than 1000 group memberships through nested
groups-in-groups
  loads Analytics in 136 seconds. By contrast, all three users can load a
  simple WCI community in a few seconds.
�
  Steps to reproduce
  =========================
  1. Create a WCI user with access to Analytics but only grant them
     membership in a few groups.
  2. Create a WCI user with access to Analytics and grant them membership in
     more than 1000 groups directly with no groups-in-groups.
  3. Create a WCI user with access to Analytics and grant them membership in
     in nested groups-in-groups that yield membership in more than 1000
groups
     total.
  4. Test loading Analytics console with each user.
  5. Test loading a simple WCI community with each user.

  Expected Behavior
  =================
  The time to verify activity rights to Analytics should be similar to checks
  for other portal resources.
�
  Workarounds
  ==========
  Increase the portlet timeout in the Analytics webservices in WCI portal
    administration.
  Consolidate groups to reduce the number of total memberships per user.
  Reduce groups-in-groups nesting.
�
  Stack Trace or Diagnostic Data
  ===========    �
  Portal spy logs will show a long pause while at the same time Analytics
ALUI
  Security spy logs many LDAP requests and ALUI Directory spy logs thousands
  of database queries.
*** 04/15/10 03:35 pm ***
*** 04/16/10 01:53 pm ***
Documented in My Oracle Support Knowledge Base Note 979305.1
*** 04/19/10 12:44 pm *** (CHG: Asg->NEW OWNER OWNER)

If, after upgrading to WCI Analytics 10.3.0.1, you’re not seeing any new events being recorded, check any one of your Analytics Views in your database. Chances are, you’re seeing a NULL value for VisitID:

Oracle has released three separate critical fixes trying to get this finally working, and from my experience you need to install the last two listed in the table below.

The real problem is, this isn’t just a matter of swapping out files, since for each month that passes before you notice this problem, more tables will continue to have these invalid values. And Oracle’s fix in the last patch set is FAR from ideal if you want to get those VISITIDs recreated. I won’t bore you with the details (trust me, you’ll be bored enough running this crazy fix), but suffice it to say that there are no less than 15 steps. And during those steps, potentially dozens of SQL scripts are generated (depending on how many months you need to created). For each month that you need to restore, you have to stop the collector, run a couple scripts, start the collector, then let it run a while to recreate everything. Wait a while, verify the number of untagged events you have, repeat:

How long does it take? Quoting from the Readme file: “Tagging old event data can take days or weeks to complete depending on the volume of your event records. Internal tests repaired the data at a rate of 150k to 300k events per hour.”. DAYS or WEEKS!?

In practice, I’ve run through each of these in about a day, going back about 6 months on moderately sized installations. But you should plan on getting these updates in there sooner than later, because the longer you wait, the longer it’s going to take you to repair all those busted tables.

Serious errors occurred during your installation.  Click OK and then click through to the end of installation to complete installation and then look at log for WebCenter Interaction in …

Recently, the same error bit me during an ALUI upgrade, and I saw pretty much the same error in the portal, Collaboration Server, and Analytics.  The errors seemed benign so I just ignored them until I realized that the WebCenter Analytics installer hadn’t created the Analytics Collector Service.

It turns out – and I have no idea why I’d never come across this issue with other installs and upgrades – that the WCI installers look for free memory on the host machine.  In some (unknown and unusual) circumstances, it can’t query the Windows OS for free memory, so it defaults to 0.  But 0GB of free RAM is less than what it needs, so the installer chokes.  In Collab and the Portal, the error is at the end of the installation process, so it seems pretty benign, but for Analytics, it gets thrown before the services are created, so you’re boned unless you fix it.

As for fixing it, check out Bill’s Blog Post, but the gist is that you need to set a fixed amount of Virtual Memory to avoid an error like…I:\apps\plumtree\uninstall\ptportal\10.3.0\register\register.xml:4979: The following error occurred while executing this line:
I:\apps\plumtree\uninstall\ptportal\10.3.0\register\macrodefs\versionpolicy.xml:690: The following error occurred while executing this line:
I:\apps\plumtree\uninstall\ptportal\10.3.0\register\macrodefs\orainventory.xml:172: Oracle Universal Installer failed to properly register your ORACLE_HOME,
I:\apps\plumtree, under name OraWCIntgHome1.  Make sure (1) that you have proper permissions (on unix you would have needed to run orainstRoot.sh as root user, on windows you need write access to registry and ability to install to %ProgramFiles% directory), (2) that, on unix, you did not run installer as root user.  You can attempt to run OUI yourself with command line “I:\apps\plumtree\uninstall\ptportal\10.3.0\register/../../../oui/cd/Disk1/install/setup.exe” -ignoreSysprereqs -attachHome “ORACLE_HOME=I:\apps\plumtree” ORACLE_HOME_NAME=OraWCIntgHome1.   If that succeeds, then you can run the installer again.

The purpose of “Master Notes” is to “provide the most important links that users will need to install and support the product”, and there are some pretty decent pages in there if you know where to look.  For example, the IDK Master Note is a collection of a bunch of documentation, KB articles, known issues, and bug fixes all in one place.

But what I really wanted to highlight here is the Webinars provided by Oracle Support – with one in particular being the best Oracle Webinar I’ve seen: the Search Webinar, by Eno Gjerasi.  Eno shows that there’s still life left from the Plumtree support group, and demonstrates a level of knowledge of the Search Server that rivals most engineers or consultants.  There was one tip in particular that I’ll focus on in upcoming posts (about how to communicate directly with Search), but I encourage you to check out all three Webinars (Search, Portal / SSO, and Analytics) and the other Master Notes – you may just find a gem in there and wonder how you made it all these years without knowing “that one thing” you never knew you needed.

Keep up the good work, Oracle support!

I got lucky on this one, and hopefully if you found this post through a Google Search, you’ll have saved yourself the headache of trying to figure out what value should be in there.  The answer is in the Analytics Configuration Worksheet (PDF link):

That’s right: it’s “saml2keystore“.  Anyone know how to reset the actual value in Java’s Keystore for Analytics?

In the release notes, it looks like kind of a pain to install, since there’s mention of having to install other software before you can install Analytics:

• Oracle WebCenter Analytics leverages Hibernate 3.0.5 for persistence. You must install Hibernate 3.0.5 before you install Oracle WebCenter Analytics. Hibernate 3.0.5 can be downloaded from http://sourceforge.net/project/showfiles.php?group_id=40712&package_id=127784.
• Oracle WebCenter Analytics leverages Cewolf 0.10.3 for its charting engine. You must install Cewolf 0.10.3 before you install Oracle WebCenter Analytics. Cewolf can be downloaded from http://sourceforge.net/projects/cewolf/files/cewolf/cewolf-0.10.3/cewolf-0.10.3.zip/download.

And, given that the versions provided are significantly further back than what’s currently available (as of this writing, Hibernate is now at version 3.5.2 and Cewolf is now at 1.0), you might be inclined to hold off on the upgrade until the next release – I know I was initially.

The good news is that now that I’ve actually been through this upgrade, it turns out that it’s not as hard as it sounds.  Contrary to the release notes, you don’t need to “install [the apps] before you install Oracle WebCenter Analytics”.  You just need to download the zips, run the installer, and point the installer at wherever you’ve saved those files.  Pretty straightforward, actually…