How do I understand and investigate site activity including automated crawling of the site?
Environment:
Oracle B2C Service, Product listing
Customer Portal, Clickstreams, Customer Session Information
Resolution:
Site activity can be viewed from the Oracle B2C Service Support portal after logging in and navigating to Site Tools > License and Compliance Dashboard.
From the agent console, site usage and statistics can be managed using Oracle B2C Service Analytics, including with the standard reports listed below. Further information on available reports can be found within the agent console from Analytics -> Report Management -> View Report Descriptions.
One helpful report is Visit Activity (report ID 37). This report gives a high-level perspective of the traffic flow on your end-user interface. This report includes the average session length in terms of time and in page turns. You can see how your users transition between the pages and how long they spend on average on each page. Though this report tracks all end-user sessions, the report does not provide data granularity for specific end-users. It is not possible for this report to link session histories to specific users. Instead, the report is designed to give more general session data about the overall usage of your site.
| Report ID | Report | Location | Description |
|---|---|---|---|
| 139 | Internet Spider Activity | /Service/Site Reports | This report will provide information about how your site is accessed by internet spiders. It includes the name of the spider, when it accessed your site, how much of the site was accessed, and whether or not the spider attempts to ignore access controls. |
| 202 | Page Summary | /Service/Site Reports/Actionable Knowledge Foundation/Pages and Actions | The report provides a list of end-user pages and actions, along with details about the visits for each. |
| 213 | Referrer Summary | /Service/Site Reports/Actionable Knowledge Foundation/Referrers | The report provides details on the end-user visits by referring domain. |
| 215 | Top Answers by Referrer | /Service/Site Reports/Actionable Knowledge Foundation/Referrers | This report provides a list of the top 10 answers most commonly viewed from the selected referrer. |
| 163 | Referring Sites | /Service/Site Reports/Customer Portal | The report provides a list of sites that referred a user to the Oracle B2C Service end-user pages, along with the percentage of users that were referred from each |
| 189 | Visits by Browser | /Service/Site Reports/Customer Portal | The report provides the percentage of visits to the Customer Portal pages by each browser type. |
| 191 | Visits by Browser and Operating System | /Service/Site Reports/Customer Portal | The report provides the percentage of visits to the Customer Portal pages by each browser and operating system combination. |
| 190 | Visits by Operating System | /Service/Site Reports/Customer Portal | The report provides the percentage of visits to the Customer Portal pages by each operating system. |
| 188 | Visits by Page Set | /Service/Site Reports/Customer Portal | This report shows the Customer Portal page set mapping usage by percentage of sessions which accessed each page set. |
| 192 | Visits by Time | /Service/Site Reports/Customer Portal | The report provides the specified time units (Month/Day by default) and the number of visits that occurred during each time unit for the date range specified at run-time. The percentage of visits per time unit is also provided to more easily compare the statistics per time unit. |
| 15 | Keyword Search | /Service/Site Reports/Keyword Search | Shows the keywords and phrases used in searches by end-users in the time period specified. This report also shows the phrase stem, the number of searches performed for each keyword, and how many answers actually contain that keyword. This report is useful for seeing what words or topics end-users are searching for most often and how many answers exist that address those topics. |
| 33 | Total Pages | /Service/Site Reports/Operations Dashboard | Shows the total number of end-user pages accessed yesterday and today. This graph is useful for determining if the overall activity on your site is increasing, decreasing, or remaining constant. |
| 37 | Visit Tracking | /Service/Visit Tracking | The Visit Tracking dashboard contains six reports that show which Customer Portal pages are visited on your site, the time spent on each page, and the information on the transitions between pages. The reports also display on where customers start their visits, end their visits, and how long their visits last. |
| 46 | Visit Activity | /Service/Site Reports/Site Effectiveness | Tracks how many out of the total number of visits have performed each one of the three actions (View Answer, Search, and Submit Question). The Count column shows this number. This report also shows the percentage of visits that performed each action. |
| 47 | Visit Comparison | /Service/Site Reports/Site Effectiveness | Compares the pages visited by end-users who did not submit an incident (self-service) with the pages visited by end-users who did submit an incident (non self-service). For each type of visit (self-service and non self-service), the percentage of the total number of visits is calculated by looking at the number of visits who accessed each page. |
| 39 | Ending Points | /Service/Site Reports/Visit Tracking | Displays statistics about the final page of the end-users visits. The summary is presented as the number of visits that ended on a page and also shows what percentage of the total visits ended on each page. This report is useful for determining if end-users receive effective answers. |
| 43 | Page Transition Time | /Service/Site Reports/Visit Tracking | Reports on the average number of minutes it took for each page transition. This could be considered as the average time spent looking at a page before changing to the new page. This statistic is computed by adding the times on each page and dividing by the number of visits that can be timed. This statistic is useful for determining where customers spend the most time. |
| 42 | Page Transitions | /Service/Site Reports/Visit Tracking | Displays the number of times an end-user transitions from one page to another. This statistic is useful for determining where end-users go from a given page. The rows correspond to the page the customer started from; the columns correspond to the pages the customer transitioned to. |
| 41 | Pages per Visit | /Service/Site Reports/Visit Tracking | Displays statistics about how often pages are viewed during a visit. |
| 40 | Starting Points | /Service/Site Reports/Visit Tracking | Displays statistics about the first page of the end-user visits. The summary is presented as the number of visits that start on each page and also shows what percentage of the total visits started on each page. This report is useful for determining entry points into the system. |
| 38 | Visit Length | /Service/Site Reports/Visit Tracking | Displays statistics about the shortest, longest, and average visit, including how many pages were viewed and the length of the visits. |
| 13088 | PAPI Meter Log | /Common/Site Administration/Public API | The report provides a log of activity on the Public API |
| 13089 | PAPI Method Meter Log | /Common/Site Administration/Public API | The report provides a log of the method invocations on the Public API |
| 13101 | Public API Meters | /Common/Site Administration/Public API | The report provides usage meters tracking activity on the public API |
| 13102 | Public API Method Meters | /Common/Site Administration/Public API | The report provides usage meters tracking method invocations on the public API |
Further, Customer Portal usage includes use of a site by customers and agents, as well as requests from web crawlers/spiders/bots used for search engine indexing or other information capturing purposes. Most of the standard reports on visits and end user activity are intended to exclude spider bot activity. We keep a global list of known spiders (which is occasionally updated) and provide a config that defines custom user agents for spiders not in that pre-defined list. Traffic that is identified as coming from a known spider is routed to the spider_track table while other traffic is routed through clickstreams.
Please note, none of that processing is retroactive. If a spider agent is identified after reviewing reports on clickstreams, listing the agent in the spider config (SEC_SPIDER_USER_AGENT) would only affect future traffic.
See the following for information on how to investigate this type of activity:
Answer ID 1723: Spike in data for site traffic or answers viewed for a given day
​Further, for information on how to control crawling of the site see
If you have questions around what generates a session and how you can prevent inaccurate session billing on your site please review Demystifying Session Usage (PDF). Some simple mis-steps in customization and configuration can increase billable sessions. For more information, see Session usage information.
