Skip Navigation
Expand
Spider bot
Answer ID 4280   |   Last Review Date 09/16/2019

Why am I seeing an unusually high number of sessions?

Environment:

Web crawler, Spider bots, Web Indexing

Resolution:

A Web crawler or spider, is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. Many sites, in particular search engines, use spidering as a means of providing up-to-date data. 

Web crawlers typically identify themselves to a Web server by using the User-agent field of an HTTP request.

For example:
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

The user-agents of many web spiders for search engines are often known and do not show up in session data.  However, there are times when a new web spider is created that is not in the Oracle Service Cloud system. In these cases, to prevent future activity from being counted in your session data, it is recommended that you add the user-agent sub-string to the configuration setting - SEC_SPIDER_USER_AGENT.

Generally, when you see dramatic spikes of traffic data, it is due to your site being accessed by a spider. Spiders, also known as robots, are automated searching utilities that come in many varieties, from search engine indexers to email address harvester scripts. For details on how to observe site activity see Answer 9693: How to understand and investigate site activity.

In the above example the user-agent sub-string is "bingbot". Web spiders usually have a unique name in the user agent that often includes "bot" or "spider". Do not add generic strings such as "gecko", "mozilla", or "applewebkit" as these terms are used in most user agents which will cause end user page reporting to be impacted significantly.

The SEC_SPIDER_USER_AGENT configuration setting defines custom User-agents that are known web spiders. Valid entries include a comma-separated list of User-agent sub-strings (For example: bingbot,MyCustomSpider,InternalGoogle - no space between entries). Browsers supplying this User-agent will not experience different behavior on end-user pages and will be considered a web spider for statistics collection. Default is blank.

Path to setting(s): Select Configuration from the navigation area > Site Configuration > Configuration Settings > and search by Key.

If you have questions around what generates a session and how you can prevent inaccurate session billing on your site please review Demystifying Session Usage (PDF). Some simple mis-steps in customization and configuration can increase billable sessions.  For more information, see Session usage information.

Available Languages for this Answer:

Notify Me
The page will refresh upon submission. Any pending input will be lost.