Collapse
Submit a Service Request
Contact Information for Technical Support
My Service Notifications
Bots and Spiders Best Practices
Answer ID 8532 |
Last Review Date 04/14/2019
What are some best practices relating to bots and spider activity?
Environment:
Bots, Spiders
Oracle B2C Service
Resolution:
What does Oracle do to prevent bot activity? And what can and should you do?
- Cloud Operations has installed a robots.txt file on each site to minimize the issues that spiders present.
- See Allowing other search engines to index the Oracle B2C Service application for more information.
- When known spiders access your site, the activity is not counted against your billable sessions.
- More information about this is available here: Large number of spiders in the Internet Spider Activity Report
- Of note from this answer:
Because of the consistent expansion of web spiders there might be times when a new and unknown spider indexes your site which does cause a spike in your answer views and or sessions. When you discover new spiders hitting your site you can always add them to the configuration setting:
SEC_SPIDER_USER_AGENT
Defines custom User Agents that are known web spiders. Valid entries include a comma-separated list of User Agent sub-strings (for example, MyCustomSpider or InternalGoogle). Values for this setting are case sensitive. Browsers supplying this User Agent will not experience different behavior on end-user pages in order to be in compliance with the major web search providers, though they will be considered a web spider for statistics collection. Default is blank.
- You can edit the SEC_INVALID_ENDUSER_HOSTS configuration to explicitly list which hosts are not allowed to access the end-user interface:
- Refer to Restricting computers or hosts that can access the Oracle B2C Service console and end-user pages for more information.
- Monitor your site activity
- See Session usage information for more information.
For additional information, please see Session usage information (includes Demystifying Session Usage.pdf).