Is active sessions counting search engine bots?


InstantForum Question
I notice today, 2 days after my forum was up. that I have 120 guests. I presume this is Search Engine bots? Does the forum not look for, detect, and discard bots as guests?
InstantForum Question

It identifies and labels them as such in Who's On

Eddy Lucast (11/13/2013)
It identifies and labels them as such in Who's On

Please tell me, then, how a brand new board on a URL that has no links to it and was out of commission for over a year constantly has 120+ active quests on it?


The forums software count shows connections over IIRC 30 minutes. If in doubt head off to a command prompt and run netstat -s -p tcp and see who's currently connected.
Edited
7 Years Ago by Eddy Lucast

Can this be adjusted somehow. I would like to have real and accurate statistics for my board.


Hi There,

Just to let you know we are adding an option in 2013-2 that will allow you to exclude search bots from your counts within the forum activity panel and within the Who's On list. This will be a new general setting you can toggle on or off via the Admin CP.

The SearchEngineSpiders.config file within the configuration directory contains a editable list of search engine spiders recognized by the the forum. If a search engine spider is indexing your board and the spider is specified within this list the spider will be identified within the Who's On page.

The SearchEngineSpiders.config file is a simple XML formatted document containing a list of HTTP_USER_AGENT tags spiders use to identify themselves and the equivalent text to display for that spider within the forum.

Identifying Bots

The best method to identify and accommodate popular bots visiting your board is to look at your raw log files. Just like browsers search engine spiders use the HTTP_USER_AGENT header to transmit their identity. The HTTP_USER_AGENT header is used by InstantForum.NET to identify spiders. For example say upon looking at your logs you identified the following entry as a search engine spider...

FAST-WebCrawler/2.2.6 (crawler@fast.no; https://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)


To accommodate for this bot you could create the following XML element within the SearchEngineSpiders.config file...

<Spiders>
<Spider Agent="FAST-WebCrawler">FastWeb</Spider>
</Spiders>


The above example performs a partial match of the HTTP_USER_AGENT tag; a partial match is useful if you don't wish to maintain the SearchEngineSpiders.config file each time a search engine slightly revising the HTTP_USER_AGENT tag. You can also provide a full match as shown below however maintaining full matches can become very time-consuming...

<Spiders>
<Spider Agent="FAST-WebCrawler/2.2.6 (crawler@fast.no; https://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)">FastWeb</Spider>
</Spiders>


A number of popular search engines are provided within the SearchEngineSpiders.config file by default.

Excluding Bots

You can stop bots from crawling your board using the robots.txt file. To learn about the robots.txt file please see the links below...
https://www.robotstxt.org/robotstxt.html

If we can assist further please don't hesitate to respond.

Thanks for all your help Eddy - your the best :)

http://www.instantasp.co.uk/images/line.gif
Kindest Regards,

Ryan Healey



ryan@instantasp.co.uk
www.instantasp.co.uk
Blog | Community | Docs

http://www.instantasp.co.uk/core/assets/images/email/facebook.png http://www.instantasp.co.uk/core/assets/images/email/twitter.png http://www.instantasp.co.uk/core/assets/images/email/google.png http://www.instantasp.co.uk/core/assets/images/email/linkedin.png

I am on 2013-2 now but am not seeing an option about search bots in the General Settings of my forum.

Sorry Eagle. This never made it into 2013-2. An option to exclude bots will be coming in a forthcoming minor update.

If I can assist further please don't hesitate to respond, 

http://www.instantasp.co.uk/images/line.gif
Kindest Regards,

Ryan Healey



ryan@instantasp.co.uk
www.instantasp.co.uk
Blog | Community | Docs

http://www.instantasp.co.uk/core/assets/images/email/facebook.png http://www.instantasp.co.uk/core/assets/images/email/twitter.png http://www.instantasp.co.uk/core/assets/images/email/google.png http://www.instantasp.co.uk/core/assets/images/email/linkedin.png

Have you considered looking thru your web server logs, finding out which robots they are and adding them so they were identified. Personally I'd rather have a complete view of whats going on but my circumstances maybe much different than yours.
GO

Login
Existing Account
Email Address:


Password:


Social Logins

Select a Forum....







InstantASP Forums


Search