1. Forum
  2. DOVE-Net
  3. Synchronet Discussion

  • webv4 and web crawlers

    From Al@VERT/TRMB to echicken on Mon Aug 10 13:12:31 2020
    Hello echicken,

    I have been messing around with the stock web interface and webv4 and I am just wondering if there is a way to block web crawlers from the message areas the way the stock interface does it with the robots.txt?

    Ttyl :-),
    Al

    ... Cursor: An expert in four-letter words

    ---
    þ Synchronet þ The Rusty MailBox - Penticton, BC Canada
  • From echicken@VERT/ECBBS to Al on Mon Aug 10 16:39:24 2020
    Re: webv4 and web crawlers
    By: Al to echicken on Mon Aug 10 2020 13:12:31

    I have been messing around with the stock web interface and webv4 and I am just wondering if there
    is a way to block web crawlers from the message areas the way the stock interface does it with the
    robots.txt?

    You can put a robots.txt file inside the document root (as on any webserver); I don't remember the exact path offhand, but the directory name is 'root' and it has an 'index.xjs' file in it.

    You may also want to configure a web 'guest' user and limit its access to message areas, giving unauthenticated bots less surface area to crawl over. (In modopts.ini->[web], set 'guest' to the username of whatever account you want guests to be treated as.)

    ---
    echicken
    electronic chicken bbs - bbs.electronicchicken.com
    þ Synchronet þ electronic chicken bbs - bbs.electronicchicken.com