From Wikipedia, the free encyclopedia

Referer spam is a kind of spamdexing (spamming aimed at search engines). The technique involves making repeated web site requests using a fake referer url that points to the site the spammer wishes to advertise. Sites that publicize their access logs, including referer statistics, will then end up linking to the spammer's site, which will in turn be indexed by the search engines as they crawl the access logs.

This benefits the spammer because of the free link, and also gives the spammer's site improved search engine placement due to link-counting algorithms that search engines use.

Some web sites receive so many referer spam hits that they amount to a denial of service attack on the server because there are not enough resources left on the server to handle legitimate traffic.

As with e-mail spam, web site operators who receive unwanted referer spam may respond using filtering and blocking.

A simple solution to render this form of spamming ineffective is to prevent the search engine spiders from crawling the site logs by moving them to a non-public area such as a password-protected area, or configure the web statistics generator to use the rel=nofollow attribute on the referer links.

An example configuration fragment for filtering using the Apache server is as follows:

 # Filter rules
 # The regexp can be refined to reduce false positives...
 # As many SetEnvIfNoCase directives can be used...
 SetEnvIfNoCase Referer "(hold-?em|poker|casino|hotel|loan|mortgage|payday|credit|pingdom)" refspam
 SetEnvIfNoCase Referer "(viagra|cialis|penis|diet|porn)" refspam
 # Whitelists can be used, too... note the !refspam vs. refspam
 SetEnvIfNoCase Referer "white-listed_site\.com" !refspam
 # Deny access to refspam (OPTIONAL: conserves resources otherwise wasted by refspam bots)
 Deny from env=refspam
 # Now separate the logs so that refspammers do not pollute your logs as they intend to
 CustomLog /var/log/apache/access.log combined env=!refspam
 CustomLog /var/log/apache/access_refspam.log combined env=refspam

The "fake" web site hits will go to access_refspam.log, whereas normal traffic goes to access.log. The "SetEnvIfNoCase" lines contain Regular expressions (more specifically, Perl regular expressions) that can be used to match any undesirable traffic.

If most of the spam is coming from a few IP addresses, or is requesting a certain page (that may no longer exist on the server) the Apache server may also be configured to deny access via the configuration file, (often named httpd.conf), based on either IP address or the name of the requested file by adding lines like the following:

 # Deny access based on the filename or path of the requested file
 
  Deny from all
 
 # Deny access based on the IP address or host name of the offending site
 
   Deny from 255.255.255.255
   Deny from example.com
 

A good statistics analysis program will allow you to target the worst offenders.

A third solution for Apache is to install ModSecurity, which allows you to deny requests based on any variable from the server environment, such as referer, request, IP, host, etc.

• Adversarial information retrieval
• Spamdexing
• Spam in blogs
• Referrer spoofing

• A proposal on addressing referer spam - technical overview and ideas for combatting the practice
• aStatSpam - Referrer-spam blacklist