Home
Non-daily weblog
Exchange | Board
Files for download
Workshop tutorials
W3 | Directory & Wiki
Artwork : AI createdSetup & config options
Suitable power supply
RasPi & sFTP file transfer
Home network print server
Home network scan server
Mesh : home Lan USB drive
Explore hard & software
UFW firewall explained
Secured by fail2ban server
Software packaging & PPA
Apache 2.4+ LAMP server
https web server : port 443
Varnish caching proxy
Module : cgi & perl
Module : geoip
Modules : php & mysql
http*s error handling
Server : conditional logging
TL-domain & dynamic DNS
Webalizer log analyser
Defeat referrer spam
robots.txt & xml sitemaps
Server : .htaccess handling
The game & not the islands
Setup & config options
North Atlantic : Macaronésia
Nine Azorean islands
Madeiran archipelago
POI on Madeira island
Streaming media
Contact details
Privacy policy & terms
Host traffic elicitation🚫 No ads & tracking
The prologue
Most of the articles, descriptions and instructions written here are applicable to the most common Debian-based Linux derivatives. Depending on the respective operating system, there may be minor or major discrepancies. 
This website is for educational purposes only. Please do not deploy anything in manufacturing plants.
No warranty or compensation is given for loss of data or hardware.
It should be also mentioned that this modest web server is hosted on a Raspberry Pi type 4B at home.
The Raspberry Pi mini-computer board as multi-purpose server deployed
A competent allrounder for domestic purposes and micro-enterprises
Raspberry Pi is a series of small single-board computers (SBCs) developed in the United Kingdom by the Raspberry Pi Foundation in association with Broadcom. The mini-computer with its armv7l processor has quickly become the favourite of hobbyists. Projects can be started with suitable Linux distributions. Even an aged RasPi e.g. the models 2B and 2B+ can definitely serve to simple tasks quite well.
Prologue
First you should consider which entries make sense and which can be skipped. Otherwise, it can result in some unpleasant surprises.
Apache v2.4+ | Contitional logging
Omit unnecessary log entries in Apache's »access.log« file
The SetEnvIf and SetEnvIfNoCase directives can be used in the following contexts in your global Apache v2.4+ configuration file. Suitable if you get lots of visits from search engines, spiders, content thefts, spammers or from certain IP addresses.
root@raspberry:# nano /etc/apache2/apache2.conf
Examples what to set to the end of this conf-file:
# Omit internal dummy connections
SetEnvIf Remote_Host "^localhost$" dontlog
# Suppress bots
SetEnvIf Remote_Host "search.msn\.com$" dontlog
SetEnvIf Remote_Host "googlebot\.com$" dontlog
SetEnvIf Remote_Host "applebot.apple\.com$" dontlog
SetEnvIf Remote_Host "yandex\.com$" dontlog
or
SetEnvIfNoCase User-Agent "Bingbot" dontlog
SetEnvIfNoCase User-Agent "Googlebot" dontlog
SetEnvIfNoCase User-Agent "Applebot" dontlog
SetEnvIfNoCase User-Agent "Yandex" dontlog
# Omit graphics and scripts
SetEnvIfNoCase Request_URI "\.(jpg|png|gif|css|js|svg)$" dontlog
# Do not log certain IPs
SetEnvIf Remote_Addr "192\.168\.0\.154" dontlog
SetEnvIf Remote_Addr "192\.168\.0\." dontlog
SetEnvIf Remote_Addr "192\.168\." dontlog
# Next line against spamming
# List is incomplete, just giving an idea what is possible
SetEnvIf Referer "\.(cc|cn|eu|io|kz|ru|su|to|tv|ua|us)" dontlog
SetEnvIf Referer "baidu\.com" dontlog
SetEnvIf Referer "semalt\.com" dontlog
SetEnvIf Referer "semrush\.com" dontlog
SetEnvIf Referer "wordpress\.com" dontlog
SetEnvIf Referer "(facebook|instagram|twitter)" dontlog
SetEnvIf Referer "anysubdomain\." dontlog
SetEnvIf Referer "app\." dontlog
SetEnvIf Request_URI "/admin" dontlog
SetEnvIf Request_URI "^/\.aws" dontlog
SetEnvIf Request_URI "^/\.git" dontlog
SetEnvIf Request_URI "^/wp-" dontlog
SetEnvIf Request_URI "(admin|aws|git|wp-)" dontlog
SetEnvIf Referer "\.app" dontlog
SetEnvIf Referer "\.(best|bid|biz)" dontlog
SetEnvIf Referer "\.(cam|chat|club|company)" dontlog
SetEnvIf Referer "\.(gdn|guru)" dontlog
SetEnvIf Referer "\.(icu|info)" dontlog
SetEnvIf Referer "\.(live|life)" dontlog
SetEnvIf Referer "\.(market|men|mobi)" dontlog
SetEnvIf Referer "\.(name|news|ninja)" dontlog
SetEnvIf Referer "\.online" dontlog
SetEnvIf Referer "\.(plus|pro)" dontlog
SetEnvIf Referer "\.(site|store)" dontlog
SetEnvIf Referer "\.top" dontlog
SetEnvIf Referer "\.(video|vip)" dontlog
SetEnvIf Referer "\.(website|work|win)" dontlog
SetEnvIf Referer "\.(xxx|xyz)" dontlog
Any terms/expressions whose are not flagged by »dontlog« are written in access.log, else dumped.
Put the essential CustomLog at the end of file apache2.conf.
CustomLog ${APACHE_LOG_DIR}/access.log combined env=!dontlog
root@raspberry:# service apache2 restart
Aspects of the request are
Remote_Host - the hostname (if available) of the client making the request
Remote_Addr - IP address of the client making the request
Server_Addr - IP address of the server on which the request was received
Request_Method - name of the method being used (GET, POST, PUT, CONNECT)
Request_Protocol - protocol name & version with which the request was made (e.g. "http/1.1")
Request_URI - the resource requested on the http request line
SetEnIf Request_Method "POST" dontlog
SetEnIf Request_Method "^(POST|CONNECT)$" dontlog
To not log requests from any clients whose hostname ends in example.com, use:
SetEnvIf Remote_Host "example\.com$" dontlog
SetEnvIf Remote_Host "(example|example2|example3)\.com$" dontlog
To not log requests from any clients whose hostname begins with example, use:
SetEnvIf Remote_Host "^example" dontlog
SetEnvIf Remote_Host "^(example|example2|example3)" dontlog
To not log certain referrals (e.g. your own domain or just to avoid, reduce spamming), use:
SetEnvIfNoCase Referer "westio\.ru" dontlog
SetEnvIfNoCase Referer "\.ru" dontlog
SetEnvIf User-Agent "zgrab" dontlog
SetEnvIf User-Agent "(hello|zgrab)" dontlog
In HTTP, »Referer« (a misspelling of Referrer) is the name of an optional HTTP header field that identifies the address of the web page i.e., the URI or IRI, from which the resource has been requested.
Virtual hosts logging
Regardless, full logging continues to run for the virtual hosts. Unless it is switched off. This can extend the lifetime of a SD card if unnecessary Read/Write processes are prevented. The sizes of these files can reach several megabytes within one day.
Comment # the line to CustomLog.
/var/log/apache2/other_vhosts_access.log
root@raspberry:# nano /etc/apache2/conf-available/other-vhosts-access-log.conf
# Define an access log for VirtualHosts that don't define their own logfile
# CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined
root@raspberry:# service apache2 restart
13-Feb 2018
Updated 02-Jul 2022
Live log monitoring with Linux's »tail« command
»tail« is a program on Unix and Unix-like systems used to display the tail end of a text file or piped data.
user@raspberry:~ $ tail -f -n +1 /var/log/apache2/access.log
user@raspberry:~ $ tail -f -n +1 /var/log/apache2/error.log
-f This option will cause »tail« will loop forever, checking for new data at the end of the file. When new data appears, it will be printed. By default, tail will output the last 10 lines of its input to the standard output.
-n +1 If you put a plus sign before num, »tail« will output all lines beginning with that line.
For example, -n +1 will print every line.
Terminate by Crtl C
Output without any line breaks. | cut -c -$COLUMNS
user@raspberry:~ $ tail -f -n +1 /var/log/apache2/access.log | cut -c -$COLUMNS
user@raspberry:~ $ tail -f -n +1 /var/log/apache2/error.log | cut -c -$COLUMNS
18-Feb 2018
Updated 02-Jul 2022
Contitional logging with examples | Suppress certain status and error codes
Regarding to the Apache 2.4+ documentation (Expressions) https://httpd.apache.org/docs/2.4+/expr.html.
The big concern and the biggest challenge was where to put the code to.
I went through so many variations until I finally got a working result.
Server version: Apache/2.4+.54 (Raspbian)
Server built: 2022-06-09T04:26:43
I make it as short as possible. At first the module mod_log_config must be enabled.
root@raspberry:# apache2ctl -M
...
log_config_module (static)
...
Examples to conditional logging.
First: this will exclude from the log file the error codes 403 (Forbidden) and 410 (Gone).
Second: this example won't write anything what is greater than or equal to 304 (Not Modified).
# Contitional logging
CustomLog /dev/null common "expr=%{REQUEST_STATUS} -in {'403','410'}"
CustomLog /dev/null common "expr=%{REQUEST_STATUS} > 304"
# Or if this does not function
CustomLog ${APACHE_LOG_DIR}/waste.log common
"expr=%{REQUEST_STATUS} -in {'403','410'}"
CustomLog ${APACHE_LOG_DIR}/waste.log common
"expr=%{REQUEST_STATUS} > 304"
/dev/null is the non-physical sink hole. The place where everything is dumped what is not needed.
Put the code in the global apache.conf at the very end of file.
Do not put the code in the 000-default.conf and default-ssl.conf at the same time.
25-May 2021
Updated 13-Jul 2022
HTTP web server on port 80
/etc/apache2/sites-available/000-default.conf => <VirtualHost *:80>
<VirtualHost *:80>
...
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog /dev/null common "expr=%{REQUEST_STATUS} > 304"
...
</VirtualHost>
HTTPS web server on port 443
/etc/apache2/sites-available/default-ssl.conf => <VirtualHost _default_:443>
<IfModule mod_ssl.c>
<VirtualHost _default_:443>
...
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog /dev/null common "expr=%{REQUEST_STATUS} > 304"
# Or if this does not function
CustomLog ${APACHE_LOG_DIR}/waste.log common
"expr=%{REQUEST_STATUS} > 304"
...
</VirtualHost>
<IfModule>
Essential after each modification
root@raspberry:# apache2ctl configtest
Syntax OK
root@raspberry:# systemctl restart apache2
Alternative contitional logging
This may be a good alternative for you, if you do not want to suppress certain status and error codes, this famous rule swaps certain log entries to an other file named waste.log.
CustomLog ${APACHE_LOG_DIR}/waste.log common
"expr=%{REQUEST_STATUS} -in {'403','410'}"
Syntax OK ? Then restart Apache for generating a new file and access to write to.
Want the both status codes 200 (OK) and 206 (Partial Content) in your log only ? Simply made.
CustomLog /dev/null common "expr=%{REQUEST_STATUS} != {'200','206'}"
root@raspberry:# systemctl restart apache2
Summary | The experimental playground figured out so far
This guesswork kept me busy for almost two weeks in total.
Funnily enough, I got it turned around so that it works halfway.
Please note, that the persistent error code 400 (Bad Request) still passes through anyway.
11-Jul 2022
Updated 19-Jul 2022
Hint | Combine dontlog rules and status code(s) selected
root@raspberry:# nano /etc/apache2/apache2.conf
This logs all entries with the status code 200 only
...
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%200{User-Agent}i\"" combined
# Another example: \"%200,206,301,302,304{User-Agent}i\"
# for more than one status code is also possible.
...
CustomLog ${APACHE_LOG_DIR}/access.log combined env=!dontlog
root@raspberry:# systemctl restart apache2
21-Jul 2023
Epilogue
Additionally to mention, it is allowed to use multiple CustomLog(s).
Instead of the standard 403 error code received, send all to any web address.
.htaccess
...
RewriteEngine On
...
ErrorDocument 403 https://startpage.com/
...
Done.
25-May 2021
Updated 30-Jun 2022