Home
Newer & updated contents
Blog. Non-daily weblog
Forum. Exchange. Exchange | Board
filetransfer Files for download
Toolbox : Workshop tutorials
Open Directory W3 Directory resources
AI revolution drives up power consumption
Raspberry Pi as server

Setup & config options

Raspberry Pi first usage
Suitable power supply
RasPi & sFTP file transfer
Home network print server
Home network scan server
Mesh : home Lan USB drive
Explore hard & software
UFW firewall explained
Secured by fail2ban server
Software packaging & PPA

Apache 2.4+ LAMP server

http web server : port 80
https web server : port 443
Varnish caching proxy
Module : cgi & perl
Module : geoip
Modules : php & mysql
http*s error handling
Server : conditional logging
TL-domain & dynamic DNS
Webalizer log analyser
Defeat referrer spam
robots.txt & xml sitemaps
Server : .htaccess handling
»Windward« server

The game & not the islands
 Setup & config options

Establish the Windward game server. About Windward

Some off-topics

Firefox web browser. Improve your Firefox

Free weather app. Weather widget

Lat. 52.27, Long. 8.01

 

Meteorological service

North Atlantic : Macaronésia

The region of the Azores islands. Nine Azorean islands
The region of the Madeiran islands. Madeiran archipelago
The point of interest on the Madeiran islands. POI on Madeira island
Streaming media, videos : Streaming media
Videos : Off-topic media
Front desk clerk
email, skype, ms-teams Contact details
Privacy policy & terms
Host traffic elicitation

🚫  No ads & tracking

Tag cloud
                                       
QRC bookmark

QR quick response code




Github VSCO Vimeo | Madeira | Madeirense Watch on Youtube

The prologue


Most of the articles, descriptions and instructions written here are applicable to the most common Debian-based Linux derivatives. Depending on the respective operating system, there may be minor or major discrepancies.
This website is for educational purposes only. Please do not deploy anything in manufacturing plants.
No warranty or compensation is given for loss of data or hardware.

It should be also mentioned that this modest web server is hosted on a Raspberry Pi type 4B at home.


The Raspberry Pi mini-computer board as multi-purpose server deployed
A competent allrounder for domestic purposes and micro-enterprises


Raspberry Pi : Apache as multiple web server. Print server, scan server, backup and NAS server. Raspberry Pi : Apache as multiple web server. Print server, scan server, backup and NAS Server.

Raspberry Pi is a series of small single-board computers (SBCs) developed in the United Kingdom by the Raspberry Pi Foundation in association with Broadcom. The mini-computer with its armv7l processor has quickly become the favourite of hobbyists. Projects can be started with suitable Linux distributions. Even an aged RasPi e.g. the models 2B and 2B+ can definitely serve to simple tasks quite well.


Prologue

First you should consider which entries make sense and which can be skipped. Otherwise, it can result in some unpleasant surprises.


Apache v2.4+ | Contitional logging

Omit unnecessary log entries in Apache's »access.log« file


Conditional logging explained.

The SetEnvIf and SetEnvIfNoCase directives can be used in the following contexts in your global Apache v2.4+ configuration file. Suitable if you get lots of visits from search engines, spiders, content thefts, spammers or from certain IP addresses.


root@raspberry:# nano /etc/apache2/apache2.conf


Examples what to set to the end of this conf-file:

        
          # Omit internal dummy connections
          SetEnvIf Remote_Host "^localhost$" dontlog
          
          # Suppress bots
          SetEnvIf Remote_Host "search.msn\.com$" dontlog
          SetEnvIf Remote_Host "googlebot\.com$" dontlog
          SetEnvIf Remote_Host "applebot.apple\.com$" dontlog
          SetEnvIf Remote_Host "yandex\.com$" dontlog

	  or

          SetEnvIfNoCase User-Agent "Bingbot" dontlog
          SetEnvIfNoCase User-Agent "Googlebot" dontlog
          SetEnvIfNoCase User-Agent "Applebot" dontlog
          SetEnvIfNoCase User-Agent "Yandex" dontlog
          
          # Omit graphics and scripts
          SetEnvIfNoCase Request_URI "\.(jpg|png|gif|css|js|svg)$" dontlog
          
          # Do not log certain IPs
          SetEnvIf Remote_Addr "192\.168\.0\.154" dontlog
          SetEnvIf Remote_Addr "192\.168\.0\." dontlog
          SetEnvIf Remote_Addr "192\.168\." dontlog
          
          # Next line against spamming
          # List is incomplete, just giving an idea what is possible
          SetEnvIf Referer "\.(cc|cn|eu|io|kz|ru|su|to|tv|ua|us)" dontlog

          SetEnvIf Referer "baidu\.com" dontlog
          SetEnvIf Referer "semalt\.com" dontlog
          SetEnvIf Referer "semrush\.com" dontlog
          SetEnvIf Referer "wordpress\.com" dontlog

          SetEnvIf Referer "(facebook|instagram|twitter)" dontlog

          SetEnvIf Referer "anysubdomain\." dontlog
          SetEnvIf Referer "app\." dontlog

          SetEnvIf Request_URI "/admin" dontlog
          SetEnvIf Request_URI "^/\.aws" dontlog
          SetEnvIf Request_URI "^/\.git" dontlog
          SetEnvIf Request_URI "^/wp-" dontlog
          SetEnvIf Request_URI "(admin|aws|git|wp-)" dontlog

	  SetEnvIf Referer "\.app" dontlog
          SetEnvIf Referer "\.(best|bid|biz)" dontlog
          SetEnvIf Referer "\.(cam|chat|club|company)" dontlog
          SetEnvIf Referer "\.(gdn|guru)" dontlog
          SetEnvIf Referer "\.(icu|info)" dontlog
          SetEnvIf Referer "\.(live|life)" dontlog
          SetEnvIf Referer "\.(market|men|mobi)" dontlog
          SetEnvIf Referer "\.(name|news|ninja)" dontlog
          SetEnvIf Referer "\.online" dontlog
          SetEnvIf Referer "\.(plus|pro)" dontlog
          SetEnvIf Referer "\.(site|store)" dontlog
          SetEnvIf Referer "\.top" dontlog
          SetEnvIf Referer "\.(video|vip)" dontlog
          SetEnvIf Referer "\.(website|work|win)" dontlog
          SetEnvIf Referer "\.(xxx|xyz)" dontlog

Any terms/expressions whose are not flagged by »dontlog« are written in access.log, else dumped.


Put the essential CustomLog at the end of file apache2.conf.

CustomLog ${APACHE_LOG_DIR}/access.log combined env=!dontlog


root@raspberry:# service apache2 restart


Aspects of the request are
Remote_Host - the hostname (if available) of the client making the request
Remote_Addr - IP address of the client making the request
Server_Addr - IP address of the server on which the request was received
Request_Method - name of the method being used (GET, POST, PUT, CONNECT)
Request_Protocol - protocol name & version with which the request was made (e.g. "http/1.1")
Request_URI - the resource requested on the http request line


          SetEnIf Request_Method "POST" dontlog
          SetEnIf Request_Method "^(POST|CONNECT)$" dontlog

To not log requests from any clients whose hostname ends in example.com, use:


          SetEnvIf Remote_Host "example\.com$" dontlog
          SetEnvIf Remote_Host "(example|example2|example3)\.com$" dontlog

To not log requests from any clients whose hostname begins with example, use:


          SetEnvIf Remote_Host "^example" dontlog
          SetEnvIf Remote_Host "^(example|example2|example3)" dontlog

To not log certain referrals (e.g. your own domain or just to avoid, reduce spamming), use:


          SetEnvIfNoCase Referer "westio\.ru" dontlog
          SetEnvIfNoCase Referer "\.ru" dontlog
          SetEnvIf User-Agent "zgrab" dontlog
          SetEnvIf User-Agent "(hello|zgrab)" dontlog

In HTTP, »Referer« (a misspelling of Referrer) is the name of an optional HTTP header field that identifies the address of the web page i.e., the URI or IRI, from which the resource has been requested.

Virtual hosts logging


Regardless, full logging continues to run for the virtual hosts. Unless it is switched off. This can extend the lifetime of a SD card if unnecessary Read/Write processes are prevented. The sizes of these files can reach several megabytes within one day.


Comment # the line to CustomLog.


/var/log/apache2/other_vhosts_access.log


root@raspberry:# nano /etc/apache2/conf-available/other-vhosts-access-log.conf


   # Define an access log for VirtualHosts that don't define their own logfile
   # CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined

root@raspberry:# service apache2 restart


13-Feb 2018
Updated 02-Jul 2022


Live log monitoring with Linux's »tail« command


»tail« is a program on Unix and Unix-like systems used to display the tail end of a text file or piped data.


tail (Unix) at Wikipedia


user@raspberry:~ $ tail -f -n +1 /var/log/apache2/access.log
user@raspberry:~ $ tail -f -n +1 /var/log/apache2/error.log


-f   This option will cause »tail« will loop forever, checking for new data at the end of the file. When new data appears, it will be printed. By default, tail will output the last 10 lines of its input to the standard output.


-n +1   If you put a plus sign before num, »tail« will output all lines beginning with that line.
For example, -n +1 will print every line.


Terminate by Crtl C



Output without any line breaks. | cut -c -$COLUMNS


user@raspberry:~ $ tail -f -n +1 /var/log/apache2/access.log | cut -c -$COLUMNS


user@raspberry:~ $ tail -f -n +1 /var/log/apache2/error.log | cut -c -$COLUMNS


18-Feb 2018
Updated 02-Jul 2022


Contitional logging with examples | Suppress certain status and error codes


Regarding to the Apache 2.4+ documentation (Expressions) https://httpd.apache.org/docs/2.4+/expr.html.


The big concern and the biggest challenge was where to put the code to.

I went through so many variations until I finally got a working result.


Server version: Apache/2.4+.54 (Raspbian)

Server built: 2022-06-09T04:26:43


I make it as short as possible. At first the module mod_log_config must be enabled.


root@raspberry:# apache2ctl -M


        ...
        log_config_module (static)
        ...

Examples to conditional logging.

First: this will exclude from the log file the error codes 403 (Forbidden) and 410 (Gone).

Second: this example won't write anything what is greater than or equal to 304 (Not Modified).


        # Contitional logging
        CustomLog /dev/null common "expr=%{REQUEST_STATUS} -in {'403','410'}"
        CustomLog /dev/null common "expr=%{REQUEST_STATUS} > 304"
        # Or if this does not function
        CustomLog ${APACHE_LOG_DIR}/waste.log common 
                                  "expr=%{REQUEST_STATUS} -in {'403','410'}"
        CustomLog ${APACHE_LOG_DIR}/waste.log common 
                                  "expr=%{REQUEST_STATUS} > 304"

/dev/null is the non-physical sink hole. The place where everything is dumped what is not needed.


Put the code in the global apache.conf at the very end of file.

 Do not put the code in the 000-default.conf and default-ssl.conf at the same time.


25-May 2021
Updated 13-Jul 2022


HTTP web server on port 80


/etc/apache2/sites-available/000-default.conf => <VirtualHost *:80>


        <VirtualHost *:80>
        ...
	ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog /dev/null common "expr=%{REQUEST_STATUS} > 304"
        ...
        </VirtualHost>

HTTPS web server on port 443


/etc/apache2/sites-available/default-ssl.conf => <VirtualHost _default_:443>


        <IfModule mod_ssl.c>
            <VirtualHost _default_:443>
            ...
            ErrorLog ${APACHE_LOG_DIR}/error.log
            CustomLog /dev/null common "expr=%{REQUEST_STATUS} > 304"
            # Or if this does not function
            CustomLog ${APACHE_LOG_DIR}/waste.log common 
                                      "expr=%{REQUEST_STATUS} > 304"
            ...
            </VirtualHost>
        <IfModule>

Essential after each modification


root@raspberry:# apache2ctl configtest


        Syntax OK

root@raspberry:# systemctl restart apache2

Alternative contitional logging


This may be a good alternative for you, if you do not want to suppress certain status and error codes, this famous rule swaps certain log entries to an other file named waste.log.


        CustomLog ${APACHE_LOG_DIR}/waste.log common 
                                  "expr=%{REQUEST_STATUS} -in {'403','410'}"

Syntax OK ? Then restart Apache for generating a new file and access to write to.


Want the both status codes 200 (OK) and 206 (Partial Content) in your log only ? Simply made.


        CustomLog /dev/null common "expr=%{REQUEST_STATUS} != {'200','206'}"

root@raspberry:# systemctl restart apache2


Summary | The experimental playground figured out so far


This guesswork kept me busy for almost two weeks in total.

Funnily enough, I got it turned around so that it works halfway.

Please note, that the persistent error code 400 (Bad Request) still passes through anyway.


11-Jul 2022
Updated 19-Jul 2022


Hint | Combine dontlog rules and status code(s) selected


root@raspberry:# nano /etc/apache2/apache2.conf


This logs all entries with the status code 200 only

...
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%200{User-Agent}i\"" combined
# Another example: \"%200,206,301,302,304{User-Agent}i\" 
# for more than one status code is also possible.
...
CustomLog ${APACHE_LOG_DIR}/access.log combined env=!dontlog

root@raspberry:# systemctl restart apache2


21-Jul 2023


Epilogue


Additionally to mention, it is allowed to use multiple CustomLog(s).

Instead of the standard 403 error code received, send all to any web address.


.htaccess

        ...
        RewriteEngine On
        ...
        ErrorDocument 403 https://startpage.com/
        ...

Done.


It's coffee time.


25-May 2021
Updated 30-Jun 2022

dosboot.org 2024 | Design and layout handmade in Northwest Europe