Just Enough Developed Infrastructure

Debug crashed apache processes with mod_log_forensics

More info can be found @ http://httpd.apache.org/docs/2.0/mod/mod_log_forensic.html

Each request is logged two times. The first time is before it's processed further (that is, after receiving the headers). The second log entry is written after the request processing at the same time where normal logging occurs.

In order to identify each request, a unique request ID is assigned. This forensic ID can be cross logged in the normal transfer log using the %{forensic-id}n format string. If you're using mod_unique_id, its generated ID will be used.

The first line logs the forensic ID, the request line and all received headers, separated by pipe characters (|). A sample line looks like the following (all on one line):

+yQtJf8CoAB4AAFNXBIEAAAAA|GET /manual/de/images/down.gif HTTP/1.1|Host:localhost%3a8080|User-Agent:Mozilla/5.0 (X11; U; Linux i686; en-US; rv%3a1.6) Gecko/20040216 Firefox/0.8|Accept:image/png, etc...

The plus character at the beginning indicates that this is the first log line of this request. The second line just contains a minus character and the ID again:


The check_forensic script takes as its argument the name of the logfile. It looks for those +/- ID pairs and complains if a request was not completed.