<!--
$Id$
-->

KONKURRENTER:

AWstats: Ikke real-time
Webalizer: Ikke real-time
Analog: Ikke real-time
Hitbot: Ikke real-time

sitestats.com <-- Hvis vi kan dette, så kan vi meget
sitestats.co.uk
livestats.com
webtrends.com
deeplog-software.com
logrover.com
s-tracking.com
clicktracks.com
instadia.com
sawmill.net
websense.com
http://www.123loganalyzer.com
CounterCentral.com
www.sitestatslive.com
webtrendslive.com
http://www.webstat.com/


TODO:

Gruppering af UserAgent after producent og OS.


Potientielt problem: hvordan virker genindlæsning af data.
Patch mod_log med Global counter, %G.

* Skal vi have en session tabel med?:
  + Start tid
  + Slut tid - opdateres ved hvert hit
  + Landekode og stat/provins
  + CountryIdentifiedBy (cidr,rev-dns,AccLanguage,UserAgent-Country)
  + UserAgent type (Mozilla/MSIE)
  + UserAgent version (4.74)
  + UserAgent OS (Linux)
  + UserAgent OS version (2.4.23)
  + ClientType (Human, robot)
  
FeedHost skal have lokalt ipadresse med så man kan se hvilke
netkort der får mest trafik. Det hedder %A i apache logformat.


mod_gzip
LogFormat "%h %l %u %t \"%r\" %>s %b
mod_gzip: %{mod_gzip_result}n
In:%{mod_gzip_input_size}n
Out:%{mod_gzip_output_size}n:%{mod_gzip_compression_ratio}npct."
common_with_mod_gzip_info2


CustomerText
============

Billing customer.

example.com	example.com	"Example Inc."
joe.com		joe.com		"Joe Inc."
black.joe.com	black.joe.com
joe.isp.com	joe
isp.com/~joe	joe		"Joe Black"


<chapter id="intro">
<title>Introduction</title>

<para>
ToughLog er logfilsanalyse og statistik, der foregår online i realtime. Det
er ikke batchanalyse. Det er interaktivt og svaret kommer med det samme -
instantaniously.
</para>

</chapter><!-- id="intro" -->

<chapter id="install">
<title>Install</title>

<para>
To make it easier to use the examples in the manual
it is a good idea to create the same hostnames.
You do not have to but it makes the manual easier to read.
Asuming your domain is <emphasis>isp.com</emphasis>
and that your <filename>/etc/resolv.conf</filename>
has the line <emphasis>search isp.com</emphasis> .
Give the database server the hostname <emphasis>toughlogdbw</emphasis> .
Give the web-server the name <emphasis>toughlog</emphasis> .
Either these names has to be added in your DNS or in 
<filename>/etc/hosts</filename> on all your computers.
An example of an <filename>/etc/hosts</filename>
where each server has it own ip address:
</para>

<programlisting>
127.0.0.1       localhost.isp.com	localhost
192.168.1.3	toughlog.isp.com	toughlog
192.168.1.4	toughlogdbw.isp.com	toughlogdbw
192.168.1.5	webhotel.isp.com	webhotel
</programlisting>

<para>
With this setup you can browse to your toughlog web-server
<xref linkend="http://toughlog/">http://toughlog/</xref> .
</para>

<para>
Minimum requirements:
PostgreSQL 8.x
Apache 1.3.30
Perl 5.8
DBI::Pg 1.46
Sys::Syslog 0.03
</para>

<screen>
<prompt>[root@toughlogdbw ~]#</prompt> <userinput>echo $HOSTNAME # Check hostname</userinput>
toughlogdbw.isp.com
<prompt>[root@toughlogdbw ~]#</prompt> <userinput>adduser -m -c "Tough Log" toughlog</userinput>
<prompt>[root@toughlogdbw ~]#</prompt> <userinput>passwd toughlog</userinput>
<prompt>[root@toughlogdbw ~]#</prompt> <userinput>su - postgres</userinput>
<prompt>[postgres@toughlogdbw ~]$</prompt> <userinput>createuser -A -D toughlog</userinput>
<prompt>[postgres@toughlogdbw ~]$</prompt> <userinput>psql -c "ALTER USER toughlog ENCRYPTED PASSWORD '$(head /dev/urandom | md5sum)'"</userinput>
<prompt>[postgres@toughlogdbw ~]$</prompt> <userinput>createdb -E UTF8 -O toughlog toughlog # Create database</userinput>
<prompt>[postgres@toughlogdbw ~]$</prompt> <userinput>vi $PGDATABASE/pg_hba.conf # Edit host permission</userinput>
<prompt>[postgres@toughlogdbw ~]$</prompt> <userinput>logout</userinput>
<prompt>[root@toughlogdbw ~]#</prompt> <userinput>su - toughlog</userinput>
<prompt>[toughlog@toughlogdbw ~]$</prompt> <userinput>psql -l # Check database created, owner and UNICODE</userinput>
        List of databases
   Name    |  Owner   | Encoding
-----------+----------+-----------
 toughlog  | toughlog | UNICODE		# Must be UNICODE!
 template0 | postgres | SQL_ASCII
 template1 | postgres | SQL_ASCII
(3 rows)
<prompt>[toughlog@toughlogdbw ~]$</prompt> <userinput>psql # Try database</userinput>
Welcome to psql 7.4.3, the PostgreSQL interactive terminal.
<prompt>toughlog=$</prompt> <userinput>\q</userinput>
<prompt>[toughlog@toughlogdbw ~]$</prompt> <userinput>tar xjf toughlog*tar.*	# Replace '*' with version</userinput>
<prompt>[toughlog@toughlogdbw ~]$</prompt> <userinput>cd sql</userinput>
<prompt>[toughlog@toughlogdbw sql]$</prompt> <userinput>make create</userinput>
<prompt>[toughlog@toughlogdbw sql]$</prompt> <userinput>cd</userinput>
<prompt>[toughlog@toughlogdbw ~]$</prompt>
</screen>

<para>
Apache webserver should have installed a special log format.
It is supplied in the <filename>apache.conf</filename>.
Place the file in <filename>/etc/toughlog/</filename>
on your webserver.
Edit <filename>/etc/httpd/conf/httpd.conf</filename>
at the bottom and add the follwing line:
</para>

<programlisting>
Include  /etc/toughlog/apache.conf
</programlisting>

<para>
If you are running Vhost (virtual hosts) you
might have to add the following line to each
host you want to watch with ToughLog.
Only the highlighted line should be added:
</para>

<programlisting>
&lt;VirtualHost *;&gt;
	Servername	example.com
	...
	CustomLog	/var/log/apache/example.com_access_log combined
	<emphasis>CustomLog	/var/log/apache/toughlog_access_log toughlog</emphasis>
	...
&lt;/VirtualHost&gt;
</programlisting>

<para>
Apache can have more than one CustomLog per webserver
so you can keep your old format also.
</para>

<para>
After configuring of Apache make a reload:
</para>

<screen>
<prompt>[root@rack01 ~]#</prompt> <userinput>/etc/init.d/apache reload</userinput>
</screen>

<para>
Test the installation of the new log format by giving your
webserver some hits and watch the log file:
</para>

<screen>
<prompt>[root@rack01 ~]#</prompt> <userinput>tail -f /var/log/apache/toughlog_access_log</userinput>
</screen>

<para>
The raw Apache logfile has to converted to SQL to get
in to the SQL-server.
This is done with the Perl script <command>2sql.pl</command>.
You can test this with the following command:
</para>

<screen>
<prompt>[root@rack01 ~]#</prompt> <userinput>cd /var/log/apache/</userinput>
<prompt>[root@rack01 apache]#</prompt> <userinput>head -n 1 toughlog_access_log | 2sql.pl</userinput>
select addhit(
	'example.com',
	...(about 30 lines comes here)
);
</screen>

<para>
The SQL command above has to be given to the SQL-server.
If you have configured your SQL-server and webserver as described
above, you can add one hit with the following command:
</para>

<screen>
<prompt>[root@rack01 apache]#</prompt> <userinput>head -n 1 toughlog_access_log | 2sql.pl | psql -t -h toughlogdbw -U toughlog</userinput>
NOTICE:  New SessionID
	1
</screen>

<para>
When the connection to the database is tested
you can start adding data SQL-server:
</para>

<screen>
<prompt>[root@rack01 apache]#</prompt> <userinput>tail -f -n 1 toughlog_access_log | 2sql.pl | psql -t -h toughlogdbw -U toughlog</userinput>
</screen>

<para>
Go to your SQL-server and login.
Test that the database is running and data is coming in:
</para>

<screen>
<prompt>[toughlog@toughlogdbw ~]$</prompt> <userinput>psql -c "SELECT COUNT(*) FROM httplog"</userinput>
  count
----------
 28210967
(1 row)
</screen>

<para>
There are some scripts which can be tried to view 
the incoming data.
The command below show the date of the latest
hit in the database.
</para>

<screen>
<prompt>[toughlog@toughlogdbw ~]$</prompt> <userinput>psql &lt; q_pipelinedelay.sql</userinput>
      hitdatetime       | Pipeline delay
------------------------+----------------
 2004-08-04 23:30:03+02 | 00:04:50.38366
(1 row)
</screen>

</chapter><!-- id="install" -->

<chapter id="reportowner">
<title>Report Owner</title>

<para>
(about how to use)
</para>

<sect1 id="internal-works">
<title>Internal works</title>
<para>
How it works
</para>

</sect1><!-- id="internal-works" -->

<sect1 id="enhanced">
<title>Enhanced reports</title>

<para>
There is several things which can be done
by the Report Owner which will give more
datiled reports.
This requires that the Report Owner installs
some extra scripts and do some configuration.
</para>


<sect2 id="track-sessions">
<title>Track sessions</title>

<para>
Install AWstats <filename>/js/awstats_misc.js</filename>
.
This file is written in JavaScript and will
only work if the end user has enabled JavaScript in the browser.
</para>

<para>
If the web hotel you are using has installed 
<filename>mod_usertrack</filename> you can enable
this by placing a <filename>.htaccess</filename>
in your root directory.
The content of <filename>.htaccess</filename>
should look like:
</para>

<programlisting># Filename: .htaccess
&lt;IfModule mod_usertrack.c&gt;
	CookieTracking on
&lt;/IfModule&gt;
</programlisting>

<para>
Please note that the should look exactly as
written with a '.' (dot) in front of the name.
When properly installed ToughLog would use this
feature immidiately.
This module requires the end user to have enabled cookies
in the browser.
</para>

<para>
If you have programmed your web site in PHP
you can add a session cookie in your PHP script.
As one of the very first lines in your
script you shoudl add <command>session_start();</command>.
</para>

<programlisting>
&lt;php
	session_start();
?&gt;&lt;!DOCTYPE....
</programlisting>

<para>
Even if the end user of your web site has
disabled cookies, PHP can be used to track
the users click-way on your site.
</para>

</sect2><!-- id="track-sessions" -->

</chapter><!-- id="reportowner" -->

<chapter id="develop">
<title>Development</title>

<para>
The back-end of ToughLog is programmed in Perl
and the web front-end is programmed in PHP.
The database used is PostgreSQL.
</para>

<sect1 id="session-trace">
<title>Session trace</title>

<para>
ToughLog uses several different ways to track a
<emphasis>click-session</emphasis>.
If the hit comes in with a cookie
toughlog will go back in the previous hits and
look for a hit with the same IP-address and
if a referer in the current hit matches the previous.
</para>

</sect1><!-- id="session-trace" -->

</chapter><!-- id="develop" -->

ReportOwner
	The person who owns a customer web-site and view the toughlog reports.
	Typically this is the site owner of <emphasis>example.com</emphasis>.

isp.com
	The domain where ToughLog is installed.

example.com
	A customer domain which is hosted at isp.com.


<table id="logfileformat">
<title>ToughLog Apache log file format</title>
<tgroup cols="4" align="char">
 <thead>
  <row>
   <entry>Number</entry>
   <entry>Code</entry>
   <entry>Sample</entry>
   <entry>Description</entry>
  </row>
 </thead>
 <tbody>
  <row>
   <entry>1</entry> 
   <entry>[0-9a-zA-Z]</entry>
   <entry>25</entry>
   <entry>ToughLog version number ID. Alpha numeric.</entry>
  </row>
  <row>
   <entry>2</entry> 
   <entry>%v</entry>
   <entry>example.com</entry> 
   <entry>ServerName. The term "ServerName" is used in Apache httpd.conf and vhost.conf.</entry>
  </row>
  <row>
   <entry>3</entry> 
   <entry>%{Host}i</entry>
   <entry>www.example.com</entry> 
   <entry>Requested hostname.</entry>
  </row>
  <row>
   <entry>4</entry> 
   <entry>%a</entry>
   <entry>12.34.56.78</entry> 
   <entry>Remote IP-address.</entry>
  </row>
  <row>
   <entry>5</entry> 
   <entry>%u</entry>
   <entry>joe</entry> 
   <entry>Auth user name.</entry>
  </row>
  <row>
   <entry>6</entry> 
   <entry>%{%F %T %z}t</entry>
   <entry>2004-12-31 23:59:59 +0200</entry> 
   <entry>Time in ISO-8601 format with time zone.</entry>
  </row>
  <row>
   <entry>7</entry> 
   <entry>%r</entry>
   <entry>GET /index.php?q=test HTTP/1.1</entry> 
   <entry>First line of request.</entry>
  </row>
  <row>
   <entry>8</entry> 
   <entry>%f</entry>
   <entry>/home/www/example.com/index.php</entry> 
   <entry>Filename served.</entry>
  </row>
  <row>
   <entry>9</entry> 
   <entry>%>s</entry>
   <entry>200</entry> 
   <entry>Status</entry>
  </row>
  <row>
   <entry>10</entry> 
   <entry>%B</entry>
   <entry>321</entry> 
   <entry>Bytes sent, excluding HTTP headers.</entry>
  </row>
  <row>
   <entry>11</entry> 
   <entry>%T</entry>
   <entry>1</entry> 
   <entry>The time taken to serve the request, in seconds. In Apache 2.0 this should be %D.</entry>
  </row>
  <row>
   <entry>12</entry> 
   <entry>%c</entry>
   <entry>+</entry> 
   <entry>Connection status when response was completed.</entry>
  </row>
  <row>
   <entry>13</entry> 
   <entry>%{Referer}i</entry>
   <entry>http://www.google.com/search?q=something</entry> 
   <entry>Referrer URI.</entry>
  </row>
  <row>
   <entry>14</entry> 
   <entry>%{User-Agent}i</entry>
   <entry>Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031007</entry> 
   <entry>Client user agent (browser).</entry>
  </row>
  <row>
   <entry>15</entry> 
   <entry>%{Accept-Language}i</entry>
   <entry>da_DK</entry> 
   <entry>A list of languages the client accepts.</entry>
  </row>
  <row>
   <entry>16</entry> 
   <entry>%{Accept}i</entry>
   <entry>text/html,image/png</entry> 
   <entry>A list of mime-types the client accepts.</entry>
  </row>
  <row>
   <entry>17</entry> 
   <entry>%{Accept-Encoding}i</entry>
   <entry>gzip,deflate</entry> 
   <entry>A list of encodings the client accepts.</entry>
  </row>
  <row>
   <entry>18</entry> 
   <entry>%{Accept-Charset}i</entry>
   <entry>ISO-8859-1,utf-8</entry> 
   <entry>A list of char-set the client accepts.</entry>
  </row>
  <row>
   <entry>19</entry> 
   <entry>%{Via}i</entry>
   <entry>1.1 PROXY</entry> 
   <entry>Via gateway. Don't trust the IP-number for user-tracking.</entry>
  </row>
  <row>
   <entry>20</entry> 
   <entry>%{Cookie}i</entry>
   <entry>PHPSESSID=705293...</entry> 
   <entry>Incoming cookie.</entry>
  </row>
  <row>
   <entry>21</entry> 
   <entry>%{Set-Cookie}o</entry>
   <entry>Apache=port72.ds1...</entry> 
   <entry>Outgoing cookie, set cookie.</entry>
  </row>
  <row>
   <entry>22</entry> 
   <entry>%{cookie}n</entry>
   <entry>37abcdf...</entry> 
   <entry>Mod UserTrack in or outgoing cookie. Can optionally be overwritten by toughlog2sql.</entry>
  </row>
  <row>
   <entry>23</entry> 
   <entry>%{Keep-Alive}i</entry>
   <entry>300</entry> 
   <entry>Number of seconds this socket connection should "keep alive".</entry>
  </row>
  <row>
   <entry>24</entry> 
   <entry>%{Connection}i</entry>
   <entry>+</entry> 
   <entry>Status of how socket connection ended. '-' = ended. '+' = keep alive. 'X' = ended during transfer.</entry>
  </row>
  <row>
   <entry>25</entry> 
   <entry>%{Transfer-Encoding}o</entry>
   <entry>chunked</entry> 
   <entry>Compression of data.</entry>
  </row>
  <row>
   <entry>26</entry>
   <entry>%{Content-Type}o</entry> 
   <entry>text/html</entry> 
   <entry>Mime-type. Set by server.</entry>
  </row>
 </tbody>
</tgroup>
</table>
