.. Licensed to the Apache Software Foundation (ASF) under one
   or more contributor license agreements.  See the NOTICE file
   distributed with this work for additional information
   regarding copyright ownership.  The ASF licenses this file
   to you under the Apache License, Version 2.0 (the
   "License"); you may not use this file except in compliance
   with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing,
   software distributed under the License is distributed on an
   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
   KIND, either express or implied.  See the License for the
   specific language governing permissions and limitations
   under the License.

.. include:: ../../common.defs

.. _admin-logging-examples:

Configuration Examples
**********************

This section provides examples for a wide range of logging scenarios. While not
exhaustive of all possibilities (|TS| logging is quite flexible), these entries
should hopefully get most administrators headed onto the right path.

Unless otherwise noted, the example configurations here are to be applied in
:file:`logging.config`.

Online Event Log Builder
========================

If you need any assistance building your event log, you can try out our
`online log builder <http://trafficserver.apache.org/logbuilder/>`_. This is a
work in progress, so any comments, critique or suggestions are most welcome.

Emulating Other HTTP Server Formats
===================================

.. _admin-logging-examples-netscape:

Netscape Common
---------------

The following figure shows a sample log entry in a Netscape Common log file.

.. figure:: /static/images/admin/netscape_common_format.jpg
   :align: center
   :alt: Sample Netscape Common log entry

The numbered sections correspond to the following log fields in |TS|:

=== ====== ====================================================================
No. Field  Description
=== ====== ====================================================================
1   chi    The IP address of the client's host machine.
2   --     This hyphen (``-``) is always present in Netscape log entries.
3   caun   The authenticated client username. A hyphen (``-``) means no
           authentication was required.
4   cqtn   The date and time of the client request, enclosed in brackets.
5   cqtx   The request line, enclosed in quotes.
6   pssc   The proxy response status code (HTTP reply code).
7   pscl   The length of the |TS| response to the client in bytes.
=== ====== ====================================================================

To recreate this as a log format in :file:`logging.config` you would define the
following format object:

.. code:: lua

   common = format {
     Format = '%<chi> - %<caun> [%<cqtn>] "%<cqtx>" %<pssc> %<pscl>'
   }

.. _admin-logging-examples-extended:

Netscape Extended
-----------------

The following figure shows a sample log entry in a Netscape Extended log file.

.. figure:: /static/images/admin/netscape_extended_format.jpg
   :align: center
   :alt: Sample Netscape Extended log entry

In addition to fields 1-7 from the Netscape Common log format, the Extended
format adds the following fields:

=== ===== =====================================================================
No. Field Description
=== ===== =====================================================================
8   sssc  The origin server response status code.
9   sshl  The server response transfer length; the body length in the
          origin server response to |TS|, in bytes.
10  cqcl  The client request transfer length; the body length in the
          client request to |TS|, in bytes.
11  pqcl  The proxy request transfer length; the body length in the |TS|
          request to the origin server.
12  cqhl  The client request header length; the header length in the
          client request to |TS|.
13  pshl  The proxy response header length; the header length in the
          |TS| response to the client.
14  pqhl  The proxy request header length; the header length in |TS|
          request to the origin server.
15  sshl  The server response header length; the header length in the
          origin server response to |TS|.
16  tts   The time |TS| spent processing the client request; the number
          of seconds between the time that the client established the
          connection with |TS| and the time that |TS| sent the last byte
          of the response back to the client.
=== ===== =====================================================================

To recreate this as a log format in :file:`logging.config` you would define the
following format object:

.. code:: lua

   extended = format {
     Format = '%<chi> - %<caun> [%<cqtn>] "%<cqtx>" %<pssc> %<pscl> %<sssc> %<sscl> %<cqcl> %<pqcl> %<cqhl> %<pshl> %<pqhl> %<sshl> %<tts>'
   }

.. _admin-logging-examples-extended2:

Netscape Extended-2
-------------------

The following figure shows a sample log entry in a Netscape Extended2 log file.

.. figure:: /static/images/admin/netscape_extended2_format.jpg
   :align: center
   :alt: Sample Netscape Extended-2 log entry

In addition to fields 1-16 from the Netscape Common and Netscape Extended log
formats above, the Extended-2 format adds the following fields:

=== ====== ===============================================================
No.  Field Description
=== ====== ===============================================================
17  phr    The proxy hierarchy route; the route |TS| used to retrieve
           the object.
18  cfsc   The client finish status code: ``FIN`` if the client request
           completed successfully or ``INTR`` if the client request was
           interrupted.
19  pfsc   The proxy finish status code: ``FIN`` if the |TS| request to
           the origin server completed successfully or ``INTR`` if the
           request was interrupted.
20  crc    The cache result code; how the |TS| cache responded to the
           request: ``HIT``, ``MISS``, and so on. Cache result codes are
           listed in :ref:`admin-logging-crc`.
=== ====== ===============================================================

To recreate this as a log format in :file:`logging.config` you would define the
following format object:

.. code:: lua

   extended2 = format {
     Format = '%<chi> - %<caun> [%<cqtn>] "%<cqtx>" %<pssc> %<pscl> %<sssc> %<sscl> %<cqcl> %<pqcl> %<cqhl> %<pshl> %<pqhl> %<sshl> %<tts> %<phr> %<cfsc> %<pfsc> %<crc>'
   }

.. _admin-logging-examples-squid:

Squid
-----

The following figure shows a sample log entry in a Squid log file.

.. figure:: /static/images/admin/squid_format.jpg
   :align: center
   :alt: Sample Squid log entry

The numbered sections correspond to the following log fields in |TS|:

=== ======== ==================================================================
No. Field    Description
=== ======== ==================================================================
1   cqtq     The client request timestamp in Squid format. The time of the
             client request in seconds since January 1, 1970 UTC (with
             millisecond resolution).
2   ttms     The time |TS| spent processing the client request. The number
             of milliseconds between the time the client established the
             connection with |TS| and the time |TS| sent the last byte of
             the response back to the client.
3   chi      The IP address of the client’s host machine.
4   crc/pssc The cache result code; how the cache responded to the request:
             ``HIT``, ``MISS``, and so on. Cache result codes are described
             in :ref:`admin-logging-cache-results`. The
             proxy response status code (HTTP response status code from
             |TS| to client).
5   psql     The length of the |TS| response to the client in bytes,
             including headers and content.
6   cqhm     The client request method: ``GET``, ``POST``, and so on.
7   cauc     The client request canonical URL; blanks and other characters
             that might not be parsed by log analysis tools are replaced by
             escape sequences. The escape sequence is a percentage sign
             followed by the ASCII code number of the replaced character in
             hex.
8   caun     The username of the authenticated client. A hyphen (``-``)
             means that no authentication was required.
9   phr/pqsn The proxy hierarchy route. The route |TS| used to retrieve the
             object.
10  psct     The proxy response content type. The object content type taken
             from the |TS| response header.
=== ======== ==================================================================

To recreate this as a log format in :file:`logging.config` you would define the
following format object:

.. code:: lua

   squid = format {
     Format = '%<cqtq> %<ttms> %<chi> %<crc>/%<pssc> %<psql> %<cqhm> %<cquc> %<caun> %<phr>/%<pqsn> %<psct>'
   }

Separating Logs by Origin
=========================

The default :file:`log_hosts.config` file is located in the |TS| ``config``
directory. To record HTTP transactions for different origin servers in separate
log files, you must specify the hostname of each origin server on a separate
line in :file:`log_hosts.config`. For example, if you specify the keyword
``sports``, then |TS| records all HTTP transactions from ``sports.yahoo.com``
and ``www.foxsports.com`` in a log file called ``squid-sports.log`` (if the
Squid format is enabled).

.. important::

   If |TS| is clustered and you enable log file collation, then you should use
   the same :file:`log_hosts.config` file on every |TS| node in the cluster.

To edit the log hosts list:

#. Enter the hostname of each origin server on a separate line in
   :file:`log_hosts.config`. ::

       webserver1
       webserver2
       webserver3

#. Run the command :option:`traffic_ctl config reload` to apply the changes.

Hourly Rotated Squid Proxy Logs
===============================

The following example demonstrates the creation of a Squid-compatible log
format, which is then applied to a log object containing an hourly rotation
policy.

.. code:: lua

   squid = format {
     Format = '%<cqtq> %<ttms> %<chi> %<crc>/%<pssc> %<psql> %<cqhm> %<cquc> %<caun> %<phr>/%<pqsn> %<psct>'
   }

   log.ascii {
     Format = squid,
     Filename = 'squid',
     RollingEnabled = 1,
     RollingIntervalSec = 3600,
     RollingOffsetHr = 0
   }

Summarizing Number of Requests and Total Bytes Sent Every 10 Seconds
====================================================================

The following example format generates one entry every 10 seconds. Each entry
contains the timestamp of the last entry of the interval, a count of the number
of entries seen within that 10-second interval, and the sum of all bytes sent
to clients:

.. code:: lua

   mysummary = format {
      Format = "%<LAST(cqts)> : %<COUNT(*)> : %<SUM(psql)>",
      Interval = "10"
   }

Dual Output to Compact Binary Logs and ASCII Pipes
==================================================

This example demonstrates logging the same event data to multiple locations, in
a hypothetical scenario where we may wish to keep a compact form of our logs
available for archival purposes, while performing live log analysis on a stream
of the event data.

.. code:: lua

   ourformat = format {
     Format = '%<chi> - %<caun> [%<cqtn>] "%<cqtx>" %<pssc> %<pscl>'
   }

   log.binary {
     Format = ourformat,
     Filename = 'archived_events'
   }
   log.pipe {
     Format = ourformat,
     Filename = 'streaming_log'
   }

Filtering Events to ASCII Pipe for Alerting
===========================================

This example illustrates a situation in which our |TS| cache contains *canary*
objects, which upon their access we want an external alerting system to fire
off all sorts of alarms. To accomplish this, we demonstrate the use of a filter
object that matches events against these particular canaries and emits log data
for them to a UNIX pipe that the alerting software can constantly read from.

.. code:: lua

   canaryformat = format {
     Format = '%<chi> - %<caun> [%<cqtn>] "%<cqtx>" %<pssc> %<pscl>'
   }

   canaryfilter = filter.accept('cqup MATCH "/nightmare/scenario/dont/touch"')

   log.pipe {
     Format = canaryformat,
     Filters = [ canaryfilter ],
     Filename = 'alerting_canaries'
   }

Summarizing Origin Responses by Hour
====================================

This example demonstrates a simple use of aggregation operators to produce an
hourly event line reporting on the total number of requests made to origin
servers (where we assume that any cache result code without the string ``HIT``
in it signals origin access), as well as the average time it took to fulfill
the request to clients during that hour.

.. code:: lua

   originrepformat = format {
     Format = '%<FIRST(cqtq)> %<COUNT(*)> %<AVERAGE(ttms)>',
     Interval = 3600
   }

   originfilter = filter.reject('crc CONTAINS "HIT"')

   log.ascii {
     Format = originrepformat,
     Filters = [ originfilter ],
     Filename = 'origin_access_summary'
   }

