..
	Copyright (c) 2010-2021 Varnish Software AS
	SPDX-License-Identifier: BSD-2-Clause
	See LICENSE file for full text of license

.. role:: ref(emphasis)

.. _vinylncsa(1):

=========
vinylncsa
=========

-------------------------------------------------------------
Display Vinyl Cache logs in Apache / NCSA combined log format
-------------------------------------------------------------

:Manual section: 1

SYNOPSIS
========

.. include:: ../include/vinylncsa_synopsis.rst
vinylncsa |synopsis|

DESCRIPTION
===========

The vinylncsa utility reads vinyld(1) shared memory logs and
presents them in the Apache / NCSA "combined" log format.

Each log line produced is based on a single Request type transaction
gathered from the shared memory log. The Request transaction is then
scanned for the relevant parts in order to output one log line. To
filter the log lines produced, use the query language to select the
applicable transactions. Non-request transactions are ignored.

The following options are available:

.. include:: ../include/vinylncsa_options.rst

MODES
=====

The default mode of `vinylncsa` is "client mode".  In this mode, the
log will be similar to what a web server would produce in the absence
of Vinyl Cache.  Client mode can be explicitly selected by using -c.

If the -b switch is specified, vinylncsa will operate in "backend
mode".  In this mode, requests generated by Vinyl Cache to the backends
will be logged.  Unless -c is also specified, client requests received
by `vinyld` will be ignored.

When running vinylncsa in both backend and client mode, it is
strongly advised to include the format specifier %{Vinyl:side}x to
distinguish between backend and client requests.

Client requests that results in a pipe (ie. return(pipe) in vcl), will
not generate logging in backend mode. This is because Vinyl Cache is not
generating requests, but blindly passes on bytes in both directions.
However, a vinylncsa instance running in normal mode can see this
case by using the formatter %{Vinyl:handling}x, which will be 'pipe'.

In backend mode, some of the fields in the format string get different
meanings.  Most notably, the byte counting formatters (%b, %I, %O)
considers `vinyld` to be the client.

It is possible to keep two vinylncsa instances running, one in
backend mode, and one in client mode, logging to different files.

.. _ncsa-format:

FORMAT
======

Specify the log format to use. If no format is specified the default log
format is used::

  %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"

Escape sequences \\n and \\t are supported.

Supported formatters are:

%b
  In client mode, size of response in bytes, excluding HTTP headers.
  In backend mode, the number of bytes received from the backend,
  excluding HTTP headers.  In CLF format, i.e. a '-' rather than a 0
  when no bytes are sent.

%D
  In client mode, time taken to serve the request, in microseconds.
  In backend mode, time from the request was sent to the entire body
  had been received. This is equivalent to %{us}T.

%H
  The request protocol. Defaults to HTTP/1.0 if not known.

%h
  Remote host. Defaults to '-' if not known.  In backend mode this is
  the IP of the backend server.

%I
  In client mode, total bytes received from client.  In backend mode,
  total bytes sent to the backend.

%{X[:first|last]}i
  The contents of request header X before any VCL processing for client side,
  and before vcl_backend_response for backend side. If the header appears
  multiple times in a single transaction, the last occurrence is used in
  backend mode and the first one in client mode.

  If an optional `:first` or `:last` is used, VCL processing will be taken into
  account and the matching rule will be first match or last match respectively.
  When using `:last`, unset headers will still be captured.


%l
  Remote logname. Always '-'.

%m
  Request method. Defaults to '-' if not known.

%{X[:first|last]}o
  The contents of response header X, as it was delivered for client mode, and
  before VCL processing for backend mode. If the header appears multiple times
  in a single transaction, the last occurrence is used in client mode and the
  first one in backend mode.

  If an optional `:first` or `:last` is used, VCL processing will be taken into
  account and the matching rule will be first match or last match respectively.
  When using `:last`, unset headers will still be captured.

%O
  In client mode, total bytes sent to client.  In backend mode, total
  bytes received from the backend.

%q
  The query string. Defaults to an empty string if not present.

%r
  The first line of the request. Synthesized from other fields, so it
  may not be the request verbatim. See the NOTES section.

%s
  Status sent to the client.  In backend mode, status received from
  the backend.

%t
  In client mode, time when the request was received, in HTTP
  date/time format.  In backend mode, time when the request was sent.

%{X}t
  In client mode, time when the request was received, in the format
  specified by X.  In backend mode, time when the request was sent.
  The time specification format is the same as for strftime(3) with
  these extensions:

  * ``%{sec}``: number of seconds since the Epoch
  * ``%{msec}``: number of milliseconds since the Epoch
  * ``%{usec}``: number of milliseconds since the Epoch
  * ``%{msec_frac}``: millisecond fraction
  * ``%{usec_frac}``: microsecond fraction

  The extensions cannot be combined with each other or strftime(3) in
  the same specification. Use multiple ``%{X}t`` specifications instead.

%T
  In client mode, time taken to serve the request, in seconds.  In
  backend mode, time from the request was sent to the entire body had
  been received. This is equivalent to %{s}T.

%{X}T
  In client mode, time taken to serve the request, in the format
  specified by X.  In backend mode, time from the request was sent to
  the entire body had been received. The time specification format can
  be one of the following: s (same as %T), ms or us (same as %D).

%U
  The request URL without the query string. Defaults to '-' if not
  known.

%u
  Remote user from auth.

%{X}x
  Extended variables.  Supported variables are:

  Vinyl:default_format
    The log format used when neither -f nor -F options are specified.
    Useful for appending/prepending with other formatters.

  Vinyl:time_firstbyte
    Time from when the request processing starts until the first
    byte is sent to the client, in seconds.  For backend mode: Time
    from the request was sent to the backend to the entire header
    had been received.

  Vinyl:hitmiss
    In client mode, one of the 'hit' or 'miss' strings, depending on whether
    the request was a cache hit or miss. Pipe, pass and synth are considered
    misses. In backend mode, this field is blank.

  Vinyl:handling
    In client mode, one of the 'hit', 'hitmiss', 'hitpass', 'miss', 'pass',
    'pipe' or 'synth' strings indicating how the request was handled. In
    backend mode, this field is blank.

  Vinyl:side
    Backend or client side. One of two values, 'b' or 'c', depending
    on where the request was made. In pure backend or client mode,
    this field will be constant.

  Vinyl:vxid
    The VXID of the `vinyld` transaction.

  VCL_Log:key
    The value set by std.log("key:value") in VCL.

  VSL:tag:record-prefix[field]
    The value of the VSL entry for the given tag-record prefix-field
    combination. Tag is mandatory, the other components are optional.

    The record prefix will limit the matches to those records that
    have this prefix as the first part of the record content followed
    by a colon.

    The field will, if present, treat the log record as a white
    space separated list of fields, and only the nth part of the
    record will be matched against. Fields start counting at 1 and
    run up to 255.

    Defaults to '-' when the tag is not seen, the record prefix
    does not match or the field is out of bounds. If a tag appears
    multiple times in a single transaction, the first occurrence
    is used.

SIGNALS
=======

* SIGHUP

  Rotate the log file (see -w option) in daemon mode,
  abort the loop and die gracefully when running in the foreground.

* SIGUSR1

  Flush any outstanding transactions.

NOTES
=====

The ``%r`` formatter is equivalent to ``%m http://%{Host}i%U%q %H``. This
differs from the Apache HTTP Server ``%r`` behavior, equivalent to ``%m %U%q
%H``.

Note that request fields are collected on a first match basis in client mode and
last match basis in backend mode. Similarly, response fields are collected on a
first match basis in backend mode and last match basis in client mode.

In other words, request headers are logged as they were received from the client
and as they were sent to the backend, while response headers are logged as they
were sent to the client and as they were received from the backend.

Furthermore, these rules also apply to items that appear multiple times in a
transaction. For example, if a header appears multiple times in a client
request, the first occurrence is logged in client mode, while in backend mode
the last occurrence is logged.

Prior to 7.7, formats %{X}i and %{X}o used to include header fields populated
from VCL. Consider %{VCL_Log:key}x instead to capture data from VCL
transactions.

EXAMPLE
=======

Log the second field of the Begin record, corresponding to the VXID
of the parent transaction::

  vinylncsa -F "%{VSL:Begin[2]}x"

Log the entire Timestamp record associated with the processing length::

  vinylncsa -F "%{VSL:Timestamp:Process}x"

Log in JSON, using the -j flag to ensure that the output is valid JSON
for all inputs::

  vinylncsa -j -F '{"size": %b, "time": "%t", "ua": "%{User-Agent}i"}'

Log the first and last values of a request header that is modified multiple
times in VCL:

  vinylncsa -F "%{x-custom-header:first}i %{x-custom-header:last}i"

SEE ALSO
========

:ref:`vinyld(1)`
:ref:`vinyllog(1)`
:ref:`vinylstat(1)`
:ref:`vsl-query(7)`
:ref:`vsl(7)`

HISTORY
=======

The vinylncsa utility was developed by Poul-Henning Kamp in
cooperation with Verdens Gang AS and Redpill-Linpro. This manual page was
initially written by Dag-Erling Smørgrav <des@des.no>, and later updated
by Martin Blix Grydeland and Pål Hermunn Johansen.


COPYRIGHT
=========

This document is licensed under the same licence as Vinly Cache
itself. See LICENCE for details.

* Copyright (c) 2006 Verdens Gang AS
* Copyright (c) 2006-2016 Varnish Software AS
