%\VignetteIndexEntry{Multivariate Generalized Kernel Smoothing and Related Statistical Methods}
\documentclass{article}
\usepackage[a4paper,top=2.6cm,bottom=3.6cm,left=3.6cm,right=3.6cm]{geometry}
\usepackage{parskip,verbatim,amsmath,amssymb,color}
\usepackage[nogin]{Sweave}
\pagestyle{myheadings}
\setlength{\parskip}{0.28cm}
\DefineVerbatimEnvironment{Sinput}{Verbatim}{xleftmargin=0.75em, formatcom=\color{rin}}
\DefineVerbatimEnvironment{Soutput}{Verbatim}{xleftmargin=0.75em,formatcom=\color{rout}}
\DefineVerbatimEnvironment{Serror}{Verbatim}{xleftmargin=0.75em,formatcom=\color{rerr}}
\newcommand {\stitle}[3]
{	\title {\vspace {-0.6cm} {\normalsize #1 #2 } \\[0.8cm] {\textbf {\huge #3} } }
	\author {\textbf {Abby Spurdle} }
	\maketitle
	\markright{Spurdle, A.\hfill #1 #2\hfill}
	\thispagestyle {empty}
}
\newcommand {\sabstract}[1]
{	\begin {center}
	\begin {minipage}{14.25cm}
		{\textsl {#1} }
	\end {minipage}
	\end {center}
	\vspace {0.06cm}
}
\definecolor{db}{rgb}{0.1, 0, 0.55}
\definecolor{rin}{rgb}{0, 0, 0.32}
\definecolor{rout}{rgb}{0, 0.14, 0}
\definecolor{rerr}{rgb}{0.5, 0.025, 0}
\SweaveOpts{keep.source=TRUE}
\SweaveOpts{eps=FALSE}
\SweaveOpts{prefix.string=temp-probhat}
\begin{document}

<<echo=false>>=
options(continue=" ", width=80)
options(SweaveHooks=list(fig=function()
par(mar=c(4.1, 4.1, 2.6, 1.6), cex=0.7, cex.main=1)))
set.seed (1)
@

\newcommand{\pnt}{$\bullet$}
\newcommand{\tmu}[1]{\textbf {\textsf {\color{db} #1}}}
\newcommand{\ind}{\hspace {0.375cm} }
\newcommand{\indf}{\vspace {-0.175cm} \hspace {0.375cm} }

\newcommand{\dks}{DKS}
\newcommand{\cks}{CKS}
\newcommand{\cat}{CAT}
\newcommand{\el}{EL}
\newcommand{\phgmix}{gMIXp}
\newcommand{\phxmix}{xMIXp}

\newcommand{\pmfuvdks}{PMF\textsubscript{(UV)}$\sim$DKS}
\newcommand{\cdfuvdks}{CDF\textsubscript{(UV)}$\sim$DKS}
\newcommand{\qfuvdks}{QF\textsubscript{(UV)}$\sim$DKS}

\newcommand{\pdfuvcks}{PDF\textsubscript{(UV)}$\sim$CKS}
\newcommand{\cdfuvcks}{CDF\textsubscript{(UV)}$\sim$CKS}
\newcommand{\qfuvcks}{QF\textsubscript{(UV)}$\sim$CKS}
\newcommand{\pdfmvcks}{PDF\textsubscript{(MV)}$\sim$CKS}
\newcommand{\cdfmvcks}{CDF\textsubscript{(MV)}$\sim$CKS}
\newcommand{\pdfccks}{PDF\textsubscript{(C)}$\sim$CKS}
\newcommand{\cdfccks}{CDF\textsubscript{(C)}$\sim$CKS}
\newcommand{\qfccks}{QF\textsubscript{(C)}$\sim$CKS}
\newcommand{\pdfmvccks}{PDF\textsubscript{(MVC)}$\sim$CKS}
\newcommand{\cdfmvccks}{CDF\textsubscript{(MVC)}$\sim$CKS}
\newcommand{\chqfcks}{ChQF$\sim$CKS}

\newcommand{\pmfuvcat}{PMF\textsubscript{(UV)}$\sim$CAT}
\newcommand{\cdfuvcat}{CDF\textsubscript{(UV)}$\sim$CAT}
\newcommand{\qfuvcat}{QF\textsubscript{(UV)}$\sim$CAT}
\newcommand{\pmfccat}{PMF\textsubscript{(C)}$\sim$CAT}
\newcommand{\cdfccat}{CDF\textsubscript{(C)}$\sim$CAT}
\newcommand{\qfccat}{QF\textsubscript{(C)}$\sim$CAT}

\newcommand{\cdfuvel}{CDF\textsubscript{(UV)}$\sim$EL}
\newcommand{\qfuvel}{QF\textsubscript{(UV)}$\sim$EL}

\newcommand{\pmfcgmix}{PMF\textsubscript{(C)}$\sim$gMIX}
\newcommand{\cdfcgmix}{CDF\textsubscript{(C)}$\sim$gMIX}
\newcommand{\qfcgmix}{QF\textsubscript{(C)}$\sim$gMIX}

\newcommand{\pdfcxmix}{PDF\textsubscript{(C)}$\sim$xMIX}
\newcommand{\cdfcxmix}{CDF\textsubscript{(C)}$\sim$xMIX}
\newcommand{\qfcxmix}{QF\textsubscript{(C)}$\sim$xMIX}

\stitle {probhat}{0.4.0}{Multivariate Generalized\\Kernel Smoothing\\and\\Related Statistical Methods}

\sabstract {Probability mass functions (PMFs), probability density functions (PDFs), cumulative distribution functions (CDFs) and quantile functions, mainly via (optionally bounded/truncated) kernel smoothing. In the continuous case, there's support for univariate, multivariate and conditional distributions, including distributions that are both multivariate and conditional. These generalize methods from the book "Kernel Smoothing", Wand and Jones (1995). Also, supports categorical distributions, mixed conditional distributions (with mixed input types) and smooth empirical-like distributions, some of which, can be used for statistical classification. There are extensions for computing distance matrices (between distributions), multivariate probabilities, multivariate random numbers, moment-based statistics and mode estimates.}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Introduction}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
This is an R package for multivariate generalized kernel smoothing, as per the title.

Kernel smoothing is generalized, by estimating:
\begin{itemize}
\itemsep -0.15cm
	\item Bounded random variables.
	\item Both discrete and continuous probability distributions.
	\item Probability mass functions (PMFs), probability density functions (PDFs), cumulative distribution functions (CDFs) and quantile functions (QFs).
	\item In the continuous case, multivariate and conditional distributions, including multivariate-conditional distributions.
\end{itemize}

Also, there are categorical distributions (univariate and conditional), smooth empirical-like distributions (univariate), and conditional distributions with mixed input types.

Discrete kernel smoothing, could also be described as frequency smoothing.

By default, univariate continuous kernel smoothing models use cubic Hermite splines as intermediate models. Corresponding quantile functions (which are spline only) are constructed by fitting a CDF using a spline, then transposing the control points.

Categorical variables are assumed to be ordinal, however, this assumption is only relevant to the interpretation of the CDF and QF. Empirical-like models are derived from empirical cumulative distribution functions. There's a small modification to the standard formula, and the resulting points are interpolated by a cubic Hermite spline, in a similar way to univariate continuous kernel smoothing models.

A list of models and some notes on terminology, are given in appendices A and B.\\
Also, there are appendices with formulae, but they need some polishing.

All models can be given frequencies or weights, and there are examples of modelling fuzzy clusters with weighted multivariate kernel density estimation, in an appendix.

There are plot methods for all univariate models, and for multivariate models but only with two or three random variables.

Often the goal of kernel smoothing is simply to plot the distribution, as an exploratory tool, however, these models can be used for a variety of purposes.

Conditional categorical distributions with mixed input types, can be used for statistical classification purposes.

Also, there are extensions for computing probabilities, random number generation and parameter-like estimation, including moment-based, order-based, robust-based and mode estimates.

Notes:
\begin{itemize}
\itemsep -0.15cm
	\item This package is based on S3-based self-referencing function objects.\\
		(Which are equivalent to the \{d, p, q, r\} approach used in R's stats package).
	\item IT'S POSSIBLE THESE MAY CHANGE TO S4-BASED FUNCTION OBJECTS, IN THE NEAR FUTURE.\\
		(In principle, you should not access attributes/slots, directly).
	\item Bivariate/trivariate plotting functions use the barsurf package, which uses the base graphics system.
	\item Variable names are taken from column names or list names.
	\item This package gives precedence to vectors and matrices, over lists and data frames.\\
		(One partial exception is categorical data which allow a list of integer/factor/character vectors).
	\item This vignette contains a small amount of non-visible R code, to produce multi-panel plots.
\end{itemize}

Currently, there are some limitations:
\begin{itemize}
\itemsep -0.15cm
	\item Bandwidth selection methods are simplistic.
	\item Parameter estimates and distribution sets, need substantial improvement.
\end{itemize}

These are high priorities for future updates, which may also support categorical smoothing.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Preliminary Code}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
I'm going to load (and attach) the probhat, fclust and scatterplot3d packages:
<<>>=
library (probhat)       #required
library (fclust)        #optional, used in appendix
library (scatterplot3d) #optional
@

Note that the probhat package imports the barsurf and kubik packages.

I will set global options for viewing PDF documents electronically, and default colors to green:

<<>>=
set.ph.options (rendering.style="pdf", theme="green")
@

And I will construct some data objects:

<<>>=
prep.ph.data ()
@

This function emulates a script.\\
The script and the resulting datasets are given in the last two appendices.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Discrete Kernel Smoothing}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We can construct a \pmfuvdks\ object, using the \tmu{pmfuv.dks} constructor.\\
Likewise, we can construct \cdfuvdks\ and \qfuvdks\ objects, using the \tmu{cdfuv.dks} and \tmu{qfuv.dks} constructors.

Here are examples, using traffic accident data, derived from the ``Traffic'' dataset in the MASS package:

<<>>=
dfh <- pmfuv.dks (traffic.bins, traffic.freq)
dFh <- cdfuv.dks (traffic.bins, traffic.freq)
dFht <- qfuv.dks (traffic.bins, traffic.freq)
@

\begin {center}
<<fig=TRUE, width=4.75, height=3>>=
plot (dfh, TRUE, main="Probability Mass Function")
@
<<fig=TRUE, width=4.75, height=3>>=
plot (dfh, TRUE, freq=TRUE, main="same as above\nbut with frequencies")
@
<<fig=TRUE, width=4.75, height=3>>=
plot (dFh, TRUE, main="Cumulative Distribution Function")
@
<<fig=TRUE, width=4.75, height=3>>=
plot (dFht, main="Quantile Function")
@
\end {center}

Here, traffic.bins is a one-column matrix of integer-indexed bins, and traffic.freq is a vector of counts.

The resulting objects are function objects.\\
By default, the PMF maps an integer vector (of quantiles) to a numeric vector (of probabilities), the CDF maps quantiles to cumulative probabilities, and the QF maps probabilities to quantiles:

<<>>=
dfh (10)
dfh (10, freq=TRUE)
dFht (c (0.25, 0.5, 0.75) )
@

Note that PMFs can be constructed, plotted and evaluated with freq=TRUE, which results in frequencies.\\
Likewise, if constructed with freq=TRUE, they can be plotted and evaluated with freq=FALSE.

Also note that this dataset is not an ideal example.\\
(Essentially, it gives frequencies of frequencies).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Continuous Kernel Smoothing:\\Univariate Probability Distributions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We can construct a \pdfuvcks\ object, using the \tmu{pdfuv.cks} constructor.\\
Likewise, we can construct \cdfuvcks\ and \qfuvcks\ objects, using the \tmu{cdfuv.cks} and \tmu{qfuv.cks} constructors.

Here are examples, using height data, derived from the ``trees'' data in the datasets package:\\
(This in metric).

<<>>=
cfh <- pdfuv.cks (height)
cFh <- cdfuv.cks (height)
cFht <- qfuv.cks (height)
@

\begin {center}
<<fig=TRUE, width=4.75, height=3>>=
plot (cfh, TRUE, main="Probability Density Function")
@
<<fig=TRUE, width=4.75, height=3>>=
plot (cFh, TRUE, main="Cumulative Distribution Function")
@
<<fig=TRUE, width=4.75, height=3>>=
plot (cFht, main="Quantile Function")
@
\end {center}

The resulting objects are function objects.\\
The PDF maps a numeric vector (of quantiles) to a numeric vector (of probabilities), the CDF maps quantiles to cumulative probabilities, and the QF maps probabilities to quantiles.

<<>>=
cfh (22)
cFht (c (0.25, 0.5, 0.75) )
@


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Continuous Kernel Smoothing:\\Multivariate Probability Distributions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We can construct  \pdfmvcks\ and \cdfmvcks\ objects, using the \tmu{pdfmv.cks} and \tmu{cdfuv.cks} constructors:

Here are examples, using earthquake data, derived from the ``quakes'' data, in the datasets package:

<<>>=
#note, depth is third variable
#(will fit a semi-bounded model)
head (quakes2, 2)
@

<<>>=
cfh2 <- pdfmv.cks (quakes2 [,1:2], smoothness = c (0.35, 1) )
cfh3 <- pdfmv.cks (quakes2 [,1:3], smoothness = c (0.35, 1, 1),
    a = c (-Inf, -Inf, 0) )
@

\begin {center}
<<fig=TRUE, width=4.75, height=3>>=
plot (cfh2,, TRUE,
    main="Bivariate PDF, 2D")
@
<<fig=TRUE, width=4.75, height=3>>=
plot (cfh2, TRUE, main="Bivariate PDF, 3D")
@
<<fig=TRUE, width=4.75, height=4>>=
plot (cfh3, main="Trivariate PDF",
    nslides=8, zlim = c (680, 40) )
@
\end {center}

The resulting objects are function objects.\\
The PDF maps a numeric vector or matrix (of quantiles) to a numeric vector (of probabilities) and the CDF maps quantiles to cumulative probabilities.\\
In PDFs and CDFs, a standard vector is equivalent to a single-row matrix.

<<>>=
cfh2 (c (180, -20) )
cfh3 (c (180, -20, 300) )
@

Note that an example of a multivariate CDF is given in the section on multivariate probabilities, later.

Also note that longitude maps to the ``x'' variable and latitude maps to the ``y'' variable.\\
(And longitude is the first variable in the derived dataset, but is the second variable in the original dataset).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Continuous Kernel Smoothing:\\Conditional Probability Distributions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We can construct a \pdfccks\ object, using the \tmu{pdfc.cks} constructor.\\
Likewise, we can construct \cdfccks\ and \qfccks\ objects, using the \tmu{cdfc.cks} and \tmu{qfc.cks} constructors.

Here are examples, using the quakes data, from the previous section:

<<>>=
conditions <- c (long=180, lat=-20)
@

<<>>=
depth.fhc <- pdfc.cks (quakes2 [,-4], smoothness = c (0.35, 1, 1),
    conditions=conditions, preserve.range=TRUE,
    a = c (-Inf, -Inf, 0) )
mag.fhc <- pdfc.cks (quakes2 [,-3], smoothness = c (0.35, 1, 1),
    conditions=conditions, preserve.range=TRUE)
@

\begin {center}
<<fig=TRUE, width=4.75, height=3>>=
plot (depth.fhc, main="conditional distribution of depth\n(long=180, lat=-20)")
@
<<fig=TRUE, width=4.75, height=3>>=
plot (mag.fhc, main="conditional distribution of mag\n(long=180, lat=-20)")
@
\end {center}

The resulting objects are function objects, similar to univariate models, given earlier.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Continuous Kernel Smoothing:\\Multivariate-Conditional Distributions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We can construct \pdfmvccks\ and \cdfmvccks\ objects, using the \tmu{pdfmvc.cks} and \tmu{cdfmvc.cks} constructors.

Here are examples, using the quakes data, from the previous sections:

<<>>=
depth.mag.fhc <- pdfmvc.cks (quakes2, smoothness = c (0.35, 1, 1, 1),
    conditions = c (long=180, lat=-20), preserve.range=TRUE,
    a = c (-Inf, -Inf, 0, -Inf) )
lat.long.fhc <- pdfmvc.cks (quakes2 [,-4], smoothness = c (0.35, 1, 1),
    conditions = c (depth=168), preserve.range=TRUE,
    a = c (-Inf, -Inf, 0) )
@

\begin {center}
<<fig=TRUE, width=4.75, height=3>>=
plot (depth.mag.fhc,
    main="conditional distribution of depth and mag\n(long=180, lat=-20)")
@
<<fig=TRUE, width=4.75, height=3>>=
plot (lat.long.fhc,
    main="conditional distribution of lat and long\n(depth=168)")
@
\end {center}

The resulting objects are function objects, similar to multivariate models, given earlier.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Categorical Distributions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We can construct a \pmfuvcat\ object, using the \tmu{pmfuv.cat} constructor.\\
Likewise, we can construct \cdfuvcat\ and \qfuvcat\ objects, using the \tmu{cdfuv.cat} and \tmu{qfuv.cat} constructors. 

Here are examples, using the number arrests per crime type, derived from the state.x77 and USArrests datasets in the datasets package: 

<<>>=
gfh <- pmfuv.cat (crime.type, n.arrests)
@

\begin {center}
<<fig=TRUE, width=4.75, height=3>>=
plot (gfh, main="Probability Mass Function")
@
<<fig=TRUE, width=4.75, height=3>>=
plot (gfh, freq=TRUE, main="same as above\nbut with frequencies")
@
\end {center}

The resulting objects are function objects, similar to discrete kernel smoothing.\\
Categorical PMFs and CDFs can be evaluated using integers, factors or characters.

<<>>=
levels (crime.type$crime.type)
@

<<>>=
gfh (1)
gfh ("Assault")
@

<<>>=
gfh ("Assault", freq=TRUE)
@

Note that the integer above (1) represents the index of the (first) category, and not the corresponding category name (Assault).\\
This is obvious here, but caution is required if a categorical distribution is constructed from integer-valued categories.

Also, it's possible to construct conditional categorical distributions from categorical data, however, I'm bypassing the examples.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Empirical-Like Distributions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We can construct a \cdfuvel\ object, using the \tmu{cdf.el} constructor.\\
Likewise, we can construct a \qfuvel\ object, using the \tmu{qf.el} constructor.

Here's an example, using the same height data, as earlier:

<<>>=
eFh <- cdf.el (height)
eFht <- qf.el (height)
@

\begin {center}
<<fig=TRUE, width=4.75, height=3>>=
plot (eFh, main="Cumulative Distribution Function")
@
<<fig=TRUE, width=4.75, height=3>>=
plot (eFht, main="Quantile Function")
@
\end {center}

These models compute a set of points, representing cumulative probabilities, and interpolate the points with a cubic Hermite spline.

The resulting functions are smooth (the property), but don't necessarily appear smooth.

Unlike continuous kernel smoothing, empirical-like models don't smooth (the method) the model.

Empirical-like models require unique x values, and a small amount of random variation is automatically added, if they're not unique.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Conditional Distributions with Mixed Input Types\\(And Statistical Classification)}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In addition to the conditional probability distributions listed so far, it's also possible to construct categorical and continuous distributions, conditional on both categorical and continuous variables, or conditional on the opposite type.

Currently, I refer to these as conditional distributions with mixed input types, and give them the extensions \phgmix\ and \phxmix.

A \phgmix\ model, fits a categorical distribution (with a categorical conditional variable) and at least one continuous conditioning variable.\\
A \phxmix\ model, fits a continuous distribution (with a continuous conditional variable) and at least one categorical conditioning variable.

A \phgmix\ model is computed by fitting one model for each category, along with two non-conditional models, and then uses Bayes' Theorem, to invert the models.

Currently, \phxmix\ models (and \phgmix\ models, if there are both categorical and continuous conditions) use a subset of data, where the categorical conditions match the categorical variables, and then fit a model to that subset, however, this may change in the future.

\begin {center}
<<fig=TRUE, width=4.75, height=3>>=
fh1a.gmix <- pmfc.gmix (species, sepal.length,
    conditions = c (sepal.length=5.5) )
plot (fh1a.gmix,
    main="Conditional Distribution of Iris Species\n(sepal.length=5.5)")
@
<<fig=TRUE, width=4.75, height=3>>=
fh1b.gmix <- pmfc.gmix (species, sepal.length,
    conditions = c (sepal.length=6.5) )
plot (fh1b.gmix,
    main="Conditional Distribution of Iris Species\n(sepal.length=6.5)")
@
\end {center}

We can use multiple categorical or continuous variables:\\
(This example uses one conditional categorical variable and two conditioning continuous variables).

\begin {center}
<<fig=TRUE, width=4.75, height=3>>=
fh2.gmix <- pmfc.gmix (species, cbind (sepal.length, sepal.width),
    conditions = c (sepal.length=6, sepal.width=3) )
plot (fh2.gmix,
    main = paste (
        "Conditional Distribution of Iris Species",
        "(sepal.length=6, sepal.width=3)",
        sep="\n")
    )
@
\end {center}

Note that the next section, on distribution sets, contains a categorical set, which is equivalent to three \phxmix\ models.

The models above can be used for statistical classification purposes:\\
(Using the last model):

<<>>=
ph.mode (fh2.gmix)
ph.mode (fh2.gmix, level.names=TRUE)
@

The distribution set objects, should be wrapped inside a call to \tmu{as.list}, with the exception of using functions specifically designed for them.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Distribution Sets\\(And Distance Matrices)}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Here, a distribution set is a set of one or more probability distributions.

Currently, there are two types:
\begin{itemize}
	\item \textbf {Categorical Set}\\One univariate probability distribution for each (categorical) level, out of many (categorical) levels.
	\item \textbf {Marginal Set}\\One univariate probability distribution for each variable, out of many variables.
\end{itemize}

Lets construct \pdfuvcks\ models of sepal length, grouped by species:

<<>>=
fh.gset <- pdfuv.gset.cks (species, sepal.length)
@

\begin {center}
<<fig=TRUE, width=4.75, height=3>>=
plot (fh.gset, main="Density Estimates of Sepal Length\n(grouped by species)")
@
\end {center}

Lets construct marginal \qfuvel\ objects:

<<>>=
Fht.mset = qfuv.mset.el (trees2)
@

\begin {center}
<<fig=TRUE, width=4.75, height=4.75>>=
plot (Fht.mset, nr=2, nc=2)
@
\end {center}

Also, we can compute a distance (or dissimilarity) matrix based on the gset object:
<<>>=
pdist (fh.gset)
@

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Multivariate Probabilities}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Here, multivariate probability refers to the probability of observing multiple random variables between pairs of lower and upper limits.

Currently, there are two sets of functions for this purpose.

The \tmu{probmv} function can be applied to almost any continuous CDF, but it's deprecated.\\ 
The \tmu{pwith} functions use a modified kernel smoothing algorithm, which is more efficient.

Using the trees data, we can compute the probability that height, girth and volume are all between certains pairs of limits.

We can use the \tmu{probmv} function, which has three arguments, the multivariate CDF, a vector of lower limits and a vector of upper limits:
<<>>=
#multivariate cdf
#note pwith function can use pdf or cdf, but only nonconditional
cFh3 <- cdfmv.cks (trees2)
@

<<>>=
xlim <- matrix (c (
    22, 24,    #height in 22 to 24
    28, 38,    #girth  in 28 to 38
    0.55, 1.05 #volume in 0.55 to 1.05
    ),, 2, byrow=TRUE, dimnames = list (colnames (trees2), c ("a", "b") ) )
xlim
@

<<>>=
#multivariate probability
probmv (cFh3, xlim [,1], xlim [,2])
pwith (cFh3, xlim=xlim)
@

Note that it's possible to compute multiple regions at once by making a and b matrices with each row representing one region. Also note that currently, variables names are ignored, so they must be in the same order as the variables used to construct the CDF.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Chained Quantile Functions\\(And Multivariate Random Number Generation)}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Standard quantile functions can be used to compute univariate random numbers via standard inversion sampling

I've created novel chained quantile functions, to compute multivariate random numbers.

It works by fitting a quantile function to the first variable's observations, and then evaluating that quantile function at the first variable's evaluation points.

Then, assuming that there are two of more variables, a sequence of conditional quantile functions are fitted to incrementing sets of variables, conditional on the previous variables' evaluations, evaluating one new variable, each time.\\
This is done for each evaluation point.

The convenience function, \tmu{rng}, takes two arguments, the univariate or chained quantile function, and the number of random numbers to generate, then evaluates the quantile function, using a vector or matrix of uniform random numbers.

<<>>=
chFht <- chqf.cks (trees2)
synthetic.data <- rng (chFht, 31)
@

<<>>=
#convenience function
plot.trees.data <- function (x, main)
{   height <- x [,"height"]
    girth <- x [,"girth"]
    volume <- x [,"volume"]
    scatterplot3d (height, girth, volume,
        main=main, type="h", angle=112.5, pch=16)
}
@

\begin {center}
<<fig=TRUE, width=4.75, height=3>>=
#original data
plot.trees.data (trees2, "original data")
@

<<fig=TRUE, width=4.75, height=3>>=
#synthetic data
plot.trees.data (synthetic.data, "synthetic data")
@
\end {center}

Note that this is computationally expensive.

For 3 variables and 31 evaluation points, the algorithm needs to fit:\\
$1 + 31 * (3 - 1) = 63$ distributions.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Parameter Estimates}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection* {Overview}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Univariate probability distributions (including univariate-conditional probability distributions) can be used to compute probabilities and parameter-like estimates, including the mean, standard deviation, variance, skewness and kurtosis, (arbitrary) moments, median, (arbitrary) quantiles and the mode.

Probabilities can be computed from the CDF.\\
(i.e. In the univariate continuous case, $\hat F (b) - \hat F (a)$, no example is given).

The mean, standard deviation, variance and higher moments, require the PMF or spline-based CDF.

The median and quantiles, require the quantile function.

The mode, requires the PMF or spline-based PDF.

In the future, I may allow automatic conversion between the PMF/PDF, CDF and QF.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection* {Moment-Based Statistics}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We can compute moment-based estimates using the high-level functions the \tmu{ph.mean}, \tmu{ph.sd}, \tmu{ph.var}, \tmu{ph.skewness} and \tmu{ph.kurtosis}.

They require a PMF or spline-based CDF.

<<>>=
ph.mean (cFh)
ph.var (cFh)
ph.skewness (cFh)
ph.kurtosis (cFh)
@

Note that currently, standard deviation, variance and higher moments, should not be regarded as accurate because the smoothing algorithm tends to inflate their values, however, they can still be used as an exploratory tool, especially for the purpose, of comparing different conditional probability distributions.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection* {Order-Based Summary Statistics}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We can compute order-based summary statistics, using the \tmu{quartiles}, \tmu{deciles} and \tmu{ntiles} functions.

All of which require a numeric vector, quantile function, or an object than can be coerced to a numeric vector.

<<>>=
quartiles (cFht)
deciles (cFht)
@

<<>>=
ntiles (8, cFht)
ntiles (8, cFht, prob=TRUE)
@

An \qfuvel\ object is automatically created, if a numeric vector is used, hence the last two results should be identical (for unique input values) or almost identical (for non-unique input values).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection* {Robust-Based Statistics}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We can compute robust-based summary statistics, using the \tmu{ph.median}, \tmu{ph.quantile} and \tmu{iqr} functions.

All of which require a numeric vector, quantile function, or an object than can be coerced to a numeric vector.

<<>>=
ph.median (cFht)
ph.quantile (cFht, c (0.25, 0.5, 0.75) )
@

<<>>=
#inter-quartile range
iqr (cFht)
@

<<>>=
#inter-quantile ranges
iqr (cFht, 2/3)
@

These function work in similar way to the order-based functions in the previous section, which the exception that they're less summary focused.

Note that calling these functions with a quantile function, is equivalent to evaluating the quantile function, directly.\\
So, the only gain is possible readability.

<<>>=
cFht (0.5)
ph.median (cFht)
@

Like the order-based functions, they can also be used on a numeric vector.
<<>>=
ph.median (height)
@

In theory, this should be more accurate than the median and quantile functions from the stats package, because these use smooth interpolation, rather than linear interpolation.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection* {Mode Estimates}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We can compute mode estimates using the \tmu{ph.mode}  and \tmu{ph.modes} functions.

The first requires a PMF or spline-based PDF, and the second which computes multiple modes, requires a spline-based PDF only.

<<>>=
ph.mode(cfh)
ph.mode(cfh, TRUE)
@

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection* {Putting it All Together}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

<<>>=
height.summary <- c (
    mean = ph.mean (cFh),         #CDF
    sd = ph.sd (cFh),             #CDF
    variance = ph.var (cFh),      #CDF
    skewness = ph.skewness (cFh), #CDF
    kurtosis = ph.kurtosis (cFh), #CDF
                                  #
    median = ph.median (cFht),    #QF
    mode = ph.mode (cfh) )        #PDF
@

<<>>=
strs <- c (c ("mean", "median", "mode") )
x <- height.summary [strs]
y <- c (0.06, 0.1, 0.14)
colors <- c ("black", "blue", "darkgreen")
adjv <- c (1.25, 0.5, -0.25)
@

\begin {center}
<<fig=TRUE, width=4.75, height=3>>=
plot (cfh)
abline (v=x, col=colors)
for (i in 1:3)
    text (x [i], y [i], strs [i], adj = adjv [i], col = colors [i])
@
\end {center}

<<>>=
height.summary
@

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {References}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
My main sources are:

\ind \tmu {R's stats Package}

\ind \tmu {Wikipedia}

Also, this package uses the following R packages, or has been influenced by them:

\ind \tmu {barsurf: Contour Plots, 3D Plots, Vector Fields and Heatmaps}

\indf Spurdle, A.

\ind \tmu {bivariate: Bivariate Probability Distributions}

\indf Spurdle, A.

\ind \tmu {mvtnorm: Multivariate Normal and t Distributions}

\indf Genz, A., Bretz, F., Miwa, T., Mi, X. \& Hothorn, T.

\ind \tmu {kubik: Cubic Hermite Splines and Related Optimization Methods}

\indf Spurdle, A.

\ind \tmu {KernSmooth: Functions for Kernel Smoothing Supporting Wand \& Jones (1995)}

\indf Wand, M. \& Ripley, B.

\ind \tmu {mgcv: Mixed GAM Computation Vehicle with Automatic Smoothness Estimation}

\indf Wood, S.

\ind \tmu {fclust: Fuzzy Clustering}

\indf Giordani, P., Ferraro, M.B. \& Serafini, A.

\ind \tmu {scatterplot3d: 3D Scatter Plot}

\indf Ligges, U., Maechler, M. \& Schnackenberg, S.

\ind \tmu {MASS: Support Functions and Datasets for Venables and Ripley's MASS}

\indf Ripley, B.

Further references:

\ind Wand, M.P. \& Jones, M.C. (1995). Kernel Smoothing.

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Appendix A:\\List of Probability Distributions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Discrete kernel smoothing models (\dks):
\begin{itemize}
\itemsep -0.15cm
	\item [\pnt] Univariate probability mass function (\pmfuvdks).
	\item [\pnt] Univariate cumulative distribution function (\cdfuvdks).
	\item [\pnt] Univariate quantile function (\qfuvdks).
\end{itemize}

Continuous kernel smoothing models (\cks):
\begin{itemize}
\itemsep -0.15cm
	\item [\pnt] Univariate probability density function (\pdfuvcks).
	\item [\pnt] Univariate cumulative distribution function (\cdfuvcks).
	\item [\pnt] Univariate quantile function (\qfuvcks).\\

	\item [\pnt] Multivariate probability density function (\pdfmvcks).
	\item [\pnt] Multivariate cumulative distribution function (\cdfmvcks).\\

	\item [\pnt] Conditional probability density function (\pdfccks).
	\item [\pnt] Conditional cumulative distribution function (\cdfccks).
	\item [\pnt] Conditional quantile function (\qfccks).\\

	\item [\pnt] Multivariate-conditional probability density function (\pdfmvccks).
	\item [\pnt] Multivariate-conditional cumulative distribution function (\cdfmvccks).\\

	\item [\pnt] Chained quantile function (\chqfcks).
\end{itemize}

Categorical models (\cat):
\begin{itemize}
\itemsep -0.15cm
	\item [\pnt] Univariate probability mass function (\pmfuvcat).
	\item [\pnt] Univariate cumulative distribution function (\cdfuvcat).
	\item [\pnt] Univariate quantile function (\qfuvcat).\\

	\item [\pnt] Conditional probability mass function (\pmfccat).
	\item [\pnt] Conditional cumulative distribution function (\cdfccat).
	\item [\pnt] Conditional quantile function (\qfccat).
\end{itemize}

Empirical-like models (\el):
\begin{itemize}
\itemsep -0.15cm
	\item [\pnt] Univariate cumulative distribution function (\cdfuvel).
	\item [\pnt] Univariate quantile function (\qfuvel).
\end{itemize}

Conditional categorical distributions with mixed input types (\phgmix):
\begin{itemize}
\itemsep -0.15cm
	\item [\pnt] Conditional probability mass function (\pmfcgmix).
	\item [\pnt] Conditional cumulative distribution function (\cdfcgmix).
	\item [\pnt] Conditional quantile function (\qfcgmix).
\end{itemize}

Conditional continuous distributions with mixed input types (\phxmix):
\begin{itemize}
\itemsep -0.15cm
	\item [\pnt] Conditional probability density function (\pdfcxmix).
	\item [\pnt] Conditional cumulative distribution function (\cdfcxmix).
	\item [\pnt] Conditional quantile function (\qfcxmix).
\end{itemize}

Distribution sets:
\begin{itemize}
\itemsep -0.15cm
	\item [\pnt] Marginal sets.
	\item [\pnt] Categorical sets.
\end{itemize}

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Appendix B:\\Notation, Terminology and Related Notes}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

When used as a prefix, the letters ``d'' and ``c'' mean discrete and continuous, respectively.\\
Likewise, DPD and CPD mean discrete probability distribution and continuous probability distribution.

PMF, PDF, CDF and QF refer to probability mass function (or just mass function), probability density function (or just density function), cumulative distribution function (or just distribution function) and quantile function, respectively.

Note that I sometimes use the notation, Fht or Fh.inv, to describe the quantile function.\\
However, quantile functions are not necessarily the exact inverse of CDFs.\\
Firstly, CDFs may have level (non-increasing) sections which are non-invertible, and secondly, the algorithm for constructing quantile splines, only transposes the control points, it does not compute an exact inverse.

When used as a suffix, the letters ``uv'', ``mv'', ``c'' and ``mvc'', mean univariate, multivariate, conditional and multivariate-conditional.

In publicly visible constructors (e.g. \tmu{pmfuv.dks}):\\
Univariate and multivariate distributions, refer to non-conditional univariate and multivariate distributions only, and conditional distributions refer to univariate conditional distributions only.\\
This also applies to the corresponding class names, and to most of the section headings in this vignette.

In other contexts, univariate and multivariate distributions, can be conditional or non-conditional, and conditional distributions can be univariate or multivariate, unless stated otherwise.

Currently, conditional distributions need at least one condition.\\
(i.e. A model with no conditions is not regarded as conditional).

The term "range", refers to the range of a random variable, unless stated otherwise.

In this package, there are three subtypes of such ranges:

\begin{itemize}
\itemsep -0.15cm
	\item The model range.
	\item The mechanistic or theoretic limits, labelled Xlim.
	\item The observed range.
\end{itemize}

Both the model and observed ranges need to be inside the theoretic limits.
The same applies to conditions, eval bins/points and the main input data.

Bounded and unbounded random variables, have finite and infinite theoretic limits, respectively.
Noting the option of being semi-bounded, which is the default for DKS models.

In the unbounded discrete case, the the model range is observed range +/- half the decremented bandwidth (which needs to be odd), at each end.
In the unbounded continuous case, the the model range is observed range +/- half the bandwidth, at each end.

In the bounded case, the model range is the same except that it is truncated, if it would fall outside the theoretic limits, which also means that a truncated smoothing algorithm is used.

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Appendix C:\\Discrete Kernel Smoothing Formulae}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Discrete kernels (without standardized intervals), take the form:
\begin {align*}
k (x; \text {bw} ) &= ...\\
K (x; \text {bw} ) &= ...\\ \\
\text {hbw} &= \frac {\text {bw} - 1}{2}
\end {align*}

Where k and K are the kernel's PMF and CDF, respectively.\\
And where bw is the (odd positive) bandwidth parameter.

Unstandardized discrete kernels have zero mass outside the interval [-hbw, +hbw].

We can define additive component-distributions as:
\begin {align*}
k^* (x; k, x_i^*, \text {bw}) &= k (x - x_i^*; \text {bw})\\
K^* (x; K, x_i^*, \text {bw}) &= K (x - x_i^*; \text {bw})
\end {align*}

Where $x_i^*$ is the (integer-valued) center of the of each additive component-distribution, and $x$ is the (integer-valued) point on the x-axis, where the function is evaluated.

Unbounded PMFs and CDFs can be computed, as follows:
\begin {align*}
\hat{f}_X (x; k, \text {bw}, n, \mathbf {x}^*, \mathbf {w}) &= \sum_i w_i k^* (x; k, x_i^*, \text {bw})\\
\hat{F}_X (x; K, \text {bw}, n, \mathbf {x}^*, \mathbf {w}) &= \sum_i w_i K^* (x; K, x_i^*, \text {bw})
\end {align*}

Where:
\begin {equation*}
w_i = \frac {h_i}{\sum_i h_i}
\end {equation*}

And where $\mathbf {x}^*$ is a vector of (integer-valued) bins and $\mathbf {h}$ is a vector of frequencies, both of which, are of length $n$, and $i \in [1, n]$.

If truncated discrete smoothing is required, then (a subset of) the data is reflected about the truncation bins.

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Appendix D:\\Continuous Kernel Smoothing Formulae}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Continuous kernels (with standardized intervals), take the form:
\begin {align*}
k (x) &= ...\\
K (x) &= ...
\end {align*}

Where k and K are the kernel's PDF and CDF, respectively.

Standardized continuous kernels have zero density outside the interval [-1, 1].

We can define additive component-distributions using:
\begin {align*}
k^* (x; k, x_i^*, \text{bw}) &= \frac {2}{\text {bw} } k (\frac {2}{\text {bw} }(x - x_i^*) )\\
K^* (x; K, x_i^*, \text{bw}) &= K (\frac {2}{\text {bw} }(x - x_i^*) )
\end {align*}

Where bw is the bandwidth, $x_i^*$ is the center of each additive component-distribution, and $x$ is a point on the x-axis, where the function is evaluated.

Univariate PDFs and CDFs can be computed, as follows:
\begin {align*}
\hat {f}_X (x; k, \text{bw}, n, \mathbf{x}^*) &= \frac{\sum_i k^* (x; k, \text{bw}, x_i^*)}{n}\\
\hat {F}_X (x; K, \text{bw}, n, \mathbf{x}^*) &= \frac{\sum_i K^* (x; K, \text{bw}, x_i^*)}{n}
\end {align*}

Where $\mathbf{x}^*$ is a vector of length $n$, and $i \in [1, n]$.

Multivariate PDFs and CDFs can be computed, as follows:
\begin {align*}
\hat {f}_\mathbf{X} (\mathbf{x}; k, \mathbf{bw}, n, m, \mathbf{x}^*)
	&= \frac{\sum_i \Big(\$f_1 \times \$f_2 \times ... \times \$f_m \Big)}{n}\\
	&= \frac{\sum_i \Big( k^* (x_1; k, \text{bw}_1, x_{[i,1]}^*) \times k^* (x_2; k, \text{bw}_2, x_{[i,2]}^*) \times ... \times k^* (x_m; k, \text{bw}_m, x_{[i, m]}^*)\Big)}{n}\\ \\
\hat {F}_\mathbf{X} (\mathbf{x}; K, \mathbf{bw}, n, m, \mathbf{x}^*)
	&= \frac{\sum_i \Big(\$F_1 \times \$F_2 \times ... \times \$F_m \Big)}{n}\\
	&= \frac{\sum_i \Big( K^* (x_1; K, \text{bw}_1, x_{[i,1]}^*) \times K^* (x_2; K, \text{bw}_2, x_{[i,2]}^*) \times ... \times K^* (x_m; K, \text{bw}_m, x_{[i,m]^*})\Big)}{n}
\end {align*}

Where $\mathbf{bw}$ is a bandwidth vector, $\mathbf{x}^*$ is a matrix with $n$ rows (observations) and $m$ columns (variables), and $\mathbf{x}$ is a vector of points on the x-plane, where the function is evaluated.

Continuous PDFs use the ratio of two multivariate expressions, with the n term cancelling out.\\
Continuous CDFs are similiar, but rather than numerically integrate the ratio of two multivariate expressions, the top expression is modified to produce a hybrid algorithm between the PDF and CDF algorithms.

If truncated continuous smoothing is required, then (a subset of) the data is reflected about the truncation points.\\
This is done is such a way, that there's a maximum of three times n, regardless of the number of variables.

Weighted versions of these formulae are created by substituting:
\begin {equation*}
\frac {\sum_i (\$ \text {SUB-EXPRESSION}) }{n}
\end {equation*}

With:
\begin {equation*}
\sum_i w_i (\$ \text {SUB-EXPRESSION})
\end {equation*}

Subject to:
\begin {equation*}
\sum_i w_i = 1
\end {equation*}


\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Appendix E:\\Empirical-Like Formulae}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
An empirical cumulative distribution function, which is a step function, can be computed by:
\begin {equation*}
\mathbb {P} (X \leq x)  = \hat {F}_X (x; n, \mathbf {x}^*) = \frac {\sum_i I (x_i^* \leq x)}{n}
\end {equation*}

Where $I$ is an indicator function, which equals 1, if the enclosed logical expression is true, and equals 0, if false.

A proto-empirical-like distribution, which is also a step function, can be computed by modifying the formula above, to give:
\begin {equation*}
\mathbb {P} (X \leq x)  = \hat {G}_X (x; n, \mathbf {x}^*) = \frac {\big( \sum_i I (x_i^* \leq x) \big) - 1}{n - 1}
\end {equation*}

This function can be used to generate a sequence of points:
\begin {equation*}
\{ \big (x_1^*, \hat {G}_X (x_1^*; n, \mathbf {x}^*) \big), \big (x_2^*, \hat {G}_X (x_2^*; n, \mathbf {x}^*) \big), ..., \big (x_n^*, \hat {G}_X (x_n^*; n, \mathbf {x}^*) \big)\}
\end {equation*}

An empirical-like distribution, which is a continuous function, can be computed by using a cubic Hermite spline to interpolate this sequence. 



\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Appendix F:\\Fuzzy Clustering\\(And Weighted Multivariate Kernel Smoothing)}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Fuzzy clustering computes a membership matrix, from some data.

The values in the membership matrix represent the membership of each data point in each cluster, with each row representing one data point and each column representing one cluster.\\
(Note that rows, not columns, sum to one).

In some situations, is may be of interest to identify the clusters, only. In other situations, it may be of interest to identify the clusters, and model the properties of one or more of those clusters.

It's possible to model each cluster using weighted kernel smoothing.

The following computes the membership matrix for three clusters:
<<>>=
membership <- FKM.gk (unemployment, k=3, seed=2)$U
@

I'm going to extract the weights of the first cluster, and transform them, so that they sum to one:

<<>>=
w <- membership [,1]
w <- w / sum (w)
@

And a weighted model:

<<>>=
wfh.1 <- pdfmv.cks (unemployment, w=w)
@

\begin {center}
<<fig=TRUE, width=4.75, height=3>>=
plot (wfh.1,, TRUE)
@

<<fig=TRUE, width=4.75, height=3>>=
k = 1 - w / max (w)
plot (unemployment, pch=16, col=rgb (k, k, k) )
@
\end {center}

And for the other two clusters:
<<>>=
w <- membership [,2]
wfh.2 = pdfmv.cks (unemployment, w = w / sum (w) )
w <- membership [,3]
wfh.3 = pdfmv.cks (unemployment, w = w / sum (w) )
@

All three:
<<eval=FALSE>>=
plot (wfh.1, main="cluster 1")
plot (wfh.2, main="cluster 2")
plot (wfh.3, main="cluster 3")
@

\begin {center}
<<echo=FALSE, fig=TRUE, width=4.75, height=4.75>>=
p0 <- par (mfrow = c (2, 2) )
plot (wfh.1, main="cluster 1")
plot.new ()
plot (wfh.2, main="cluster 2")
plot (wfh.3, main="cluster 3")
par (p0)
@
\end {center}

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Appendix G:\\Data Preparation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
<<>>=
prep.ph.data (eval=FALSE, echo=TRUE)
@

\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section* {Appendix H:\\Datasets}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

<<>>=
ht <- function (object)
{   print (head (object, 2) )
    print (tail (object, 2) )
}
@

<<>>=
ht (cbind (traffic.bins, freq=traffic.freq) )
@

<<>>=
ht (trees2)
ht (quakes2)
ht (crimes)
ht (data.frame (crime.type=crime.type$crime.type, n.arrests) )
ht (data.frame (species, sepal.length, sepal.width) )
ht (unemployment)
@

\end{document}
