\documentclass[12pt]{article}    

%\VignetteIndexEntry{WiSEBoot Vignette}
%\VignetteEngine{knitr::knitr}

\usepackage{amsmath}    % need for subequations
\usepackage{graphicx}   % need for figures
\usepackage{verbatim}   % useful for program listings
\usepackage{color}      % use if color is used in text
\usepackage{hyperref}   % use for hypertext links, including those to external documents and URLs
\usepackage{amsfonts}
\usepackage{bm}


\setlength{\parskip}{11pt}           % sets spacing between paragraphs
\setlength\parindent{0pt}            % sets leacing space for paragraphs


\title{\texttt{WiSEBoot} Vignette}
\date{\today}
\author{Megan Heyman \url{heyma029@umn.edu}}
\begin{document}


\maketitle
%-------------------------------------------------------------------------------------------------
% Model Selection Description
\section{WiSE Bootstrap Model Selection}
\label{ModSel}
Wild Scale-Enhanced (WiSE) bootstrap is a variant of the wild bootstrap where the model residuals are multiplied by an additional scaling factor.  Theoretical details of the general WiSE bootstrap methodology may be found in (Chatterjee, 2015).  This package is an implementation of the WiSE bootstrap for a specific case of the partial linear model.  Namely, given an equally-spaced time series of length $T=2^J, J\in \mathbb{I}^+$, we assume that a partial linear model adequately describes the time series.
\begin{equation}
\bm{Y(t)} = \gamma_0 \bm{1} + \gamma_1 \bm{t} + \bm{\mu(t)}  + \bm{e(t)}
\label{eqn:init}
\end{equation}
$\bm{Y}\in \mathbb{R}^T$ is the observed data, $\bm{t}$ is a vector time indices, and $\bm{\mu(t)}$ is a general function (with some additional assumptions) in time.  The nonparametric component, $\bm{\mu(t)}$, may contain the interesting cycles or signals within the data, so it is of interest to estimate.  However, current methodology does not provide the ability to fully estimate this function.  Thus, it is proposed that this function be approximated by a fixed basis.  If  $W$ is a $T-$ row matrix of a fixed wavelet basis for the discrete wavelet transform (DWT), and $\gamma$ contains the scaling and filter wavelet coefficients, then our approximate model is   
\begin{equation}
\bm{Y(t)} \approx \gamma_0 \bm{1} + \gamma_1 \bm{t} + W \bm{\gamma}  + \bm{e(t)}
\label{eqn:1}
\end{equation}
Note, in many cases of the DWT, especially those implemented within \texttt{wavethresh}, the scaling coefficient is equivalent to $\gamma_0$.  Thus, only the scaling coefficient or $\gamma_0$ should be estimated in any models where this occurs.

We propose that models of this type are useful descriptors of various time series, including climate model output.  The quantity $W \bm{\gamma}$ is the approximation of $\bm{\mu(t)}$, which hypothetically contains `interesting' signal within the data.  Thus, the estimates of $\bm{\gamma}$ are of relevance to us.  The parametric, linear component ($\gamma_0 \bm{1} + \gamma_1 \bm{t}$), would necessarily change by data application, but we claim it is adequate in the examples presented here.  This parametric component is important to estimate, but may not contain interesting information for our application.  Thus, in this approximation of the partial linear model, the set of population parameters $\gamma_0$, $\gamma_1$ and $\bm{\gamma}$ are estimated. 

Estimating the full vector of wavelet coefficients, $\bm{\gamma}$, is problematic with current methodologies.  However, it is typical that the wavelet filter coefficients are sparse, containing many zero or nearly zero-valued entries.  The proposed methodology takes advantage of the sparsity and chooses a strong threshold criteria for the coefficients.  With data of length $2^J$, filter wavelet coefficients exist for levels 0, 1, ..., $J-1$.  Call $J_0$ the threshold.  Then all coefficients occuring at levels greater than $J_0$ are set to 0.  Essentially, our models assume all fine-level filter coefficients are 0-valued in expectation.  Thresholding $\bm{\gamma}$ in this fashion decreases the number of parameters to estimate which implies consistent estimation is possible.  Of course, this is subject to certain assumptions and conditions on the data and model, which are not discussed here.

Assume fixing the threshold, $J_0$, provides an adequate model.  Then, we may estimate the population parameters $\gamma_0$, $\gamma_1$ and $\bm{\gamma}$ consistently.  Using least squares estimation, $\hat{\gamma}_0$ and $\hat{\gamma}_1$ are obtained and residuals are defined as $\bm{r(t)} = \bm{Y(t)} - \hat{\gamma}_0 \bm{1} - \hat{\gamma}_1 \bm{t}$.  The DWT is performed on $\bm{r(t)}$ to find a length-$T$ vector of wavelet coefficients.  The threshold is then applied and $\bm{\gamma}$ is estimated by $\bm{\hat{\gamma}}_{J_0}$.  Wavelet residauls are defined by $\bm{r_w(t)} = \bm{r(t)} - W \bm{\hat{\gamma}}_{J_0}$.  

The discussion thus far has concerned parameter estimation.  It is usually of interest to also estimate the error associated with the parameters in the model so that inference is possible.  Although all of the assumptions of \ref{eqn:1} have not been stated in this document, it is important to note that the following bootstrap methodology does \emph{not} require that the errors, $\bm{e(t)}$, be homoscedastic or come from a normal distribution.  The WiSE bootstrap is an excellent tool for estimation in situations where the typical, strong assumptions over models do not hold.  Not only do we obtain consistent estimates of the parameters, but also their errors.

To perform the WiSE bootstrap for model \ref{eqn:1} with set threshold $J_0$, create bootstrap series for $b=1, 2, ..., B$
\begin{equation}
\bm{Y_b(t)} = \hat{\gamma}_0 \bm{1} + \hat{\gamma}_1 \bm{t} + W \bm{\hat{\gamma}}_{J_0}  + \tau U_b \bm{r_w(t)}
\label{eqn:2}
\end{equation}
where $\tau$ is a scaling parameter such that $\tau^2/T \to 0$ and $\tau \to \infty$.  Define $U_b=diag(u_1, u_2, ..., u_T)$ for $b=1,...,B$.  The $u_i, i=1,2, ..., T$ are independent, mean 0, variance 1, finite 8$^{th}$ moment random variables, which are also independent across each bootstrap sample.  The $\bm{Y_b(t)}$ are used to find the bootstrap estimates $\hat{\gamma}_{0b}$, $\hat{\gamma}_{1b}$, and $\bm{\hat{\gamma}}_{J_0 b}$.  The set of $B$ parameter estimates allows the user to examine the variability in each parameter.  The WiSE bootstrap also allows for estimation of the variance associated with the errors, but this is not implemented within \texttt{WiSEBoot}.  All details of theoretical conditions and consistency results are contained in (Chatterjee, 2015).


This WiSE bootstrap re-sampling technique is novel, as it allows for a model selection while concurrently providing (asymptotically) consistent estimates of the model parameters (Chatterjee, 2015).  All prior description sets the threshold level, $J_0$, at a fixed level.  The \texttt{WiSEBoot} package provides an automatic process to select the wavelet coefficient threshold level.  The choice of the threshold determines the number of wavelet coefficients to be estimated, and thus, a model selection.  Technically, speaking the models considered for selection follow a sieve selection scheme (i.e. estimate ALL wavelet coefficients at any levels less than or equal to $J_0$).

The model selection process is conducted as follows.  For a data series of length $2^J$, a WiSE bootstrap sample is created for thresholds of $J_0 \in \{0, 1, ..., J-2\}$ as well as setting all filter wavelet coefficients to 0.  The selected model minimizes the mean of the mean squared-error between the bootstrap estimated model and the provided data.  That is, for any $J_0$, we calculate the mean of the mean-squared error
\begin{equation*}
\overline{MSE}_{J_0} =\frac{1}{TB} \sum\limits_{b=1}^B <\bm{Y(t)}-\hat{\gamma}_{0b}\bm{1} -\hat{\gamma}_{1b}\bm{t} -W\bm{\hat{\gamma}}_{J_0 b}, \hspace{1em} \bm{Y(t)}-\hat{\gamma}_{0b}\bm{1} -\hat{\gamma}_{1b}\bm{t} -W\bm{\hat{\gamma}}_{J_0 b}>
\end{equation*}
where $< \bm{x}, \bm{y} >$ is defined as the inner product of vectors $\bm{x}$ and $\bm{y}$.  The selected wavelet coefficient threshold in the model is $J_0^*$ where
\begin{equation*}
\overline{MSE}_{J_0^*} = \min_{j} \overline{MSE}_j
\end{equation*}
%-----------------------------------------------------------------------------------------------














%-----------------------------------------------------------------------------------------------
% Model Selection Simulation
\section{WiSE Bootstrap Model Selection:  Simulation Example}
To demonstrate the wavelet coefficient threshold selection, this example uses some simulated wavelet signals.  What may be considered as a population-level signal is contained in \texttt{SimulatedSmoothSeries}, which is a matrix. 

{\footnotesize
<<sel0, eval=TRUE, echo=FALSE, message=FALSE>>=
library(WiSEBoot)
@

<<sel1, eval=TRUE, echo=TRUE>>=
data(SimulatedSmoothSeries)
dim(SimulatedSmoothSeries)
SimulatedSmoothSeries[1:3, ]
@
}



All signals in this matrix were generated using the \texttt{wavethresh} package with a \texttt{"DaubLeAsymm"} (Daubechies Least Asymmetric) wavelet with \texttt{8} vanishing moments and a \texttt{"periodic"} boundary condition.  The column names of \texttt{SimulatedSmoothSeries} indicate the true wavelet coefficient threshold level.  Let's consider the 4$^{th}$ column.  The wavelet coefficients were thresholded at the $J_0=3$ level.  Thus, all finer-level coefficients (levels 4,5,6,7,8, and 9) were set to 0, to create this smooth signal.


\newpage
Here is a plot of the signal in the 4$^{th}$ column:

{\footnotesize
<<sel2, eval=TRUE, echo=FALSE, out.width='12cm', out.height='8cm'>>=
plot(seq(1, 2^10), SimulatedSmoothSeries[,4], ty="l", 
    main=bquote(paste("Smooth Series with Threshold ", J[0],"=3")), 
    xlab="Time", ylab="Observation")
@
}

\sloppy If data were observed from a population signal such as this, the data would be noisy.  The matrices \texttt{SimulatedSNR5Series}, \texttt{SimulatedSNR9Series}, \texttt{SimulatedSNR15Series}, and \texttt{SimulatedSNR25Series} contain data series with population signals from \texttt{SimulatedSmoothSeries} and varying levels of added noise.  For data of length $2^J$ and a signal, $\bm{\mu(t)}$, with threshold $J_0$, the signal-to-noise ratio (SNR) is defined as 
 \begin{equation}
 SNR = \dfrac{<\bm{\mu(t)}, \bm{\mu(t)}>/(2^{J_{0}+1}-1)}{(2^J)\sigma^2/(2^J-2^{J_{0}+1} + 1)}
 \end{equation}
when $\sigma^2$ is the variance of the noise.  Our SNR is similar to an F-ratio, and in other contexts, the user may wish to take the square-root of the SNR in discussion.  The provided data matrices contain series with SNR=5, 9, 15 and 25.  It seems reasonable to assume that the signal would be more apparent in the SNR=25 series than the SNR=5 series.  

\newpage
Here is a look at the 'noisy data':

<<sel3, eval=TRUE, echo=FALSE>>=
data(SimulatedSNR5Series); data(SimulatedSNR9Series); data(SimulatedSNR15Series); data(SimulatedSNR25Series)
par(mfrow=c(2,2))
plot(seq(1, 2^10), SimulatedSNR5Series[,4], ty="l", 
    main=bquote(paste("SNR=5 Data, Threshold ", J[0],"=3")), 
    xlab="Time", ylab="Observation", col="gray", ylim=c(-.5, .6))
lines(seq(1, 2^10), SimulatedSmoothSeries[,4], col="red", lwd=2)
plot(seq(1, 2^10), SimulatedSNR9Series[,4], ty="l", 
    main=bquote(paste("SNR=9 Data, Threshold ", J[0],"=3")), 
    xlab="Time", ylab="Observation", col="gray", ylim=c(-.5, .6))
lines(seq(1, 2^10), SimulatedSmoothSeries[,4], col="red", lwd=2)
plot(seq(1, 2^10), SimulatedSNR15Series[,4], ty="l", 
    main=bquote(paste("SNR=15 Data, Threshold ", J[0],"=3")), 
    xlab="Time", ylab="Observation", col="gray", ylim=c(-.5, .6))
lines(seq(1, 2^10), SimulatedSmoothSeries[,4], col="red", lwd=2)
plot(seq(1, 2^10), SimulatedSNR25Series[,4], ty="l", 
    main=bquote(paste("SNR=25 Data, Threshold ", J[0],"=3")), 
    xlab="Time", ylab="Observation", col="gray", ylim=c(-.5, .6))
lines(seq(1, 2^10), SimulatedSmoothSeries[,4], col="red", lwd=2)
@

We can see that the true signal is somewhat hidden in each of these data series, but the lower SNR series are noisier.  Typically, the analyst would not know the true signal threshold level.  The \texttt{smoothTimeSeries} function allows for a quick visualization of all possible wavelet coefficient threshold levels.  This is especially useful if the analyst would like to manually choose a wavelet coefficient threshold level.  Using the SNR=15 series, we can demonstrate this capability.  Note, the wavelet settings for these smooths generated below exactly match the true signals from \texttt{SimulatedSmoothSeries}.  The user could change the wavelet family, filter, and boundary.

{\footnotesize
<<sel4, echo=TRUE, eval=TRUE, out.width='1.1\\textwidth', out.height='17cm'>>=
smoothPlot <- smoothTimeSeries(SimulatedSNR15Series[ ,4], plot="all")
@
}

Without knowing the truth, an analyst may guess that the wavelet threshold level could be any of $J_0 \in \{ 3, 4, 5 \}$.  We can use the \texttt{WiSEBoot} function to remove the guess-work and automatically choose a wavelet threshold.  As discussed in Section \ref{ModSel}, the model which achieves the minimum of the mean-squared error is selected.  Let's leave the default wavelet settings again (which exactly match the true signal wavelet here) and look at the MSE criteria for B=\texttt{R=10} bootstrap samples.  Typically, a larger number of bootstrap samples is used, as these may be generated in parallel.  We choose a lower number in the vignette for the sake of time.

{\footnotesize
<<sel5, echo=TRUE, eval=TRUE>>=
set.seed(1414)
SNR15Boot <- WiSEBoot(SimulatedSNR15Series[ ,4], R=10)
SNR15Boot$MSECriteria
@
}

Even in this small bootstrap sample, the correct model was selected.  We can see at $J_0 + 1=4$ the mean of the MSE is minimized with a value of \Sexpr{prettyNum(SNR15Boot$MSECriteria[6,2], digits=3)}.  The reader should keep in mind that this is a simulation example, so a desirable result is not surprising.



Now, we have an automatically selected model with threshold $J_0=3$.  The output \texttt{WiSEBoot} object, called \texttt{SNR15Boot}, contains not only the model selection, but also the estimated parameters from each bootstrap sample in the selected model.  Thus, the distributions of the $\gamma_0$, $\gamma_1$ and $\gamma$ parameters may be examined by visualizing the bootstrap estimates.  

The boxplots below show the empirical distributions of the parametric linear parameter estimates ($\hat{\gamma}_{0b}, \hat{\gamma}_{1b}$).  Because the data was simulated, the population parameters are known to be $\gamma_0=0, \gamma_1=0$.

{\footnotesize
<<sel6, echo=TRUE, eval=TRUE, out.width='13cm', out.height='7cm'>>=
par(mfrow=c(1,2))
boxplot(SNR15Boot$BootIntercept, 
        main=expression(paste("R=10 Bootstrap Estimates of ", gamma[0])),
        ylab=expression(hat(gamma)[0][b]))
abline(h=0, col="red")

boxplot(SNR15Boot$BootSlope, 
        main=expression(paste("R=10 Bootstrap Estimates of ", gamma[1])),
        ylab=expression(hat(gamma)[1][b]))
abline(h=0, col="red")
@
}

As always in any type of bootstrap, the bootstrap distribution will be centered around the original estimate from the data.  We can see that these boxplots aren't exactly centered at 0 because of this centering issue associated with the bootstrap.  (Keep in mind there are only \texttt{R=10} bootstrap samples too.)

Likewise, any individual non-thresholded filter wavelet coefficient may be visualized.  Below are the bootstrap distributions of the level=1 filter coefficients.

{\footnotesize
<<sel7, echo=TRUE, eval=TRUE, out.width='15cm', out.height='7cm'>>=
par(mfrow=c(1,2))
boxplot(SNR15Boot$BootWavelet[,3], 
        main=expression(paste("R=10 Boot. Est. of lvl=1, coef=1, ", gamma)),
        ylab=expression(hat(gamma)[b]))
boxplot(SNR15Boot$BootWavelet[,4], 
        main=expression(paste("R=10 Boot. Est. of lvl=1, coef=2, ", gamma)),
        ylab=expression(hat(gamma)[b]))
@
}
%-----------------------------------------------------------------------------------------------








%-----------------------------------------------------------------------------------------------
% Hypothesis Test Theory
\newpage
\section{WiSE Bootstrap Hypothesis Test}
Next we provide a brief introduction to a hypothesis test of the wavelet coefficients from two data series of equal lengths.  This methodology is discussed in detail in (Braverman, 2015).  Here, the theoretical discussion is meant to give users of the \texttt{WiSEHypothesisTest} and \texttt{WiSEConfidenceRegion} a general idea of what mathematics are running in the background.

Consider two equally-spaced data series of length $T=2^J, J \in \mathbb{I}^+$.  Each data series may be modeled by eqn. \ref{eqn:init} and approximated by eqn. \ref{eqn:1}.  That is,
\[
\bm{Y(t)} = \gamma_{y0}\bm{1} + \gamma_{y1} \bm{t} + W \bm{\gamma}_{y}  + \bm{e(t)}_y
\]
\[
\bm{X(t)} = \gamma_{x0}\bm{1} + \gamma_{x1} \bm{t} + W \bm{\gamma}_{x}  + \bm{e(t)}_x
\]

Notice, the same wavelet basis, $W$, is used in each approximation, but separate parameters are allowed.  Users of the \texttt{WiSEHypothesisTest} function may wish compare the `interesting' signals within these two series. Recalling the discussion from Section \ref{ModSel}, we may decide that the `interesting' signals are $\bm{\mu(t)}_y$ and $\bm{\mu(t)}_x$.  With the wavelet basis approximation, the user tests the linear relationship between the two sets of wavelet coefficients:  $\bm{\gamma}_{y} = \alpha \bm{1} + \beta \bm{\gamma}_{x}$.  The null hypothesis in \texttt{WiSEHypothesisTest} is
\[ H_0:  \alpha = m, \beta= n, \hspace{1em} m,n \in\mathbb{R}\]

In (Braverman, 2015), values of $m=0$ and $n=1$ are tested.  If the null hypothesis in this specific scenario is not rejected, then we may say that there is not sufficient evidence to conclude that the signal within the climate model output does not match the observed climate.
The algorithm to generate the WiSE bootstrap sample changes slightly so that we are sampling under the null hypothesis.  Please see (Braverman, 2015) for the details.
%-------------------------------------------------------------------------------------------------






%------------------------------------------------------------------------------------------------
% Hypothesis Test Analysis
\section{WiSE Bootstrap for Hypothesis Testing: Climate Model Signals}
An example analysis of climate model data is presented here.  The full analysis of the climate models at some specific grid-cells is available in (Braverman, 2015).  Within the \texttt{WiSEBoot} package, two sets of climate model outputs and observed climate are available.  Here, we will look at the data in \texttt{CM20N20S60E}. 

{\footnotesize
<<hyp1, echo=TRUE, eval=TRUE>>=
data(CM20N20S60E)
CM20N20S60E[1:3,]
@
}
This data matrix contains data from AIRS (what we call `observed climate'), 4 runs of the IPSL model, and 6 runs of the MIROC5 model.  Each of the model runs is obtained by choosing different starting parameters.  Specifically, this matrix contains a set of specific humidity observations/outputs between 20N and 20S at 60E and an altitude of 500 hPa.  Observations are daily, from October 1, 2002 to December 29, 2010.

In this analysis, we'll compare the signals in AIRS and MIROC5, run 5.  Here are some plots of the raw data.

{\footnotesize
<<hyp2, echo=TRUE, eval=TRUE, out.width='15cm', out.height='15cm'>>=
par(mfrow=c(2,1))
plot.ts(CM20N20S60E[,1], main="AIRS", ylab="Obs. Climate")
plot.ts(CM20N20S60E[,10], main="MIROC5, run 5", ylab="Model Climate")
@
}

Before it is possible to use the WiSE bootstrap methodology with a wavelet basis approximation, the data must be of length $T=2^J$ for a positive integer, $J$.  The raw data contains 3012 observations.  Thus, we begin by using the \texttt{padMatrix} function to lengthen each series simultaneously.  Conversely, the user could truncate the data to the closest power of 2 in length, which may be more highly recommended when it is OK to exclude some of the data.

{\footnotesize
<<hyp3, echo=TRUE, eval=TRUE>>=
pad60E <- padMatrix(CM20N20S60E)
dim(pad60E$xPad)
@
}

The code above uses the default options within \texttt{padMatrix} of padding at both sides of the data series by reflecting.  The linear trend is not replaced (default) to the padded data matrix because we may easily input the estimated linear (parametric) parameters to the hypothesis testing function.  This data is now of length $2^{12}$.  

The hypothesis testing function requires that the user choose a wavelet coefficient threshold level.  This may be done automatically with the \texttt{WiSEBoot} function first, by inputting both series as a 4096 x 2 matrix.  Here, we choose to just set a threshold of $J_0=5$, as this corresponds to a cycle of 128 days.

Now that the data is of correct length, we may test the hypothesis
\[ H_0: \alpha=0, \beta=1\]
(i.e. the climate model signal matches the observed climate).  Our `X' series is AIRS and the `Y' series is MIROC5, run 5.  For demonstration purposes, we will choose to take \texttt{R=10} bootstrap samples.  A higher number of samples is generally recommended.

{\footnotesize
<<hyp4, ehco=TRUE, eval=TRUE>>=
hypObj <- WiSEHypothesisTest(pad60E$xPad[,1], pad60E$xPad[,10], R=10, J0=5, 
                             XParam=pad60E$linearParam[,1], 
                             YParam=pad60E$linearParam[,10])
hypObj$AsymptoticPValue
hypObj$BootstrapPValue
@
}

The function calculates two p-values.  The asymptotic p-value is based upon the distribution of Hotelling's T$^2$ and the test statistic utilizes the variance-covariance matrix from the bootstrap sample.  The bootstrap p-value computes a Hotelling's T$^2$ test statistic for each bootstrap sample, and the quantile of the data-based Hotelling's T$^2$ is computed from the bootstrap sample.  A plot of the bootstrap sample and observed data parameter estimates is also generated optionally.

We can see that both the asymptotic and bootstrap p-values indicate that the null hypothesis should be rejected.  This is not surprising, especially after examining the generated plot.  The $\hat{\alpha}$ and $\hat{\beta}$ estimated from the data (red point) is clearly outside of the cloud of bootstrap sample points (black).  The gray vertical and horizontal lines represent the parameter values under the null hypothesis.  


\section*{Acknowledgments}
Thank you to all of the collaborators on this project.  Specifically, Amy Braverman (NASA-JPL) and Matthew Gunson (NASA-JPL) provided the re-gridded climate data within \texttt{CM20N20S60E} and \texttt{CM20N20S150W}.  My thesis advisor, Snigdhansu Chatterjee (University of Minnesota), not only helped test this package's code, but also gave me much feedback and advice.  Noel Cressie (Univeristy of Wollongong) and Stephen LeRoy (Harvard) also provided useful feedback and innovation to this project.


\section*{References}
Chatterjee, S. and Heyman, M. ``WiSE Bootstrap for model selection", 2015 (in progress).

Braverman, A.,  Cressie, N., Chatterjee, S., et al. ``Probabilistic Climate Model Evaluation", 2015 (in progress).




\end{document}