\documentclass{article}

\usepackage{amsmath}
\usepackage{amscd}
\usepackage[tableposition=top]{caption}
\usepackage{ifthen}
\usepackage{hyperref} 
\usepackage[utf8]{inputenc}
\usepackage[margin=1in]{geometry}

\date{}
\title{Bayesian experimental design for systems biology: reproduce results}
%\author{Edouard Pauwels}
\author{}

%\VignetteIndexEntry{Bayesian experimental design for systems biology: reproduce results}

\begin{document}
\SweaveOpts{engine=R,eps=FALSE}

<<preliminaries,echo=FALSE,results=hide>>=
#options(prompt = " ")
#options(continue = " ")
options(width=70)
@
\maketitle

\abstract{We present details about how the functions provided in \verb@pauwels2014@ can be used to reproduce the results presented in \cite{pauwels2014bayesian}. We also describe how the system, model specifications and experimental settings were represented. The goal is to provide one with the capacity to reproduce research results to potentially adapt the method to other settings.}

\tableofcontents

\section{Introduction}

The purpose of \verb@pauwels2014@ package is to reproduce the experimental results presented in \cite{pauwels2014bayesian}. We focus on reproducing the comparison between different simulations based on a small network made of $3$ genes with $6$ variables kinetics explicitly modeled and 9 parameters to infer. 

The first section shows how to reproduce the graphics based on pre-computed simulations. We provide pre-computed results because of the high computational cost of repeating experiments multiple times. We also describe the commands and scripts that were used to compute these results. We used parallel processes to run multiple simulations involving different parameter sampling and design strategies. This considerably reduces the total computational time of getting these results. We provide scripts and commands that were used to run multiple simulations.

In a second section, we describe more specifically the encoding of the dynamical system, noise model and estimation schemes that we considered. The purpose is to ease the adaptation of the content of this package to other dynamical systems, noise models, parameter estimation algorithms, sampling schemes and design strategies in order to perform reproducible comparisons.

The package makes extensive use of \verb@ode@ solver provided in package \verb@deSolve@ \cite{deSolve}. We use the graphics libary \verb@ggplot2@ \cite{ggplot} to produce our graphics. Those two packages should be installed to use \verb@pauwels2014@ package, for example in the \verb@R@ interpreter
\begin{verbatim}
install.packages("ggplot2")
\end{verbatim}
\verb@pauwels2014@ is distributed as an \verb@R@ package source archive. The installation requires a working \verb@C@ compiler correctly interfaced with \verb@R@.

\section{Results of the paper}
\subsection{Plot precomputed simulation results}

The following code will load all the precomputed datasets and plot the associated results as presented in figure \ref{fig:one}. 

<<label=figPlot,include=FALSE>>=
library(pauwels2014)
data(knobjs)
sapply(	
			 1:length(knobjs),
			 function(k){
				 assign(names(knobjs)[k], knobjs[[k]], envir = .GlobalEnv)
			 }
)
data(exps)

mean_risks_act_mult <- read_knobjs( sprintf("knobjActMult%s", 1:10) ) 
mean_risks_dream6_mult <- read_knobjs( sprintf("knobjDream6Mult%s", 1:10) )  
mean_risks_rand_mult <- read_knobjs( sprintf("knobjRandMult%s", 1:10) ) 

mean_risks_act_mult <- compute_mean_risks(mean_risks_act_mult, "Bayesian active")
mean_risks_dream6_mult <- compute_mean_risks(mean_risks_dream6_mult, "Dream6")
mean_risks_rand_mult <- compute_mean_risks(mean_risks_rand_mult, "Random")

data_to_plot <- rbind(mean_risks_act_mult,mean_risks_dream6_mult, mean_risks_rand_mult)

ggplot(data = data_to_plot, aes(x=cost, y=risk)) + 
  theme_bw() + 
  facet_grid(.~type) + 
  scale_y_log10() + 
  geom_line(aes(group = chain), alpha = 0.2) + 
  geom_point() + 
  stat_smooth(method = "loess", colour = "black") + 
  theme(axis.text.x = element_text(angle = 45, hjust = 1))
@

The \verb@knobj@ section of the reference manual describes what these datasets are and gives the list of the datasets of this kind available in the package. \verb@knobj@ stands for knoledge list, \verb@Act@ stands for active design, \verb@Rand@ stands for random design, \verb@Mult@ stands for multimodal, \verb@Sing@ stands for unimodal and the integer is a random number generator seed used to run the simulation.   

\begin{figure}[h]
\begin{center}
<<label=fig1,fig=TRUE,echo=FALSE>>=
<<figPlot>>
@
\end{center}
\caption{graphical representation of experimental design simulation results. See the article for details. Note that the figure is slightly different from that of the article. Indeed, this figure is based on subsamples of the samples used to plot the figure of the article. This is to keep the size of the package reasonable.}
\label{fig:one}
\end{figure}

\subsection{Re-run the simulations}
<<preliminaries2,echo=FALSE,results=hide>>=
rm(list = ls())
@
\subsubsection{Direct approach}
The simulation can be re-run easily using the functions provided in the package. The default parameters allow to reproduce the pre-computed simulations that were used in the last section. However, this can be quite long. For example \verb@knobjActMult1@ can be re-computed using the following commands

<<re-run, eval=FALSE, echo=TRUE, keep.source=TRUE>>=
## Load datasets
data(exps)
data(experiment_list1)
data(observables)

## Initialize a knowledge list
knobj <- generate_our_knowledge(transform_params)
knobj$datas[[1]] <- list(
  manip = experiment_list1$nothing,
  data = add_noise(
   simulate_experiment(
     knobj$global_parameters$true_params_T, knobj, 
     experiment_list1$nothing)[
       knobj$global_parameters$tspan %in% observables[["mrnaLow"]]$reso, 
       observables[["mrnaLow"]]$obs
     ]
  )
)
knobj$experiments <- paste("nothing", "mrnaLow")

## Update the knowledge list
knobjActMult1 <- active_design(knobj, sample_function_multi_mod_weight, seed = 1, credits = 5000)
@
In total, there were $30$ simulations which are represented in figure \ref{fig:one}. They can all be re-run using similar scripts, choosing different design strategies, sampling strategies and random seeds.
\subsubsection{Using scripts}
We describe here the two scripts that were used to launch multiple simulation. First we define a script that we call \verb@perform_simu_one.R@ and that allows to control the behaviour of the simulation through the shell.

<<perform_simu_one, eval = FALSE, echo = TRUE, keep.source=TRUE>>=
## Simulates the active learning experimental design with
## A simplified network
## Arguments to be passed
## active: O or 1
## multimodal search: 0 or 1
## random seed: integer
## Results are saved during the process

args <- commandArgs(trailingOnly=TRUE)
if(length(args) != 3){
  stop("Three argument should be passed: bool active (0 or 1), bool multimod (0 or 1), int seed")
}


active <- as.integer(as.numeric(args[1]))
multi_mod <- as.logical(as.numeric(args[2]))
seed <- as.integer(args[3])


## Define the home directory
home <- "some_working_directory"


## Where to save the results
file_to_save <- paste(
  home,
  "results_subdirectory/",
  "knobj",
  c("Rand","Act", "Dream6")[active +1], c("Sing","Mult")[multi_mod +1],
  seed,
  sep=""
)


## Load datasets and library
set.seed(seed)
library(pauwels2014)
data(exps)
data(experiment_list1)
data(observables)


## Initialize a knowledge list
knobj <- generate_our_knowledge(transform_params)
knobj$datas[[1]] <- list(
  manip = experiment_list1$nothing,
  data = add_noise(
  simulate_experiment(
    knobj$global_parameters$true_params_T, 
    knobj, 
    experiment_list1$nothing)[
      knobj$global_parameters$tspan %in% observables[["mrnaLow"]]$reso, 
      observables[["mrnaLow"]]$obs
    ]
  )
)
knobj$experiments <- paste("nothing", "mrnaLow")


## Chosse a sample function
if(multi_mod ){
  sample_function <- sample_function_multi_mod_weight
}else{
  sample_function <- sample_function_single_mod
}


## Update the knowledge list with the chosen strategy
if(active==1){
	knobj <- active_design(knobj, sample_function, seed, credits = 5000, file_to_save = file_to_save)
}
if(active==0){
	knobj <- random_design(knobj, sample_function, exps, seed, credits = 5000, file_to_save = file_to_save)
}
if(active==2){
	knobj <- dream6_design(knobj, sample_function, exps, seed, credits = 5000, file_to_save = file_to_save)
}

@

This script is available from the package and can be called from the shell. The \verb@knobj@ lists available from the package were computed using the following shell commands that could be passed to a resource manager for example.

<<commands>>= 
home <- "some_home/"
path_to_R <- "some_path/R"

n_seed <- 10
seeds <- 1:n_seed
active <- c(rep(2,n_seed),rep(1,2*n_seed),rep(0,n_seed))
multi_mod <- c(rep(1,2*n_seed),rep(0,n_seed),rep(1,n_seed))

commands <-
  sprintf(
    "%s --vanilla --args '%s' '%s' '%s' < %s",
    path_to_R,
    active,
    multi_mod,
    seeds,
    paste(home, "perform_simu_one.R", 
    sep = ""
  )
)

print(commands)
@

\section{Problem specifications}
In this section, we describe more some important aspects of the problem and related functions from the package. This should ease the process of adapting this code to different contexts.

\subsection{Design strategy}
A design strategy consists in choosing experiments sequentially. They consists in a molecular perturbation from \verb@experiment_list1@ and an observable from \verb@observables@. Once this is chosen, the corresponding noisy simulation of the studied dynamical system is provided as a dataset and the simulation can continue. This is run until an initial credit budget has been spent. The objective is to estimate kinetic parameters of the underlying system as well as possible.

\subsubsection{Random design}
This is a baseline strategy. The experiments are chosen randomly among all possible experiments. The only task to perform at each stage is to estimate the parameters based on the data at hand. \verb@random_design@ simulate the process of following this strategy.

\subsubsection{Active design}
In order to perform active design, we implemented the Bayesian experimental design strategy described in the paper. The only difference with the previous strategy is that the experiment are chosen based on the estimation of an expected risk which is provided by the \verb@estimate_risk_out_all@ function. The parameter sampling schemes used for this strategy are the same as in the case of the random design strategy. The function \verb@active_design@ simulates the proposed active design strategy.

\subsubsection{Dream6 design}
We implemented the experimental design criterion given in \cite{steiert2012experimental}. We use the same sampling scheme as for the two previous strategies and estimate an expeced value of the criterion using function \verb@estimate_risk_dream6@. \verb@dream6_design@ simulates this design strategy.


\subsubsection{Organisation of this section}
We present parts of the problem that could easily be modified by the user in order to adapt to a new setting. We first specify the probabilistic model and inference strategies in \ref{sec:model}. Section \ref{sec:syst} contains a desciption of how dynamics, molecular perturbations, observables, and stochastic perturbations were encoded in order to perform our simulation.

\subsection{Model specificication and inference}
\label{sec:model}
\subsubsection{Probabilistic model}
\label{sec:proba}
We describe here the probabilistic model that was used for parameter inference. We first specify a \verb@log_prior@ function which is a large variance Gaussian prior.
<<log_prior>>=
log_prior
@
We also need to specify a likelihood function. We use the same noise model as the one used in the generative noise process. The likelihood function allows to add prior information by penalizing large variations and large values of the kinetic variable which dynamics correspond to a chosen parameter value. It is preferable to use this kind of options when local maximization of the posterior is performed in order to avoid areas of the parameter space which lead to numerical instabilities in solving the corresponding initial value problems (see paper and supplementary information for details).
<<log_prior>>=
log_likelihood
@
Details about the arguments and the value of those functions are found in the manual of this package.

\subsubsection{Parameter inference and sampling}
\label{sec:sampling}

Details are given in annex B of the paper. The first step of the generation of a sample is to find a local posterior maximum using \verb@BFGS_special@ which implements the BFGS quasi-newton method. The local maximum is used as an initialization point for an implementation of the Metropolis Hasting algorithm in function \verb@generate_sample@. The process is repeated several time and a gaussian mixture model approximation is used to recombine samples. The function that performs all these steps is \verb@sample_function_multi_mod_weight@. The parameters governing the behaviour of the local maximization function and the sampling function are found in \verb@knobj$global_parameters@. The manual provides more details about those functions and their parameters.

In order to change the sampling strategy, a function of the same kind as \verb@sample_function_multi_mod_weight@ (e.g. \verb@sample_function_single_mod@) should be saved as \verb@sample_function@.

\subsubsection{Risk}
The last element of the decision process is the risk function. The risk is a dissimilarity function between two parameter values. In our simulations we implemented the following risk function.
<<risk_theta_fun>>=
risk_theta_fun
@

\subsection{System specifications}
\label{sec:syst}
We describe here how are encoded the simulation functions that define system dynamics, how they can be modified and what can be observed from this system.

\subsubsection{Dynamics of the network}
\label{sec:dyn}
The dynamics of the small network we simulate obeys the following ordinary differential equation.

\begin{eqnarray}
{\mathrm{[as7]}} \; &=& \;  {\frac { { \, \left(\frac{\mathrm{[p6]}}{\mathrm{r12_{Kd}}}\right)^ {{\mathrm{r12_h}} }  } }  {{1} \, + \, \left(\frac{\mathrm{[p6]}}{\mathrm{r12_{Kd}}}\right)^ {{\mathrm{r12_h}} }  }  } \nonumber \\
{\mathrm{[as9]}} \; &=& \;  {\frac { {\left(\frac{\mathrm{[p6]}}{\mathrm{r11_{Kd}}}\right)^ {{\mathrm{r11_h}} }  }  }  {{1} \, + \, \left(\frac{\mathrm{[p6]}}{\mathrm{r11_{Kd}}}\right)^ {{\mathrm{r11_h}} }  }  } \nonumber \\
{\mathrm{[rs7]}} \; &=& \;  {\frac{1} {{1} \, + \, \left(\frac{\mathrm{[p7]}}{\mathrm{r6_{Kd}}}\right)^ {{\mathrm{r6_h}} }  }  } \nonumber \\
{\mathrm{[g6]}} \; &=& 1 \nonumber \\
{\mathrm{[g7]}} \; &=& \;  \mathrm{[as7]} \cdot \mathrm{[rs7]}  \nonumber \\
{\mathrm{[g9]}} \; &=& \mathrm{[as9]}  \nonumber \\
\frac {\mathrm{d}\left( {{\mathrm{[p6]}} } \right) }  {\mathrm{d}{t} }  \; &=& \;  {\mathrm{rbs6_{strength}}} \, \cdot \, {\mathrm{[v6_{mrna}]}}   -{\mathrm{p6_{degradationRate}}} \, \cdot \, {\mathrm{[p6]}}  \nonumber \\
\frac {\mathrm{d}\left( {{\mathrm{[p7]}} } \right) }  {\mathrm{d}{t} }  \; &=& \;  {\mathrm{rbs8_{strength}}} \, \cdot \, {\mathrm{[v7_{mrna}]}}   -{\mathrm{p7_{degradationRate}}} \, \cdot \, {\mathrm{[p7]}}  \nonumber \\
\frac {\mathrm{d}\left( {{\mathrm{[p8]}} } \right) }  {\mathrm{d}{t} }  \; &=& \;  {\mathrm{rbs7_{strength}}} \, \cdot \, {\mathrm{[v8_{mrna}]}}   -{\mathrm{p8_{degradationRate}}} \, \cdot \, {\mathrm{[p8]}}  \nonumber \\
\frac {\mathrm{d}\left( {{\mathrm{[v6_{mrna}]}} } \right) }  {\mathrm{d}{t} }  \; &=& \;  {\mathrm{pro6_{strength}}} \, \cdot \, {\mathrm{[g6]}}   -{\mathrm{v6_{mrnaDegradationRate}}} \, \cdot \, {\mathrm{[v6_{mrna}]}}  \nonumber \\
\frac {\mathrm{d}\left( {{\mathrm{[v7_{mrna}]}} } \right) }  {\mathrm{d}{t} }  \; &=& \;  {\mathrm{pro7_{strength}}} \, \cdot \, {\mathrm{[g9]}}   -{\mathrm{v7_{mrnaDegradationRate}}} \, \cdot \, {\mathrm{[v7_{mrna}]}}  \nonumber \\
\frac {\mathrm{d}\left( {{\mathrm{[v8_{mrna}]}} } \right) }  {\mathrm{d}{t} }  \; &=& \;  {\mathrm{pro9_{strength}}} \, \cdot \, {\mathrm{[g7]}}   -{\mathrm{v8_{mrnaDegradationRate}}} \, \cdot \, {\mathrm{[v8_{mrna}]}}  \nonumber 
\end{eqnarray}

With kinetic variables $\mathrm{[p6]}, \mathrm{[p7]}, \mathrm{[p8]}, \mathrm{[v6_{mrna}]}, \mathrm{[v7_{mrna}]}, \mathrm{[v8_{mrna}]}$ and kinetic parameters
\begin{eqnarray}
\mathrm{v6_{mrnaDegradationRate}} \nonumber\\
\mathrm{v7_{mrnaDegradationRate}} \nonumber\\
\mathrm{v8_{mrnaDegradationRate}} \nonumber\\
\mathrm{p6_{degradationRate}} \nonumber\\
\mathrm{p7_{degradationRate}} \nonumber\\
\mathrm{p8_{degradationRate}} \nonumber\\
\mathrm{pro6_{strength}} \nonumber\\
\mathrm{pro7_{strength}} \nonumber\\
\mathrm{pro9_{strength}} \nonumber\\
\mathrm{rbs6_{strength}} \nonumber\\
\mathrm{rbs7_{strength}} \nonumber\\
\mathrm{rbs8_{strength}} \nonumber\\
\mathrm{r6_{Kd}}, \mathrm{r6_h} \nonumber\\
\mathrm{r11_{Kd}}, \mathrm{r11_h} \nonumber\\
\mathrm{r12_{Kd}}, \mathrm{r12_h} \nonumber
\end{eqnarray}
Among them, we suppose that we have 
$$\mathrm{v6_{mrnaDegradationRate}} = \mathrm{v7_{mrnaDegradationRate}} = \mathrm{v8_{mrnaDegradationRate}} = 1$$  
and 
$$\mathrm{p6_{degradationRate}} = \mathrm{p7_{degradationRate}} = \mathrm{p8_{degradationRate}}.$$
We suppose also that we know the $\mathrm{r_h}$ parameters, 
$$\mathrm{r6_h} = 4$$
$$\mathrm{r11_h} = 2$$
$$\mathrm{r12_h} = 2$$ as well as $\mathrm{r12_{Kd}} = 0.2$. See also the \verb@transform_params@ function of the package. The parameter value used for simulation is found in \verb@knobj$global_parameters$true_params_T@.
<<tr_params>>=
knobj <- generate_our_knowledge(transform_params)
knobj$global_parameters$true_params_T
transform_params(knobj$global_parameters$true_params_T)
@

We encode this system in a compiled shared object which can be passed to the ode solver of the package \verb@deSolve@. See the vignette \cite{deSolveVignette2} for more details about this. The \verb@C@ code used to represent these dynamics is available in \verb@src/model0_simplified_mrna_rates.c@ of the source of this package. It contains the follwing instructions:

\begin{verbatim}
/* file model0_simplified_mrna_rates.c */
/* Encodes a simplified network dynamics and structure(gene 6,7,8)*/
   
#include <R.h>
#include <math.h>
static double parms[16];
#define mrna6_degradation_rate parms[0]
#define mrna7_degradation_rate parms[1]
#define mrna8_degradation_rate parms[2]
#define p_degradation_rate parms[3]
#define r6_Kd parms[4]
#define r6_h parms[5]
#define r11_Kd parms[6]
#define r11_h parms[7]
#define r12_Kd parms[8]
#define r12_h parms[9]
#define pro6_strength parms[10]
#define pro7_strength parms[11]
#define pro9_strength parms[12]
#define rbs6_strength parms[13]
#define rbs7_strength parms[14]
#define rbs8_strength parms[15]

/* initializer */
void initmod(void (* odeparms)(int *, double *))
{
    int N=16;
    odeparms(&N, parms);
}

/* Derivatives and 1 output variable */
void derivs (int *neq, double *t, double *y, double *ydot,
             double *yout, int *ip)
{   
    if (ip[0] <1) error("nout should be at least 1");
    
    double as7, as9;
    double rs7;
    double g6, g7, g9;
    
    /* Rules (skipping the pro* and rbs* variables which are constant and do not enter */
    /* the equations anyway */
    /* y[0] = g6=cte  y[1]=p6  y[2]=p7  y[3]=p8,  */
    /* y[4] = v6_mrna  y[5] = v7_mrna  y[6]=v8_mrna  */

    as7 = pow( y[1]/r12_Kd , r12_h ) /(1+ pow( y[1]/r12_Kd , r12_h ) );
    as9 =  pow( y[1]/r11_Kd , r11_h ) /(1 +  pow( y[1]/r11_Kd , r11_h ) );
    rs7 = 1/(1+ pow( y[2]/r6_Kd , r6_h ) );
    g6 = y[0];         /* cte */
    g7 = as7 * rs7;
    g9 = as9;

    /* mrna update (be carrefull with promotor numerotation !!) */

    ydot[4] = pro6_strength * g6 - mrna6_degradation_rate * y[4]; /* d(v6) */
    ydot[5] = pro7_strength * g9 - mrna7_degradation_rate * y[5]; /* d(v7) */
    ydot[6] = pro9_strength * g7 - mrna8_degradation_rate * y[6]; /* d(v8) */

    /* protein update (g9=y[0]=cte). ribosome and promoteur numbering  */

    ydot[0] = 0.;
    ydot[1] = rbs6_strength * y[4] - p_degradation_rate * y[1];  /* d(p6) */
    ydot[2] = rbs8_strength * y[5] - p_degradation_rate * y[2];  /* d(p7) */
    ydot[3] = rbs7_strength * y[6] - p_degradation_rate * y[3];  /* d(p8) */
    yout[0] = 1;
}
\end{verbatim}

Note that the order of the parameters given by \verb@transfom_params@ function is the same as the order of the parameters expected by the \verb@C@ representation of the dynamics.

\subsubsection{Obervables and perturbations}
\label{sec:obsPerturb}
Observables are quantities that can be observed. The description of an observable is made through a list. Details are provided in the manual. The source code used to generate the \verb@observables@ data file from the package is as follows.

<<obserevables, eval=FALSE, echo=TRUE, keep.source=TRUE>>=
superHighRes <-  0:200/2
highRes <- 0:50 * 2
lowRes <- 0:25 * 4

observables <- list(
  list(name = "p6", obs = c("time", "p6"),reso = superHighRes, cost = 400),
  list(name = "p7", obs = c("time", "p7"),reso = superHighRes, cost = 400),
  list(name = "p8", obs = c("time", "p8"),reso = superHighRes, cost = 400),
  list(name = "mrnaHigh", obs = c("time", "v6_mrna", "v7_mrna", "v8_mrna"), reso = highRes, cost = 1000),
  list(name = "mrnaLow", obs = c("time", "v6_mrna", "v7_mrna", "v8_mrna"), reso = lowRes, cost = 500)
)

names(observables) <- sapply(observables, FUN = function(x){x$name})
@

Perturbations are modeled through functions that act on parameter and initial condition values. Details about these functions can be found in the manual of the package. The source code that was used to generate \verb@experiment_list1@ and \verb@exps@ data file is as follows.

<<experiments, eval=FALSE, echo=TRUE, keep.source=TRUE>>=
delete_gene6 <- function(theta, init){
  theta[names(theta) %in% c("pro6_strength","rbs6_strength")] <- 0
  init[names(init) == "p6"] <- 0
  res <- c()
  res$theta <- theta
  res$initial_conditions <- init
  res$cost <- 800
  res
}

delete_gene7 <- function(theta, init){
  theta[names(theta) %in% c("pro7_strength","rbs8_strength")] <- 0
  init[names(init) == "p7"] <- 0
  res <- c()
  res$theta <- theta
  res$initial_conditions <- init
  res$cost <- 800
  res
}

delete_gene8 <- function(theta, init){
  theta[names(theta) %in% c("pro9_strength","rbs7_strength")] <- 0
  init[names(init) == "p8"] <- 0
  res <- c()
  res$theta <- theta
  res$initial_conditions <- init
  res$cost <- 800
  res
}



knockdown_gene6 <- function(theta, init){
  theta[names(theta) %in% "mrna6_degradation_rate"] <- 10
  res <- c()
  res$theta <- theta
  res$initial_conditions <- init
  res$cost <- 350
  res
}

knockdown_gene7 <- function(theta, init){
  theta[names(theta) %in% "mrna7_degradation_rate"] <- 10
  res <- c()
  res$theta <- theta
  res$initial_conditions <- init
  res$cost <- 350
  res
}

knockdown_gene8 <- function(theta, init){
  theta[names(theta) %in% "mrna8_degradation_rate"] <- 10
  res <- c()
  res$theta <- theta
  res$initial_conditions <- init
  res$cost <- 350
  res
}



decrease_rbs_gene6 <- function(theta, init){
  theta[names(theta) %in% "rbs6_strength"] <- theta[names(theta) %in% "rbs6_strength"] / 10
  res <- c()
  res$theta <- theta
  res$initial_conditions <- init
  res$cost <- 450
  res
}

decrease_rbs_gene7 <- function(theta, init){
  theta[names(theta) %in% "rbs8_strength"] <- theta[names(theta) %in% "rbs8_strength"] / 10
  res <- c()
  res$theta <- theta
  res$initial_conditions <- init
  res$cost <- 450
  res
}

decrease_rbs_gene8 <- function(theta, init){
  theta[names(theta) %in% "rbs7_strength"] <- theta[names(theta) %in% "rbs7_strength"] / 10
  res <- c()
  res$theta <- theta
  res$initial_conditions <- init
  res$cost <- 450
  res
}

nothing <- function(theta, init){
  list(theta = theta, initial_conditions = init, cost = 0)
}

experiment_list1 <- c()
experiment_list1$delete_gene6 <- delete_gene6
experiment_list1$delete_gene7 <- delete_gene7
experiment_list1$delete_gene8 <- delete_gene8
experiment_list1$knockdown_gene6 <- knockdown_gene6
experiment_list1$knockdown_gene7 <- knockdown_gene7
experiment_list1$knockdown_gene8 <- knockdown_gene8
experiment_list1$decrease_rbs_gene6 <- decrease_rbs_gene6
experiment_list1$decrease_rbs_gene7 <- decrease_rbs_gene7
experiment_list1$decrease_rbs_gene8 <- decrease_rbs_gene8
experiment_list1$nothing <- nothing

exps <- lapply(names(experiment_list1), function(name){
    t(sapply(observables, function(obs){
        cbind(obs$name, obs$cost + experiment_list1[[name]](0,0)$cost, name)
      }
    ))
  }
)

exps <- Reduce(exps, f = function(s,t){rbind(s,t)}) 

exps <- data.frame(Measurement = exps[,1], Cost = as.numeric(exps[,2]), exp = exps[,3])
@
\subsubsection{Noise generative process}
This describes how noise affects the true dynamics of the system. The noise model we have is an independant heteroscedastic centered Gaussian noise which variance has the expression $0.01 + 0.04 x^2$ where $x$ is the non corrupted data. The function \verb@add_noise@ specifies this process.
<<add_noise>>=
add_noise
@
\section{Conclusion}
The purpose of this package is to allow users to reproduce the results reported in our article. One of the objectives of the development of active experimental design strategies is to provide methods that could be used by non specialists, we consider that reproducibility is a key issue in this realm. 

In addition, this package could be used as a benchmark to compare newly proposed design strategy. As implementation is readilly available, it should not cost much to compare new parameter estimation schemes or design strategies provided that they can be implemented in a way intelligible for a computer.

%% this adds References to the PDF-Index without adding an obsolete section
\phantomsection
\addcontentsline{toc}{section}{References}
\bibliographystyle{plain}
\bibliography{refs}

\end{document}

