# Uncertainty and Sensitivity Analysis

## Overview

This app allows exploration of the concept of uncertainty and sensitivity analysis. For this purpose, we use the basic bacteria infection model introduced in the app of that name. If you haven’t done so yet, familiarize yourself and work through that app first.

## The Model

### Model Overview

The model is the continuous time model in the ‘basic bacteria model’ app. See the documentation there for the model description. For convenience, here is a quick summary and the equations again.

We model 2 compartments:

- B - bacteria
- I - immune response

We specify the following processes/flows:

1. Bacteria grow/divide at some maximum rate (which we label g) and saturate as they approach some maximum carrying capacity, Bmax.
2. Bacteria die at a natural death rate (which we label dB).
3. Bacteria are killed by the immune response at some rate k.
4. The immune response grows proportional to the number of bacteria and itself at some rate r.
5. The immune response decays at some rate (which we label dI).

The equations are given by:

\[\dot B = g B (1-\frac{B}{B_{max}}) - d_B B - k BI\] \[\dot I = r B I - d_I I\]

### Uncertainty and Sensitivity analysis

Often, for a given system we want to model, we only have rough estimates for the model parameters and starting values. Instead of specifying fixed values (which results in a single time-series), we can instead specify parameter ranges, choose sets of parameter values from these ranges, and run the model for multiple sets of parameters.

The simplest way of specifying parameter ranges is to set an upper and lower bound (based on what we know about the biology of the system) and randomly choose any value within those bounds. We can almost always set bounds even if we know very little about a system. Assume we want to model the death rate of some cell type (e.g. NK cells) in humans. We might not know anything, but we can still be fairly confident that their lifespan is at least 1 second and less than 100 years. That’s of course a wide range and we should and usually can narrow ranges further, based on biological knowledge of a given system.

If we are fairly certain that values are close to some quantity, instead of specifying a uniform distribution, we can choose one that is more peaked around the most likely value. Normal distributions are not ideal since they allow negative values, which doesn’t make sense for our parameters. The gamma distribution is a better idea, since it leads to only positive values.

To run the model for this app, we need to specify values for the 2 initial conditions, B0 and I0, and the 6 model parameters g, Bmax, dB, k, r, dI. All initial conditions and parameters are sampled uniformly between the specified upper and lower bound, apart from the bacteria growth rate, which is given by a gamma distribution, with user-specified mean and variance. For this teaching app, there is no biological reason for making bacterial growth different, I just picked one parameter and decided to make it non-uniformly distributed to show you different ways one can implement distributions from which to draw parameter samples.

The way the samples are drawn could be done completely randomly, but that would lead to inefficient sampling. A smarter method exists, known as Latin Hypercube sampling (LHS). It essentially ensures that we sample the full range of possible parameter combinations in an efficient manner. For more technical details, see e.g. (Saltelli et al. 2004). For this app, we use LHS.

Once we specify the ranges for each parameter, the sampling method, and the number of samples, the simulation draws that many samples, runs the model for each sample, and records outcomes of interest. While the underlying simulation returns a time-series for each sample, we are usually not interested in the full time-series. Instead, we are interested in some summary quantity. For instance in this model, we might be interested in the maximum/peak level of bacteria during the infection, the level of bacteria at the end (the steady state) of the infection, and the level of the immune response at steady state. This app records and reports those 3 quantities as Bpeak, Bsteady and Isteady.

Results from such simulations for multiple samples can be analyzed in different ways. The most basic one, called uncertainty analysis only asks what level of uncertainty we have in our outcomes of interest, given the amount of uncertainty in our model parameter values. This can be graphically represented with a boxplot, and is one of the plot options for this app.

In a next step, we can ask ‘how sensitive is the outcome(s) of interest to variation in specific parameters’ - that part is the sensitivity analysis. When you run the simulations, you essentially do both uncertainty and sensitivity analysis at the same time, it’s just a question of how you further process the results. We can graphically inspect the relation between outcome and some parameter with scatterplots. If we find that there is a monotone up or down (or neither) trend between parameter and outcome, we can also summarize the finding using a correlation coefficient. For this type of analysis, using the Spearman rank correlation coefficient is useful, which is what the app produces below the figures.

### A note on randomness in computer simulations

This simulation (as well as some of the others) involves sampling. This leads to some level of randomness. In science, we want to be as reproducible as possible. Fortunately, random numbers on a computer are not completely random, but can be reproduced. In practice, this is done by specifying a random number seed, in essence a starting position for the algorithm to produce pseudo-random numbers. As long as the seed is the same, the code should produce the same pseudo-random numbers each time, thus ensuring reproducibility.

## What to do

First, familiarize yourself with the setup of the app, it looks different from most others. Parameters are not set to specific values. Instead, most parameters have a lower and upper bound. For each simulation that is run, random values for the parameter are chosen uniformly between those bounds. The parameter g does not have a uniform but instead a gamma distribution, you can specify its mean and variance to determine the distribution from which values are sampled.

The default outcome plots are boxplots, which show the distribution of the 3 outcomes of interest for the different parameter samples. You can set the number of samples you want to run. Samples are constructed using the latin hypercube method to efficiently span the space of possible parameter values. In general, more samples are better, but of course take longer to run.

### Task 1:

Since the creation of parameter samples involves some element of uncertainty, we need to make use of random numbers. We still want results to be reproducible. That’s where the random number seed comes in. As long as the seed is the same, the code should produce the same pseudo-random numbers each time, thus ensuring reproducibility. Let’s explore this.

- Leave all settings as they are, run 20 samples twice with the same random number seed, check to make sure you get exactly the same result twice.
- Now change the random number seed to a different value, run again. You should see results changed. (It doesn’t matter if you change the seed by just a bit or a lot.)
- The more samples you have, the more robust the results are to changes in the underlying sample generation (determined by the random number seed). Try checking this by running 10 samples with 2 different random number seeds, then running 100+ samples (or as many as you can do without waiting too long) with 2 different seeds. You should see less variability in the central quantities (mean, median) for the larger sample size.

Note that each sample means one simulation of the underlying dynamical model, so as sample numbers increase, things slow down. Also note the ‘system might not have reached steady state’ message. If for too many of the samples steady state has not been reached, the results for Bsteady and Isteady are not correct. In that case you need to increase the simulation time to allow the system to settle into steady state. For some parameter combinations, that can take very long.

### Task 2:

- Recall the underlying dynamical model and its behavior. If you can’t, revisit the “Basic Bacteria” app and go through it. Use your understanding of the model to predict what happens if you increase both lower and upper bound for the immune response activation rate, r.
- Increase lower/upper bounds by a factor of 10, from 10-5/10-4 to 10-4/10-3. Run simulations, see how results change.
- Now go the opposite, lower the initial lower/upper bounds by a factor of 10. Run simulations, see how results change.

### Task 3:

- Now let’s explore what happens if we change ranges for the bacteria carrying capacity, Bmax. If we increase it, which of the outcomes do you expect to change, and in which direction?
- Test your assumption by increasing and decreasing lower/upper bounds for Bmax by a factor of 10. Set r back to the initial values, leave all other parameters as before.

### Task 4:

- Continue exploring by changing ranges for different parameters, see what you find. It is likely that for some settings you’ll see warning or error messages on the `R` console. That generally means that the parameters for a given simulation are such that the differential equation solver can’t properly run the model. That usually corresponds to biologically unrealistic parameter settings. We’ll ignore them, but if you did a research project and you got such warning or error messages, you’d have to figure out why you get them and only once you fully understand why is it maybe ok to ignore them.

### Task 5:

The above approach of exploring the impact of a parameter on results by varying bounds is tedious. Also, often we have bounds that are specified by biology, and not subject to us changing them. It would still be useful to know how a given parameter impacts the results. This is where sensitivity analysis comes in. We run the same simulations, but now instead of plotting outcomes as a boxplot, we produce scatterplots for outcomes as function of each varied parameter.

- Set values back as in task 1. If you can’t remember, close the app and reopen.
- Switch the plot type from boxplot to scatterplot, run the simulation.
- Take a close look at the scatterplots to investigate the relation between different parameters and the various outcomes.
- Look at the text below the plots. For each parameter-output pair, the code computes a rank correlation coefficient. Numbers close to 0 mean there is essentially no correlation, close to 1 or -1 means a large positive or negative correlation. (One could compute p-values for these correlations, but they are somewhat meaningless since the values will get smaller the more samples you use, so you can basically produce any p-value you want.)
- Increase sample size to 100+ or whatever number runs within a reasonable amount of time. With more samples, the patterns of correlation are clearer in the plots.

### Task 6:

Since our model is rather simple, we can actually determine relations between parameters and some of the outcomes analytically. Specifically, it is possible to compute the steady state values for B and I, Bsteady and Isteady. If you don’t know what steady states are and how to compute them, go through the “Bacterium Model Exploration” app, where this is explained.

- If you haven’t done it yet or can’t remember the equations for the steady states, compute them. You should find that \(B_{steady} = d_I / r\) and a similar equation for Isteady.
- Compare the scatterplots and correlation coefficients with the results from these equations. For instance based on the equation, you should see a linear correlation between _Bsteady and dI. You might find that there is a lot of scatter in the data, too much to see clear patterns. One could always increase sample size which should help detect patterns, but it takes longer to run. Another option is to restrict the variability to a subset of parameters, which we’ll do next.

### Task 7:

- Let’s explore in more detail how different parameters impact results by making the system less ‘noisy’. To do so, we’ll impose no variability for some parameters. For the following parameters, set both their lower and upper bound to the specified value: B0 = I0 = 1, Bmax = 1e5, dB = 1, k = 10-7, r = 10-4.
- Let dI vary between 1 and 2 and give g a mean of 5 and variance of 1. Run the simulation. You’ll get a bunch of warning messages from the function that computes the correlations, ignore them.
- You should now see nice patterns of correlation between g and dI and the different outcomes. Confirm that - as expected from the steady state equations - that Bsteady depends linearly on dI and that it has no correlation with g. Both the scatterplots and the correlation coefficient should give you that information.
- Also note the distribution for g and dI. The former has more points around its mean and less for lower/higher values, while dI values are uniformly distributed along the x-axis. This comes from the underlying assumption about how the parameters are distributed, gamma-distribution versus uniform distribution.

### Task 8:

- Above, you should have found for the Isteady equation that \(I_{steady} = g(1-d_I/(r*B_{max})-d_B)/k\). Let’s compare the results from the equation with the simulation. Run the simulations several times, once with dI ranging 0.1 to 2, then from 1.5 to 2, and then from 0.1 to 0.6. Investigate the plot and correlation coefficient for dI and Isteady for both settings. What differences do you note based on the values? How could those be explained?

The important take-home message from this task is that the influence of a parameter on some outcome can be different over different ranges. For instance in range A-B, the parameter might have a major influence, but once the parameter value goes above B, the parameter does not further influence the result. If you have large uncertainty in your parameters, it might be worth considering both the full range, and dividing the range into smaller areas to see how the parameter behaves.

## Further Information

- This app (and all others) are structured such that the Shiny part (the graphical interface you see and the server-side function that goes with it) calls an underlying R script (or several) which runs the simulation for the model of interest and returns the results.
- For this app, the underlying function running the simulation is called `simulate_usanalysis`. You can call them directly, without going through the shiny app. Use the `help()` command for more information on how to use the functions directly. If you go that route, you need to use the results returned from this function and produce useful output (such as a plot) yourself.
- You can also download all simulator functions and modify them for your own purposes. Of course to modify these functions, you’ll need to do some coding.
- For examples on using the simulators directly and how to modify them, read the package vignette by typing `vignette('DSAIRM')` into the R console.
- Good papers explaining uncertainty and sensitivity analysis in a bit more detail are (Hoare, Regan, and Wilson 2008; Marino et al. 2008).

### References

Hoare, Alexander, David G Regan, and David P Wilson. 2008. “Sampling and Sensitivity Analyses Tools (Sasat) for Computational Modelling.” Theor Biol Med Model 5: 4. https://doi.org/10.1186/1742-4682-5-4.

Marino, Simeone, Ian B. Hogue, Christian J. Ray, and Denise E. Kirschner. 2008. “A Methodology for Performing Global Uncertainty and Sensitivity Analysis in Systems Biology.” J. Theor. Biol 254 (1): 178–96. https://doi.org/10.1016/j.jtbi.2008.04.011.

Saltelli, Andrea, Stefano Tarantola, Francesca Campolongo, and Marco Ratto. 2004. Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models. John Wiley & Sons.