Gaussian Distribution

Parametrization

The probability density function (PDF) for the Gaussian distribution is expressed as:

\[\label{eq:gaussian-pdf} f(\pmb{y}) = \left( \frac{\sqrt{s\tau}}{\sqrt{2\pi}} \right)^n \, \exp\!\Bigl( -\tfrac{1}{2} \, s\tau \, (\pmb{y} - \pmb{\mu})^T (\pmb{y} - \pmb{\mu}) \Bigr),\]

where:

• \(\pmb{y} \in \mathbb{R}^n\) is a vector of observed values.

• \(\pmb{\mu} \in \mathbb{R}^n\) is the vector of means corresponding to each element of \(\pmb{y}\).

• \(s\) is a positive scaling factor that modifies the overall precision of the distribution.

• For \(s = 1\), \(\tau\) is the precision, defined as the inverse of the variance \(\tau = 1/\sigma^2\).

• \(n\) is the number of observations (the dimension of \(\pmb{y}\)).

To illustrate how different parameter settings (mean \(\mu\), variance \(\sigma^2\), precision \(\tau\), and scale \(s\)) influence the shape of the Gaussian distribution, Figure 1 displays several overlaid Gaussian curves.

As seen in the figure, increasing \(\sigma^2\) (i.e., decreasing \(\tau\)) broadens the Gaussian curve, while decreasing \(\sigma^2\) makes it narrower and taller. Likewise, shifting \(\mu\) moves the location of the peak along the x-axis. The additional scale factor \(s\) can also expand or contract the distribution as needed for certain modeling contexts.

Mean and Variance

The mean vector \(\pmb{\mu}\) is directly linked to the linear predictor \(\pmb{\eta}\) via an identity link function:

\[\pmb{\mu} = \pmb{\eta}.\]

The variance of the distribution, \(\sigma^2\), is defined as:

\[\sigma^2 = \frac{1}{s \tau},\]

where \(\tau\) is the precision, controlling the spread or dispersion of the distribution. In the simplest case (isotropic covariance), the covariance matrix is \(\sigma^2 I_n\).

Hyperparameters

In a Bayesian framework for Gaussian likelihood models, hyperparameters typically control the precision \(\tau\) (inverse of the variance) or other variance-related quantities. These hyperparameters often appear as log-transformed parameters to ensure they remain within valid (positive) ranges and to improve numerical stability during inference.

Within pyINLA, the Gaussian likelihood collects these in the hyperparameter vector \(\boldsymbol{\theta} = (\theta_1, \theta_2)\). The first entry, \(\theta_1\), controls the main precision term and is always present. The second entry, \(\theta_2\), is optional and contributes an offset precision component; when not needed, it is omitted or fixed so that its effect on \(\tau\) is negligible.

Hyperparameter \(\theta_1\) (precision)

Key: prec

By default, the model uses: \[\tau = \kappa_1,\] where \(\kappa_1\) is a positive parameter that we often represent via the log-transformation: \[\theta_1 = \log(\kappa_1).\] Thus, \(\theta_1\) can take on any real value, ensuring \(\kappa_1>0\). In practice, \(\theta_1\) is often assigned a prior distribution (e.g., a log-Gamma) to capture prior beliefs about the likely range of \(\tau\). In the default configuration (captured in the dictionary below), this hyperparameter is labelled “log precision”, starts at 4, is estimated rather than fixed, and sets prior="loggamma" with param=[1, 5e-5], which corresponds to a log-gamma prior with those shape and rate values. These defaults imply that, initially, \(\kappa_1 = e^{\,4}\approx 54.6\), with a broad log-Gamma prior that allows the precision \(\tau\) to adapt during model fitting.

Hyperparameter \(\theta_2\) (precision offset)

Key: precoffset (fixed by default)

Some models introduce a second hyperparameter \(\kappa_2\) to incorporate additional variance or offset terms in the precision. Its log-transformed version is: \[\theta_2 = \log(\kappa_2).\] This leads to a precision defined by \[\frac{1}{\tau} = \frac{1}{\kappa_1} + \frac{1}{\kappa_2},\] which equivalently becomes \[\tau = \frac{1}{(1/\kappa_1) + (1/\kappa_2)}.\] Hence, \(\tau\) depends on both \(\kappa_1\) and \(\kappa_2\). For instance, if \(1/\kappa_2\) is negligible (i.e., effectively zero), we recover \(\tau \approx \kappa_1\). When this hyperparameter is used, the default configuration (see dictionary below) labels it “log precision offset”, assigns the large initial value 72.087, fixes it during optimisation, and leaves prior and parameters unset, reflecting that it is only there to guard against numerical issues.

In many implementations, a practical threshold classifies \(1/\kappa_2\) as zero if it falls below machine epsilon. Numerically, this is equivalent to \(\theta_2 \geq 36.05\), beyond which the contribution of \(\kappa_2\) to \(\tau\) is effectively nil. For this reason precoffset must always remain fixed: explicitly setting fixed=False for it raises a safety error.

The offset component is treated as a safety valve: the default keeps it fixed with prior and parameters left unset. Attempting to supply a non-None prior, supply param, or set fixed=False for precoffset raises a safety error.

When translated into control['family']['hyper'], the default entries look like this:

control = {
    'family': {
        'hyper': [
            {
                'id': 'prec',
                'prior': 'loggamma',
                'param': [1.0, 5e-5],
                'initial': 4.0,
                'fixed': False,
            },
            {
                'id': 'precoffset',
                'initial': 72.087,
                'fixed': True,
            },
        ]
    }
}

In summary, by tuning or estimating \(\theta_1\) and \(\theta_2\), one can flexibly control the precision \(\tau\) in a Gaussian likelihood model, allowing for richer or more constrained variance structures based on the application’s requirements.

Validation Rules

pyINLA enforces several validation rules for Gaussian models to ensure correct specification:

Hyperparameter Configuration

Key: control['family']

For the Gaussian family, control['family'] only accepts the key hyper. Any other key raises a safety error.

Each entry in control['family']['hyper'] may only contain these keys:

• id - Hyperparameter identifier (prec or precoffset)

• prior - Prior distribution name

• param - Prior parameters

• initial - Initial value on log scale

• fixed - Whether to fix the hyperparameter

Allowed Priors for Precision

Key: control['family']['hyper'] entry with 'id': 'prec'

The precision hyperparameter (prec) supports these priors:

• loggamma - Log-gamma prior with param=[shape, rate], where both shape and rate must be strictly positive

• pc.prec - Penalized complexity prior with param=[u, alpha]

It can also be fixed at a known value by setting fixed=True (no prior required).

# With loggamma prior
control = {
    'family': {
        'hyper': [{
            'id': 'prec',
            'prior': 'loggamma',
            'param': [1.0, 0.01],
            'initial': 2.0
        }]
    }
}

# With pc.prec prior
control = {
    'family': {
        'hyper': [{
            'id': 'prec',
            'prior': 'pc.prec',
            'param': [1.0, 0.01]
        }]
    }
}

# Fixed at a known precision value
control = {
    'family': {
        'hyper': [{
            'id': 'prec',
            'initial': 4.0,  # log(prec), so prec = exp(4.0) ~ 55
            'fixed': True
        }]
    }
}

Precision Offset Restrictions

Key: control['family']['hyper'] entry with 'id': 'precoffset'

The precoffset hyperparameter has special restrictions:

• Must remain fixed=True

• Prior must be None, the string "none", or omitted

• Cannot specify custom param

import math

# Fixed known variance component var0
control = {
    'family': {
        'hyper': [{
            'id': 'precoffset',
            'initial': math.log(1.0 / var0),  # log(1/var0)
            'fixed': True
        }]
    }
}

# Combined: estimated prec + fixed precoffset
control = {
    'family': {
        'hyper': [
            {
                'id': 'prec',
                'prior': 'loggamma',
                'param': [1.0, 5e-5],
                'initial': 4.0
            },
            {
                'id': 'precoffset',
                'initial': math.log(1.0 / var0),
                'fixed': True
            }
        ]
    }
}

Scale Vector

Key: scale

When using the scale argument, all values must be strictly positive (> 0).

# With observation-specific scale
model = {'response': 'y', 'fixed': ['1', 'x']}
result = pyinla(model=model, data=df, family="gaussian", scale=df["weights"])

Specification

• family="gaussian"

• Required arguments:

  • \(\pmb y\): observed values (\(y_i \in \mathbb{R}\) for each observation).

• Gaussian Regression: Log-Gamma Prior
• Gaussian Regression: Observation Scale
• Gaussian Regression: Precision Offset