Evaluate the accuracy of multiple (candidate) classifiers in several subgroups

Assess classification accuracy of multiple classifcation rules stratified by subgroups, e.g. in diseased (sensitivity) and healthy (specificity) individuals.

Usage

evaluate(
  data,
  contrast = define_contrast("raw"),
  benchmark = 0.5,
  alpha = 0.05,
  alternative = c("two.sided", "greater", "less"),
  adjustment = c("none", "bonferroni", "maxt", "bootstrap", "mbeta"),
  transformation = c("none", "logit", "arcsin"),
  analysis = c("co-primary", "full"),
  regu = FALSE,
  pars = list(),
  ...
)

Arguments

data: (list)
of n_g x m binary matrix or data.frame (n_g observations of m binary decisions), g is the index of subgroups/classes, usually created via compare.
contrast: (cases_contrast)
specified via define_contrast
benchmark: (numeric)
value to compare against (RHS), should have same length as data.
alpha: (numeric)
significance level (default: 0.05)
alternative: (character)
specification of alternative hypothesis
adjustment: (character)
specification of statistical adjustment taken to address multiplicity. The default 'none' does not perform any adjustment for multiplicity.
transformation: (character)
define transformation to ensure results (e.g. point estimates, confidence limits) lie in unit interval ("none" (default), "logit", or "arcsin" (sqrt))
analysis: (character)
"co-primary" or "full"
regu: (numeric | logical)
vector of length 3, specify type of shrinkage. Alternatively, logical of length one (TRUE := c(1, 1/2, 1/4), FALSE := c(0, 0, 0))
pars: (list)
further parameters given as named list list(type="pairs", nboot=2000)
...: (any)
additional named parameters, can be used instead of (in in conjunction with) pars

Value

(cases_results)
list of analysis results including (adjusted) confidence intervals and p-values

Details

Adjustment methods (adjustment) and additional parameters (pars or ...):

"none" (default): no adjustment for multiplicity

"bonferroni": Bonferroni adjustment

"maxt": maxT adjustment, based on a multivariate normal approximation of the vector of test statistics

"bootstrap": Bootstrap approach

nboot: number of bootstrap draws (default: 2000)
type: type of bootstrap, "pairs" (default) or "wild"
dist: residual distribution for wild bootstrap, "Normal" (default) or "Rademacher"
proj_est: should bootstrapped estimates for wild bootstrap be projected into unit interval? (default: TRUE)
res_tra: type of residual transformation for wild boostrap, 0,1,2 or 3 (default: 0 = no transformation) (for details on res_tra options, see this presentation by James G. MacKinnon (2012) and references therein)

"mbeta": A heuristic Bayesian approach which is based on a multivariate beta-binomial model.

nrep: number of posterior draws (default: 5000)
lfc_pr: prior probability of 'least-favorable parameter configuration' (default: 1 if analysis == "co-primary", 0 if analysis == "full").

Examples

#
data <- draw_data_roc()
evaluate(data)
#> [cases] evaluation results:
#> $specificity
#>    parameter hypothesis estimate  lower  upper   tstat   pval reject pval_all
#> 1     model1     == 0.5     0.76 0.6428 0.8772  4.3476 0.0000   TRUE   0.0000
#> 2     model2     == 0.5     0.74 0.6196 0.8604  3.9075 0.0001   TRUE   0.0001
#> 3     model3     == 0.5     0.80 0.6902 0.9098  5.3561 0.0000   TRUE   0.0000
#> 4     model4     == 0.5     0.84 0.7394 0.9406  6.6231 0.0000   TRUE   0.0000
#> 5     model5     == 0.5     0.88 0.7908 0.9692  8.3510 0.0000   TRUE   0.0000
#> 6     model6     == 0.5     0.96 0.9062 1.0138 16.7640 0.0000   TRUE   0.0018
#> 7     model7     == 0.5     0.68 0.5520 0.8080  2.7557 0.0059   TRUE   0.0059
#> 8     model8     == 0.5     0.84 0.7394 0.9406  6.6231 0.0000   TRUE   0.0000
#> 9     model9     == 0.5     0.94 0.8748 1.0052 13.2312 0.0000   TRUE   0.0000
#> 10   model10     == 0.5     0.96 0.9062 1.0138 16.7640 0.0000   TRUE   0.0000
#>    reject_all
#> 1        TRUE
#> 2        TRUE
#> 3        TRUE
#> 4        TRUE
#> 5        TRUE
#> 6        TRUE
#> 7        TRUE
#> 8        TRUE
#> 9        TRUE
#> 10       TRUE
#> 
#> $sensitivity
#>    parameter hypothesis estimate  lower  upper   tstat   pval reject pval_all
#> 1     model1     == 0.5     0.78 0.6663 0.8937  4.8271 0.0000   TRUE   0.0000
#> 2     model2     == 0.5     0.88 0.7908 0.9692  8.3510 0.0000   TRUE   0.0001
#> 3     model3     == 0.5     0.78 0.6663 0.8937  4.8271 0.0000   TRUE   0.0000
#> 4     model4     == 0.5     0.90 0.8177 0.9823  9.5219 0.0000   TRUE   0.0000
#> 5     model5     == 0.5     0.88 0.7908 0.9692  8.3510 0.0000   TRUE   0.0000
#> 6     model6     == 0.5     0.70 0.5742 0.8258  3.1168 0.0018   TRUE   0.0018
#> 7     model7     == 0.5     0.92 0.8455 0.9945 11.0559 0.0000   TRUE   0.0059
#> 8     model8     == 0.5     0.86 0.7648 0.9552  7.4093 0.0000   TRUE   0.0000
#> 9     model9     == 0.5     0.84 0.7394 0.9406  6.6231 0.0000   TRUE   0.0000
#> 10   model10     == 0.5     0.80 0.6902 0.9098  5.3561 0.0000   TRUE   0.0000
#>    reject_all
#> 1        TRUE
#> 2        TRUE
#> 3        TRUE
#> 4        TRUE
#> 5        TRUE
#> 6        TRUE
#> 7        TRUE
#> 8        TRUE
#> 9        TRUE
#> 10       TRUE
#>