Presented by: Shouhao Zhou, PhD, Assistant Professor, Division of Biostatistics and Bioinformatics; and Zizhong Tian, MS, PhD Candidate

The accurate assessment of inter-rater agreement (IRA) for multiple-rater binary-response data is a pivotal task in psychological, social, and medical sciences. In this work, we introduce an advanced framework exploiting multivariate Bernoulli distribution to encapsulate the statistical nature of binary rating responses. Leveraging a versatile f-parameterization, we propose the agreement factor, a new and natural cross-product ratio measure for effective IRA assessment. Simulation studies are conducted to demonstrate agreement factor’s estimation accuracy and its superiority in terms of the empirical performances over the Phi coefficient, Cohen’s k, and Gwet’s AC₁ under the most common two-rater case, especially the extreme-prevalence scenario. Furthermore, we demonstrate the flexibility of the proposed framework to address the missing data scenarios, or to extend into more complex models that can assess overall IRA while accounting for covariate information or hierarchical data structures. The practical application of our method is illustrated through real-world examples.