Analysis of reporting quality for oral presentations of observational studies at 19 National Surgical Congress: Proposal for a national evaluation system.

Abstract

To compare the quality of oral presentations presented at the 19 National Surgical Congress with a national evaluation system with respect to the applicability of systems, and consistency between systems and reviewers.

Fifty randomly selected observational studies, which were blinded for author and institute information, were evaluated by using the Strengthening the Reporting of Observational Studies (STROBE), Timmer Score, and National Evaluation System by two reviewers. Abstract scores, evaluation periods, and compatibility between reviewers were compared for each evaluation system. Abstract scores by three different evaluation systems were regarded as the main outcome. Wilcoxon matched-pairs signed rank and Friedman tests for comparison of scores and times, kappa analysis for compatibility between reviewers, and Spearman correlation for analysis of reviewers based on pairs of evaluation systems were used.

There was no significant difference between abstract scores for each system (p>0.05). A significant difference for evaluation period of reviewers was detected for each system (p<0.05). Compatibility between reviewers was the highest for the Timmer Score (medium, κ=0.523), and the compatibility for STROBE and National Evaluation System was regarded as acceptable (κ=0.394 and κ=0.354, respectively). Assessment of reviewers for pairs of evaluation systems revealed that scores increased in the same direction with each other significantly (p<0.05).

The National Evaluation System is an appropriate method for evaluation of conference abstracts due to the consistent results between the referees similarly with the current international evaluation systems and ease of applicability with regard to evaluation period.