Challenge Events and Scoring

CPSC 2019 is comprised of two events related to scoring: QRS location and HR estimation. Only test set will be used for event scoring. QRS annotations in the training and test sets are labeled and approved by cardiologists and trained volunteers. These reference annotations have been processed to derive the reference RR interval time series (the intervals between successive QRS annotations). HR is further derived from the RR interval with a 4 s time window ranging from 5.5 s to 9.5 s in each of 10-s ECG segment.


Event 1: QRS detection

In this event, the goal is to produce a set of QRS annotations that can matches the reference QRS annotations. For each reference QRS annotation, a matched QRS annotation should lie in 75 ms duration centered by the reference QRS annotation [9]. Noted that the reference QRS annotations appear in the first and last half seconds are omitted. As shows in Figure 1, detected QRS must be within 75 ms from the reference ones. For each 10-s ECG segment, the scoring rules are:

• complete matching scores one point
• a false positive (FP) detection scores 0.7 points
• a false negative (FN) detection scores 0.3 points, since from a clinical perspective, missed diagnosis is more serious than misdiagnosis, thus we penalize FN detection here
• other situations score 0

            

The final score for Event 1 (QRSacc) can be calculated as:

                 

Event 2: HR estimation

In this event, the goal is to produce a real-time estimate for HR for a 4-s ECG episode, i.e., ECG signals from 5.5 s to 9.5 s. Figure 2 demonstrates the evaluation method for Event 2, i.e., how HRscore for each 10-s ECG segment can be obtained.



Figure 2. Demonstration of the evaluation method for Event 2. In this situation, reference HR can be calculated from the reference QRS annotations between 5.5 s and 9.5 s ECG episode (HRref = 96), and estimated HR can be calculated from the detected QRS annotations between the same 4-s ECG episode (HRtest = 76).
The detailed scoring rules for HR estimation in each 10-s ECG segment are summarized as:

                  

The final score for Event 2 (HRacc) can be calculated as:

               

where HRtest and HRref represent heart rate calculated by competitor and reference, respectively. HR estimation is usually derived from the QRS detection but we encourage participant to develop robust HR estimation without QRS location.