After a candidate submits a prediction file, we assign a score or evaluate the model performance only on 50% of the test data set. The evaluation is based on the evaluation metrics that we define for a problem.
In this case, we have defined the evaluation metric as, Score = No. of correct predictions/total rows.
In the file that is submitted by the candidate during the online phase, we consider the first 50% of the test data i.e. IDs 1 and 2. There is only 1 correct prediction i.e. ID 1.
Therefore, the score (based on the defined formula) for the first two rows is calculated by using the formula Score online = ½ = 0.5.
After the contest is over, we re-evaluate the file and assign a score to the complete data set. The score is calculated by using the formula Score offline = ¼ = 0.25.
Therefore, the score after the offline evaluation is subject to change.
HackerEarth evaluates only 50% of the data set during the online phase. We do this because of the overfitting practice where candidates may try to maximize their score during the online phase and their models may not be generic enough to perform well on data outside of what was used in the training set. The practice of evaluating only 50% of the test data set during the online phase is to discourage overfitting.