Forum Replies Created
-
Posts
-
If you can give out this information — does the subset change on subsequent scorings?
E.g. if our score goes up from one day to the next, can we be confident that the new submission performed better, or could small variations possibly be attributed to changes of the subset used to calculate the displayed score?
I am guessing this relates to my most recent entry (since it certainly could be seen that way). By way of explanation along with a couple of questions — I am focusing first on just the 2nd sub-problem, the “shear” dataset, so each of my submissions so far has had invalid values for the rows corresponding to the “pairs” dataset.
My first submission was something of a throwaway effort; my second was a very conservative estimator that ended up reporting very few errors. It received a score of 40 (out of a possible 255 for that data set), although it only reported a handful of errors. This confused me, because it would seem to imply that well over half of the groups in this dataset have errors, which seems unlikely.
To try to double-check of this, for my third submission I submitted a file with invalid values for all the “pairs” data, and 1 (at least one failure) for all of the “shear” data. I think this is the submission that could be seen as trying to “game” the system.
Actually, in re-reading the instructions, I think I might have fundamentally misunderstood one aspect of the provided data. I suppose my only question for the moment is whether it is legitimate to focus on one data set at a time, and to make submissions with invalid values for the rows corresponding to other data set.
