Prescriptive competitors may register on the web page and download datasets. Every dataset consists of two parts: training data - set of known (labeled) samples, and test data - set of samples with unknown classification (unlabeled). Competitors may analyze labeled data, build classifiers and try to classify unlabeled samples from test data file. Tests results and description of methodologies will be published on this web page and will be the subject of presentation during BTAS 2012 (The Fifth IEEE International Conference on Biometrics: Theory, Applications and Systems, September 23-27, Washington DC, USA).
Results of classification should be sent in a text file. The file should contain a line for every sample in test file (every test file for Datasets C and D) containing the result in format below:
taskid:sid1:prob,sid2:prob [...]
where:
taskid
- sample number for datasets A and B or classified file name (like task_001
) for datasets C and D
sid
- subject's identifier
prob
- calculated probability that this sample belongs to sid
(in range 0-1)
Number of sid:prob
is not specified. It may be just one sid:1
for strong classifiers or a list of all sids
and probabilities for weak classifiers. We encourage competitors to send sets of sid:prob
pairs to enable participation in metrics other than simple accuracy. The probabilities don't need to sum up to 1.
The main metric used for evaluation will be ACC1. ACC1 is defined as the number of test samples classified correctly to the whole number of test samples. Correct classification is when correct subject_id gets the highest probability.
There will be several additional metrics tested as well. It will be ACCn - the number of samples for which correct sid is among the N sids with the highest probability. Results with several sids
will be also used to emulate users verification (checking if the given sample belongs to a given subject with true
or false
as the result).
If more than one sid
has the same probability assigned, none is taken.
If the result for sample is: sidA:0.7,sidB:0.7,sidC:0.3,sidD:0.3