|
Performance is evaluated given a predefined series of user queries and manually checked results (the GROUND-TRUTH). In our case those Carte de Visite front images were taken of which exactly 2 printed copies happen to be present in our integrated Dutch Carte de Visite Portrait database.In turn each of the copy pair images is used to retrieve its counterpart. Therefore the size of the cluster of relevant answers is always 1 for each query in the testset The testset of copy pairs can be embedded into a database of irrelevant items of varying size (when the embedding consists of like images one speaks about a NARROW-DOMAIN EMBEDDING; when embedded into dissimilar images one speaks about a wide-domain embedding. In our case the total number of front images in the Leiden 19th-Century Portrait Database is used for a narrow-domain embedding. Each image in the database is characterized and indexed by a set of features (like histogram, projections, trigrams, local binary patterns etc) and the differences of pairwise comparisons of the feature elements are transformed into a distance (several metrics are possible here). Hence a DISTANCE MATRIX of all images versus all other images is obtained. This METHOD determines how the distance matrix will be filled. To determine the answer given to a content-based query with one of the database images, its row of distance values is sorted into a RANKING list of which the top (SCOPE) is shown to the user. We evaluate the succes of image indexing and retrieval by presenting results to the user as if a TOTAL RECALL IDEAL SYSTEM was used; such a system would present the user the relevant images, no more, no less. Hence a scope of 1 will be used for the retrieval of copy pair counterparts. Under these circumstances PRECISION=RECALL and its exact average value for the complete testset will depend on the embedding and method used. Since the embedding is fixed, it will only depend on the method. The embedding is characterized by its GENERALITY value (1/databasesize). Performance by the fraction of TRIS performance. In the following method comparison the testpairs where only embedded within the total testset size which is 436. |
|
One can use either a single method and see a list of pairwise matching ranks plus performance figure or one can select several methods (by holding CTRL-key down) to obtain a table with performance figures for selected method |
|
|
test set
|
|---|
Back to home page