Gratzl, S., A. Lex, N. Gehlenborg, H. Pfister, and M. Streit. 2013. “LineUp: Visual Analysis of Multi-Attribute Rankings.” IEEE Transactions on Visualization and Computer Graphics 19 (12): 2277–86. doi:10.1109/TVCG.2013.173.
“Abstract—Rankings are a popular and universal approach to structuring otherwise unorganized collections of items by computing a rank for each item based on the value of one or more of its attributes. This allows us, for example, to prioritize tasks or to evaluate the performance of products relative to each other. While the visualization of a ranking itself is straightforward, its interpretation is not, because the rank of an item represents only a summary of a potentially complicated relationship between its attributes and those of the other items. It is also common that alternative rankings exist which need to be compared and analyzed to gain insight into how multiple heterogeneous attributes affect the rankings. Advanced visual exploration tools are needed to make this process ef?cient. In this paper we present a comprehensive analysis of requirements for the visualization of multi-attribute rankings. Based on these considerations, we propose LineUp – a novel and scalable visualization technique that uses bar charts. This interactive technique supports the ranking of items based on multiple heterogeneous attributes with different scales and semantics. It enables users to interactively combine attributes and ?exibly re?ne parameters to explore the effect of changes in the attribute combination. This process can be employed to derive actionable insights as to which attributes of an item need to be modi?ed in order for its rank to change. Additionally, through integration of slope graphs, LineUp can also be used to compare multiple alternative rankings on the same set of items, for example, over time or across different attribute combinations. We evaluate the effectiveness of the proposed multi-attribute visualization technique in a qualitative study. The study shows that users are able to successfully solve complex ranking tasks in a short period of time.”
“In this paper we propose a new technique that addresses the limitations of existing methods and is motivated by a comprehensive analysis of requirements of multi-attribute rankings considering various domains, which is the ?rst contribution of this paper. Based on this analysis, we present our second contribution, the design and implementation of LineUp, a visual analysis technique for creating, re?ning, and exploring rankings based on complex combinations of attributes. We demonstrate the application of LineUp in two use cases in which we explore and analyze university rankings and nutrition data. We evaluate LineUp in a qualitative study that demonstrates the utility of our approach. The evaluation shows that users are able to solve complex ranking tasks in a short period of time.”
Rick Davies comment: I have been a long time advocate of the usefullness of ranking measures in evaluation, because they can combine subjective judgements with numerical values. This tool is focused on ways of visualising and manipulating existing data rather than elicitation of the ranking data (a seperate and important issue of its own). It includes lot of options for weighting different attributes to produce overall ranking scores
Free open source software, instructions, example data sets, introductory videos and more available here
