Evaluating Information Visualization in Large Companies

Researchers

Summary

This project is a result of the PhD work of Michael Sedlmair. It summarizes a number of challenges and recommendations arising in the context of information visualization evaluation in large companies with several thousand employees. Our papers describe a collection of evaluation challenges, discuss our own experiences conducting information visualization evaluation within the context of a large automotive company, and present a set of recommendations derived from our experiences. The set of challenges and recommendations can aid researchers and practitioners in preparing and conducting evaluations of their products within a large company setting.

Challenges and Recommendations

In the paper we list and explain the following challenges and recommendations for evaluating information visualization or visual analytics tool in a large company context:

Challenges:

  • Integrating Tools in Daily Work Processes
  • Getting the Data
  • Choosing an Evaluation Context
  • Finding Domain Expert Participants
  • Attachment to Conventional Techniques
  • Confi dentiality of Information
  • Complex Work Processes
  • Convincing the Stakeholders
  • Publishing

Recommendations

  • Familiarize yourself with currently used software environments
  • Overcome technical obstacles of data integration
  • Choose your study environment with care
  • The magic one hour limit
  • Convince your target audience
  • Delight with usability and aesthetics
  • Learn from the experts
  • Try to get a license, do studies in any case
  • Be in constant, close cooperation
  • The magic metric: Money
  • Factor in high skill with current techniques
  • Clarify publishing conditions upfront

References

Michael Sedlmair, Petra Isenberg, Dominikus Baur, and Andreas Butz (2011) Information Visualization Evaluation in Large Companies: Challenges, Experiences and Recommendations. Information Visualization Journal, 10(3):248–266, July 2011.
×

BibTeX entry:


@ARTICLE{Sedlmair:2011:EIV, author = {Michael Sedlmair and Petra Isenberg and Dominikus Baur and Andreas Butz}, title = {I}nformation {V}isualization {E}valuation in {L}arge {C}ompanies: {C}hallenges, {E}xperiences and {R}ecommendations, journal = {Information Visualization Journal}, year = {2011}, volume = {10}, number = {3}, month = jul, pages = {248--266}, doi = {10.1177/1473871611413099}, doi_url = {https://doi.org/10.1177/1473871611413099}, oa_hal_url = {https://hal.inria.fr/inria-00638541}, pdf = {../publications/papers/Sedlmair_2011_IVE.pdf}, }
Michael Sedlmair, Petra Isenberg, Dominikus Baur, and Andreas Butz (2010) Evaluating Information Visualization in Large Companies: Challenges, Experiences and Recommendations. In Proceedings of the CHI Workshop Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV). ACM Press, New York, pages 79–86, 2010. Received the best paper award at BELIV 2010.
×

BibTeX entry:


@INPROCEEDINGS{Sedlmair:2010:EIV, author = {Michael Sedlmair and Petra Isenberg and Dominikus Baur and Andreas Butz}, title = {E}valuating {I}nformation {V}isualization in {L}arge {C}ompanies: {C}hallenges, {E}xperiences and {R}ecommendations, booktitle = {Proceedings of the CHI Workshop Beyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV)}, year = {2010}, pages = {79--86}, publisher = {ACM Press}, address = {New York}, doi = {10.1145/2110192.2110204}, doi_url = {https://doi.org/10.1145/2110192.2110204}, oa_hal_url = {https://hal.inria.fr/inria-00587680}, pdf = {../publications/papers/Sedlmair_2009_EIV.pdf}, }

This is a related paper which talks about the evaluation process in context of a specific project:

Michael Sedlmair, Petra Isenberg, Dominikus Baur, Michael Mauerer, Christian Pigorsch, and Andreas Butz (2011) Cardiogram: Visual Analytics for Automotive Engineers. In Proceedings of the Conference on Human Factors in Computing Systems (CHI). ACM, New York, pages 1727–1736, 2011.
×

BibTeX entry:


@INPROCEEDINGS{Sedlmair:2011:CVA, author = {Michael Sedlmair and Petra Isenberg and Dominikus Baur and Michael Mauerer and Christian Pigorsch and Andreas Butz}, title = {Cardiogram: Visual Analytics for Automotive Engineers}, booktitle = {Proceedings of the Conference on Human Factors in Computing Systems (CHI)}, year = {2011}, pages = {1727--1736}, publisher = {ACM}, address = {New York}, doi = {10.1145/1978942.1979194}, doi_url = {https://doi.org/10.1145/1978942.1979194}, oa_hal_url = {https://hal.inria.fr/inria-00638544}, pdf = {../publications/papers/Sedlmair_2011_CAV.pdf}, }

back to the projects page