Fireman's Fund Usability Report Sample

Introduction

This document details findings from an integrated EPAS/BASS usability study conducted between February 25 – March 4 of 2011. In the study, 11 users with prior exposure to the Enterprise Policy Administration System (EPAS) participated in one of two usability sessions:

• General Market home (five participants)
• General Market auto (six participants)

As time permitted, tasks related to the new Online Bill Pay feature were incorporated into one of the two primary testing sessions. Five participants in all worked through test scenarios related to Online Bill Pay.

Intended Audience

It is assumed that readers of this document already have familiarity with the tested applications. This report provides a detailed accounting of the observed usability issues over the course of the two primary studies.

The document begins with a high-level, executive summary of the usability findings, followed by a number of video files containing audio and screen captures of actual session highlights that underscore some of the more important findings.

Following this, the report delves into page level accountings of the issues encountered. Where usability issues were observed, a screenshot of the relevant application page is included for reference, along with a tabular presentation of the observed issues and recommendations as to whether individual issues should be addressed in pre-rollout training materials and whether the issue should be addressed via a future application enhancement. Issues are categorized according to the following criteria:

• Serious – Issues that indicate a fundamental conceptual disconnect on the part of users, or underlying problem states that will inhibit real-world users from successfully completing tasks using the underlying systems.
• Moderate – Slightly less severe conceptual disconnects. Users in most cases are able to successfully complete system tasks, but express some degree of confusion.
• Minor – Issues that don’t interfere with users’ ability to complete system tasks and are not indicative of a substantial conceptual disconnect. Many of these issues are related to terminology or less important cosmetic/fit-and-finish/industry standards.

As many pages are similar or identical across both the Home and Auto tests, and many are repeated in each test across the Quote and Policy work-flows, there is a fair amount of redundancy in the reported page level findings. While not ideal, this redundancy seemed preferable to forcing document consumers to page back to an earlier section of the report in cases of redundant findings. While this does contribute to a substantially larger report, the tradeoff in increased ease-of-use for document consumers seemed to warrant this approach.

Session Participants

With assistance from our EPAS, BASS, and DSM partners, user profiles were developed and candidates were screened according to these user profiles. All participants were extensively vetted by our DSM partners to ensure their suitability for the testing process. All are Fireman’s Fund champions and advocates, but were also extremely forthright and candid in their assessment of the current state of our various FFIC web-based applications.

The following table provides information about the eleven subjects who ultimately participated in one of testing sessions.

Testing Protocol

All subjects participated using a remote usability protocol, meaning that the usability facilitators hosted the sessions from the San Marin office and participants participated from their offices via a Web Ex connection and AT&T call-in number. Participants were granted remote control of the facilitator’s test system, and drove prototype versions of the EPAS, BASS, and Online Bill Pay systems. A more detailed accounting of the testing protocol is provided in the Recruiting Backgrounder companion document.

Executive Summary

The following sections detail high-level takeaways (both good and bad) from the sessions. A more detailed, page level accounting that correlates to these high-level findings can be found in the ensuing sections.

Based on agent feedback from the sessions, we anticipate that agency users who are familiar with EPAS will be able to use the new capabilities with the support of communications and training. Without supporting communication and training, we expect significant agent confusion and inability to complete certain key tasks using the new systems.

Positive Findings

• Those users with prior experience using EPAS to generate Auto lines of business noticed and responded favorably to the E3 enhancements.
• Those users with significant prior experience using EPAS noted solid improvement in the application over the last couple of release.
• While not fully conscious of what the differences were, users responded favorably to the improved readability and less overwhelming volume of data on the redesigned Quote Summary page in EPAS Homeowners.
• Users who triggered error conditions in the BASS workflows noticed the error messages and were able to use them to fix the errors.
• Users responded favorably to the general concept of the new Online Bill Pay feature.
• Agents acknowledged/confirmed that their job responsibilities include performing billing-related activities and answering bill questions for customers.
• One CSR reported she must see exactly what policyholders can see because:
• She needs to be able to walk them through billing-related tasks.
• She wants to make payments on their behalf.

Areas for Improvement

Three issues emerged as dominant trends over the course of the sessions:

• Users seemed completely unaware that they were working with three different internal systems, which led to mistaken assumptions and confusion when the systems behaved slightly differently.
• At various points in the study, users expressed a complete lack of confidence with respect to where they were in the overall process of creating a quote; issuing a policy; and creating a billing account.
• One particularly problematic area was Mortgagee Search.
• Users didn’t like having iLog errors displayed on the Action pages in EPAS.
• All expressed a strong preference for having these displayed in Quote, or (if in Policy) presented before the Action page is reached.
• Users firmly believed that billing account setup should occur prior to policy issuance.
• Key elements required for issuing in both the Property and Auto tests were hidden on secondary pages.
• Participants expressed a strong preference for integrating these pages into the primary work-flows.
• In general, all users felt too many pages were required to complete the tasks; too much redundant data was displayed; pages were too busy/wordy; and too many extraneous mouse clicks were required.