Ongoing app optimization

Timeline - Ongoing since January 2020

Project overview

Lumin Digital provides a white-label digital banking platform for credit unions, giving members access to a modern banking mobile app and website. There are currently 14 credit unions on Lumin's platform with over several hundred thousand active members. Lumin wanted a way to track the health of their app through credit union member feedback.

Solution

I created a system of tracking and tagging customer feedback to keep a qualitative pulse on the app's health. This system has allowed us to easily analyze common themes that members are experiencing in order to fix bugs, address usability issues, identify feature requests, and constantly improve the user experience of the app.

Goals

Identify themes between individual credit unions, as well as across all credit unions.
Provide qualitative data to inform product decisions.

My role

User research

The set-up

Lumin utilized Apptentive as their tool to collect in-app feedback and to request app store ratings from members. Apptentive also uses a natural language processor to identify feedback sentiment–positive, negative, or neutral. While all of these features are helpful, using Apptentive to synthesize the data wasn't feasible because each instance of Lumin's product is set up as a stand-alone app within Apptentive.

I needed to find a solution that would allow me to conduct thematic analysis across all of the credit unions on the platform.

Screenshot of Appentive. Each instance of the credit unions on Lumin's platform is displayed on the left.

I required a tool that would allow me to easily aggregate, tag, sort, and filter the data exported from Apptentive. I chose to use Airtable primarily because of how easy it is to create multi-select options. This was key for me in order to accurately tag feedback.

Member feedback is often about more than one issue, making it necessary to have multiple tags.

Creating the structure

The next step was to set up how the data would be structured within the table. I knew that I wanted to be able to slice the data by Date, Credit Union, OS, Sentiment, and Theme. These attributes would be the columns in the table, along with a column for the feedback.

Using this structure, I can easily filter the data to answer questions related to app performance.

Questions like:

How often has this issue occurred over the past quarter?
How is sentiment tracking for each credit union over time?
What area of the app are members having the most trouble with this month?
Is this bug specific to users on Android or iOS?

Example of a graph tracking sentiment across all credit unions over 4 months.

Tagging feedback

I set up two types of tags in order to track feedback and identify common themes.

Section/Theme

The first type of tag categorizes the feedback at a high level, grouping the feedback by either the section or user experience of the app.

User experience type feedback includes themes such as "Layout/UI", "Usability", and "Readability".

Sections of the app include areas like, "Accounts", "Bill Pay", "Transfers", and "Mobile Deposit". One benefit of categorizing the feedback at a section level is that I can create specific views for product managers. This allows them to keep a pulse on the sections that they are responsible for.

List of tags within the Section/Theme column.

Sub-theme

The second type of tag adds a level of granularity to the feedback. Sub-theme tags are the specific features, functionality, or details, within a section of the app.

For example, there might be a lot of problems with users being able to log into the app. Tagging the feedback by sub-theme can let us know how many login issues are related to features like TouchID versus FaceID and help the development team pinpoint bugs.

I'm able to graph the themes and sub-themes each month to get an overview of where members had the most negative feedback.

Reporting

There are several outputs from the member feedback that get delivered to the Lumin product team.

Monthly reporting

I create monthly reports for Lumin's Design Director and Director of Product Management. These reports include a summary of the sentiment for the month and highlight themes that the team should be aware of. The monthly reports are shared with the rest of the PMs and help them to keep track of the health of the app.

New credit union launches

New credit union launch reports track member feedback over the first month when a credit union switches its digital banking platform provider to Lumin. We have found that there is usually a spike in negative feedback from members in the first three months while they adjust to a new interface. These reports also allow Lumin to proactively address issues in future launches.

Section views

Section views are created in Airtable and shared with the product managers. Product managers can filter and slice the data in these views to research feedback on their own time. Below is an example of member feedback for the Accounts section of the app.