Formative and validative human factors studies for automated insulin dosing app

Assessing task performance for the use of critical safety-related features ahead of clinical trials

Screenshot of Tidepool Loop automated insulin dosing app. The app has a blood glucose reading, a green status circle indicator, several graphs including blood glucose and active insulin, and a toolbar at the bottom with five icons.
Journey map points icon
Flexible Problem-Solving

Re-wrote protocols and moderators guides on a short deadline to change a formative study to a validation study, redesigning tasks in order to produce data that could be submitted to the FDA

Magnifying glass icon
Root Cause Analysis

Rapid scorekeeping and note-taking during sessions, conducting root cause probing and analysis

List icon
Project Management

Project management and research operations for in-person study including consent forms, prototype management and troubleshooting, and running recordings and live streams including webcams and screen mirroring

Report icon
Report Writing

Drafted and submitted materials for 510(k) submission for FDA approval ahead of a clinical trial

Impact

Number 3 icon
Drafted documentation of risk analysis for successful FDA submission; approved to move forward with clinical trial
Number 2 icon
Recommended design updates to the app for tasks with an unacceptably high number of use errors
Number one icon
Ensured that studies were organized, well-documented, and ran on schedule
Diagram of three components of the Tidepool Loop Automated Insulin Dosing system. There is an icon of an iPhone with the app open, and it's in a circle with two icons, one which looks like a continuous glucose monitor and one which looks like an insulin patch.

Overview: Tidepool Loop Automated Insulin Dosing App

The Tidepool Loop app is interoperable with different pumps and CGMs, so it needs to go through human factors validation studies with each new medical device partner in order to validate device-specific tasks such as filling the pump with insulin or changing out the CGM sensor.

Tidepool Loop helps users to stay in their preferred blood glucose range by taking in blood glucose readings from a bluetooth-paired continuous glucose monitor (CGM) and adjusting the amount of insulin the bluetooth-paired insulin pump is delivering, updating as often as every five minutes

Research goal: confirm whether users can successfully complete a series of usability-related and risk-related tasks in the app
Magnifying glass icon
2 formative studies conducted over zoom
1 validation study in-person
DESIGN RECOMMENDATIONS FOR CRITICAL SAFETY RISKS
Diagram of three components of the Tidepool Loop Automated Insulin Dosing system, with broken "x" symbols between parts. There is an icon of an iPhone with the app open, and it's in a circle with two icons, one which looks like a continuous glucose monitor and one which looks like an insulin patch. There is an "x" cutting off the circle between the phone and the insulin patch, and the insulin patch and the CGM.
Pausing Insulin Delivery

Several users hesitated to take action to suspend all insulin delivery, due to negative transfer from previous experiences navigating a different insulin pump app. Participants ultimately resolved their difficulty after exploration or a simulated call to customer support.

The risk of a use event was evaluated with the regulatory team and there was an acceptable level of post-mitigation risk, so we did not recommend any further design changes.

Diagram of three components of the Tidepool Loop Automated Insulin Dosing system. There is an icon of an iPhone with the app open, and it's in a circle with two icons, one which looks like a continuous glucose monitor and one which looks like an insulin patch. There is no line between the insulin pump and the CGM, signifying that they are not communicating with each other.
Recognizing when automation is turned off

During formative studies, several users experienced perception difficulties with the terminology for automation, because they did not associate it with “closed loop,” a term that comes from DIY Loop. We recommended adding a pop-up when a user taps on the icon explaining if automation is currently on or off.

During validation, there were zero use errors and only one close call with a participant who ultimately read the pop-up and self-corrected. Because this event is unlikely to occur after initial use, we did not recommend any further design changes.

Image of tidepool loop app with an x-ed out circle and alarm symbol, indicating that the phone is silenced.
Muting sounds and alerts coming from the app

During formative testing, several users expressed the belief that they would still hear some high-priority critical alerts with “mute sounds” enabled, due to negative transfer from other diabetes devices which do not allow the user to mute critical alerts.

The in-app text explaining “mute sounds” is small and gray, and users did not perceive it. The risk of a use event was evaluated with the regulatory team and there was an acceptable level of post-mitigation risk with the design recommendation to make the explanatory text more visually salient.

Study Details

Human factors testing setup of several items on a desk underneath a webcam. The items include a sharps container, an iphone, several paper booklets, insulin vials, hand sanitizer, and a CGM sensor and transmitter.
Study planning

During the second formative testing I co-authored the protocol and moderator’s guide for the third and final study

Adapting to new time constraints

The deadline to start clinical trial was moved earlier, so we decided that there was a low enough level of errors to adapt the third formative study into a validation study, which allowed us to submit the results of that study to the FDA

Getting ready to conduct a validation study

I coordinated with the risk and regulatory team to determine the appropriate level of risk for any use events that might arise during the study tests according to the URRA (use related risk analysis) and ensured consistency across documentation

Running an in-person validation study

I coordinated and tracked the pre-study training sessions and user testing sessions with 3 certified diabetes education specialists, 4 moderators, 4 note-takers and 18 participants. I set up all study session recording and streaming, including several camera angles and iPhone screen mirroring

Screenshot of a recording from a study session. In the top left corner there is an overhead view of someone's hand on a phone, in the bottom left there is a view of the room with the participant and the moderator, and on the right side there is a large view of the phone screen which is directly mirrored.
Image of participant wearing a pink shirt sitting in a user testing lab with materials including an insulin pump and an instruction manual on the table in front of her.
Screenshot of a recording from a study session. In the top left corner there is an overhead view of someone's hand on a phone, in the bottom left there is a view of the room with the participant, and on the right side there is a large view of the phone screen which is directly mirrored.
Screenshot of a recording from a study session. In the top left corner there is an overhead view of someone's hand on a phone, in the bottom left there is a view of the room with the participant, and on the right side there is a large view of the phone screen which is directly mirrored.

Results and Report Writing

Screenshot of HFE report. It reads "Table 8-34. Close call(s) on tasks associated with the potential for serious harm," and "Table 8-35. Related risk and potential harms." The contents of the table are blocked out. Below the tables it reads "Tidepool's root cause analysis: negative transfer when using the Product from previous experience with insulin pump. Navigation was unintuitive based on expectations."
Number one icon
510(k) submission

Drafted a description of use events with participant-reported root causes, and conducted a root cause analysis of each use error and close call.

For tasks associated with potential for serious harm, identified current risk mitigation efforts and residual risk commentary.

Described any planned changes to the UI ahead of clinical trials and whether or not the changes would require further validation studies

Number 2 icon
Stakeholder presentations

Uploaded and organized 80+ hours of recordings and transcripts from study sessions

Aggregated video and audio highlight reels to demonstrate overall positive feedback, areas for improvement, and any insights related to tasks with a high number of use events

RELATED PROJECTS

Screenshot of a usability testing session. The screen is split. On the left half of the screen, there is a prototype of an automated insulin dosing app which has an error reading "pump expired" in the top right corner. On the right side of the screen, there are video boxes with Caroline, the moderator, and a parent-child dyad research participants whose faces have been covered to protect their privacy.
Diabetes app usability testing with caregivers

Rapid Iteration Testing and Evaluation (RITE), usability testing, moderating parent-child dyads
Photograph of a woman wearing all black and sitting in a red wheelchair, using a reacher-grabber tool with an outstretched arm and grabbing a can of Goya kidney beans.
Assistive grabbing device for wheelchair users

Interviewing, competitor usability testing, identifying personas, prototyping