Giving Residents a Voice, Remotely: Civic User Testing Amid COVID

As our lives move increasingly online, the pandemic has underscored the need for new technology to keep us connected and deliver efficient services. The success of new public-facing technologies, however, relies on direct resident engagement and community-focused design. Pre-COVID, City Tech generally conducted our resident engagement programs in person at places such as libraries and community centers, but closures, shelter-in-place orders, and public health concerns have added new access complications.

Amid the outbreak, City Tech Collaborative and Civic User Testing group (CUTgroup) Cleveland had been working with the civic tech group unBail to test an app that helps court defendants and their families navigate their pre-trial information; originally planned for in-person testing in March 2020, the team pivoted to remote testing in late August 2020. Although conducting testing remotely raises some barriers, including residents in the development and deployment of these solutions is more important than ever.

City Tech believes that public-facing technology must serve all residents; through our resident engagement program, we offer guidance, convene residents, and conduct specialized usability testing to support the development and roll-out of digital tools, websites, technology plans, and civic products. Our 1,600+ member Civic User Testing group (CUTgroup) invites Chicago residents to contribute to emerging technology while providing public, private, and social sector partners with feedback to improve product design and deployment; the success of this program has led to City Tech launching and coordinating the new Civic User Testing Network, a national network of Civic User Testing groups to share resources and develop best practices, thus building a more robust framework to make local technology more user-friendly, accessible, and relevant. Civic User Testing Network members include Detroit, Toronto, Miami, CUTgroup Cleveland, and City Tech’s Chicago CUTgroup.

CUTgroup Cleveland began when the Cleveland Foundation approached City Tech to help develop their own Civic User Testing group. City Tech worked with the Cleveland Foundation and Neighborhood Connections, a nationally recognized community-building program, to establish the program. CUTgroup Cleveland now has 132 members. Organizers of CUTgroup Cleveland envision the initiative as an authentic means to engaging the community to address and overcome specific barriers to equity and justice.

City Tech partnered with the newly established CUTgroup Cleveland in conducting their inaugural usability test. The team tested the wireframe of a new app from unBail, an organization originating from a scope-a-thon hosted by Cleveland Foundation and Hack Cleveland. Before engaging with the app, many testers reported that they found the legal process confusing and difficult to navigate; while some did not have issues accessing information, whether through a combination of court websites and law offices/attorneys, most wished there were an easier way. The app seeks to solve this by delivering valuable and relevant legal information to defendants and their families and will help defendants in both municipal and county court systems facing felony or misdemeanor charges navigate their pre-trial information. The app is also meant to serve family members and friends of a defendant who is being held in jail while awaiting trial.

Originally scheduled for in-person testing in March 2020, the team pivoted to remote testing in August 2020 due to circumstances of the pandemic. While remote testing creates some access concerns, it removes others. No longer reliant on a central community space to conduct tests, remote testing offers flexibility to those who would otherwise not be able to attend due to issues such as lack of transportation access, lack of childcare, or conflicting working schedules.

Public-facing technology must serve all residents, regardless of their background and skills, and therefore should include feedback from a variety of users. To facilitate this, CUTgroup does not require participants to have specific digital skills, technology access, or specific level of connectivity; although these aspects are important to understand how different users interact with a technology when testing it, these can also be barriers when needing to conduct tests remotely. To address these concerns for testing the unBail app, CUTgroup Cleveland dropped off sanitized technology kits to testers who did not have access to the needed technology or lacked connectivity. Each kit contained laptops to complete the tests and internet hotspots.

Usability testing not only gains verbal responses from the tester, but it also relies on visual observations from the proctor of how the tester is interacting with the app, including the completion of key tasks to assess how well the tester can navigate the site and find and understand information. To mitigate this, the team trained proctors in advance and conducted tests via Zoom with screenshare; sessions were also recorded with consent so that results could be reviewed more closely during analysis.

“This initial test with unBail displayed the power of community engagement in the design process for social impact applications. Residents provided concrete feedback that will make the app even more accessible to those who will use it,” said Curt Williams, Digital Innovation Fellow at the Cleveland Foundation. “As we move forward, we hope to expand from this initial test to fully include the community in the solution design process. We cannot move forward with our work to address causes of systemic injustice without understanding how that injustice is affecting the communities we hope to serve.”

Focusing on Cleveland residents who had experience with municipal or county court systems and the criminal legal system, CUTgroup Cleveland recruited and tested the app with 13 diverse testers who provided insight on their experience with the legal system.

All but one tester had familiarity with the criminal legal system, with six being personally convicted of a felony or misdemeanor. Testers commented on their experienced with the criminal legal system, reviewed the app, completed specific tasks within the app, and shared their overall thoughts.

After the test, City Tech and CUTgroup Cleveland analyzed the results and were able to present unBail with a summary, details on each of the specific tasks, and key recommendations based on tester feedback.

• Overall, testers were very excited about unBail’s app. Many testers expressed frustration in their past experiences navigating the system and over half of the testers did not or could not access pre-trial information at the time of the test. Those that did, however, found the process difficult.
• On a 1-5 scale with 1 being “Very Difficult” and 5 being “Very Easy”, testers rated the app an average of 4 for ease of use.
• 61% of testers said that they were “Very Likely” to use the website if it existed because the information is important, accessible, and there are not many other sources for it. All but one of the remaining testers were “somewhat likely” to use the app, but it would depend on individual case needs.
• Overall, the testers like the ease of the layout, but wanted a few tweaks to more easily access important information and some personalization to better understand how they fit into the timeline of the case.
• City Tech and CUTgroup Cleveland recommended achieving this by organizing information by case timeline so it will be available when they need it, with dedicated sections for the most important information. Another suggestion included markers that correspond to which step of the process their case is in, so they know where to go for current information needs. Additionally, due to the vast amount of information in the app, testers suggested adding some sort of onboarding or directions so users could utilize the app to the fullest.

The unBail team has already incorporated the feedback from the user test into the app prototype, and is now preparing to release a request for proposal to developers in order to translate the prototype into a cross-platform mobile app. The team may also conduct additional usability testing for a beta version of the unBail app with end users.

“CUTgroup provided us our first opportunity to get our product in front of potential users. They conducted a thorough and precise user test that provided us with detailed feedback,” said Cait Kennedy, Executive Director and Co-Founder of unBail. The user test significantly increased our organizational capacity, but more importantly, affirmed that we are moving in the right direction to best serve our users.”

Resident engagement ensures that technology is accessible and created with the end user in mind; this is especially important as COVID-19 creates the need for new technology-enabled solutions. While traditional resident engagement programs are conducted in person, the CUTgroup Cleveland test showed that there is room to pivot and allow for remote testing. Through the wireframe testing, unBail will be able to improve their app to be more accessible, user-friendly and create more resources that align with their users’ needs. The success of this test has shown that we must be even more mindful of technology access, connectivity barriers, and digital skills and provide resources and support to ensure inclusivity when conducting remote testing. It is difficult to know when we will be able to move back to conducting tests in community locations, but we have seen that perhaps a combination of in-person testing and remote testing holds potential for the future.

To learn more about City Tech’s resident engagement program and the unBail case study, visit CityTech.org/Resident-Engagement.

Click here to review the full CUTgroup Cleveland test results.

To learn more about CUTgroup Cleveland and to join their group of testers, click here .

About City Tech Collaborative (City Tech): City Tech is an urban solutions accelerator that tackles problems too big for any single sector or organization to solve alone. City Tech’s work uses IoT sensing networks, advanced analytics, and urban design to create scalable, market ready solutions. Current initiatives address advanced mobility, healthy cities, connected infrastructure, and emerging growth opportunities. City Tech was born and raised in Chicago, and every city is a potential partner. Visit www.CityTech.org and follow us on Twitter and LinkedIn.

About CUTgroup: City Tech's Civic User Testing group (CUTgroup) is a 1,600+ member civic engagement program that invites Chicago residents to contribute to emerging technology while providing public, private, and social sector partners with feedback to improve product design and deployment. Learn more at www.CityTech.org/CUTgroup.