OONI Run Usability Study: Findings

OONI Run is a platform (launched in September 2017) that you can use to generate mobile deep links and widget code to coordinate OONI Probe testing. To enable community members to more effectively use OONI Run, we researched its limitations and collected structured community feedback through an OONI Run usability study.

This report outlines the objectives, personas, assumptions, and methods of this study.

We also share the outcomes based on the analysis of information collected through our survey and interviews with community members, as well as some of the next steps towards improving OONI Run.

OONI Run usability study

Objectives

The overarching objective of the OONI Run usability study (conducted in April and May 2020) is to better support decentralized community efforts around the world in coordinating OONI Probe censorship measurement efforts.

More specifically, through this usability study, we aim to improve our understanding of:

Based on community feedback collected through this usability study, we aim to release an improved version of OONI Run that is more usable and useful to the internet freedom community, supporting the coordination of censorship measurement campaigns and promoting rapid response to emergent censorship events around the world.

Personas

To meet the aforementioned objectives, this study involved the following 3 user personas:

  1. Measurement campaign coordinator. Experienced OONI Probe users who generate and share OONI Run links.This includes researchers, human rights defenders, journalists, and technologists who are part of the broader internet freedom community and who are long-term OONI Probe users. This persona has experience coordinating OONI Probe censorship measurement campaigns by generating and sharing OONI Run links with other OONI Probe users, often leading up to and during political events (such as elections and protests).
  2. Measurement campaign volunteer. OONI Probe users who receive OONI Run links (from measurement campaign coordinators) and participate in censorship measurement campaigns by running the tests and/or testing the URLs included in the shared OONI Run link(s). This persona may include individuals that participate in the broader internet freedom community, or who otherwise care about human rights.
  3. Internet freedom community member. Researchers, human rights defenders, journalists, lawyers, and technologists who are part of the broader internet freedom community and who use OONI Probe to measure internet censorship. They don’t have experience using OONI Run, but could potentially be interested in using OONI Run to generate and share custom links, or to receive OONI Run links to perform customized testing.

The main thing that the three user personas have in common is that they have all used OONI Probe to measure internet censorship. The first two user personas participate in censorship measurement campaigns, whether by coordinating them or by participating in them as a volunteer. The third user persona includes individuals who have not participated in a censorship measurement campaign before, but may potentially be interested in doing so (which is why it’s important to collect their feedback too).

Assumptions

We started off by taking note of the assumptions we have around the challenges that users currently face when using the OONI Run platform. By being aware of these assumptions we are able to validate if they are true and, more importantly, challenge them to learn if they are misguided.

Based on past community feedback, the main challenges that we suspect OONI Run users face are the following:

  1. Issues related to sharing OONI Run links due to their length;
  2. Issues related to triggering the mobile app deep link and opening the OONI Probe mobile app;
  3. Issues related to OONI Run links breaking or getting mangled by specific messaging clients;
  4. Tracking and accessing the measurements collected by measurement campaign volunteers.

In addition the the above, we also have the following assumptions:

Over the last years, we mainly received feedback from experienced OONI Probe users who used OONI Run to generate and share links.

As part of this study, our aim was to improve our understanding of the challenges that they encounter, and to receive feedback from OONI Probe users who make use of OONI Run links as well. We also aimed to better understand the needs of OONI Probe users who have not necessarily used OONI Run yet (but who could be interested in using the platform), and to understand what challenges they encounter (if any) in using the platform.

Methods

Between 23rd April 2020 to 15th May 2020, we conducted an OONI Run usability study with community members to explore the limitations to the current OONI Run link format and collect community feedback for improvement.

This study involved the following two methods:

Once we concluded the interviews on 15th May 2020, we analyzed the information collected from both the interviews and survey responses. The findings and next steps are documented below.

Findings

Survey

Unlike the OONI Probe mobile app (which is widely used by tens of thousands of users in more than 200 countries every month), OONI Run appears to have a much smaller user base. This is suggested by the fact that our previous survey for the revamp of the OONI Probe mobile app received 287 responses, whereas our recent OONI Run survey only received 30 complete responses.

The following factors though need to be taken into consideration:

Our OONI Run survey was therefore primarily completed by community members with both experience and interest in using OONI Run. We summarize the survey findings below.

OONI Run usage: Overview

Most survey respondents (60%) reported that they have primarily used OONI Run to generate and share OONI Run links, as well as to tap on an OONI Run button to perform testing. A smaller percentage (40%) reported having received an OONI Run link from others to perform coordinated testing, while only 20% of participants reported having used OONI Run for the purpose of generating an OONI Run button.

When asked how frequently they use OONI Run, 33.3% of survey respondents reported having used OONI Run only a few times per year, while 26.7% of respondents reported using OONI Run a few times per month. Only 10% of participants reported that they use OONI Run daily or weekly.

The vast majority (76.7%) of survey respondents reported that their primary reason for using OONI Run is to measure the blocking of websites, while less than 30% of the participants reported that they use the platform to run the other available tests (only 26.7% of participants reported using OONI Run for the middlebox tests, while only 20% reported using the performance tests).

Monitoring internet censorship during political events appears to be an important motivation for using OONI Run, as reported by more than half of the survey participants (53.3%).

Many participants also reported that they use OONI Run to:

Website testing appears to be the primary reason why OONI Run is used, as most survey respondents (53.3%) reported that they have generated and shared an OONI Run link for the testing of websites.

When asked whether they have generated and shared an OONI Run link based on the speed and performance tests, 80% of respondents said that they have not. Similarly, 86.7% of respondents said that they have never used OONI Run to generate and share an OONI Run link based on the middlebox tests. When asked whether they have found it useful to generate OONI Run links based on the middlebox and performance tests, none of the 30 survey respondents provided an answer.

The platforms that were reportedly used to share and/or receive OONI Run links include email, Twitter, WhatsApp, Telegram, Facebook Messenger, Slack, IRC, Matrix, and online forums, with most (50%) reporting that they used email to share/receive OONI Run links.

Most respondents (33.3%) reported using OONI Run to generate links for the testing of 4-20 URLs. Quite similarly, many reported using OONI Run to generate links for the testing of 1-3 URLs (26.7% of respondents) and for the testing of 21-50 URLs (23.3% of respondents). Meanwhile, only 10% of respondents reported using OONI Run to generate links for the testing of more than 50 URLs, and only 6.7% for the testing of more than 500 URLs. This suggests that OONI Run is primarily used for the testing of relatively short lists of URLs. However, it is possible that OONI Run is primarily used for the testing of shorter lists of URLs due to UI challenges associated with adding long lists of URLs (an issue expressed by multiple interview participants, as well as survey respondents, as discussed further in the next sections), rather than lack of interest in testing long lists of URLs.

When asked whether they have ever added the URLs that they have tested with OONI Run to the Citizen Lab test lists, the vast majority of survey participants (70%) responded that they have not. Out of these respondents, many reported that they don’t know how to add URLs to the Citizen Lab test lists and some reported that they find this process too complicated. Some of the respondents reported that they didn’t know what the Citizen Lab test lists are, and that they didn’t know they should contribute the URLs they were testing (with OONI Run) to the Citizen Lab test lists.

Our survey provided an open-ended question that encouraged participants to describe the challenges they encountered in using OONI Run links.

Several respondents reported that when they include many URLs for testing, the generated OONI Run links are far too long to share. They reported that these long OONI Run links break when shared through various messaging platforms, and that some messaging platforms have a character limit that prevents sharing very long OONI Run links. As a result, OONI Run users have had to generate shorter OONI Run links based on shorter lists of URLs. However, having to coordinate testing based on multiple OONI Run links is reportedly more complicated (than coordinating testing based on a single OONI Run link), particularly since it results in unequal measurement coverage.

Furthermore, a few respondents mentioned that the process of adding URLs to the OONI Run platform (i.e. typing URLs in the URL slots) can be slow and sluggish, as it can take a few seconds before the typed letters appear in the URL slots. They also reported that they expected OONI Run to serve as a browser-based version of OONI Probe, and that they were disappointed and confused when they weren’t able to run tests and receive test results through the platform. This suggests that the purpose and functionality of OONI Run (i.e. to generate mobile deep links and widget code) is not clearly communicated through the platform.

A few survey participants also mentioned that when coordinating censorship measurement campaigns, another challenge is associated with users reporting errors when tapping/clicking on the OONI Run links that they receive. As a result, the censorship measurement coordinators need to invest time in trouble-shooting, which often is challenging to do remotely.

Another reported challenge is associated with the requirement of having the OONI Probe mobile app installed. Many respondents mentioned that the people that they share OONI Run links with often don’t have OONI Probe already installed, and that requesting them to install the app can be a challenge. Furthermore, the requirement of having to explain OONI Probe (and all other relevant information) in order to convince people to install the app was also reported as a challenge. This, reportedly, is especially challenging when social media platforms are blocked, limiting the ability to share links and encourage app installation.

Challenges associated with using OONI Run buttons

Our survey provided an open-ended question that encouraged participants to describe the challenges they encountered in using OONI Run buttons.

Most survey respondents did not report having extensive experience using OONI Run to generate widget code. Out of those who shared feedback on this, the main issue that they reported was not having an easy and quick way to update the list(s) of URLs included in OONI Run buttons that they create. They expressed the need to be able to quickly and easily update OONI Run buttons, by easily viewing and editing the included URLs.

Similarly to the challenges expressed with using OONI Run links, some of the survey respondents mentioned that requiring people to have OONI Probe installed (in order to be able to make use of the OONI Run buttons) is a challenge. Others mentioned that embedding OONI Run buttons based on a long list of URLs is a challenge as well, as the large length of such widget code is not always easily supported by various websites.

Other survey respondents reported not having faced any issues and that they find the process of generating and using OONI Run buttons “simple and straightforward”.

Overall: Challenges associated with using OONI Run

At the end of our survey, we provided an open-ended question that encouraged participants to describe what they find most challenging about OONI Run (overall).

Many of the same challenges (previously discussed) were reiterated in these answers, as summarized below:

Suggestions for improving OONI Run

The last survey question asked participants to describe their “wish list” features for OONI Run.

The suggestions and feature requests provided by survey respondents are summarized below:

Interviews

As part of our OONI Run usability study, we interviewed 16 community members to collect more in-depth feedback.

The interview participants included experienced OONI Run users (measurement campaign coordinators and volunteers), as well as internet freedom community members who use OONI Probe to measure internet censorship, but who have not used OONI Run before.

As these different user personas have different experiences (or lack thereof) with OONI Run, we prepared separate interview questions for each group. In all cases, the second part of the interviews (following a set of more high-level questions) involved hands-on use of the OONI Run platform, so that we could observe the challenges and needs associated with using the tool in practice.

In the following sections, we share the feedback provided during the interviews by measurement campaign coordinators and volunteers (i.e. experienced OONI Run users) and more general internet freedom community members.

Measurement campaign coordinators and volunteers

Through the interviews with measurement campaign coordinators and volunteers, our goal was to better understand their needs and the specific challenges that they have encountered when using OONI Run.

As part of this study, we interviewed 9 measurement campaign coordinators and volunteers, most of whom have extensive and long-term experience running measurement campaigns with large groups of volunteers.

During the interviews, we started off by asking the following questions:

Through these questions, we aimed to introduce participants to the study, learn more about their goals, motivations, and expectations around using OONI Run, how OONI Run has been useful to them so far, and what is the biggest challenge they have encountered with using OONI Run (that stands out in their memory, without interacting with the platform).

The second part of the interview was more hands-on. We requested participants to access the OONI Run website (run.ooni.io) and share their screen with us, so that we could observe how they interact with the platform in practice. We asked participants to show us how they go about using the platform, during which we asked the following questions:

In the following sections, we share the main interview findings around OONI Run usage, associated challenges, and suggestions for improvement.

OONI Run usage

When asked what are their primary reasons for using OONI Run, the interview participants shared (in summary) that they use OONI Run to:

All participants mentioned that they almost always use OONI Run solely for the purpose of testing websites (though the Web Connectivity test), and that they rarely (if ever) use OONI Run to coordinate testing based on the other available tests. They explained that this is partly due to the fact that their research interests center more around website censorship, but also because the other tests are already included in the OONI Probe mobile app (and since OONI Run links require use of the app anyway, they usually just ask volunteers to run the tests directly inside the app). While several participants mentioned that they have made use of widget code generated through OONI Run, they primarily use OONI Run for its mobile deep links.

When asked what has been the most rewarding outcome of using OONI Run, interview participants mentioned that OONI Run has been particularly helpful with rapid response, enabling them to coordinate the testing of specific websites and gain access to relevant measurements quickly. Several interview participants elaborated how OONI Run has helped them collect evidence of internet censorship during time-sensitive events, such as during a temporary 2-hour block, or during political events, such as elections and protests.

Evidence of internet censorship, collected through OONI Run campaigns, has supported journalistic reporting and advocacy efforts. Some interviewees mentioned that collecting measurements that can potentially serve as evidence of internet censorship – especially when it is being denied by governments – has helped inform their advocacy work.

Several interview participants mentioned that, for them, the most rewarding outcome has been being able to create a country-wide censorship measurement system, through the use of OONI Run. This has been possible because OONI Run enabled them to monitor the blocking of websites across different regions and networks in somewhat real-time, with the help of volunteers. OONI Run has also helped increase relevant measurement coverage (across network and regions), providing more conclusive results and stronger evidence (for the broader internet freedom community, since all measurements are openly published).

As participants interacted with the OONI Run platform during the interview, they said that they add URLs either by manually typing each URL (usually only when it involves a few websites), or by copy-pasting their own (longer) list of URLs from a text editor, Excel file, or Google spreadsheet.

Interestingly, when some participants manually typed URLs in the OONI Run website during the interview (while they shared their screen with us), we noticed that they accidentally mistyped the URL, didn’t notice the error, but proceeded to generate an OONI Run link anyway. The fact that even experienced OONI Run users run the risk of generating (and sharing) OONI Run links based on broken URLs – due to the limitations of the OONI Run platform – highlights the need to improve the platform in such a way that such errors are more easily noticeable or avoidable.

Most participants mentioned that they primarily share OONI Run links via email, because email clients are more reliable than instant messaging apps in terms of not breaking long OONI Run links. In other cases (particularly when they aim to reach larger audiences), they share OONI Run links using Twitter, Facebook, WhatsApp, Facebook Messenger, Telegram, and Signal. Some participants also mentioned that they sometimes embed OONI Run widget code into their website, as they find it easier to point volunteers to a button on a website, than to share incredibly long OONI Run links.

Challenges

Interview participants (who are experienced OONI Run users) shared that they have encountered the following challenges when using the OONI Run platform:

Out of all of these challenges, those that were highlighted the most (as the biggest challenges) include the issues associated with long OONI Run links breaking and not having an easy way to track, identify, aggregate, and access relevant measurements during a censorship measurement campaign.

Suggestions and “dream features”

Based on their experience using OONI Run, interview participants offered many detailed suggestions for the improvement of the platform.

These include:

When asked what “dream features” they would like to see in future versions of OONI Run, interview participants shared the following:

Internet freedom community members

Apart from understanding and tailoring to the needs of existing OONI Run users (measurement campaign coordinators and volunteers), we also aim to enable and encourage OONI Run usage by new users as well.

To this end, we interviewed internet freedom community members who could be interested in using OONI Run as part of their work, but who haven’t used the platform yet. Through these interviews, we aimed to gain fresh perspective from new users and explore their needs and challenges in using the platform.

As part of this study, we interviewed 7 individuals who are active in the broader internet freedom community and who use OONI Probe and/or OONI data to examine internet censorship.

During the interviews, we started off by asking the following questions:

Through these questions, we aimed to introduce participants to the study, learn more about their goals, motivations, and expectations around using OONI Probe, and explore whether they have experienced needs that OONI Run is designed to meet.

The second part of the interview was more hands-on. We shared the OONI Run website (run.ooni.io) with participants, asked them to access the site and share their screen with us, so that we could observe how they interact with the platform in practice.

While they interacted with the OONI Run platform, we asked the following questions:

In the following sections, we share the main interview findings around OONI Run usage, associated challenges, and suggestions for improvement.

OONI Run usage

Interview participants mentioned that they use OONI Probe for research and advocacy purposes, primarily with the goal of measuring the blocking of websites and apps. Some interview participants mentioned that they also run OONI Probe for its middlebox and performance tests, as part of their broader research efforts to examine net neutrality violations. A few of the interview participants mentioned that they facilitate OONI workshops and engage their local communities with censorship measurement research and advocacy.

When asked whether they ever had the need to coordinate censorship measurement (such as the testing of specific websites in a particular country, region, or network), several participants mentioned that they did, but that they weren’t familiar with OONI Run. These interview participants shared that they would find it useful to coordinate testing among their partner networks, particularly leading up to and during political events (such as elections). Quite similarly, other interview participants mentioned that they have had the need to coordinate OONI Probe testing on several occasions when specific websites were reported blocked in foreign countries (and they wanted to collect relevant measurements). Other interview participants mentioned that they would find the coordination of testing useful in order to perform tests on more networks and regions (that they don’t personally have easy access to).

Once we shared the OONI Run website (run.ooni.io) with participants, we asked them to access the site and share their screen with us. The goal of this exercise was to observe, in practice, how a new user interacts with the OONI Run platform for the first time (and acquire feedback from a fresh perspective).

While most interview participants understood that the OONI Run website somehow pertained to OONI Probe testing, their first reaction was that they mostly found the platform rather confusing. At first glance, many interview participants said that the workflow of the OONI Run website is not straightforward, and that without guidance or instructions, it’s unclear what the website is about, what they’re required to do, and why. In short, they did not find the platform intuitive at all. One interview participant highlighted that even though they use OONI Probe and they’re very familiar with OONI, they don’t understand, contextually, what they are supposed to do when they land on the OONI Run website.

Most interview participants eventually proceeded to manually type URLs into the URL slots of the OONI Run website, while a few copy-pasted URLs directly from a browser. Some interview participants were careful to ensure that the HTTP prefix in the inserted URLs was accurate, while other interview participants did not fix the prefix or deleted it entirely. A few participants were unsure which URLs they should add, and it was unclear to them how many URLs they could add. None of the interviewed participants assumed that they could copy-paste a long list of URLs. When asked how they would go about adding a longer list of URLs, they responded that they would have to manually type them one by one.

Interestingly, several interview participants assumed that OONI Run was a browser version of OONI Probe (similarly to the OONI Probe mobile and desktop apps), expecting to be able to run tests directly through the OONI Run website and gain access to results (similarly to the OONI Probe apps). When asked to interact with the OONI Run website, these participants tried running tests by clicking on the test options (“Web Connectivity”, “NDT Speed Test”, etc.) and by clicking the “Generate” button. They were confused by the pop-up window that appeared instead, as it was unclear what they were meant to share or otherwise do. Having assumed that they ran tests, it was unclear how they could gain access to the results through the OONI Run website, leading to visible disappointment.

Upon clicking the “Generate” button, several interview participants found the pop-up window confusing. It was unclear what they were supposed to share, why, and how. More specifically, it was unclear to them if they would be sharing the OONI Probe app (to encourage installation), or if they would just be sharing the websites they previously inserted. To some participants, it was initially unclear that they had to share anything at all.

Interestingly, several participants clicked on the “Done” button inside the pop-up window, expecting something to happen. When the window just closed and brought them back to an empty (URL-wise) OONI Run website, they were confused. By clicking “Done”, they had originally assumed that they would run tests, or that they would gain access to test results (quite similarly to the OONI Probe apps).

Image: OONI Run pop-up window upon generating a mobile deep link and widget code, https://run.ooni.io/

Upon clicking the “Link” button inside the pop-up window, many interview participants were confused by the page that they landed on (accessed from a desktop browser). That page told them to install OONI Probe, but they were puzzled because they already had the OONI Probe app installed. Some participants even had both the OONI Probe desktop and mobile apps installed, finding this page particularly confusing. When clicking on the “Run” button on that page (from a desktop browser), nothing happened. It was therefore unclear to them how to proceed.

Image: OONI Run page (when accessed from a desktop browser) based on generated OONI Run link, https://run.ooni.io/

Since the purpose and functionality of OONI Run was not immediately clear, the use of the generated widget code was not clear either. However, once participants understood the goal and functionality of OONI Run, several participants understood how they would use the generated widget code as well.

Overall, many interview participants mentioned that for first-time users, OONI Run is not intuitive at all, as the platform lacks contextual information and it’s unclear what users are supposed to do. But they highlighted that once you know what OONI Run is about, the platform is easy and simple to use. Some participants mentioned that, while they would appreciate links to guides and further information, at the same time they appreciate that the current OONI Run website is quite “minimalist” and doesn’t “drown users with too much information”.

Challenges

While interacting with the OONI Run platform for the first time, interview participants experienced the following challenges:

Suggestions and “dream features”

Interestingly, many of the suggestions (for the improvement of the OONI Run platform) offered by first-time OONI Run users were similar to those shared by experienced measurement campaign coordinators and volunteers. But they also shared several new ideas.

The suggestions they offered include:

When asked what “dream features” they would ideally like to see in a future version of OONI Run, interview participants shared the following:

Conclusion

Overall, many of the same challenges and suggestions were not only shared by both survey respondents and interview participants, but also by both experienced OONI Run users and first-time users.

Some of the main challenges expressed by most participants include issues related to generating long OONI Run links (which often break when shared on popular messenger platforms), requiring volunteers to have OONI Probe installed (as a prerequisite to using OONI Run links), not being able to quickly and easily edit URLs included in OONI Run links, and not having an easy way to track, aggregate, and access measurements generated based on specific OONI Run links.

To improve the OONI Run platform, most participants requested the shortening of OONI Run links, having the ability to save and easily edit URLs included in OONI Run links, and having an easy way to coordinate the testing of popular social media sites and apps. Many participants pointed out the missing OONI Probe tests (particularly the instant messaging app tests) which should get added to the platform, as well as the need for OONI Run links to provide users with an estimated runtime. To enable adoption, participants proposed linking to relevant guides that provide step-by-step instructions, as well as to a relevant FAQ and other contextual information. To enable contributions to the Citizen Lab test lists, participants proposed providing an easy way to do so in the OONI Run website, or automating the process entirely.

To promote rapid response to emergent censorship events, participants emphasized the need for having an easy way to identify, access, and aggregate relevant measurements pertaining to specific OONI Run links, used as part of censorship measurement campaigns.

Next Steps

We thank all those who participated in this OONI Run usability study, both those who shared feedback by completing our survey and those who participated in interviews. We also thank all other community members who have shared feedback over the years, and who have used OONI Run as part of their work (despite the challenges).

The feedback we gathered through this study is incredibly useful, as it introduced us to many user challenges and needs that we were not previously aware of, and to many brilliant suggestions and ideas that we had not thought of. We will therefore do our best to address all feedback shared through this study (and documented in this report).

We will start off by opening tickets on our OONI Run and OONI Probe GitHub repositories (which are publicly available), based on all of the feedback collected through this study. Through these tickets, you will be able to track our discussions around each piece of feedback (and participate in those discussions), which decisions we make (and why), and how we proceed with implementation. If you have further feedback you would like to share, we encourage you to open a ticket or write to us.

We aim to release an improved version of OONI Run by the end of 2020. Stay tuned!