Year in Review: OONI in 2020

In light of the global COVID-19 pandemic, 2020 has been a challenging year for everyone.

Yet, several exciting things happened in the censorship measurement world. In this post, we share some OONI highlights from 2020, as well as some upcoming OONI projects for 2021!

OONI Probe

New OONI Probe Desktop App for Windows and macOS

OONI Probe Desktop

We launched an OONI Probe desktop app for macOS and Windows!

With our new desktop app, you can easily install OONI Probe and measure network performance and various forms of internet censorship with the click of a button.

This is a major milestone because this is the first time that OONI Probe is officially supported on Windows.

Windows is not only the most widely used desktop operating system in the world, but it is also commonly used by human rights defenders around the world. Supporting OONI Probe on Windows has therefore been a popular community request over the last years, and we are thrilled that this has now been achieved!

With the new OONI Probe desktop app, you can run all of the same tests included in the OONI Probe mobile app. As soon as you run tests, you will immediately have access to the results – and relevant network measurement data that can potentially serve as evidence of internet censorship – in your app.

Thanks to the Localization Lab community, the OONI Probe desktop app has been translated to the following 17 languages: Chinese, Russian, Spanish, Catalan, French, Hindi, Indonesian, Thai, Turkish, Italian, Portuguese, German, Romanian, Slovak, Albanian, Icelandic, and Greek. The app has also been translated to Arabic and Farsi, but those translations will be shipped once we’ve added RTL support.

During 2020, we also revamped the design of the OONI Probe mobile app dashboard.

OONI Probe Mobile

As part of our ongoing efforts to ensure feature parity and a seamless experience between the OONI Probe mobile and desktop apps, we aim to ship a new OONI Probe desktop release in early 2021 with the same new dashboard design as OONI Probe mobile.

Warm thanks to the Localization Lab community for having translated the OONI Probe mobile app to the following 20 languages: Arabic, Farsi, Chinese (simplified), Chinese (traditional), Russian, Spanish, French, Portuguese, Hindi, Thai, Indonesian, Turkish, Italian, Greek, Catalan, Icelandic, German, Slovak, Romanian, and Albanian.

New OONI Probe measurement engine

The OONI Probe apps now rely entirely on our new golang OONI Probe measurement engine!

Previously, the OONI Probe mobile app was powered by the C++ Measurement Kit library. Now, both the OONI Probe mobile and desktop apps are powered entirely by our new Go engine (and we re-wrote all of the OONI Probe tests in Go).

This is an important milestone because it means that we can write (and integrate) new network measurement tests faster and with greater confidence in code quality, enhancing our ability to more rapidly respond to emergent censorship events around the world.

Our new Go engine is also easier for us to maintain (since it enables us to share more code between the engine and the mobile apps), it’s easier for community members to review and contribute to our codebase (and they have already contributed many new tests!), improving the overall robustness and sustainability of our apps and experiments.

OONI Run usability study

OONI Run

To support decentralized censorship measurement efforts and promote rapid response to emergent censorship events, we carried out a usability study to improve OONI Run.

OONI Run is a platform that you can use to generate mobile deep links and widget code to coordinate OONI Probe testing. This is especially used by community members worldwide to coordinate the testing of specific websites during political events (such as elections and protests), when censorship events tend to emerge.

To enable community members to more effectively use OONI Run, we researched its limitations and collected structured community feedback through a usability study, which involved a survey and in-depth interviews.

We documented all of the OONI Run usability study findings in a report, and plan to release an improved version of OONI Run in 2021!

Measurement methodologies

New circumvention tool tests

OONI Probe Tor result

When censorship events emerge, circumvention tools often get blocked in many countries (likely as part of attempts to limit censorship circumvention).

We therefore developed and released new circumvention tool tests to enable the OONI community to check the reachability of specific circumvention tools: Tor and Psiphon. We chose these tools because they’re open source, commonly used by human rights defenders in censored environments, and we were able to collaborate directly with the developers of these tools. We had also previously developed Tor and Psiphon tests, and so these new tests build and expand upon our previous methodologies.

Through the OONI Probe mobile and desktop apps, you can now run the following new tests:

As soon as you run these tests, you will immediately have access to the results in your OONI Probe app (along with the relevant network measurement data), and (unless if you have opted-out) these test results will be openly published in real-time. You can therefore refer to OONI data to track whether Tor and Psiphon work around the world.

New experiments

Censorship measurement is like a cat and mouse game.

As internet censorship becomes more sophisticated, the need to improve and expand upon our censorship measurement capabilities increases. We therefore work continuously on improving our censorship measurement methodologies, and the types of tests that we prioritize on developing are informed by emerging censorship trends and community needs.

Throughout 2020, we developed the following new experiments:

We are grateful to community members who volunteered to run these experiments in censored environments, enabling us to check whether these tests work as expected and to improve upon them.

We are also thrilled that community members contributed many new network measurement tests for OONI Probe! Now that the OONI Probe apps are powered by our new Go engine, integrating new tests has become much easier (in comparison to the C++ Measurement Kit library that OONI Probe previously relied on).

New tests contributed by community members include:

We are extremely grateful for these contributions, and we aim to ship these new experiments as part of the OONI Probe apps in 2021!

Smart URL list system

Website blocking remains an ongoing - and increasingly worsening - problem, often affecting marginalized communities the most.

To improve the monitoring of website censorship around the world, we aim to create a smart URL list system which will help ensure “smarter” testing of websites. More specifically, we will prioritize the testing of websites that are of public interest, whose blocking would have a greater impact on human rights, and which are frequently blocked (such as social media services). To help ensure the safety of the tested URLs, we will continue to run them through the usual Citizen Lab URL review process on GitHub.

“Smarter” URL testing will enable us – and the broader internet freedom community – to more effectively monitor, analyze, and respond to cases of website censorship around the world.

Throughout 2020, we worked on building OONI’s smart URL list system. We started off by creating a policy for URL prioritization. The goal of this policy is to determine the criteria based on which the OONI Probe testing of certain types of URLs will be prioritized over others. Through URL prioritization, we aim to optimize the value of collected measurements, ensure regular testing of the same URLs for consistency, ensure that the tested URLs are relevant to OONI Probe users, and to improve the monitoring of website censorship globally.

We added backend support for URL prioritization based on the Citizen Lab test list categories, as well as for the testing of specific URLs (included in the Citizen Lab test lists). To help increase testing coverage, we added support to the OONI Probe mobile app for configuring push notifications to solicit testing, and so that OONI Probe mobile app users can receive push notifications and run experiments. This will be particularly useful during emergent censorship events when fast coordination of targeted URL testing is crucial.

To ensure that the measurements are more actionable, we developed data analysis capabilities aimed at examining results from a website-centric perspective. To present measurements, we created an initial version of a frontend to present results on OONI Explorer in a website-centric way, and we made progress on creating relevant user accounts.

We have completed most work towards building a smart URL list system and we aim to release it in 2021!

Real-time analysis and publication of global measurements

OONI Explorer

Starting from November 2019, we are analyzing and openly publishing OONI measurements collected from around the world within seconds!

Our new fastpath pipeline made the real-time analysis and publication of global OONI measurements possible. Throughout 2020, we made significant progress on further advancing our data analysis capabilities and improving our server infrastructure.

We replaced the batch OONI data processing pipeline entirely with our new fastpath pipeline, we built a new OONI PostgreSQL metaDB (powered by the latest version of PostgreSQL 11), we refactored the OONI API codebase to use the fastpath pipeline, we implemented a fastpath pipeline-based API to support OONI Explorer, and we improved our infrastructure by migrating to new servers and including better monitoring and alerting.

All of our backend work not only ensured the robustness of real-time analysis and publication of OONI measurements, but it also significantly boosted the performance of OONI Explorer: the web interface that human rights defenders rely on to search through OONI measurements and track censorship events worldwide.

Throughout the summer of 2020, we had the opportunity to host a great Google Summer of Code (GSoC) student (Krona Emmanuel) from Pakistan, who worked on making improvements related to social media sharing on OONI Explorer. We are very grateful for the great work Krona accomplished throughout his GSoC internship with OONI!

Research reports

In collaboration with community members, we published the following 11 research reports documenting internet censorship events around the world:

We also had the opportunity to serve as the host organization for OTF Information Controls Fellow, Chinmayi SK, who researched internet censorship in Manipur, India (through the use of OONI Probe and OONI data) and its impact on womxn in the region. We published Chinmayi’s research report, “*Those Unspoken Thoughts: A study of censorship and media freedom in Manipur, India*”, on our website.

Community

Internet Measurement Village 2020

Internet Measurement Village 2020

Between 10th June 2020 to 3rd July 2020, we organized and hosted the Internet Measurement Village (IMV) 2020, an online community event aimed at sharing skills, knowledge, and resources on internet measurement in defense of a free and open internet.

As part of the Internet Measurement Village (IMV), we live-streamed a total of 18 presentations: one presentation almost every day over four weeks.

The presentations featured:

To encourage participation during the IMV, we hosted a live chat on several platforms (YouTube chat, Slack, IRC) and, on average, each presenter received around 10 quality questions from the viewers which were addressed as part of the stream Q/A.

As all the live-streamed sessions of the IMV 2020 will continue to live on the OONI YouTube channel, we hope that these recordings will serve as a valuable resource for the internet freedom community.

Internet Measurement Village 2020

Internet Shutdown Measurement Training for Advocates

Internet Shutdown Measurement Training

Between 12th October 2020 to 20th November 2020, we served as the lead partner on Internews’ Internet Shutdown Measurement training program for human rights defenders in Sub-Saharan Africa.

During this 6-week training program, we had the opportunity to work with inspiring human rights defenders from across Africa, learn from them, and engage them with censorship measurement methodologies. Our goal was to train human rights defenders to use measurement tools and datasets as part of their research and advocacy efforts.

More specifically, we created video webinars and led the trainings for the following 2 modules:

We also helped facilitate the following modules:

Throughout the training program, we provided mentorship, created and reviewed homework assignments, and supported participants with the development of their research and advocacy projects.

OONI partnerships

OONI partners

Since 2016, we have had the opportunity to form partnerships with 36 organizations from around the world on the study of internet censorship.

We published an OONI Partners page on our website to feature these organizations, highlight their important work, and share the research projects we have collaborated on.

Throughout 2020, we formed several new partnerships, and we continued to support our partners primarily through the coordination of censorship measurement efforts, reviewing and updating test lists, and by providing OONI data analysis support.

OONI workshops and presentations

In light of the COVID-19 pandemic, most conferences were hosted online this year (or cancelled). Online events though provided a great opportunity to meet and form collaborations with people who we may have not otherwise met in person (because they previously did not have the opportunity to travel to attend such events).

Throughout 2020, we presented OONI at the following conferences and events:

We also discussed OONI’s censorship measurement tools as part of the following podcasts:

On 15th December 2020, we attended Internews’ workshop, “*Building capacity for internet shutdown advocacy: Community needs assessment*”, where we facilitated the discussion on community needs around censorship measurement data. This workshop builds upon Internews’ study which found that the vast majority of advocates reported that (out of several network measurement tools) they are most familiar with (and have experience using) OONI Probe. The community feedback collected through this workshop will help inform the development and improvement of our work.

OONI-verse

In 2020, the OONI community continued to contribute many measurements from most countries and territories around the world. They also made use of OONI data in many interesting ways!

Some highlights of community use of OONI data in 2020 include:

We are excited to see how community members make use of OONI data in 2021! If you’re interested in working with OONI data and need support, please don’t hesitate to reach out.

2021

OONI

We have many exciting projects lined up for 2021!

Some highlights include:

Our above priorities for 2021 have been informed by community feedback collected over the years (as well as in response to the dynamic censorship environment worldwide). If there are additional areas that you think we should prioritize, please don’t hesitate to let us know.

Warm thanks to the global OONI community for supporting our work throughout 2020!

We’re grateful to every OONI Probe user out there, and we’re excited for 2021. Stay tuned! And above all, stay safe.