Making the OONI Probe Android app more resilient
Simone Basso
2021-05-27
We recently made OONI Probe Android
more robust against accidental or deliberate blocking of our backend
services. Specifically, we implemented support for specifying a proxy
that speaks with OONI’s backend services. We also improved the build
process to influence the TLS Client Hello fingerprint, which helps with
avoiding accidental blocking.
Adding support for a proxy
Since late 2020, community members have been reporting specific OONI
Probe Android failures. The symptom is that all tests fail with the “all
available probe services failed” error. This issue has been publicly
documented, for example, in
ooni/probe#1324. The
following screenshot shows what a user would see under this specific
error condition.
The error in the above screenshot means that OONI Probe attempted to
contact all the public OONI API endpoints (which we call “probe
services”), but to no avail.
We first discussed implementing a proxy during the January 2020 OONI
community meeting. After this meeting, we opened
ooni/probe#985 to
describe the implementation of this work. My initial thought
on this matter was that we could automatically start a
Psiphon tunnel upon failure.
But after discussing this with a few advanced users (thank you!), I
refined my view on this issue. There will always be an advanced user who
knows the local context better than us. These users will already have
well-configured circumvention technology in place (e.g., an instance of
Orbot running on the same device using bridges and pluggable
transports).
These discussions and the urgency to solve the “all available probe
services failed” issue also convinced me that we wanted to provide users
with options first. Automatically detecting whether the ISP is
censoring the probe services would then be implemented later.
The result of these conversations was a plan
to allow users to choose whether they wanted to:
You can read through
ooni/probe#985 and
ooni/probe#1324 for more
low-level details on what we did to make this possible. Here, for
brevity, I will just mention the two most significant moments. On
April 3, 2021, we added support for embedding an encrypted Psiphon
config file into OONI Probe builds.
This improvement enabled us to start a Psiphon tunnel without depending
on the probe services for fetching this configuration file. On April
19, 2021, we discovered what was likely the root cause of the blocking.
As I will further explain in the next section, it was the TLS Client
Hello fingerprint. We needed to change how we build OONI Probe’s support
libraries for Android devices to remediate this problem.
With these low-level details out of the way, my colleague Lorenzo and I
started working on the user interface for configuring a proxy on Android
(ooni/probe-android#423).
A few Android users also assisted us, providing feedback, testing, and
code patches.
This work resulted in adding an OONI backend proxy section in the
settings of the OONI Probe app:
By tapping on the “OONI backend proxy” row, we will direct you to
another screen, which allows you to choose which proxy you want to use:
The default (None) is that no proxy is being used. If you choose
Psiphon, OONI Probe will create a Psiphon
tunnel using the encrypted configuration file. If you select Custom,
you can set the hostname and port of a SOCKS5 proxy. I have filled the
hostname and port with 127.0.0.1 and 9050 in the above
screenshot. These are the settings you can use if you have an
Orbot
instance running on your device (or tor inside Termux). Instead, if you
are running Tor Browser,
the IP address is 127.0.0.1 and the port is 9150.
Changing our Android TLS fingerprint
In mid-April 2021, we discovered the likely cause of blocking. V2Ray
developers previously documented this issue in
v2fly/v2ray-core#557.
The root cause is that the TLS Client Hello generated by Golang on
Android devices changes depending on Golang’s understanding of the
hardware capabilities. Golang’s TLS library uses a hardware
implementation of AES when possible.
Otherwise, it falls back to AES code written in pure Go. According to
the official docs,
such code is not constant over time. Probably, for this reason, when
Golang thinks that there is no hardware support for AES, it reorders
the preferred cipher list in the Client Hello.
This reordering aims to significantly reduce the probability that the
server will select AES ciphers by moving such ciphers towards the bottom
of the list.
As documented by V2Ray developers in the aforementioned
issue, some ISPs
only allow specific TLS Client Hello fingerprinting.
When AES algorithms sit at the top of the preferred ciphers list, TLS
works. Instead, when they are near the bottom, TLS fails.
We noticed this issue when debugging OONI Probe Android on one such
network. Very interestingly, the symptom was that a debug build of our
app was working. At the same time, a release build with “all the
available probe services” failed. Our analysis of
the problem is documented more in detail in the
ooni/probe#1444
issue. Golang fails to read the hardware capabilities because
/proc/self/auxv is not readable on Android in release mode.
(At the moment of writing this blog post, it is unclear whether
/proc/self/auxv is not readable on all arm64 Android devices or
whether we were just lucky with our devices.)
To fix this issue for OONI Probe, we forked Golang. We modified the src/runtime
to call the C library’s getauxval function to get the correct
hardware information. This patch seems a bit hacky and does not necessarily feel right in
general. Golang’s src/runtime should not depend on the C library.
Still, our solution works for us because we need to link the C library
on Android anyway.
When reviewing the diff, my colleague Arturo suggested that a better way
to fix this issue is patching the
golang/mobile repository instead.
Patching golang/mobile feels like
a more proper solution for this problem, but we have not had time to
explore it yet.
For now, our mk build script uses ooni/go when building for Android.
While this may not be the best solution, it nonetheless provides us a
way to ship this crucial fix to users within a reasonable time frame.
Future improvements
Regarding the proxy functionality, we are also planning on implementing
proxy support for OONI Probe iOS
(ooni/probe#1470), the
OONI Probe Command Line Interface
(ooni/probe#1488), and
the OONI Probe desktop app
(ooni/probe#1489). We
chose to prioritize Android because Android users have been impacted by
this issue the most, as far as we know. If you see the “all available
probe services failed” error on other platforms (beyond Android), please get in touch! (So far, we have received a
single report of a similar problem occurring on iOS. If you also see
this issue, please let us know by commenting on the
ooni/probe#1491 issue.)
We are also planning on enabling users to choose HTTP proxies. We are
considering the possibility of enabling users to specify a username and
a password for SOCKS5/HTTP proxies. We are also discussing the option of
passing Psiphon a downstream SOCKS5 proxy with optional authentication.
We documented these future changes in
ooni/probe#1465. If you
have specific proxy needs, please let us know (perhaps on our
Slack channel), and we will plan for them!
Regarding the fix for Golang, we need to figure out whether we can fix
the issue inside of the
golang/mobile repository
(ooni/probe#1486). We
also need to explore what happens on older arm devices
(ooni/probe#1487). If
you have insight on how we can fix the
golang/mobile issue or ways to
test old Android arm-based devices, please ping us!