Improve the reproducible build process.

* Moved stuff into it's own `reproducible-builds` directory.
* Improved reproducible build by using a debian snapshot and more clearly listing dependencies.
* Removed signing from assembleReelase.
* Updated README.

.gitignore

@@ -23,6 +23,5 @@ ffpr
 test/androidTestEspresso/res/values/arrays.xml
 obj/
 jni/libspeex/.deps/
-*.sh
 pkcs11.password
 dev.keystore

Dockerfile

@@ -1,25 +0,0 @@
-FROM ubuntu:17.10
-
-RUN dpkg --add-architecture i386 && \
-    apt-get update -y && \
-    apt-get install -y software-properties-common && \
-    apt-get update -y && \
-    apt-get install -y libc6:i386=2.26-0ubuntu2.1 libncurses5:i386=6.0+20160625-1ubuntu1 libstdc++6:i386=7.2.0-8ubuntu3.2 lib32z1=1:1.2.11.dfsg-0ubuntu2 wget openjdk-8-jdk=8u171-b11-0ubuntu0.17.10.1 git unzip opensc pcscd && \
-    rm -rf /var/lib/apt/lists/* && \
-    apt-get autoremove -y && \
-    apt-get clean
-
-ENV ANDROID_SDK_FILENAME android-sdk_r24.4.1-linux.tgz
-ENV ANDROID_SDK_URL https://dl.google.com/android/${ANDROID_SDK_FILENAME}
-ENV ANDROID_API_LEVELS android-28
-ENV ANDROID_BUILD_TOOLS_VERSION 28.0.3
-ENV ANDROID_HOME /usr/local/android-sdk-linux
-ENV PATH ${PATH}:${ANDROID_HOME}/tools:${ANDROID_HOME}/platform-tools
-RUN cd /usr/local/ && \
-    wget -q ${ANDROID_SDK_URL} && \
-    tar -xzf ${ANDROID_SDK_FILENAME} && \
-    rm ${ANDROID_SDK_FILENAME} 
-RUN echo y | android update sdk --no-ui -a --filter ${ANDROID_API_LEVELS}
-RUN echo y | android update sdk --no-ui -a --filter extra-android-m2repository,extra-android-support,extra-google-google_play_services,extra-google-m2repository
-RUN echo y | android update sdk --no-ui -a --filter tools,platform-tools,build-tools-${ANDROID_BUILD_TOOLS_VERSION}
-RUN rm -rf ${ANDROID_HOME}/tools && unzip ${ANDROID_HOME}/temp/*.zip -d ${ANDROID_HOME}

app/build.gradle

@@ -454,16 +454,6 @@ task signProductionWebsiteRelease {
     }
 }
 
-tasks.whenTaskAdded { task ->
-    if (task.name.equals("assemblePlayRelease")) {
-        task.finalizedBy signProductionPlayRelease
-    }
-
-    if (task.name.equals("assembleWebsiteRelease")) {
-        task.finalizedBy signProductionWebsiteRelease
-    }
-}
-
 def getLastCommitTimestamp() {
     new ByteArrayOutputStream().withStream { os ->
         def result = exec {

reproducible-builds/Dockerfile

@@ -0,0 +1,28 @@
+FROM debian:stretch
+
+COPY docker/ docker/
+COPY docker/apt.conf docker/sources.list /etc/apt/
+
+RUN dpkg --add-architecture i386
+
+RUN apt-get update -y && apt-get install -y apt-utils
+
+RUN apt-get update -y && apt-get install -y $(cat docker/dependencies.txt)
+RUN docker/print-versions.sh docker/dependencies.txt
+
+ENV ANDROID_SDK_FILENAME        android-sdk_r24.4.1-linux.tgz
+ENV ANDROID_API_LEVELS          android-28
+ENV ANDROID_BUILD_TOOLS_VERSION 28.0.3
+
+ENV ANDROID_HOME /usr/local/android-sdk-linux
+ENV PATH         ${PATH}:${ANDROID_HOME}/tools:${ANDROID_HOME}/platform-tools
+
+RUN cd /usr/local/ && \
+    wget -q "https://dl.google.com/android/${ANDROID_SDK_FILENAME}" && \
+    tar -xzf ${ANDROID_SDK_FILENAME} && \
+    rm ${ANDROID_SDK_FILENAME} 
+RUN echo y | android update sdk --no-ui -a --filter ${ANDROID_API_LEVELS}
+RUN echo y | android update sdk --no-ui -a --filter extra-android-m2repository,extra-android-support,extra-google-google_play_services,extra-google-m2repository
+RUN echo y | android update sdk --no-ui -a --filter tools,platform-tools,build-tools-${ANDROID_BUILD_TOOLS_VERSION}
+RUN rm -rf ${ANDROID_HOME}/tools
+

reproducible-builds/README.md

@@ -3,30 +3,33 @@
 
 ## TL;DR
 
-You can just use these [instructions](https://signal.org/blog/reproducible-android/) from the official announcement at Open Whisper Systems's blog:
+```bash
-```
 # Clone the Signal Android source repository
-$ git clone https://github.com/signalapp/Signal-Android.git && cd Signal-Android
+git clone https://github.com/signalapp/Signal-Android.git && cd Signal-Android
 
 # Check out the release tag for the version you'd like to compare
-$ git checkout v[the version number]
+git checkout v[the version number]
+
+# Build the Docker image
+cd reproducible-builds
+docker build -t signal-android .
+
+# Go back up to the root of the project
+cd ..
 
 # Build using the Docker environment
-$ docker run --rm -v $(pwd):/project -w /project whispersystems/signal-android:1.3 ./gradlew clean assembleRelease
+docker run --rm -v $(pwd):/project -w /project signal-android ./gradlew clean assembleRelease
 
 # Verify the APKs
-$ python3 apkdiff/apkdiff.py build/outputs/apks/project-release-unsigned.apk path/to/SignalFromPlay.apk
+python3 apkdiff/apkdiff.py build/outputs/apks/project-release-unsigned.apk path/to/SignalFromPlay.apk
 ```
 
-Note that the instructions above use a pre-built Signal Docker image from [Docker Hub](https://hub.docker.com/u/whispersystems/). If you wish to compile the image yourself, continue reading the longer version below.
-
-
 ***
 
 
 ## Introduction
 
-Since version 3.15.0 Signal for Android has supported reproducible builds. This is achieved by replicating the build environment as a Docker image. You'll need to build the image, run a container instance of it, compile Signal inside the container and finally compare the resulted APK to the APK that is distributed in the Google Play Store.
+Since version 3.15.0 Signal for Android has supported reproducible builds. The instructions were then updated for version 5.0.0. This is achieved by replicating the build environment as a Docker image. You'll need to build the image, run a container instance of it, compile Signal inside the container and finally compare the resulted APK to the APK that is distributed in the Google Play Store.
 
 The command line parts in this guide are written for Linux but with some little modifications you can adapt them to macOS (OS X) and Windows. In the following sections we will use `3.15.2` as an example Signal version. You'll just need to replace all occurrences of `3.15.2` with the version number you are about to verify.
 
@@ -34,24 +37,19 @@ The command line parts in this guide are written for Linux but with some little
 ## Setting up directories
 
 First let's create a new directory for this whole reproducible builds project. In your home folder (`~`), create a new directory called `reproducible-signal`.
-```
+
-user@host:$ mkdir ~/reproducible-signal
+```bash
+mkdir ~/reproducible-signal
 ```
 
 Next create another directory inside `reproducible-signal` called `apk-from-google-play-store`.
-```
+
-user@host:$ mkdir ~/reproducible-signal/apk-from-google-play-store
+```bash
+mkdir ~/reproducible-signal/apk-from-google-play-store
 ```
 
 We will use this directory to share APKs between the host OS and the Docker container.
 
-Finally create one more directory inside `reproducible-signal` called `image-build-context`.
-```
-user@host:$ mkdir ~/reproducible-signal/image-build-context
-```
-
-This directory will be used later to build our Docker image.
-
 
 ## Getting the Google Play Store version of Signal APK
 
@@ -61,27 +59,30 @@ First make sure that the Signal version you want to verify is installed on your
 
 Plug your device to your computer and run this command to pull the APK from the device:
 
-```
+```bash
-user@host:$ adb pull $(adb shell pm path org.thoughtcrime.securesms | grep /base.apk | awk -F':' '{print $2}') ~/reproducible-signal/apk-from-google-play-store/Signal-$(adb shell dumpsys package org.thoughtcrime.securesms | grep versionName | awk -F'=' '{print $2}').apk
+adb pull $(adb shell pm path org.thoughtcrime.securesms | grep /base.apk | awk -F':' '{print $2}') ~/reproducible-signal/apk-from-google-play-store/Signal-$(adb shell dumpsys package org.thoughtcrime.securesms | grep versionName | awk -F'=' '{print $2}').apk
 ```
 
 This will pull a file into `~/reproducible-signal/apk-from-google-play-store/` with the name `Signal-<version>.apk`
 
 Alternatively, you can do this step-by-step:
 
+```bash
+adb shell pm path org.thoughtcrime.securesms
 ```
-user@host:$ adb shell pm path org.thoughtcrime.securesms
+
-```
 This will output something like:
-```
+
+```bash
 package:/data/app/org.thoughtcrime.securesms-aWRzcGlzcG9wZA==/base.apk
 ```
 
 The output will tell you where the Signal APK is located in your device. (In this example the path is `/data/app/org.thoughtcrime.securesms-aWRzcGlzcG9wZA==/base.apk`)
 
 Now using this information, pull the APK from your device to the `reproducible-signal/apk-from-google-play-store` directory you created before:
-```
+
-user@host:$ adb pull \
+```bash
+adb pull \
   /data/app/org.thoughtcrime.securesms-aWRzcGlzcG9wZA==/base.apk \
   ~/reproducible-signal/apk-from-google-play-store/Signal-3.15.2.apk
 ```
@@ -90,12 +91,12 @@ We will use this APK in the final part when we compare it with the self-built AP
 
 ## Identifying the ABI
 
-Since v4.37.0, the APKs have been split by ABI, the CPU architecture of the target device. Google play will serve the correct one to you for your device.
+Since v4.37.0, the APKs have been split by ABI, the CPU architecture of the target device. Google Play will serve the correct one to you for your device.
 
 To identify which ABIs the google play APK supports, we can look inside the APK, which is just a zip file:
 
-```
+```bash
-user@host:$ unzip -l ~/reproducible-signal/apk-from-google-play-store/Signal-*.apk | grep lib/
+unzip -l ~/reproducible-signal/apk-from-google-play-store/Signal-*.apk | grep lib/
 ```
 
 Example:
@@ -135,41 +136,28 @@ In the following sections we will assume that your Docker installation works wit
 
 
 ## Building a Docker image for Signal
-
+First, you need to pull down the source for Signal-Android, which contains everything you need to build the project, including the `Dockerfile`. The `Dockerfile` contains instructions on how to automatically build a Docker image for Signal. It's located in the `reproducible-builds` directory of the repository. To get it, clone the project:
-#### Grabbing the `Dockerfile`
-
-First you will need the `Dockerfile` for Signal Android. It comes bundled with Signal's source code. The `Dockerfile` contains instructions on how to automatically build a Docker image for Signal. You just need to run it and it builds itself.
-
-Download the `Dockerfile` to the `image-build-context` directory.
 
 ```
-user@host:$ wget -O ~/reproducible-signal/image-build-context/Dockerfile_v3.15.2 \
+git clone https://github.com/signalapp/Signal-Android.git signal-source
-                    https://raw.githubusercontent.com/signalapp/Signal-Android/v3.15.2/Dockerfile
 ```
 
-Note that the `Dockerfile` is specific to the Signal version you want to compare to. Again you have to adjust the URL above to match the right version. (Though sometimes the file might not be up to date, see the [Troubleshooting section](#troubleshooting))
+Then, checkout the specific version you're trying to build:
-
-
-#### Building the image
-
-Now we have everything we need to build the Docker image for Signal. Go to the `image-build-context` directory:
 
 ```
-user@host:$ cd ~/reproducible-signal/image-build-context
+git checkout --quiet v5.0.0
 ```
 
-And list the contents.
+Then, to build it, go into the `reproducible-builds` directory:
+
 ```
-user@host:$ ls
+cd ~/reproducible-signal/signal-source/reproducible-builds
-```
-The output should look like this:
-```
-Dockerfile_v3.15.2
 ```
 
-Now in this directory build the image using `Dockerfile_v3.15.2`:
+...and run the docker build command:
+
 ```
-user@host:$ docker build --file Dockerfile_v3.15.2 --tag signal-android .
+docker build -t signal-android .
 ```
 
 (Note that there is a dot at the end of that command!)
@@ -181,61 +169,39 @@ Wait a few years for the build to finish... :construction_worker:
 :calendar: :sleeping:
 
 After the build has finished, you may wish to list all your Docker images to see that it's really there:
+
 ```
-user@host:$ docker images
+docker images
 ```
+
 Output should look something like this:
+
 ```
 REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
 signal-android      latest              c6b84450b896        46 seconds ago      2.94 GB
-ubuntu              14.04.3             8693db7e8a00        9 weeks ago         187.9 MB
 ```
 
 
 ## Compiling Signal inside a container
 
-Next we will run a container of the image we just built, grab Signal's source code and compile Signal.
+Next we compile Signal.
 
-First go to the `reproducible-signal` directory:
+First go to the directory where the source code is: `reproducible-signal/signal-source`:
 
 ```
-user@host:$ cd ~/reproducible-signal/
+cd ~/reproducible-signal/signal-source
 ```
 
-To run a new ephemeral container with an interactive terminal session execute the following long command:
+To build with the docker image you just built (`signal-android`), run:
-```
-user@host:$ docker run \
-              --name signal \
-              --rm \
-              --interactive \
-              --tty \
-              --volume $(pwd)/apk-from-google-play-store:/signal-build/apk-from-google-play-store \
-              --workdir /signal-build \
-              signal-android
-```
-Now you are inside the container.
 
-Grab Signal's source code from GitHub and go to the repository directory:
 ```
-root@container:# git clone https://github.com/signalapp/Signal-Android.git
+docker run --rm -v $(pwd):/project -w /project signal-android ./gradlew clean assemblePlayRelease
-root@container:# cd Signal-Android
 ```
 
-Before you can compile, you **must** ensure that you are at the right commit. In other words you **must** checkout the version you wish to verify (here we are verifying 3.15.2):
-```
-root@container:# git checkout --quiet v3.15.2
-```
-
-Now you may compile the release APK by running:
-```
-root@container:# ./gradlew clean assemblePlayRelease --exclude-task signProductionPlayRelease
-```
 This will take a few minutes :sleeping:
 
 
-#### Checking if the APKs match
+### Checking if the APKs match
-
-After the build has completed successfully we can finally compare if the APKs match. For the comparison we need of course the Google Play Store version of Signal APK which you copied to the `apk-from-google-play-store` directory in the beginning of this guide. Because we used that directory as a `--volume` parameter for our container, we can see all the files in that directory within our container.
 
 So now we can compare the APKs using the `apkdiff.py` tool.
 
@@ -247,17 +213,20 @@ See [Identifying the ABI](#identifying-the-abi) above if you don't know the ABI
 
 Once you have determined the ABI, add an `abi` environment variable. For example, suppose we determine that `armeabi-v7a` is the ABI google play has served:
 
-```
+```bash
-root@container:# export abi=armeabi-v7a
+export abi=armeabi-v7a
 ```
 
-And the diff script to compare:
+And run the diff script to compare (updating the filenames for your specific version):
-```
+
-root@container:# python3 apkdiff/apkdiff.py \
+```bash
+python3 reproducible-builds/apkdiff/apkdiff.py \
         build/outputs/apk/play/release/*play-$abi-release-unsigned*.apk \
-                   ../apk-from-google-play-store/Signal-3.15.2.apk
+        ../apk-from-google-play-store/Signal-5.0.0.apk
 ```
+
 Output:
+
 ```
 APKs match!
 ```
@@ -277,7 +246,6 @@ If the build environment (i.e. `Dockerfile`) has not changed, you don't need to
 If you cannot get things to work, please do not open an issue or comment on an existing issue at GitHub. Instead, ask for help at https://community.signalusers.org/c/development
 
 Some common issues why things may not work:
-- some pinned packages in the `Dockerfile` are not available anymore and building of the Docker image fails
 - the Android packages in the Docker image are outdated and compiling Signal fails
 - you built the Docker image with a wrong version of the `Dockerfile`
 - you didn't checkout the correct Signal version tag with Git before compiling

reproducible-builds/docker/apt.conf

@@ -0,0 +1,6 @@
+Acquire::Check-Valid-Until "false";
+Acquire::Languages "none";
+Binary::apt-get::Acquire::AllowInsecureRepositories "false";
+
+APT::Install-Recommends "false";
+APT::Immediate-Configure "false";

reproducible-builds/docker/dependencies.txt

@@ -0,0 +1,8 @@
+libc6:i386=2.31-2
+libncurses5:i386=6.2-1
+libstdc++6:i386=10.2.0-3
+lib32z1=1:1.2.11.dfsg-2
+wget=1.20.3-1+b3
+openjdk-8-jdk=8u265-b01-0+deb9u1
+git=1:2.28.0-1
+unzip=6.0-25

reproducible-builds/docker/print-versions.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+while read dep; do
+  dep_name=$(echo $dep | cut -f1 -d '=')
+  version=$(dpkg -s $dep_name | grep 'Version: ' | cut -f2 -d ' ')
+  echo "$dep_name=$version"
+done < $1

reproducible-builds/docker/sources.list

@@ -0,0 +1,3 @@
+deb http://httpredir.debian.org/debian/ stretch main
+deb http://security.debian.org/ stretch/updates main
+deb http://snapshot.debian.org/archive/debian/20200731T211026Z/ unstable main