Flutter Integration Testing: Scheduled Tests on Firebase Test Lab using Bitrise
In my last blog where we scheduled Integration Test runs on virtual simulator devices, today we’ll be taking a look at how Bitrise and Test Lab come together to let you use real physical devices and make your testing routine even more powerful.
Setting up the Project
Same setup as the last blog, check out the Setup here. We’re using the default Counter App that comes with Flutter and I wrote a simple Integration Test for it. I then registered it on Bitrise.
Pretty basic, you can check the code here.
Preparing the App to run on Test Lab
Test Lab uses Android Instrumentation Tests and iOS XCTests under the hood for Flutter and so we’ll need to create Test Runners in our app’s native code to be able to run them on Test Lab. These native test runners then interact with our Flutter Integration Tests and are responsible for running them.
We’ll be using Android for this particular example, check out the setup steps for iOS here.
For Android, first, create the Instrumentation Test MainActivityTest.java under android/app/src/androidTest/java/YOUR_APP_PACKAGE_NAME
Configure app/build.gradle with espresso and JUnit dependencies.
We then run and verify the Instrumentation Test we just created on our local Android Emulator.
Once we’ve verified that the Instrumentation Test runs locally, move on to Bitrise and run it on Firebase Test Lab.
Setting up Firebase and Google Cloud
Create a Firebase Project.
Once You’ve done that, Go to Project Settings > General, and make a note of the Project ID you see here.
Next, we need to go to the Google Cloud Console and under our project:
- Create a service account.
- Add an Editor role to it.
- Enable Cloud Tool Results API.
These steps are explained really well by Majid Hajian in his video.
Configuring Firebase Secrets in Bitrise
The Project ID and service account credentials (contents of your <service_account>.json file from the previous step) will be stored as Bitrise secrets. To do so, navigate to the Workflow Editor > Secrets tab and save them as FIREBASE_PROJECT_ID and FIREBASE_TEST_LAB_SERVICE_ACCOUNT respectively.
And Finally the Workflow
The first few steps of the workflow are pretty standard stuff. We clone the GitHub repo, install Flutter and build the app. We follow it up with a 5-step shell script:
- Install gcloud CLI.
- Read the Bitrise secrets and create a
gcloud_key_file.jsonfile on the CI machine for gcloud auth. - Authenticate gcloud CLI with the service account.
- Build the Instrumentation Test.
- Run it on Firebase Test Lab using gcloud.
Here’s the complete workflow.
Test it Out!
Save and run the workflow on Bitrise. Assuming everything is properly configured, the workflow should function correctly and the build should pass!
You can scroll down and see the test results and a link to the test matrix inside the build logs.
And the scheduling!
The final and arguably the most crucial bit is scheduling your integration tests to run automatically at a designated time.
Click on the “Start/Schedule build” button, and specify, at what time, and how frequently you’d like your integration tests to run (daily, weekdays, every alternate day). Here’s a short clip from my previous blog, showing you how to do it:
Once you do that, just wait for the build to run at the scheduled time.
- Integration tests can often be long-running, so a general rule of thumb is to schedule these tests to run during the night and monitor their reports the next morning.
And that’s it!
Scheduling integration tests in Flutter on Bitrise can save you a lot of QA time and effort.
It ensures that your app has a predictable behavior, and also allows you to detect potential bugs or issues promptly.
Happy testing!
Checkout the complete code with the workflow at:
Have any questions? Reach out to me on Twitter:
