Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Temporarily move content from redirect pages #2445

Merged
merged 2 commits into from
Dec 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 130 additions & 1 deletion docs/ci/firebase-test-lab.mdx
Original file line number Diff line number Diff line change
@@ -1,3 +1,132 @@
---
redirect: /documentation/ci/firebase-test-lab
title: CI - Firebase Test Lab
---

There are many device lab providers. Below we're showing how to run Patrol tests
on [Firebase Test Lab], because it's popular in the Flutter community, but the
instructions should be similar for other device farms, such as [AWS Device
Farm][aws_device_farm].

<Info>
Before you proceed with the steps listed below, make sure that you've
completed the [native setup] guide.
</Info>

<Tabs
defaultValue="android"
values={[
{ label: "Android", value: "android" },
{ label: "iOS", value: "ios" },
]}
>
<TabItem value="android">
To run the integration tests on Android, you need 2 apps: the app itself
(often called the "app under test") and the test intrumentation app.

To build these apps, run:

```
patrol build android --target integration_test/example_test.dart
```

Once you have built the apks, use the [gcloud] tool to run them on Firebase
Test Lab:

```
gcloud firebase test android run \
--type instrumentation \
--use-orchestrator \
--app build/app/outputs/apk/debug/app-debug.apk \
--test build/app/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--timeout 1m \
--device model=MediumPhone.arm,version=34,locale=en,orientation=portrait \
--record-video \
--environment-variables clearPackageData=true
```

<Info>
You must [install the gcloud tool] first. [Here][gcloud_android] you can learn
more about all available options and flags.
</Info>

<Info>
The environment variable `clearPackageData=true` tells orchestartor to clear the
package data between test runs. Keep in mind that it clears only the data of your
app, not other data on the device, e.g. Chrome.
</Info>

It's convenient to create a shell script to avoid typing that long command
every time. You might want to take a look at Patrol example app's
[run_android_testlab script][example_android_script].

<Info>
On Android, all permissions are granted by default. This behavior can be
changed using the [alpha version of the gcloud tool].
</Info>

</TabItem>
<TabItem value="ios">

To run the integration tests on iOS, you need 2 apps: the app itself
(often called the "app under test") and the test intrumentation app.

First, build your Flutter app, choosing the integration test file as target:

```
patrol build ios --target integration_test/example_test.dart
```

`patrol build ios` outputs paths to the built app binaries, for example:

```
$ patrol build ios -t integration_test/example_test.dart --release
• Building app with entrypoint example_test.dart for iOS device (release)...
✓ Completed building app with entrypoint example_test.dart for iOS device (31.5s)
build/ios_integ/Build/Products/Release-iphoneos/Runner.app (app under test)
build/ios_integ/Build/Products/Release-iphoneos/RunnerUITests-Runner.app (test instrumentation app)
build/ios_integ/Build/Products/Runner_iphoneos16.2-arm64.xctestrun (xctestrun file)
```

Firebase Test Lab requires these files to be packaged together in a zip
archive. To create the archive:

```
pushd build/ios_integ/Build/Products
zip -r ios_tests.zip Release-iphoneos Runner_iphoneos16.2-arm64.xctestrun
popd
```

Finally, upload the `ios_tests.zip` to Firebase Test Lab for execution:

```
gcloud firebase test ios run \
--test build/ios_integ/Build/Products/ios_tests.zip \
--device model=iphone8,version=16.2,locale=en_US,orientation=portrait
```

<Info>
You must [install the gcloud tool] first. [Here][gcloud_ios] you can learn
more about all available options and flags.
</Info>

If your `.xctestrun` file has different iOS version in its name than the
device you're running on, simply rename the `.xctestrun` so that the version
matches.

It's convenient to create a shell script to avoid typing that long command
every time. You might want to take a look at Patrol example app's
[run_ios_testlab script][example_ios_script].

</TabItem>
</Tabs>

[native setup]: /documentation
[gcloud]: https://cloud.google.com/sdk/gcloud
[example_android_script]: https://github.com/leancodepl/patrol/blob/master/dev/e2e_app/run_android_testlab
[example_ios_script]: https://github.com/leancodepl/patrol/blob/master/dev/e2e_app/run_ios_testlab
[firebase test lab]: https://firebase.google.com/products/test-lab
[aws_device_farm]: https://aws.amazon.com/device-farm
[install the gcloud tool]: https://cloud.google.com/sdk/docs/install
[gcloud_android]: https://cloud.google.com/sdk/gcloud/reference/firebase/test/android/run
[gcloud_ios]: https://cloud.google.com/sdk/gcloud/reference/firebase/test/ios/run
[alpha version of the gcloud tool]: https://cloud.google.com/sdk/gcloud/reference/alpha/firebase/test/android/run#--grant-permissions
9 changes: 8 additions & 1 deletion docs/ci/overview.mdx
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
---
redirect: /documentation/ci/overview
title: CI - Overview
---

This section of the documentation is focused on running Patrol tests as part of
your Continuous Integration workflows.

Having tests doesn't bring you any benefits if you don't automatically verify
that they pass. We know this too well, and we're putting a lot of work into
making it easy to do so.
152 changes: 151 additions & 1 deletion docs/ci/platforms.mdx
Original file line number Diff line number Diff line change
@@ -1,3 +1,153 @@
---
redirect: /documentation/ci/platforms
title: CI - Platforms
---

In this document, we'll outline a few ways to run Patrol UI tests of Flutter
apps.

Generally, the solutions for running UI tests of mobile apps can be divided into
2 groups:

- Device labs - platforms that provide access to mobile devices in the cloud. You
upload an app binary with tests to the device lab, which runs the tests and
reports the results back to you.

- Traditional – containers or VMs, either managed or self-hosted. In this
approach, you get access to the shell, so everything is possible. You manually
script what you want to achieve, which is usually: installing the Android/iOS
SDK tools, creating a virtual device, running tests, and collecting results.

There are quite a few solutions in each of these groups, and each is unique, but
generally, **device labs trade flexibility for ease of use**. They're a good fit
for most apps but make certain more complicated scenarios impossible.

# Device labs

### Firebase Test Lab

[Firebase Test Lab] is one of the most popular device labs. It is a good choice
for most projects.

You upload the app main app, the test app, select devices to run on, and after a
while, test results along with a video recording are available.

Firebase Test Lab has a large pool of physical and virtual devices.

See also:

- [Firebase Test Lab pricing]

### emulator.wtf

[emulator.wtf] is a fairly new solution created by Madis Pink and Tauno Talimaa. It
claims to provide a 2-5x speedup compared to Firebase Test Lab, and 4-20x
speedup compared to spawning emulators on CI machines. It works similarly to
Firebase Test Lab - you upload your main apk, test apk, select emulators to run
on, and the rest is up to emulator.wtf - it runs the tests and outputs results.

The emulators are indeed rock stable. Emulator.wtf automatically records videos
from test runs, and it presents the test results nicely.

It's a solid choice if you can accept that your tests will run only on Android
emulator.

Reports are available in JUnit.

See also:

- [emulator.wtf pricing]

### Xcode Cloud

[Xcode Cloud] is a CI/CD platform built into Xcode and designed expressly for
Apple developers. It doesn't support testing on Android.

Since integration tests written with Patrol are also native `XCTest`s, it should
be possible to run Patrol on Xcode Cloud. We plan to research it soon and share
our findings here.

### Other

Another popular device lab is [AWS Device Farm].

If your use-case is highly specific, you might want to build an in-house device
farm. A project that helps with this is [Simple Test Farm].

### Limitations

We mentioned above that device labs make certain scenarios impossible to
accomplish.

An example of such a scenario scanning a QR code. One of the apps we worked on had
this feature, and we wanted to test it because it was a critical part of the user
flow. When you have access to the shell filesystem (which you do have in the
"manual" approach, and don't have in the "device lab" approach), you can easily
[replace the scene that is visible in the camera's viewfinder][so_viewfinder].

This is not possible on device labs.

# Traditional

### GitHub Actions

[GitHub Actions] is a very popular CI/CD platform, especially among open-source
projects thanks to unlimited minutes.

Unfortunately, running Flutter integration tests on GitHub Actions is not a
pleasant experience.

**Android**

We used the [ReactiveCircus/android-emulator-runner] GitHub Action to run
Android emulator on GitHub Actions. Our takeaway is this: Running an Android
emulator on the default GitHub Actions runner is a bad idea. It is slow to start and
unstable (apps crash randomly) and very slow. Really, really slow. We tried to
mitigate its instability by using [Test Butler], but it comes with its own
restrictions, most notably, it doesn't allow for Google Play Services.

**iOS**

We use the [futureware-tech/simulator-action] GitHub Action to run iOS simulator
on GitHub Actions is stable. But given that the iOS simulator is just that – a
simulator, not an emulator – the range of cases it can be used for is reduced.
For example, there's no easy way to disable an internet connection, which makes it
very hard to test the behavior of an app when offline.

Bear in mind that to run an iOS simulator on GitHub Actions, you have to use a
macOS runner. 1 minute on macos-latest counts as 10 minutes on ubuntu-latest.
You can also use a custom runner – more on that below.

Custom Runners Workflows on GitHub Actions can run on external runners, in
addition to default runners such as ubuntu-latest and macos-latest.

One example of such a custom runner provider is BuildJet. We tried running
Android emulator on it, hoping that the performance benefits it brings would
help with the abysmal stability, but we've found that, even though the emulator
works faster and is more stable, it sometimes just crashes with no actionable
error message.

### Other

There are many more CI/CD platforms. Some of the most popular include
[CircleCI], [CirrusCI], and [GitLab CI/CD]. There are also CI providers that are
focused specifically on mobile apps, for example [Bitrise] and [Codemagic]. If
you used these platforms, we (and other Patrol users) will be happy to hear
about your experiences!

[github actions]: https://github.com/features/actions
[aws device farm]: https://aws.amazon.com/device-farm
[emulator.wtf]: https://emulator.wtf
[emulator.wtf pricing]: https://emulator.wtf/pricing
[firebase test lab]: https://firebase.google.com/docs/test-lab
[firebase test lab pricing]: https://firebase.google.com/docs/test-lab/usage-quotas-pricing
[xcode cloud]: https://developer.apple.com/xcode-cloud
[test butler]: https://github.com/linkedin/test-butler
[reactivecircus/android-emulator-runner]: https://github.com/ReactiveCircus/android-emulator-runner
[futureware-tech/simulator-action]: https://github.com/futureware-tech/simulator-action
[simple test farm]: https://github.com/DeviceFarmer/stf
[so_viewfinder]: https://stackoverflow.com/questions/13818389/android-emulator-camera-custom-image
[circleci]: https://circleci.com
[cirrusci]: https://cirrus-ci.org
[gitlab ci/cd]: https://docs.gitlab.com/ee/ci
[bitrise]: https://bitrise.io
[codemagic]: https://codemagic.io/start
25 changes: 24 additions & 1 deletion docs/compatibility-table.mdx
Original file line number Diff line number Diff line change
@@ -1,3 +1,26 @@
---
redirect: /documentation/compatibility-table
title: Compatibility table
---

The following table describes which versions of `patrol`
and `patrol_cli` are compatible with each other.
The simplest way to ensure that both packages are compatible
is by always using the latest version. However,
if for some reason that isn't possible, you can refer to
the table below to assess which version you should use.

| patrol_cli | patrol | Min Flutter Version |
| -------------- | -------------- | ------------------- |
| 3.4.1+ | 3.13.1+ | 3.24.0 |
| 3.4.0 | 3.13.0 | 3.24.0 |
| 3.3.0 | 3.12.0 | 3.24.0 |
| 3.2.1 | 3.11.2 | 3.24.0 |
| 3.2.0 | 3.11.0 - 3.11.1| 3.22.0 |
| 3.1.0 - 3.1.1 | 3.10.0 | 3.22.0 |
| 2.6.5 - 3.0.1 | 3.6.0 - 3.10.0 | 3.16.0 |
| 2.6.0 - 2.6.4 | 3.4.0 - 3.5.2 | 3.16.0 |
| 2.3.0 - 2.5.0 | 3.0.0 - 3.3.0 | 3.16.0 |
| 2.2.0 - 2.2.2 | 2.3.0 - 2.3.2 | 3.3.0 |
| 2.0.1 - 2.1.5 | 2.0.1 - 2.2.5 | 3.3.0 |
| 2.0.0 | 2.0.0 | 3.3.0 |
| 1.1.4 - 1.1.11 | 1.0.9 - 1.1.11 | 3.3.0 |
37 changes: 36 additions & 1 deletion docs/debugging-patrol-tests.mdx
Original file line number Diff line number Diff line change
@@ -1,3 +1,38 @@
---
redirect: /documentation/debugging-patrol-tests
title: Debugging Patrol tests
---

If you want to debug your application during patrol tests,
you can do in Visual Studio Code by attaching a debugger to the running process.
Here is how you can do it:

1. In your `launch.json` file, add a new configuration for attaching debugger to a process:

```json
{
"name": "attach debugger",
"request": "attach",
"type": "dart",
"cwd": "integration_test",
"vmServiceUri": "${command:dart.promptForVmService}"
}
```

2. Run your patrol tests using `develop` command with the same arguments as you would normally do.

3. When the tests will start running, at some point you will see a message with the VM service URI.
Copy the URI from the message.

`The Dart VM service is listening on http://127.0.0.1:63725/57XmBI_pwSA=/`

4. From "Run and Debug" tab in Visual Studio Code, select the configuration you have created in step 1.
You will be prompted to enter the VM service URI. Paste the URI you copied in step 3.

5. Once the debugger is attached, you can set breakpoints and debug your application as you would normally do.

<Info>
Intellij/Android Studio does not support attaching a debugger to a running process via Observatory Uri.
Therefore you cannot achieve the same behavior in those IDEs (See this [issue]).
</Info>

[issue]: https://github.com/flutter/flutter-intellij/issues/2250
Loading
Loading