Is it possible to run multiple test files on FTL at the same time?

[prolificcoder]

Is there an existing issue for this?

• I have searched the existing issues

Steps to reproduce

1. Follow https://patrol.leancode.co/ci/firebase-test-lab
2. https://github.com/leancodepl/patrol/blob/master/packages/patrol/example/run_android_testlab#L21

Actual results

First, I know this is not a problem with patrol but directly with firebase test lab but wondering if you have a workaround or a solution for this.

From the firebase docs: https://docs.flutter.dev/testing/integration-tests, they ask us to create assembleDebug
./gradlew app:assembleDebug -Ptarget=integration_test/<name>_test.dart for each test file.

And you are doing the a similar example https://github.com/leancodepl/patrol/blob/master/packages/patrol/example/run_android_testlab#L21

But after a while you can't have all tests in the same file, we now have about 25 test files and I am trying to figure out how can we run all of these at the same time in firebase test lab.

Things I tried so far

1. Create a import file with a single main called all_tests.dart and invoke each test main() in it.
2. Create assembleDebug with multiple -Ptarget arguments eg:
   ./gradlew app:assembleDebug -Ptarget=integration_test/first_test.dart -Ptarget=integration_test/second_test.dart
   But in first case its running tests only in the first file and in second case only the last file.

I will also post this on flutter integration_test github issues but usually its easier to talk with folks on smaller projects
also let me know if you have a different method to contact for questions and this is reserved more for issues

Logs

Logs

n/a

Patrol version

n/a

Patrol Doctor output

n/a

Flutter Doctor output

Flutter Doctor output

[!] Flutter (Channel stable, 3.7.9, on macOS 13.3.1 22E772610a darwin-arm64, locale en-US) ! Warning: `dart` on your path resolves to /opt/homebrew/Cellar/dart/2.19.5/libexec/bin/dart, which is not inside your current Flutter SDK checkout at /Users/smalugu/fvm/versions/3.7.9. Consider adding /Users/smalugu/fvm/versions/3.7.9/bin to the front of your path. [✓] Android toolchain - develop for Android devices (Android SDK version 33.0.2) [✓] Xcode - develop for iOS and macOS (Xcode 14.2) [✓] Chrome - develop for the web [✓] Android Studio (version 2022.1) [✓] VS Code (version 1.78.1) [✓] Connected device (4 available) [✓] HTTP Host Availability

[bartekpacia]

Hi @prolificcoder,

This is a design issue in the official integration_test package. I reported this to flutter many months ago, but looks like they don't have the incentive/throughput to fix it.

I'm happy to say we were aware of this issue and are working on fixing this using the test bundling approach. Actually, we already did the hard part and can confirm it works :) It's already available on our develop branch (but it's not yet stable enough to recommend). A few more weeks of polishing, testing, and docs updates, and we'll likely release this as Patrol v2.

[prolificcoder]

That's awesome I will give this a try soon. I just want all my tests running in firebase test lab.

Also instead of test files, I also tried test groups and created on giant test file but firebase test lab only counts the first group and discards the rest :(

[bartekpacia]

Yeah it's a shame how it works currently.

I'm really happy we're doing something no one has done before with Flutter :)

[prolificcoder]

I see you are very active on this project and raising issues to flutter, I am happy help.
Let me know what would be some simple tasks to pick up. I will first try to setup my small POC project with patrol.

[bartekpacia]

Sure, just please make sure you follow the setup docs thoroughly since it's kinda tricky to set up right (it's kinda sad but that's how things are – you have to touch some native code and config file). But that's one-time setup, you do it, and once it works, it works :)

Regarding the simple issue to contribute: I think a cool candidate for a "good first issue" can be #1217. Thanks!

[prolificcoder]

Just started with the setup on an existing project and running into an issue: prolificcoder/house_stats#16 will check back tomorrow. Looks like I do not have a correct gradle version?

Also I can easily upgrade this project to 3.10

[bartekpacia]

Ah yes, before running patrol test, do a flutter run. This will generate a few files, including gradlew.

[prolificcoder]

Thanks yes, I missed that part, after a few tweaks I am able to run my test locally and on FTL

[prolificcoder]

@bartekpacia if you get a chance can you take a look at this: prolificcoder/house_stats#16

I think I am doing everything correctly but running into an issue with creating the assembleDebug build. You can see the log in this comment prolificcoder/house_stats#16 (review)

[prolificcoder]

i can create an issue on this project too

[bartekpacia]

I can create an issue on this project too

Thanks, that's not necessary.

First of all, I recommend you do dart pub cache clean.

Then, make sure you're using at least Flutter 3.3. We do not provide any support for older Flutter versions.

Third, I recommend you to set min API to 21 instead of 19 (so you won't have to use multidex). The loss of users is negligible, but you get "saner" build and dependencies.

[prolificcoder]

oh that's exactly what i did and was about to message. Sorry for the false alarm.

Pushing through it

[prolificcoder]

Also I am on flutter 3.7, changing to API 21.. on it.

[bartekpacia]

If you want to use a specific Dart version, you have to declare it in pubspec.yaml. So if you're using Flutter 3.7.x (which uses Dart 2.19.x), you have to put this in your pubspec:

environment:
  sdk: '>=2.19.0 <3.0.0'
  flutter: '>=3.7.0'

[prolificcoder]

ok. I am running into an odd issue. Looks like the PatrolTestRunner on the android side is not being compiled. May be I should clean my gradle cache.. this is the error

I am able to flutter run and flutter test integration_test/non_patrol_test.dart

Patrol test logs

/Users/smalugu/code/house_stats/android/app/src/androidTest/java/com/example/house_stats/MainActivityTest.java:5: error: cannot find symbol import pl.leancode.patrol.PatrolTestRule; ^ symbol: class PatrolTestRule location: package pl.leancode.patrol /Users/smalugu/code/house_stats/android/app/src/androidTest/java/com/example/house_stats/MainActivityTest.java:6: error: cannot find symbol import pl.leancode.patrol.PatrolTestRunner; ^ symbol: class PatrolTestRunner location: package pl.leancode.patrol /Users/smalugu/code/house_stats/android/app/src/androidTest/java/com/example/house_stats/MainActivityTest.java:11: error: cannot find symbol public PatrolTestRule rule = new PatrolTestRule<>(MainActivity.class); ^ symbol: class PatrolTestRule location: class MainActivityTest /Users/smalugu/code/house_stats/android/app/src/androidTest/java/com/example/house_stats/MainActivityTest.java:8: error: cannot find symbol @RunWith(PatrolTestRunner.class) ^ symbol: class PatrolTestRunner /Users/smalugu/code/house_stats/android/app/src/androidTest/java/com/example/house_stats/MainActivityTest.java:11: error: cannot find symbol public PatrolTestRule rule = new PatrolTestRule<>(MainActivity.class); ^ symbol: class PatrolTestRule location: class MainActivityTest 5 errors

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:compileDebugAndroidTestJavaWithJavac'.
> Compilation failed; see the compiler error output for details.

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 2s
: 68 actionable tasks: 20 executed, 48 up-to-date

✗ Failed to build apk with entrypoint test_bundle.dart (Gradle build failed with code 1) (22.9s)
Exception: Gradle build failed with code 1