My favorite testing approach in Flutter

How golden testing saved me hours of headache

[FlutterBountyHunters]

Ignoring the variance will definitely bite you. I've had goldens that fail with a 0.0% difference where the mismatch was actually relevant.

In Super Editor we've taken two steps to deal with this. First, we created a golden runner that uses Docker to get as close to matching platforms no matter who runs the tests. Second, for tests that still don't match, we created a golden matcher that lets you specify the number of mismatched pixels instead of mismatched percent, because in practice, percent diffs are unpredictable. If your golden mismatch is 103 pixels due to platform differences, then you can allow up to exactly 103 pixels difference. Otherwise it fails.

[maef]

I agree with you on option 2- setting threshold.... especially because if you are on the free tier on GitHub, then your CI can't run on Mac... that's a paid feature.

///edit: but I would say it very much depends on the project. On my project we were fine dealing with the threshold and we took extra care on PRs if we saw a changed golden

[tayabfarooq]

Your followers eagerly look for your content that you post! Please take care of them by uploading something for them as much as possible!

[atreeon]

Yes to Golden tests! was waaay too slow to adopt them. Unit tests for UI stuff - yes. Also integration tests for UI stuff - better. All three? perfect!

[jayKrollin]

I struggle with the exact same issue but I would highly recommend to not use a tolerance. E.g. if any dev changes a padding by 1px the related test results could be false positives.

[tbm]

the pain is the result still different between OS

[coffee-is-power]

you should've just changed the implementaton to instead of comparing the bytes, just open the two images and compare the *pixels* not the bytes which is what flutter should do by default but it doesn't