I found the problem!
Actually, @schlonzo reported the same problem in this post:
Turns out, removing the SetMaskFilter + Blur from the class makes everything run nice and smooth again ;) looks like using MaskFilters inside a ForEach causes some kind of memory leak.
But thank you @bjoern for jumping in and offering help!