Excess flour in banneton after baking

I finally got around to buying bannetons for my bread baking and just used one for the first time. Worked great! However, I used rice flour to coat the interior and, after releasing the dough, there is quite a bit in the grooves throughout the bowl and it doesn't fall out by shaking or tapping. So do I leave it there to build up for future proofing or should I get a brush and get it all out?