Are you sure it`s the brakes chattering,could be the suspension making the noise. Seems like if the brakes were the problem it wouldn't quit when you applied them. But I`m no mechanic either.
Quite certain it's the brake pads making the chatter. That's why it stops (the chatter) when they are applied.
I'm not intending to address the issue unless it begins to become more annoying. It is only slightly annoying now.
Unless this can lead to a catastrophic failure. I don't think it can. The pads are just bumping around a bit over rough roads.