I have been using a Gill Skiff harness with a quick release button for about 5 years now and have never experienced such a problem. The only issue I had was when I first got it and sailing on a Laser 5000 my helm couldn't resist the temptation of the red button! I'll leave the rest to your imagination!
I could only suggest re adjusting all of the straps to see if it will make a difference to where the button is in relation to any buckle that may cause and unexpected swim!
Let me how your getting on