I agree, however; if you retry, get kicked, leave, or w/e the voteban does NOT get reset for a while... That's how you catch the assholes who come in, kill EVERYONE, then leave, then come back under a new name. If you type voteban in console, it shows you everyone connected, and how many votes against them, so if someone enters with... Let's say 5... Chances are they're the asshole you just tried votebanning that left.
I think the votes last as long as the person who initiated the vote is in the server and has not voted on someone new.