An answer to a recent question attracted a handful of downvotes, prompting the answerer to comment that the downvotes are "misleading" and as they were not accompanied by a comment were also "invalid". These (and some other) comments have since been removed by moderators, and the user has now edited their answer to begin with this notice: declaring that "all downvotes are wrong and misleading", and again declaring the downvotes "invalid".
I'm not linking to the answer (to try to avoid driving meta-effect downvotes). My initial instinct was to simply edit it and remove this text, as in my opinion this is in no way part of an answer to the question (no matter how much the author may claim otherwise). I'd be inclined to allow it if were accompanied by some explanation of why the downvotes are wrong (i.e. something of a form like "people downvoted because they believe x, but regulation y shows otherwise", with appropriate evidence). But this provides no such explanation, it's merely a declaration that downvotes are "wrong and misleading" accompanied by an assertion that complaining about downvote the answerer perceives to be "wrong and invalid" is "a legitimate part of the answer".
What's the community opinion? Is this critical commentary on votes received acceptable as part of an answer?