This is all part of the "slippery slope" logical fallacy, which states that "once X happens, how will you stop Y?" And the answer is, the same way you stop Y now. X has nothing to do with Y.
Legalizing marriage between two unmarried adults in no way changes bars to polygamy. Sorry, uh-uh, just doesn't do it. Just as marriage between a man and a woman does not promote marriage between a man and many women, or between a brother and sister (all of which are man-woman marriages.)
I also found this article comparing (again) the movement against inter-racial marriage to same-sex marriage. And, yes, the slippery slop argument was used there too:
It is clear from the most recent available evidence on the psycho-sociological aspect of this question that intermarried families are subjected to much greater pressures and problems then those of the intermarried and that the state's prohibition of interracial marriage for this reason stands on the same footing as the prohibition of polygamous marriage, or incestuous marriage or the prescription of minimum ages at which people may marry and the prevention of the marriage of people who are mentally incompetent.Same argument.
BTW, I am a firm believer that marriage is between two people.