Whenever you add a player you risk incompatibility. In a home game, you’ve usually screened the players, and usually only deal with one new variable–one new player–at a time. In con games, you’re not just adding one person to a table–you’re building a whole table from scratch, often meeting your players for the first time.
In both cases, it can be difficult to figure out how to deal with difficult or obnoxious players. In someone’s home, particularly when the GM is the host, social expectations align to give the group the right to remove someone who doesn’t fit. It can be difficult to fight expectations–particularly since so many gamers are friends first. The infamous geek social fallacies  remind us that there are widespread assumptions about gaming, friends, and expectations of tolerance.
In public, there’s a lower barrier to entering games–and too often, an expectation that a GM will put up with any player. For a one shot, it’s often more trouble to root out a player than it’s worth–unless their problems go well beyond roleplaying. (A jerk who makes everyone miserable has to be tossed, even if it takes precious minutes out of the group’s limited time.)
Over a Longer Timeframe
For an ongoing game, public or private, there’s an expectation that the game will be more intense, that the interplay between the GM and players matters more. (A good plot won’t save you if there’s a guy who is like fingernails on a chalkboard session after session.)
Martin’s Problem Players  article has a great approach for figuring out when there are problems and guidelines for addressing them. It is written from a home game perspective, and works best for groups with a strong gaming charter/social contract.
For public games, it’s a little trickier. Often the GM and players don’t see each other outside of the session–the GM may not even have contact info for her players. The situation may be more difficult to assess, especially since the some players might never have a chance to give personal feedback without the problem player being present.
Another complication to public games is that many are designed as open–with explicit rules on who can join, how their characters are designed, and expectations for play. Advance signups and steady groups are rare in D&D Encounters , and are missing from some Pathfinder Society  and living game (like Living Forgotten Realms  or Living Arcanis ) tables. GMs may not have the power to preemptively boot a player who registered properly–it might be up to the event organizer or society bylaws.
Spoiling adventures by acting on the written module is reprehensible and earns players a bad reputation as quickly as almost any other table behavior. Even if only the GM is tipped off by your actions, it will skew the whole table’s play experience when she reacts, whether overtly or subtly. Worse for the player, GM networks tend to be tight and stories travel quickly. You don’t want to go into an adventure with the GM wondering which actions you took because you read ahead, any more than you want them wondering which 20s are real.
Cheating by players is universally frowned on. Sometimes you’ll garner sympathy from other players–being removed from play is disappointing, especially if it’s due to one failed saving throw–but cheating warps a game, and the reputation you garner will last longer than your character. It sucks, but hopefully the GM has a plan for your petrified character. After all, they don’t want you unable to participate for 3 hours of the 4 hour time slot.
Many GMs are frustrated by min-maxed  characters, particularly when the amped up character is backed by a player who expects more than their share  of spotlight time. In home campaigns, a group that enjoys breaking the system together can have a great time, with challenges that scale appropriately. In thrown together groups, too often the “well designed” characters just make everyone else feel that their characters are extras in the superhero’s story. Players can mitigate this–by being good players, stepping back and supporting others when the spotlight turns to them. Often it’s not that Bob’s character is better that frustrates them–it’s feeling that their character is useless, particularly if Bob’s character dominates their character’s role .
What’s to be Done?
If you have a negative history with a player, let the organizer know. Often they’ll be sympathetic and will follow your lead, dropping or reassigning the player as you recommend. It’s hard enough to get people to volunteer to GM–if organizers don’t back GMs, fewer GMs volunteer the next time.
If a player socially dominates the group and frustrates the other players, call out the behavior at the table. Remind them that each player is responsible for their character’s actions, enforce anti-table talk rules as necessary, or ensure that everyone has a chance to have a say. Going around the table and asking for orderly input can help establish expectations and set boundaries. If taking turns and intrusive input remains an issue, a talking stick  can be a tactile and visual reminder. (Improvise something from among your possessions if you didn’t anticipate needing a talking stick.)
Talk as much as possible. Get phone numbers or email addresses and discuss issues between games. Often repetition makes something minor feel much bigger–leading to a blow up that could have been headed off with one direct conversation. Even if you can’t communicate between games, take advantage of a game that wraps up early to get feedback .
How do you Deal with It?
My list is horribly incomplete on all sides. What behavior frustrates you when you play in public games? When you run them? How have you dealt with players who cheat, hog the spotlight, or just can’t get along with their fellows? Do you concentrate on rehabilitating problem players, or are there always more players clamoring for a chair, so there’s no reason to put up with dodgy or pushy behavior? Please share your tips in comments!