The confinements since the start of the pandemic have increased the use of smartphones and mobile apps tenfold. Amazon’s interactive video game and streaming platform, Twitch, has been popular with young people. Confronted for months with the problem of cyberbullying and criticized for its lack of moderation, the platform announced a few days ago the creation of an artificial intelligence system called “Suspicious User Detection”. The latter should make it possible to identify malicious users who create new accounts after having been excluded.

Reports of sexist, racist or homophobic harassment on Twitch are numerous. Accounts of cyber harassment have been widely shared, including the hashtag #TwitchDoBetter (#FaisMieuxTwitch) started by RekItRaven who said:

“I started #TwitchDoBetter because I’m tired of having to fight to exist on a platform that claims to be diverse and inclusive but has remained silent in the face of pleas from marginalized creators asking for more protections from hate raids.”

The Twitch team responded and announced new security measures on August 11 to better detect the circumvention of account bans and improve account verification, which were considered insufficient by streamers. It stated:

“We’ve seen a lot of conversations about the use of bots, hate raids and other forms of harassment aimed at marginalized creators. You’re asking us to do better, and we know we need to do more to address these issues.”

So Suspicious User Detection is Twitch’s latest response to cyber harassment. The tool was shared by the team, who state:

“In order to continue to make Twitch a safe place for everyone, we’re introducing a new tool that will allow moderators to take action against accounts that try to avoid bans.”

On Twitch, you can actually create multiple accounts with a single email address, which leaves banned people with the ability to return to the chats very easily and resume their harassment. It’s to try and address this problem that Twitch has just rolled out this new AI-based tool.

How Suspicious User Detection Works

In its November 30 press release, Twitch explains that the tool will:

“identify suspicious accounts through machine learning by analyzing a number of signals; thanks to them, this tool will report suspicious users to moderators, who will be able to take the necessary actions.”

Twitch does not specify the nature of the signals analyzed but specifies that two scenarios will be possible: the reporting of a“probable” case and a“possible” case.

• If a user is flagged as“probably” trying to circumvent a ban, they will be automatically hidden in the chat, but their messages will be displayed for creators and moderators “so they can decide whether to keep the restriction, monitor the user, or ban them from the channel.
• In the event that this is deemed only “possible” and not “likely,” the person will be able to post in the chat as normal, but their posts will be reported to moderators, “so they can monitor the user and prevent them from posting in the chat if necessary.”

The feature will be activated automatically on all channels and can be disabled by streamers. It will also offer the possibility to restrict the messages of suspicious users so that they do not appear in the chat. It will also be possible to deactivate the tool from the channel’s moderation settings page or to manually add users that the moderators deem suspicious.

A tool that struggles to convince

The lack of details about the signals used to define a banned or unbanned user, as well as Twitch’s warning about the risk of errors in its tool, leave room for doubt. Its effectiveness in the event of a raid is also up for debate. Twitch warns:

“machine learning will never be 100% effective, which means there are risks of false positives, or false negatives”

Many female streamers, daily targets of sexist harassment, have said they are completely unconvinced by using artificial intelligence rather than simply banning the creation of multiple accounts from one email address.

Translated from Cyberharcèlement : Twitch présente son nouvel outil de modération basé sur de l’intelligence artificielle