Instagram uses 'I will rape you' post as Facebook ad in latest algorithm mishap

Instagram used a user’s image which included the text “I will rape you before I kill you, you filthy whore!” to advertise its service on Facebook, the latest example of social media algorithms boosting offensive content.

Guardian reporter Olivia Solon recently discovered that Instagram, which is owned by Facebook, made an advertisement out of a photo she had posted of a violent threat she received in an email, which said “Olivia, you fucking bitch!!!!!!!” and “I Will Rape You”.

Instagram selected the screenshot, which she posted nearly a year ago, to advertise the photo-sharing platform to Solon’s sister this week, with the message, “See Olivia Solon’s photo and posts from friends on Instagram”.

The ad has surfaced at a time when Facebook is facing intense scrutiny over the ethical failings of its algorithms and advertising tools. Last week, ProPublica reported that Facebook was allowing advertisers to target users interested in the topic of “Jew hater” and “How to burn Jews” – categories that the social media site had automatically created. Journalists were able to pay $30 to target “promoted posts” to the antisemitic groups.

Others quickly discovered that there was a range of bigoted and derogatory terms that Facebook allowed for ad targeting and that Google and Twitter had similar problems.

Facebook chief operating officer Sheryl Sandberg issued a mea culpa on Wednesday and said the company was changing its policies as a result. Facebook disabled the targeting system that created the offensive categories, and Sandberg said the site would only allow targeting options reviewed by humans in the future.

It’s unclear why Instagram chose to highlight Solon’s hate mail to friends on Facebook. When she posted the screenshot last year, she wrote: “This is an email I received this afternoon. Sadly this is all too common for women on the internet. I am sure this is just an idiot rather than any kind of credible threat but it’s still pretty vile.”

The photo received three likes and more than a dozen sympathetic comments. It’s possible that Instagram’s algorithm considered it an “engaging” post because of the number of responses. But given existing technology that can analyze words embedded in images, it’s unclear if Instagram has any systems in place that would detect the violent and abusive text and flag it as an inappropriate choice for an ad.

An Instagram spokesperson apologized and claimed that the image was not used in a “paid promotion”. “We are sorry this happened – it’s not the experience we want someone to have,” the statement said. “This notification post was surfaced as part of an effort to encourage engagement on Instagram. Posts are generally received by a small percentage of a person’s Facebook friends.”

The spokesperson said these types of posts were designed to motivate people who aren’t on Instagram or hadn’t been on the site recently to visit the platform by showing them content from their friends. The company did not answer questions about how widely the post was shared, but said it would have surfaced to some of Solon’s Facebook friends.

Facebook’s algorithms have long faced backlash for inadvertently causing its users pain and harm. In 2014, the company was forced to apologize over its “Year in Review” clips, which created an automated series of posts that highlighted hurtful memories of deceased loved ones and tragic incidents – posts that presumably had a lot of “engagement”. The company’s “On This Day” nostalgia feature, which resurfaces old posts, has faced similar criticisms.

Facebook also admitted this month that an influence operation probably based in Russia bought $100,000 worth of ads to promote divisive political messages. On Wednesday, 20 Democratic senators and representatives urged the Federal Election Commission to “develop new guidance for advertising platforms on how to prevent illicit foreign spending in US elections”.

Last week, Facebook was also mocked after it sent a ProPublica reporter behind the “Jew hater” investigation an automated email suggesting that she buy an ad to promote her story that exposed Facebook’s embarrassing ad practices.