How the news media censors itself

What I learned by building a news app.

Several years ago I built a simple local news aggregator called LocalReader. It scanned RSS feeds to download local news from different sources and show them all on a single page, which was pretty much the limit of technology at the time. With new breakthroughs in AI technology, I decided to build a new type of news aggregator - one that would check news feeds, download articles and use AI to deconstruct those articles into just the basic facts. My goal was to help fix what I saw as the primary issue of the news media, which I believed was biased reporting and dishonesty. By stripping the articles of bias, and collecting verifiable facts from numerous sources, I thought the app would present a more honest accounting of the story.

I built the app (True North News), which actually worked well, but I found that the facts that were being collected from mainstream news sources were sometimes missing critical context that was present on social media. I could often get a better understanding of a major story by reading social media than by reading the facts collected from several mainstream sources. Why?

Missing Statistics

After watching news feeds for months, scanning thousands of articles and comparing them to social media coverage, what I found is that while dishonesty is actually very rare in the mainstream news, lying by omission is somewhat common. I noticed that these omissions are often statistics that are shared on social media but are absent in the mainstream news.

Missing data is important if it changes the framing of a story. Consider the recent controversy of the massive social services fraud in Minnesota. This has been an ongoing issue for years, and was extensively covered in the local news media. What was missing from the mainstream news coverage was accurate data on the involvement by members of the Somali immigrant community, which was necessary to fully understand the issue. This became a scandal when an article in City Journal discussing alleged terrorist funding via welfare fraud went viral, and then a week later President Donald Trump issued a controversial Thanksgiving message blasting Somali immigrants. The day after Trump’s message there was an article in The New York Times discussing the explosion of fraud in Minnesota which specifically included statistics on the involvement by members of the Somali community. Would the New York Times have published this article if Trump hadn’t made his comment? So, until Thanksgiving, if you wanted to try and figure out why fraud had exploded in Minnesota and you relied only on the mainstream news media, you were left without key information.

For example, here is a typical news report on the fraud from a year ago. Notice what is missing?

While some of these omissions might have an idealogical cause, we also need to consider the role that experts play in our modern society. The mainstream news relies heavily on experts in their reporting. The problem is that some of the experts quoted in mainstream reporting are not disinterested parties - they are activists with a vested interest in a certain theory or perspective. So, instead of accurate journalism, what we sometimes get is data from biased experts presented as the whole truth (while any data that contradicts their theory is left out). For example, how many times have you read in the news that experts had “debunked” a certain claim only for it to be later proven true, or read in the news about some fashionable theory touted by experts, which later ended up being based on poor quality research?

Once I realized this, the obvious next step for me was to try and add this missing data to news stories in my app (using AI) to see if how much the framing of the story changes. I created a new “Relevant Data” section for each story, where I add relevant data that was not included in the mainstream reporting. I was actually surprised by how much data is available (click here to see a report on all the data I added to stories). For most stories, this data just added some additional facts that weren’t necessary to understand the story but were interesting. In a few cases, adding missing data significantly changed how an average reader would understand the story.

The goal of my news app has now gone from aggregation of the news to improving on the actual reporting by adding missing data. I have also added social media discourse and opinion pieces by public intellectuals in an effort to broaden understanding of the stories.

It is still a work in progress, but it is amazing how much AI has already helped improve the integrity of news coverage.

I predict that AI will enable new and extremely important historical analysis of the news media in the near future. We already have easy access to digital news archives as well as vast archives of digitized books and research from various time periods. It would be fairly easy with AI to figure out what information was missing in reporting on historical events and how our understanding was subtly changed by those exclusions.