Treegoat presents an AI model that measures how interesting speeches are

Voicebot Research today published a report on an analysis of a new AI model introduced by tree goat. The model analyzes the text of the speeches and assigns an “interest” score between 0.0 and 1.0 for each sentence of the text of a speech. You can then add the measures to provide a general interest score for the entire speech. What you have is a new way of evaluating speeches based solely on their text.

Treegoat asked Voicebot Research to evaluate the model output for an independent assessment of what we can learn from how AI evaluates speech. You can download the full report with Voicebot analysis here.

What the AI ​​model sees

Martin Luther King, Jr.’s “I Have a Dream” speech is a good example to demonstrate what the model produces. It’s worth noting that this famous speech is one of the highest-scoring ones we evaluated in terms of its “interest” score. You can see from the model output that this speech starts with some text that Treegoat’s model finds very interesting and continues through most of the speech until the end.

King’s most famous speech started with an interest score above 0.8 and changed a bit throughout the speech, but overall it stayed high until the end. The AI ​​model generated an overall average interest score of 0.88. We can compare this to a lesser-known speech by the former First Lady of the United States, Barbara Bush. Her Wellesley College commencement speech in 1990 shows a lower general interest score of 0.33 with a much wider variance.

Franklin Delano Roosevelt, the 32nd president of the United States, fared even worse in arguably his most famous speech. The speech, known for the famous line, “a quote that will live in infamy,” earned just an interest score of 0.18. You may notice that Roosevelt’s speech is shorter horizontally than King’s. The horizontal x-axis reflects the number of sentences in the speeches evaluated. “Pearl Harbor Address to the Nation” is only 28 sentences compared to 78 for “I Have a Dream.”

There are a couple of good reasons for Roosevelt’s low performance on the scale of interest in the speech about the attack on Pearl Harbor that launched the United States into World War II. The report breaks this down in detail and shows why it was consistent with the former president’s other speeches at the time.

These were different speakers at different time periods, delivering different types of speeches. However, the Treegoat model allows us to directly compare how interesting the speech text was without the intrusion of opinion. That is important. To date, opinion has been the only method for evaluating the quality of speeches. More on that topic is below.

Download Report

The origins of the AI ​​model

“The creation of the speech analytics model grew out of the work Treegoat was doing creating models to analyze and identify the most interesting moments in podcasts for our Marbyl app, which will be released on the iOS and Android app stores in late of the year,” said Matthew Groner, Treegoat’s director of product. He added: “Working with various training data sets and large blocks of audio, we began to investigate speech and its similarities and differences to podcasts and then built separate models specifically to analyze speech.”

We’ve seen a lot lately about the power of AI models to generate text. Open AI GPT-3 is the most famous, and many new applications and entire businesses are being built on the famous AI model. There are also models like the one provided by grammatically, which will evaluate compliance with the grammar rules of the text and recommend editions. The application of AI models to evaluate text based on how interesting the language is in an entire speech or any document is a novel application.

The power of subjective language

The analysis includes the results of more than 120 speeches. Voicebot had no role in the development or training of Treegoat’s AI model; however, we were given complete freedom to submit pitches for AI model evaluation and to evaluate the results independently.

In addition to interest, the model also assesses the level of subjective language used in each sentence and then adds this into an overall “subjectivity” score that is also presented on a scale of 0.0 to 1.0. This allowed us to trace the relationship between “interest” and “subjectivity” for the discourses.

You can see from the graph above that there is a strong correlation between the use of subjective language and how interesting a speech is rated. While there are some outliers in the data set, the pattern is easy to identify through visualization. There is also a branch of the results, which has an interesting explanation highlighted in the full report, where the chart data calls are addressed individually.

Four dimensions of discourses

While most of our analysis focused on the model output, we also identified two important factors about discourses that go beyond what the AI ​​model tries to measure. First, we learned that all speech assessments today are completely opinion based. There is no objective or mathematical method to measure how interesting a speech is. Influential people in the media, finance, academia, and government tell us if a speech is good, interesting, or not. The idea that Treegoat could inject a more objective assessment of speeches that decouples from preconceptions and individual biases is intriguing.

Second, the way people evaluate speeches goes well beyond the text of a speech. Their opinions are made up of four dimensions including context, speaker, delivery, and text. The text is unique because it has objective elements that the other dimensions lack. Eliminates the influence of opinion bias. It is also the only element that is under the full control of the speaker.

While you can’t control what preconceptions and prejudices listeners have about the speaker, topic, or other characteristics, you can determine what words you say in a speech and how they’re arranged. It makes sense that speakers would want to write more interesting speeches. One way they can do this is to optimize the text. Treegoat’s Groner added:

There are so many possible applications of this model, including for researchers or educators who want to dig deeper into political speech analysis, speechwriters who could use the model output to pretest and refine speeches, and in training other AIs to create speeches. attractive.

You’ll find more than 30 charts and 25 pages of analysis in the full report. You can download it at no cost by clicking the button below. Let me know what you think.


OpenAI Leads a $50 Million Raise for AI Video Editor

The latest details on the Amazon layoffs and the impact on Alexa

GitHub preview Speech function coding for Copilot AI programming assistant

Leave a Reply

Your email address will not be published. Required fields are marked *