ADS Capstone Chronicles Revised

9

Table 5 Average Bert Score per LLM-assigned Topic Label

stances/actions on climate change (topic 3), and climate change on a global stage (topic 4). Several subcategories of climate change were also identified such as greenhouse gas emissions (topics 1 and 3), Al Gore, who played a significant role in raising awareness for climate change (topic 2), and the paris (agreement) (topic 4). Other keywords identified suggest that discourse surrounding climate change is dominated by politics. Notably, the prominent presence of political figures such as donald trump (topic 0), white house (topic 2), and president obama (topic 3) highlight the pervasiveness of political ideologies in how climate change is discussed in the media. It is important to note that these findings may be influenced by the methodology used to obtain the data, which will be explained further in the discussion section. To optimize resources and minimize the call frequency to the OpenAI API, the labeling procedure was split into three distinct runs. Each run contained a batch of 50 news snippets. Empirical observations showed the LLM successfully assigned labels to approximately 50% of snippets. The remainder of the snippets were labeled unknown , reflecting the limitations in generating definitive labels from a given text. We evaluate GPT model outputs against LDA-derived topic labels using BERTScores, which return precision, recall, and F-1 scores. The evaluation yielded average F-1 scores (see Table 5) as follows: 0.87 for carbon emissions reduction, 0.85 for climate action efforts, 0.84 for climate change impact, 0.88 for climate crisis, and 0.89 for global warming impact. 5.1 EvaluationofResults

Topic Labels

Avg. F-1 Score

carbon emissions reduction

0.867 0.854 0.844 0.876 0.890

climate action efforts climate change impact

climate crisis

global warming impact

6 Discussion 6.1 Strengths and Weaknesses

6.1.1 Multiple Subjects The snippets were pulled from news broadcast transcriptions using a 15-second window in which one of the six climate phrases exists in the snippet. The window creation methodology needs to be clarified, which results in the topic models being analyzed for multiple and unrelated subjects. 6.1.2 Reporting Bias Climate change is a polarizing issue, and these scenarios are discussed through the lens of those with political bias. In many snippets, climate change is discussed in the context of a political platform or government policies, rather than the main subject of the conversation. 6.1.3 LLM topic model evaluation There is no generally agreed-upon method for calculating the coherence of topic labels generated by GPT-3.5-turbo. The recommended method is to create a mock LDA model to fit the Gensim coherence model API, and then use the topic labels generated by ChatGPT. 6.1.4 BERT Score with LDA as reference We evaluated the GPT topic labels using the BERTScore which computes a similarity score between each token in the reference sentence or

25