Bot for summarising Reddit comment threads
I want to create a bot that distills the distinct threads of thought within Reddit comment sections and then summarises them.My initial approach is to use BERTopic to extract the individual topics present in the comments section for a given post, group individual users’ comments by topic, and then pass those comments to PEGASUS for abstractive summarisation. I want to do this for each comment group of the most dominant topics discovered by BERTopic, constituting p% of the total comment pool in the same way you might select the top n principal components that are responsible for p% of a dataset’s variability.My initial approach is quite simple and naive, so I would be very grateful for any guidance that might help improve the approach and produce higher-fidelity text summaries for each distinct topic.