Have you tried using AI to improve how you write? This time, we focus on an experiment in editing a research article which was originally condensed from various parts of a thesis. I attempted to simplify some writing by asking the AI to summarise it. Not surprisingly, the results needed the inclusion of many of the original ideas as they were too summarised.
What did I do?
For this experiment I used a draft article on the development of active sports tourism in developing countries. The author identified some key success factors for rock climbing tourism in one such country and devised a framework of interrelated key success factors. The author had already condensed information from several chapters to form an article of approximately 13250 words (excluding the reference list). The journal’s word limit is 10000 words. This experiment was designed to assess the way in which this AI handled summarising parts of the article which are included in most journal articles as distinct sections.
I used the free version of Chat Generative Pre-training Transformer (Chat GPT) and made no attempt to explore the subscription-based version. Each time, I fed the simple command “summarise this:” followed by one section of the article. I trialled the AI with the literature review, research context, data collection, data analysis, research findings (in two parts) and finally a section on the discussion and conclusion. Because the author had crafted the introduction very well, and I only needed to fine tune some sentences, I did not ask for a summary of this part. I analysed two versions of the literature review and the final section.
What did the AI do?
In this experiment, the AI did not include references even when asked to and could not distinguish a quote from the body of the text, although it might be trained to do this. The AI removed any examples in all sections.
The most comprehensive summaries came from the data collection, data analysis and research findings sections. The main points were smoothly set out, with redundant language eliminated and a logical flow of ideas (just as in the original text). Chat GPT “summarised” the first and very concise paragraph of the findings section by reproducing the original, practically word for word.
The least helpful summaries were of the literature review and the discussion and conclusion section. Chat GPT offers to write an alternative version after the first version it provides. For the literature review, the first version included the language “is a crucial aspect,” “plays a crucial part,” “are necessary” and “are essential.” For a post-graduate publication, this seemed quite simplistic so I checked the second literature review summary. This was more integrated than the first, with less repetition of words and ideas. However, there was no nuancing of the arguments and the hard work of the researchers included in the literature review was entirely lost in a fairly simple listing of concepts. The author’s meticulous analysis was, of course, also lacking in this summary of the literature review. One example is “Social mechanisms and structures of community capacity building are opportunities to nurture a wider range of active sports tourists (Jones et al., 2018).” In Version 2, this was reduced to “Community capacity building plays a crucial role in sport tourism development.”
As for the discussion and conclusion section, the AI ignored any comparisons of the results with the current literature in the field. Interestingly, while Version 2 seemed a more sophisticated attempt, it would have been improved by including some points present in Version 1 which had been removed. Both versions of the summary consisted of a structure offering a list of key success factors for rock climbing tourism with some elaboration of each point. Some remarks followed about how they apply to fostering active sports tourism in developing countries.
Surprisingly, part of Version 2 in this section made less sense than its corresponding part in Version 1. Awkward language here included “the human element, led by the sport-based community and pioneers, acts as a catalyst for active sports tourism development.”
What did I learn?
While Chat GPT can be useful for summarising, this works better for sections which deal with data collection, data analysis and findings. I do not recommend this AI to summarise a literature review. If you want to develop articles from your thesis, select one aspect of your broader literature review and include that in each article. I doubt that Chat GPT has the capacity to do this effectively yet for sophisticated texts like academic literature reviews, even if you develop very accurate and specific prompts. You would need to add a great deal of nuancing to what it produces (including checking references). For a discussion and conclusion, you might like to try using AI as a first step, then insert other ideas, sentences and paragraphs to convey your true meaning in depth. For instance, in the summarised conclusion for the case study article, details about current support from government bodies and sports associations for active sports tourism could be put back in. This adds colour and, by extension, illustrates underlying mechanisms within the country which promote active sports tourism, which is useful for policy development.