Bumping Into Walls
Or One Way to Build a House
Another day, another approach to testing what GPT can actually do in terms of understanding fiction and readers and finding productive links between them.
A different experiment today- I pulled out 500 word excerpts from my Hugo finalist short stories, broken into a set of high ranked training stories, then smaller sets of testing stories split into high, mid and low rank.
I presented a training story to GPT as an example of my tastes, then asked it to select which of the three test stories it thought I would enjoy the most (one each of high, mid and low ranked).
Out of the four tests I ran, it selected the mid ranked story three times and the top ranked story once. Out of interest I also asked GPT to explain it’s reasoning. A typical example is below:
***
Based on the example story provided, it seems that the reader enjoys stories that have a unique and imaginative premise, as well as a strong protagonist with depth and an emotional arc. The use of humor and playful language is also present, and the reader seems drawn to stories with supernatural or fantastical elements. With this in mind, the story that I predict the reader would enjoy the most is A. This story has a clear and intriguing premise about a woman on a journey with magical shoes. The protagonist is well-developed and has a strong emotional arc as she reflects on her past and struggles to move forward. The use of humor is also present in the narrative, such as the playful pun about numbers. Additionally, the use of fantastical elements such as nanite technology and alien planets adds to the appeal of the story. Story B may also be of interest to the reader, as it also has elements of imaginative technology and a unique premise. However, the focus seems to be more on the world-building and descriptions, rather than the protagonist's emotional journey. Story C also has a unique premise and fantastical elements, but the focus seems to be more on the relationships between characters, rather than the protagonist's personal journey. In conclusion, story A seems to align most closely with
***
This got me thinking more about how GPT does this kind of analysis. Prose quality seems limited to picking up spelling/grammar mistakes in unpolished stories, some sense of tone/atmosphere based on word frequency, and an ability to identify key themes and content elements. The problem then is GPT can only handle small chunks of “context” before you have to start from scratch again. Its focus and its scope are in some ways incompatible.
I tried feeding it a whole short story to see if it could summarise the most important elements, but it could only handle half a short story for this purpose. The summary it produced mostly focused on plot elements. Maybe the future release of GPT4-32k will overcome some of these issues. It might be possible to summarise a novel chapter by chapter, then summarise the chapter summaries, down to a scale that would then allow decent comparisons of multiple novels. My gut feeling is that LLMs have the potential to connect readers and writers in a completely new and disruptive way, but the details are still undefined.
Part of the problem is the Hugo test set is too narrow in quality and subject matter. I have asked GPT to do something that most humans would struggle to do reliably for each other. So I think I will shift focus a bit instead.
Plan D (E? what letter am I up to?) is to pivot to focusing on blurbs for ebooks recently published on Amazon. I will gather a set of these for testing purposes, but still limited to sci-fi since there are plenty of subgenres at least. If this kind of functionality works then it could be a useful tool to filter through the hundreds of new ebooks released every month on the platform.
