I never did get around to explaining this. Two years is probably long enough to wait before giving out the answer key.
This is likely why people without discernment will often put those who have it in their “wizard” mental box since they note that, like Gandalf, their articulations of difficult abstractions tend to be at once sensible, insightful, and of mysterious origin. By what art or science, they wonder, did Gandalf’s heart sense that Gollum had a part yet to play? Gollum very clearly had no utility at the time Gandalf said that to Frodo. If you can figure out the answer to that question on your own, you probably have discernment. (I’ll put a hint in the comments, but try it by yourself first.)https://aeolipera.wordpress.com/2020/06/10/notes-on-estimation-as-a-skill/
The reason Gandalf knew Gollum had a part to play is he sensed that Frodo had a blind spot he would need to overcome. At that time, Frodo couldn’t sympathize with Gollum’s wretchedness because he didn’t understand the significance yet of being the ringbearer. It’s like I said yesterday about setting out on a 100-mile race, he didn’t really understand that by the end he stood a very good chance of becoming exactly like Gollum. If he did understand it, he would have pitied Gollum from the beginning. “There but for the grace of Eru go I.” Or maybe more concretely, to get the mortality salience flowing: “That’s probably what I’m going to be like by the end of the year.” Imagine if you knew that odds were you’d transform into Gollum over the remainder of 2022 (so you’d be halfway there by the end of Summer).
So even though Gandalf couldn’t say why, he knew that Gollum’s existence at least served a purpose as a cautionary tale, like a figure from Galadriel’s mirror. And Gollum did in fact end up being Frodo’s understanding of his own darkness and wretchedness, an embodiment of what he could become very easily under the influence of the Ring. In Jungian terms we’d call it his Shadow (but we have to take Tolkien at his word that he hated allegory and this symbolism was unintentional). Frodo came to understand this as he realized he wasn’t up to the task of fighting the Ring, and even at his best moments he could only slow its progress as it ate away at his soul and sanity. That’s why the sympathy angle was important: Frodo would eventually come to see himself in Gollum, and by showing mercy to Gollum live with himself.
But the cautionary tale was just as important, because despite his mercy Frodo didn’t want to become Gollum. Unlike Jung’s admonition to accommodate the Shadow, I think this was Tolkien saying to literally “master” it (in mythical language). You may recognize this idea from my #1 all-time best post. Gollum as Shadow also explains why he was the guide in the evil lands. The wickedness in us can predict what wicked people will do. Same idea as “it takes a thief to catch a thief”. But you don’t want to do as Jung advised and accommodate your Shadow like Jordan Peterson and so many others have tried, you have to master it if you’re dumb enough to go into Mordor and try to save the world.
If all this is too abstract for you, the takeaway is that if you don’t enjoy Book 6 it’s because you have smol bren.