Author, author

Having become increasingly unsettled by the perils of faction, I decided to seize the opportunity to stir up a debate on the subject when I was invited to give the 2011 Roy Jenkins lecture at the Royal Society of Literature.

Last year at the Guardian Hay Festival, the simmering discussion about the uses and abuses of faction began to bubble merrily. Niall Ferguson argued that historical fiction "contaminates historical understanding". That is too sweeping. There are novels which can raise interesting historical questions, because they are able to go where historians should never dare to tread. Jonathan Littell's The Wellwishers, initially published in France as Les Bienveillantes – he is the first American to win the Prix Goncourt – has been highly controversial. But Littell's extraordinary research turned his psychological speculations on the mentality of the SS elite involved in the final solution into a useful adjunct to history, even though it lacked historical validity.

Some novelists want to give people in history a voice, because they have been denied it in the past. Andrea Levy, whose novel The Long Song is set in Jamaica in the 1830s, said that the almost complete absence of accounts of the period by enslaved people allowed fiction to come "into its own in this type of story", with the novelist's imagination filling in the blanks of history. Nobody could disagree with that. A problem only comes, I think, when real historical characters are introduced with invented speech and thoughts. Helen Dunmore, whose novel The Siege is about wartime Leningrad – its sequel, The Betrayal, is about the period immediately before Stalin's death – said that novelists stray into "dangerous territory" when they fictionalise real people.

One can argue that historical fiction set further away in the past poses less of a dilemma, if only because of the lack of verbatim accounts. Hilary Mantel, who wrote the highly praised Wolf Hall about Thomas Cromwell, wrote: "For a novelist, this absence of intimate material is both a problem and an opportunity." This very well-researched and magnificently written book is compelling, and Mantel makes absolutely clear that it is a work of fiction. "Unlike the historian," she argues, "the novelist doesn't operate through hindsight. She lives inside the consciousness of her characters for whom the future is blank." In fact, the historian should do both – first see the world as it appears to protagonists at the time, and then analyse with hindsight. But the key point is that when a novelist uses a major historical character, the reader has no idea what he or she has taken from recorded fact and what has been invented in their re-creation of events.

Restorers of paintings and pottery follow a code of conduct in their work to distinguish the original material from what they are adding later. The question is: should writers do the same if they are going to take a piece of history and then fill in the blanks, to allow the reader to know what is fact and what is the invention of the writer? If novelists do not want to make this distinction then they could at least change the names of real historical characters to emphasise that their version is at least one step away from reality. The novelist Linda Grant argues that the roman à clef also gives the writer a much greater freedom of invention. She pointed out that Curtis Sittenfeld's American Wife, which is obviously based on Laura Bush, benefitted greatly from the change of names. Sittenfeld thus was able to go where it would have been unthinkable to go otherwise. Keeping real names shackles the imaginative writer perhaps more than they realise. In War and Peace, the most convincing and interesting characters are those Tolstoy made up, not the historical figures he introduces.

A blend of fact and fiction has been used in various forms since the dawn of creative writing, starting with sagas and epic poems. Yet the appeal of faction to writers and readers has recently increased in a dramatic way. Is this due to a poverty of imagination, as some critics argue? Is it a marketing-driven phenomenon, perhaps catering to the modern desire in a fast-moving world to learn and to be entertained at the same time? Is it influenced by a curious need for authenticity, even in works of fiction? Is there a prurient compulsion to fill in those gaps in our knowledge about the private lives of great figures, which history has failed to cover? Movies and TV now revel in the speculative biopic.

The power of historical fiction for bad and for good can be immense in shaping consciousness of the past. In what was then West Germany, the TV miniseries Holocaust, broadcast over four nights in a single week in January 1979, attracted vast audiences. This story of parallel families, one of assimilated Jews and one Nazi, seized the imagination of the younger generation. Members of the older generation suddenly found themselves on the defensive as they were asked: "What did you know about it at the time?" Young Germans felt a surge of moral outrage that their parents' generation had buried the truth of the second world war.

We now live in an age in which "entertainment history" forms the historical perceptions of most of the population. Of course, many claim that even if it distorts the material, it at least gives a taste for the subject. But although some will be encouraged to read a truly factual account, most will not. And the compulsion of movie and TV directors to change the story for dramatic effect means that the result is bound to be incompatible with accurate history. Film-makers simplify historical subjects according to set formulas. Their films have to have heroes and baddies – there are seldom shades of grey. And the hero has to have the "arc of character" treatment in which he or she goes through a form of moral metamorphosis. Endings have to be upbeat, even for the Holocaust. Look at Schindler's List and the sentimentality of its finale, revealing that, in movies, only the survivors count.

I am certainly not trying to imply that novelists who use a historical setting are attempting to hoodwink their readers, but I do think that the growth of faction has consequences that should be recognised and discussed.