Robert Kagan’s latest book is a short but powerfully written argument about the return of great power conflict and the danger of believing that history is moving towards a world of liberal democracies living at peace with one another. The prospect of “a new era of international convergence” has faded. “History has returned,” he announces, and — however embattled the democracies may be — they “must come together to shape it, or others will shape it for them”.

Kagan somewhat overstates his case when he suggests that great power competition has been on the increase in recent years and that a 19th-century diplomat would instantly recognise the “elaborate dances and shifting partnerships” of today’s great power competition. Great power competition did not disappear with the end of the Soviet Union, but it is not clear that it is getting worse in recent years.

Kagan is most convincing — and this is ­really the central point of his book — when he points to the return of a kind of ideological conflict, this time in a form reminiscent of the 19th century, rather than the Cold War. “The rulers of Russia and China,” he writes, “like the rulers of autocracies in the past, do have a set of beliefs that guides them in both dom­estic and foreign policy.” Even though they have abandoned Marxist ideology, it would be a mistake to think that they had become mere pragmatists, pursuing selfish interests and believing in nothing. To the contrary, these autocratic rulers “believe in the virtues of a strong central government and disdain the weaknesses of the democratic system”.

Thanks to their substantial economic success — and, in the case of Russia, thanks also to the economic disaster in the 1990s — autocracy in both countries has acquired a kind of legitimacy. “It would be a mistake,” Kagan asserts, “to believe that autocracy has no international appeal.” But today’s autocrats lack the legitimacy conferred in the 19th century by the doctrine of “divine right”, and this, in a way, makes them more dangerous. “Today’s autocracies,” Kagan says, “struggle to create a new kind of legitimacy, and it is no easy task.” Integration into a world dominated by democratic ideology is thus threatening to them, and they will try to push back violently.

Kagan agrees that in the long run, rising prosperity in China and Russia may well produce political liberalism, but this, he points out, “may be too long to have any strategic or geopolitical relevance”. Tellingly, he cites a sardonic joke about how “Germany launched itself on a trajectory of economic modernisation in the late 19th century and within six decades became a fully fledged democracy.” But, of course, the intervening 60 years were quite a different story.

This well-aimed humour makes it all the more surprising that Kagan devotes so little attention to what he calls “the hopeless dream of radical Islam”. His contention, not entirely persuasive, is that radical Islam will be unable ultimately to resist the forces of modernisation. However, like the long run, “ultimately” can be much too long a time, particularly ­given the possibility, which Kagan himself acknowledges, that the connection between terrorists and nuclear weapons may soon be made.

This dismissal of radical Islam is at least partly tactical. Kagan does not hide his concern that placing too much emphasis on the danger of Islamic terrorism may promote “illusions” about the possibilities of cooperation with Russia and China. Elsewhere he has argued that basing foreign policy on US political principles can attract more support abroad than appealing to the danger of terrorism. But even if one believes that managing the rise of new major powers is a greater challenge for the future than Islamic extremism, there is no way to avoid dealing with both at the same time.

Kagan is by no means pessimistic about the outcome of the competition between the democracies and the autocracies. He argues that the US still has an “indispensable” role to play, not because it is better than other countries, but simply because it is the world’s only ­superpower, even if a “flawed” one. There is, Kagan acknowledges, “an American problem”, due to “errors of commission and omission, not only in recent years but throughout America’s history” — a tendency towards unilateralism and a “proclivity to use force”, alongside “generosity of spirit” and “enlightened self-interest”. Nevertheless, much of the world looks to the US for support, and even Russia may some day do so against a powerful China. And it is in America’s interest, Kagan argues, to play the role of the “keystone in the arch”. That is true, but the US could probably do a better job of leveraging help from others.

Kagan concludes with his most controversial suggestion, although advanced somewhat tentatively: that we take steps, “moving informally at first”, towards a “league of democracies” that would provide more legitimacy than unilateral action — and more effectiveness than the United Nations version of multilateralism. This strategy has been criticised as likely to promote, rather than reduce, conflict between the democracies and the autocracies — and there would be some grounds for that fear if the lines were drawn too sharply, in the spirit of a new Cold War. Kagan’s own analysis of the autocrats’ defensiveness and sense of fragility argues for caution. But that doesn’t mean that no lines should be drawn at all. An imperfect but perhaps useful analogy is the way that the line between members and non-members of the European Union has effectively pulled the excluded countries in the direction of European values and interests.

A very different objection to Kagan’s proposal is that a “league of democracies” will be a fragile basis for collective action. When crises develop over local problems — in Asia or Eastern Europe or the Persian Gulf — democratic solidarity is not likely to trump individual countries’ strategic or commercial interests. But although the idea of “coalitions of the willing” has taken a beating in recent years, there is no obvious substitute for it. While shared democratic values may help to draw such a coalition together, they are not necessarily the strongest link. A shared sense of threat — along with the confidence that there is will and capability to resist it — is not only the strongest motivating factor bringing countries together, but also the strongest deterrent to aggressive action by any regional power.

How much will the Iraq experience affect America’s ability to lead in the future? Kagan doesn’t address that question but his answer is implied when he says that the democratic world will still look to the sole superpower for leadership, no matter how “flawed”. It is striking how US leadership recovered after Korea’s unpopular and stalemated war – and even after Vietnam. Whether Kagan believes that history’s return will include the return of US confidence, and how quickly, is unclear.

History’s answer to that question will depend on several factors, including the leadership capacity of the next US administration and whether Iraq ultimately comes to be viewed as a failure or a success – albeit a costly one. But America’s future leadership role may depend even more on how threatening the world appears. Historically, that leadership role has often emerged out of a compelling crisis: Pearl Harbour, the Soviet invasion of Afghanistan and the Iran hostage crisis, Iraq’s invasion of Kuwait, or the attacks of 9/11. Paradoxically, the relative security which Americans have enjoyed since 2001 makes it easier to doubt the necessity of shouldering the burden of leadership. One hopes it will not take another calamity to convince us of the need for a vigilant foreign policy.