Shinzo Abe surprised his ruling Liberal Democratic Party twice in the past few weeks. First, he refused to resign after the LDP's devastating defeat in the July 29 Upper House elections, saying that he would not back away from a fight. Then he did just that Tuesday out of a combination of frustration and fatigue, and only two days after he had given a major policy speech at the opening of the Diet session.

At that time, he had promised to stake his political life on securing an extension beyond Nov. 1 for the Anti-Terrorism Special Measures Law under which the Japanese Navy is supplying fuel for Pakistani, American and other forces fighting terrorists in Afghanistan.

Much has been written about Abe's misjudgments, which began with his decision last year to run for prime minister — despite his relative inexperience, the difficulty of following a popular and charismatic predecessor (Junichiro Koizumi), and the likelihood, even then, that the LDP would lose seats in the Upper House election. One can feel sympathy for Abe, who might have gone into a different and more successful career, except that his grandfather and father were leading political leaders and this gave him little choice but to accept the mantel.

At first it seemed he might do well. Early on, Abe reaped benefits of repairing the breech in relations with China and South Korea created by Koizumi's repeated visits to the controversial Yasukuni Shrine. But then his popularity was progressively undermined by misjudgments — including creating a Cabinet based on friendships rather than capabilities, inviting back into the party some of the old guard drummed out by Koizumi, and emphasizing a patriotism and constitutional revision agenda rather than economics. A series of Cabinet scandals reflected Abe's weakness and further contributed to it. When Abe announced his resignation, his approval rate was probably close to 20 percent, and the media was threatening to target some of his own fundraising transactions.

In the coming days, the LDP will choose a new party president, who automatically becomes prime minister thanks to the party's majority in the more powerful Lower House. This may be Taro Aso, a conservative nationalist who speaks fluent English, or possibly Yasuo Fukuda, a senior politician of distinguished political pedigree. Either will be transitional. Moreover, the "who" is less important than how the Japanese political system as a whole comes to grips with some fundamental national challenges. Three come immediately to mind.

The first is to devise a political culture that embraces the possibility of "divided" government as a norm, allowing for compromise on issues of basic national interests while tolerating a considerable degree of constructive competition. For Japan, the current situation of Lower House dominance by one party and Upper House dominance by another is entirely new, and requires some new rules of the game. It is neither in the interests of the LDP nor the opposition Democratic Justice Party (whose leader, Ichiro Ozawa, and many members were once in the LDP) that political competition overwhelm their ability to work together on key issues.

The second is to develop a new social contract in Japan, which is at the global frontier of demographic change. Japanese are rightly concerned about pensions since within a very short 30-year period Japan has gone from having the youngest labor force among the advanced countries to having the oldest one. This transition continues. Projections suggest by midcentury, 23 percent of the Japanese population could be 75 years old or older. Even Koizumi's inspirational but vague commitments to "structural reform" did not begin to address the aging issue. Immigration policy is one aspect, as is the future of the economic reform process. While Japan needs desperately to raise productivity to provide for its elderly, some in the LDP are anxious to accommodate the party's economically less efficient but politically disaffected rural base by restoring large-scale subsidies and government spending in these areas.

A third challenge concerns Japan's foreign policy. There is deep commitment across the political spectrum to the U.S.-Japan security relationship, which more than ever is rooted in basic U.S. and Japanese national interests. Nonetheless, the Japanese alliance, unlike that with Britain, is not fully solid because it is not yet grounded in a broader public and elite vision of Japan's international role, nor is there a feeling of full confidence in Japan about U.S. commitment. Domestic maneuvering rather than careful debate seems likely to end, at least temporarily, the Indian Ocean refueling effort. Emotional issues, like North Korean abductions and a permanent seat on the United Nations Security Council, have dominated the foreign policy debate. Japan's China policy remains a mix of confused attitudes and positions. One would hope that in the longer run, the LDP and the DJP could devise a foreign policy that starts at the water's edge.

Charles E. Morrison is president of the East-West Center. He wrote this commentary for The Advertiser.