A leadership challenge now could damage the party and would do little to improve the government's chances of winning the next election, the political analysts say.
"Miliband is obviously one to watch, but he's a work in progress," said John Curtice, a professor of politics at Strathclyde University.
"If you're going to upset the apple cart, you've got to have someone who can put things together again afterwards, and there's not really an obvious person to do that."
Support for Labour is at its lowest in more than 20 years, an opinion poll showed on Sunday. At just 26 percent, Brown's party is 23 points behind the main opposition Conservatives.
Rising energy prices and the global credit crisis, which has hit the economy and made homeowners nervous, has dented confidence in the government and hit the Scot's popularity.
The economy grew by 0.4 percent in the first quarter from three months earlier, the slowest rate in three years. Inflation rose to 3.3 percent in May, the highest rate since Labour took power in 1997, making it harder for the central bank to cut interest rates and boost growth.
Nearly 45 percent of those surveyed in Sunday's poll said Brown should resign now.
To force a leadership contest, 20 percent of Labour MPs - a total of 70 - would have to sign up.
"The rules are stacked against," said Worcester. "For a start, there aren't 70 people willing to put their heads above the parapet."
A challenge can take place only if requested by a majority of the party conference on a card vote. Robert Rogers and Rhodri Walters, the authors of "How Parliament Works", say this is virtually impossible to achieve.
"I think it's just mid-term blues writ large," Worcester said, suggesting "it certainly will not be the same come early 2010."
Last month Labour lost a parliamentary seat to the Conservatives for the first time in 30 years. On June 11, Brown won a parliamentary vote his plans to extend the time terrorism suspects can be held with a greatly reduced majority, and he had to rely on Northern Irish MPs to secure victory.
But barring a disaster or scandal, political experts say Labour is likely to limp on, hoping for an improvement between now and May 2010 to improve its election chances.
