George Charamba, Mugabe's spokesman, dismissed Britain's objections to attending saying: "The British fear a handshake. We can't expect timid characters to be where men are."
"FOOTNOTE"
Only Czech Prime Minister Mirek Topolanek would join Brown in staying away. Denmark, Sweden and the Netherlands, which had hesitated, have informed the Portuguese presidency that their prime ministers will attend, the source added.
The EU presidency spokeswoman said the full list of those attending was still not available.
The EU source said a plan to send an EU special envoy to study the situation in Zimbabwe would not be carried out until after the summit to avoid any risk of undermining South African President Thabo Mbeki's efforts to broker a deal between Mugabe and opposition parties on holding free elections next year.
Portugal has come under fire from human rights activists for the invitation to Mugabe, but African leaders have said they would not attend the Lisbon meeting if Zimbabwe were excluded.
Portugal's Foreign Minister Luis Amado, who invited Mugabe, recently said he would prefer him not to come because his presence could distract from the essential point of the summit.
"There is no kind of diplomatic embarrassment. We deeply regret that what is being done to innovate and truly transform the relationship between Europe and Africa is being dampened by some obsession by the media around the president of Zimbabwe," Joao Cravinho, Portuguese secretary of state for foreign affairs, told TSF radio in Lisbon.
"When the history of the summit in Lisbon is made this issue, which currently attracts a lot of attention, will only be mentioned as a footnote," he added.
Western critics accuse Mugabe of ruining Zimbabwe's economy, rigging elections and violently suppressing opposition.
Mugabe denies he has wrecked the economy with policies such as seizing white-owned farms for blacks with little experience, and he blames Western pressure for hyperinflation and hunger.
The EU agreed earlier this month to send a "clear and tough" message to Mugabe on human rights at the summit.
