Former rebel leaders told the BBC that they posed as merchants in meetings with charity workers to get aid money.
They used the cash to fund attempts to overthrow the government of the time.
One rebel leader estimated $95m (£63m) – from Western governments and charities including Band Aid – was channelled into the rebel fight.
The CIA, in a 1985 assessment entitled Ethiopia: Political and Security Impact of the Drought, also alleged aid money was being misused.
Its report concluded: “Some funds that insurgent organisations are raising for relief operations, as a result of increased world publicity, are almost certainly being diverted for military purposes.”
The crisis in 1984 prompted a huge Western relief effort, spearheaded by pop star Bob Geldof’s Band Aid campaign and Live Aid concerts.
Although millions of people were saved by the aid that poured into the country, evidence suggests not all of the aid went to the most needy.
At the time, the Ethiopian government was fighting rebellions in the northern provinces of Eritrea and Tigray.
Much of the countryside was outside of government control, so relief agencies brought aid in from neighbouring Sudan.
Some was in the form of food, some as cash, to buy grain from Ethiopian farmers in areas that were still in surplus.
Max Peberdy, an aid worker from Christian Aid, carried nearly $500,000 in Ethiopian currency across the border in 1984.
He used it to buy grain from merchants and believes that none of the aid was diverted.
“It’s 25 years since this happened, and in the 25 years it’s the first time anybody has claimed such a thing,” he says.
He insists that to the best of his knowledge, the food went to feed the starving.
But the merchant Mr Peberdy dealt with in that transaction claims he was, in fact, a senior member of the Tigray People’s Liberation Front (TPLF).
“I was given clothes to make me look like a Muslim merchant. This was a trick for the NGOs,” says Gebremedhin Araya.
Underneath the sacks of grain he sold, he says, were sacks filled with sand.
He says he handed over the money he received to TPLF leaders, including Meles Zenawi – the man who went on to become Ethiopia’s prime minister in 1991.
Mr Meles, who is still in office, has declined to comment on the allegations.
But Mr Gebremedhin’s version of events is supported by the TPLF’s former commander, Aregawi Berhe.
Now living in exile in the Netherlands, he says the rebels put on what he describes as a “drama” to get the money.
“The aid workers were fooled,” he says.
He says that some $100m went through the hands of the TPLF and affiliated groups.
Some 95% of it was allocated to buying weapons and building up a hard-line Marxist political party within the rebel movement.
Both Mr Aregawi and Mr Gebremedhin fell out with the TPLF leadership and fled the country.
Much of the money that ended up in the TPLF’s hands was channelled through affiliated groups such as the Relief Society of Tigray.
Band Aid’s accounts show that it gave almost $11m to the society and other groups close to the rebels, but the charity has declined to comment.
It should not be forgotten that this all took place at the height of the Cold War.
The Soviet Union had poured $4bn into Ethiopia, and provided Soviet officers to direct Ethiopia battles against the rebels.
In January 1983, President Ronald Reagan issued National Security Directive 75, which aimed to confront the Soviet Union across the developing world.
“US policy will seek to limit and destabilise activities of Soviet Third World allies and clients,” it said.
In a November 2009 speech, US Secretary of Defence Robert Gates – who was deputy head of the CIA during Mr Reagan’s time in office – said that the president’s approach was to “impose ever stiffer costs on the Soviet Union for its Third World adventurism”.
He included Ethiopia among the states in which “Soviet surrogates soon faced their own lethal insurgencies”.
Mr Gates was unwilling to expand on whether the US backed the Ethiopian insurgents.
But since there were only a limited number of rebel movements, the suggestion cannot be ruled out that the CIA not only knew about, but supported, the diversion of aid funds to the TPLF.