Neal Gray called for a public inquiry into the child's death as he told BBC Radio 4's Today programme that his granddaughter was terrified of Ben Butler and Jennie Gray before she went back to live with them.
Mr Gray and his late wife Linda were Ellie's guardians for five years after Butler was convicted of child abuse before a judge exonerated the father and returned the child to her parents.
Butler was convicted in 2009 for shaking Ellie as a baby, although this was later quashed on appeal.
But in 2012 the couple then won a High Court judgment to have Ellie returned to their care.
Asked how he tried to keep her, Mr Gray said: "As hard as we could over a period from 2009-2010, to 2011-2012, in 2011 the local authority asked my wife and I whether we would like to adopt Ellie because we passed all the criteria.
"But a social worker blocked it and advised the parents that we were going to adopt Ellie and the application was stopped. We were backed by the London Borough of Sutton's children's services to adopt Ellie.
"We found it very devastating , we tried to fight it tooth and nail but every time we were protested we were told we were troublemakers and we were elderly people and we weren't worthy of looking after children."
Asked why the authorities got it so wrong, he said: "I think they wanted a quick fix. I don't think they were that interested. The local authority fought tooth and nail to back myself and my wife, who is now departed from this world, to keep Ellie.
"But Mrs Justice Hogg had had it in her mind that we were unworthy to look after Ellie."
Mr Gray said he and his wife used to take Ellie to a children's centre for troubled families every few weeks to see her parents - and in two and a half years her father went twice and her mother, four times.
"She was absolutely petrified of the social workers," Mr Gray said. "She told them she didn't want to go back and she started to have night dreams where she was scared they [her parents] were going to come and take her away in the night. She used to hide behind the curtains or under the duvet every time they turned up at our house."
"She hadn't known them as parents," he said. "But her mother turned around and said 'If you don't come back and live with mummy and daddy we won't love you anymore'."
The youngster went back to live with her parents 11 months before her death.
Mr Gray recalled the last time he saw Ellie, on October 27, the day before she died, in a McDonald's restaurant in Sutton.
Describing her as bedraggled and bruised, and wearing odd shoes and socks, he said: "She wasn't the Ellie we knew.
"She had bruises on her forehead and her face and scratches, her hair was all bedraggled, she had odd shoes, odd socks, clothes and she looked as though she'd been dragged through the back of a hedgerow."
Asked if Ellie resisted returning with her parents at the end of the meeting, Mr Gray said: "No. But you could see her eyes were sunk in and there was sadness in her eyes."
He added: "Now the court case is over I hope to pursue a judicial review or an article 2 inquiry."
Mr Gray went on: "In layman's terms, a public inquiry surrounding this case to find out why the judiciary didn't do their job correctly, why the social services failed Ellie - especially the private side.
"I want it all open and above board, and I want everybody to answer because everybody failed Ellie completely and utterly."Timeline | Ellie Butler
Ben Butler rushes his seven week-old baby Ellie to St Helier Hospital, south London after she becomes limp and pale. Butler is accused of shaking Ellie after doctors diagnose bleeding on the brain. She makes a full recovery.
Butler is convicted of grievous bodily harm and cruelty at Croydon Crown Court. He is jailed for 18 months. Ellie taken to be cared for by other people.
Butler's conviction is quashed by the Court of Appeal after Lord Justice Moses ruled the trial judge's summing-up to the jury was defective and the conviction was unsafe. Sutton Council continue to insist that Ellie is at risk and allow the parents only brief supervised visits.
Butler and Ellie's mother Jennie Gray are finally told they can care for Ellie again after a seven-week family court battle. The judge said the couple had "shown tenacity and courage" during the "storm" of the case.
Ellie, now aged six, dies after being found with severe head injuries at her home in Sutton, south London
Butler, now 34, is charged with Ellie's murder and appears at Croydon Magistrates' Court.
Gray is charged with cruelty and intent to pervert the course of justice and appears at Croydon Magistrates' Court.
Butler and Gray go on trial at the Old Bailey. He is charged with murder and cruelty and she with cruelty in connection with a broken arm Ellie sustained in the weeks leading up to her death.