Donald Soffer; James Caan.

Donald Soffer is rememberingJames Caanas a tough guy who would do anything for those he loved.
In an exclusive chat with PEOPLE, the 89-year-old Miami businessman and philanthropist recalls first meeting Caan, whodied Wednesday at age 82, “in the early 1980s, when I was fishing off of Bimini in the Bahamas.”
“He was a rodeo rider and wearing cowboy boots. We hit it off and became friends,” says Soffer.
He says they loved to hang out in Europe, California and Miami: “We were two single guys for many of those years, and had a great time in Europe.”
Soffer is known for having developed the city of Aventura, Florida, including Turnberry Isle, and tells PEOPLE Caan had his own place there.
“We hung out and did things together all the time,” he says.
Never miss a story — sign up forPEOPLE’s free daily newsletterto stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human-interest stories.
James Caan, Tom Brady and Donald Soffer.Courtesy of Donald Soffer

According to Soffer, Caan “loved to meet girls and take them out on dates,” and sometimes “he would take one out and tell me he would be right back. Then I didn’t see him until three weeks later!”
“I remember how hard he took his sister Barbara’s death from cancer in 1981,” Soffer tells PEOPLE. “It tore him apart.”
Soffer says what he admired most about Caan was hisloyalty to his friends, often to a fault. He acted like a tough guy, especially if someone bad-mouthed a pal.
“One time, someone made a wisecrack about me, and Jimmy took him to the ground,” says Soffer.
RELATED VIDEO: James Caan Dead at 82
But while Caan “played a criminal” inThe Godfatherand “liked to act that way in real life,” Soffer says “he was a good guy who was a friend when he said he was a friend.”
“In fact, Jimmy was always a straight-up guy,” he adds.
Case in point? Caan once appeared onThe Tonight Showand said during the broadcast, " ‘I owe all I have to Don Soffer, who flew me here on his plane,’ " remembers Soffer.
“Before [hostJohnny] Carsonstopped him, he tried to sell apartments in my Turnberry Isle development to members of the audience because he was my friend,” he adds of Caan. “He never stopped being loyal.”
James Caan.Matt Carr/Getty

Atweet from Caan’s official accountThursday read, “It is with great sadness that we inform you of the passing of Jimmy on the evening of July 6.”
“The family appreciates the outpouring of love and heartfelt condolences and asks that you continue to respect their privacy during this difficult time,” the tweet added.
A cause of death was not made immediately available.
source: people.com