Photo: John Raoux/AP/Shutterstock

Tampa Bay Buccaneers quarterback Tom Brady (12) leaves the field after an NFL wild-card football game

Tom Bradyis making it clear that he has yet to make a decision about whether to return for his 24th season in the NFL.

“Jim, if I knew what I was going to f—ing do, I’d have already f—ing done it,” a clearly annoyed Brady said. “I’m taking it a day at a time.”

“You’re scratching, " Brady responded. “I appreciate your asking, thank you.”

Brady and his Tampa Bay Buccaneers lost their Wild Card playoff game against the Dallas Cowboys last week, bringing an end to a disappointing season for the future Hall of Famer whose frustrations were evident on the sidelines throughout the fall and into recent months.

Fromtossing tabletstoscreaming at teammates, Brady’s struggles were top of mind — all against the backdrop of his divorce from his wife of 13 years,Gisele Bündchen.

Julio Aguilar/Getty Images

Tom Brady #12 of the Tampa Bay Buccaneers warms up prior to a game against the Dallas Cowboys in the NFC Wild Card playoff game

Last month, he addressed the elephant in the room on his podcast.

“I think next time I decide to retire, that’s it for me,” Brady said with a laugh. “So whenever that day comes, we’ll figure it out.”

As for whether Brady can truly picture himself stepping away from the NFL, he replied, “I really don’t. I think what I really realized last year was you’ve got to be really sure to do that. And for me, you know, a lot of people have kind of gone through this situation.”

“I’m going to take my time, whenever that time does come,” the athlete expanded.

Never miss a story — sign up forPEOPLE’s free daily newsletterto stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Brady said on his podcast last month that he admires other athletes who remain “composed” in the media’s bright lights, as he has struggled to do the same during his current football season.

“A lot of them are very personal that I really don’t care to share. I really don’t,” Brady said on the SiriusXM podcast about his issues. “We all have physical, mental, emotional aspects to us that are challenged each year. Some years you are challenged a little more physicallyand you gotta overcome adversitiesphysically.”

source: people.com