What does Armie Hammer’s life look like years after allegations of sexual and physical abuse derailed his Hollywood career?

Following the launch of The ASrmie HammerTime Podcast, sources tell PEOPLE that the actor, 38, is living a humble life in Los Angeles and navigating the fallout from a series of claims against him that began in January 2021.

“He still gets frustrated at specific sensational allegations but he’s trying to be at peace about it,” says one source.

After Los Angeles police first identified Hammer as a suspect in an investigation into an alleged 2017 sexual assault, the Los Angeles District Attorney found in May 2023 that there was not sufficient evidence for a case, resulting in no charges for Hammer. Exes Courtney Vucekovich and Julia Morrison have also claimed that the Call Me by Your Name star sent messages detailing violent and cannibalistic fantasies about them. A lawyer for Hammer said in a statement at the time, “All interactions between Mr. Hammer and his former partners were consensual.”

These days, Hammer is “not living in some lap of luxury,” says the source, noting that he “got zero money from family” and lives in a “very small” L.A. apartment.

“He wants to show people he’s actually working for it… He feels like he’s admitted to the wrongs he made and he is determined to start over. He’s doing it in what he thinks is ‘the right way,’” they add. “He definitely wants to have his career back and he’s taking it step by step.”

As a result of the allegations, Hammer was dropped by his talent agency, William Morris Endeavor, in February 2021. Following his last acting credit, 2022’s Death on the Nile, he next stars in an upcoming Western titled Frontier Crucible.

A second source tells PEOPLE that Hammer has a lawyer “handling everything” as he continues to have no Hollywood representation. “He is super stoked about everything happening and really thinks his career is about to have a second wave. He has the podcast and two or three projects coming up.”

Those close to Hammer “are being cautiously optimistic,” adds the insider. “They’re basically telling him, ‘Yes, this is all great, but we don’t want you to get your hopes up and you get let down when people aren’t ready to take you back yet.’”

Although “skeptical” about the Armie HammerTime Podcast, his inner circle is “happy that Armie’s happy and very much leading with the ‘we’ll see’ mentality… He thinks he’s paid his dues and spent his time in the doghouse.”

Through it all, Hammer continues to co-parent his two kids, daughter Harper, 9, and son Ford, 7, with ex-wife Elizabeth Chambers. The couple announced their separation in 2020 amid the allegations and finalized their divorce in 2023.

“He really is a great dad,” says the first source. “He’s very hands-on and he and Elizabeth are in a great co-parenting routine now. They still consider each other ‘family’ and support each other and only want the best for the other.”

Following the series of allegations, Hammer entered rehab for addictions to drugs, alcohol and sex, as he detailed in a July interview on Bill Maher’s Club Random. The actor continues to be “committed to his sobriety” as he co-parents, notes the source. “There’s a lot of grace, and it’s because their kids are the priority.”