Oliver Hudson is sharing how he and his sister Kate Hudson felt when Kurt Russell expressed an interest in adopting them.

On the Sunday, Nov. 24 episode of Oliver and Kate’s Sibling Revelry podcast, Oliver, 48, hosted Miley Cyrus’ mother Tish Cyrus-Purcell and sister Brandi Cyrus. As the group discussed life with adoptive parents versus biological parents, Oliver inquired how Brandi, 37, felt when Billy Ray Cyrus adopted her during his marriage to Tish.

“I only say this because my dad — he was around for a little bit, then he bailed. We have a good relationship now, but it took a long, long, long, long time, but we’re there,” Oliver said, referencing his and Kate’s biological father Bill Hudson, whom their mother Goldie Hawn was married to from 1976 to 1982. Goldie, 79, has been in a relationship with Russell, 73, since 1983.

“Kurt came into my life when I was six, five or six and essentially he raised me. I’m the man I am today because of him, right?” Oliver said. “There was a moment when he asked us, myself and Kate, if we wanted to be adopted, and we said no.”

“Not that we were — you know, very aware of probably what it even f—— meant,” he added. “We were six and 12 or however, I don’t remember the age, but we just said, ‘Well, we don’t need it. The love is right there.’ ”

Oliver and Kate, 45, have been open in the past about their at-times strained relationship with their father Bill. With Kurt, Goldie shares son Wyatt Russell as well as Kurt’s son Boston Russell. The blended family spend a significant amount of time together, with Goldie even suggesting in recent months that they should all costar in an ensemble comedy.

“At the same time, when you don’t have your biological [parent] around, it’s hard to replace that, no matter what,” Oliver admitted during the podcast episode. “The love was there with Kurt, there’s no doubt that he is my dad. I call him Pa. But there was always — not something missing, but there’s that other side.”

Just as Kurt never actually adopted Kate and Oliver, he and Goldie have never gotten married despite their decades-old romance. “For people like us, the marriage certificate wasn’t going to create anything that otherwise we wouldn’t have,” Russell told PEOPLE during a joint interview with Goldie back in 2020. “I don’t know. 40 years isn’t enough to finally say, ‘Well I guess…’ “