Frankie Muniz is opening up about an under-discussed aspect of child stardom.

On the Nov. 26 episode of Inside of You with Michael Rosenbaum, Muniz, 38, explained that in the past year, he’s been reflecting on the sacrifices his parents and sister made so he could be a child actor. His parents divorced when he was about 11 or 12, he explained, and he had already started acting in North Carolina at age 8.

As a preteen, he and his mom headed to New York City “because we’d kind of maybe outgrown what we could do in North Carolina, like as an actor,” he explained. “And so that separated my family for the first time. My sister stayed with my dad in North Carolina. I went with my mom up to New York and we were going on like six auditions a day and kind of doing the whole grind.”

“And I started booking things and never went back to North Carolina,” he said. He and his mom settled in New Jersey, where his family was from. “I remember we tried to get my sister to move up, but she was going to school at the time, [and] didn’t really want to come.”

When they “finally” convinced his sister to move to New Jersey, “like a month later that I got Malcolm and we moved to L.A.” His sister refused to move again, and she stayed with their grandparents in New Jersey.

“It was kind of this really weird dynamic to look at now,” he said. “It actually makes me a little bit sad to realize that me following my dream and the things that I got to do affected every single member of my family.”

He said that he and his sister have talked about this time in their life now and added, “When it’s happening you don’t realize the sacrifice that every single person is making for you.” He noted that he was particularly moved that his elderly grandparents “were so willing to help out and do whatever they could do” to fulfill his dreams.

“This year, it was the first time I thought about the past enough to think about the sacrifice that people made,” he said. “And it made me really sad. Like it made happy that I had people in my life like that. But it’s just a crazy thought that without that support, probably none of that happens.”

Muniz starred as the title character in Malcolm in the Middle from 2000 to 2006. He also starred in movies like 2002’s Big Fat Liar and 2003’s Agent Cody Banks. In 2008, he put his acting career on hold to pursue race car driving.

In October, he announced he was joining Reaume Brothers Racing as a full-time driver of the No. 33 F-150 in the upcoming 2025 season of the NASCAR Craftsman Truck Series. He told PEOPLE at the time, “I don’t want to say it’s a long time coming, but this has been a dream of mine for forever.”