I'm not actually sure why they didn't have Lou Ferrigno do the growls. Sure, he was hard of hearing and had a speech impediment, but he was totally capable of doing voice work. He even did the voice for the Hulk in The Incredible Hulk (2008).

It could be they just wanted someone with a deeper, bigger voice. When Cassidy passed away partway through the show's run, they replaced him with Charles Napier. Even he didn't have a voice as deep and booming as Cassidy's, though.