I’ve heard a number of people say that the complexity of the rules of many modern RPGs are one of the major things keeping casual gamers away from the game. While rules complexity certainly is part of the problem, I don’t believe it is as big a part of the problem as some do.
Look at the first edition of AD&D. It was a relatively complex game, but it has lots of causal players — because while its rules were complex, you really did not have to know many of them to play. A lot of rules knowledge — let alone system mastery — wasn’t needed to play. The only person who really needed a good knowledge of the complex rules of first edition AD&D was the GM.
New players could create a fighter or a thief in minutes and really did not need to know any rules to play the character. They could just tell the GM what the character was doing in regular terms and the GM or other players could easily translate “I try to hit the zombie with my sword” or “I try to sneak down the corridor without the guard in the side passage seeing me” into whatever die roll was needed. Unless the player wanted a magic-using character, little to no knowledge of the rules was needed to play.
The player never needed to buy or study rules books to play well. All he had to do was describe what his/her character was doing and roll the dice when told to. Neither their lack of rules knowledge or lack of system mastery did not really hurt them — nor did it hurt the rest of the party.
Games that more or less require people to buy rulebooks and study them to master the game system are unlikely to attract many casual gamers, in my experience. Worse, they often tend to make casual gamers unwelcome at a table that includes non-casual gamers as their lack of knowledge of the rules and their lack of interest in system mastery hurts the other players.
In my experience, complex rules don’t turn off casual RPG players unless they are expected to read them and master them. If they can just say what their character is trying to do in non-game terms and roll the dice when they need to do, casual gamers do fine with complex games.