The scanner should be able to be configured on what actions to apply when matching rules. e.g. you would usually want to skip whitespace and tabs.
def default_handler(matched: str, rule) -> Token:
return Token(rule, matched)
def whitespace_handler(matched: str, rule) -> None:
return
scanner.set_handler(whitespace_rule, whitespace_handler)