as per my knowledge
given pattern constitutes a token or not
depends upon the source language.
if the pattern is in source language, then it should constitute a token, otherwise why that pattern exist?
Compiler ( Specific to a language ) should be implement all patterns ( in that source language )