A group of lexemes are declared as tokens if they match the pattern of a token.(tokens can be identifiers , numbers etc)
comments , white space and preprocessor directives are not declared as tokens in C.
a token is declared after seeing the next input string. why ?
Since “&&” is also a token, so we can’t declare “&” as a token unless we see next input string.
Since “==” is also a token, so we can’t declare “=” as a token unless we see next input string.
“%=” is also a token (for ex: a% = y; equivalent to a = a%y;) so “%” can’t be declare as token without looking at next symbol.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
a>>=1;
a
we see 'a' first and make it as token (variable) but we don't declare it as token and see the next symbol first before declaring it as token. the next lexeme is '>' and since 'a>' does not match any pattern we declare 'a' as a token.
>>=
we see '>' (greater than) then we see next symbol '>' and so '>>' (right shift) it becomes a token but we don't declare it as token and see the next symbol first before declaring it as token
then we see '=' and check '>>=' ( this is shorthand operator ) and make it as a token but we don't declare it as token and see the next symbol first before declaring it as token.
then we see '1' so '>>=1' is not matching the pattern of any token so we finally declare '>>=' as a token.
1
we see '1' and make it as token (digit) but we don't declare it as token and see the next symbol first before declaring it as token. the next lexeme is ';' and since '1;' does not match any pattern we declare '1' a as a token.
';' is a token.
so answer is 4.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
similarly a! are two tokens.
