Find Number Of Tokens and Lexemes

Question

// comment
printf("string %d ",++i++&&&i***a);
return(x?y:z)

Q1]  Find Number of Tokens and Lexems in the above code snippet
Q2 ]  ***  -> counted as 1 token or 3 tokens ?  why it is so
Q3 ] ' printf ' is lexeme whose token is id , ' i ' is lexeme and token is id are they both countes as single token or separate token
Q4 ] If the Question would have been count the number of unique lexemes and tokens what would have been the answer ?

Comments

i think  ***  -> should be taken as 3 tokens because our lexical analyser when see * will wait for one more *..if it comes it should count it as 1 token only.....if we go this way then

++i++&&&i****a

here i think ++ is one token && is one token and ** is one token...

dont know whether m right or not..but i think this should be the way because we previoulsy give regular expressions before hand for every keyword,identifier etc to our lexical analyser..so it should match accordingly...

---

i think && and ** will be considered as single token each

---

What is rhe difference between number of tokens and number of lexemes??

What are the cases where they are not equal?

Answer 1

// comment
printf("string %d ",++i++&&&i***a);
return(x?y:z)

I am applying whitespaces in the code to seperate tokens.

// comment
printf   (  "string %d "  ,  ++  i  ++  &&  &   i  *   *   *   a  )  ;
return   (  x  ?  y  :  z  )

Total number of tokens will be = 24 

Distinct number of tokens will be = 18 

How && and & are seperated ?

Compiler's Lexical Analysis phase uses greedy tokenization approach. It always matches with the biggest pattern available .

Like + is there, but ++ is also used in C. Hence, whenever i have ++, i count it as single token.

Now, i think its easy to count tokens in &&&. Like, & is used in C as Bitwise AND while && is used in C as Logical AND, so we use biggest possible pattern which is matched . Hence, &&& has 3 tokens like && and &. 

We have learnt * in C. Is there ** defined in C ? No, so *** is 3 tokens .

Difference between Lexeme and a token ?

When a source program is fed into the lexical analyzer, suppose 

return(x?y:z)

which yields the lexemes : return       (        x          ?            y      :       z      )

With corresponding tokens : <id , 0>   <(>   <id,1>   <?>   <id,3>   <:>   <id,z>    <)>

Comments

&&& has two tokens which are && and &.

---

? : 

Is a ternary operator, shouldn't we count it as a single operator token ?

---

I was completely frustrated in this topic .it's a perfect explanation.And i got full marks in my mid exam. Now lexemes and token difference concept also clear.

Thank you @Kapil sir.

Answer 2

total 24 tokens. tokens are as follows: *,*,*  and &&,& are considered separately.

1. printf 
2. ( 
3. "string %d " 
4. ,
5. ++
6. i
7. ++
8.  &&
9.  &

10. i
11. *
12. *
13. *
14. a
15. )
16. ;
17. return
18. (
19. x
20. ?
21. y
22. :
23. z
24. )

Comments

answer will be 24 && ist one oprator.

---

ya! u r right. i missed it.

---

Q1] ** is not considered as single token because there is no nuch operator . is that the reason ? Why did you take it as separate ?

Q2] if the question would have been count the number of unique tokens . Would we count printf and i as separete ? because they fall under same tokens id .