Overflow condition can be written in any of the three ways –
- $C_{n} \oplus C_{n-1}=1$ which says that either 1st MSB or 2nd MSB should give carry but not both.
- $A_{n}B_{n}\bar{C_{n-1}} + \bar{A_{n}}\bar{B_{n}}C_{n-1}=1$ where $A_n$ and $B_n$ are the MSB (which are sign bits) and $C_{n-1}$ is the carry from the 2nd MSB.
- $A_{n}B_{n}\bar{R_{n}} + \bar{A_{n}}\bar{B_{n}}R_{n}=1$ where $R_{n}$ is the MSB (sign bit) of result of the addition.
From the third condition above, we can see that overflow will happen when:
1. $A_n,B_n$ is 0 (which means positive numbers) and $R_n$ is 1 (which means a negative number).
2. $A_n,B_n$ is 1 (which means both are negative numbers) and $R_n$ 0 (which means a positive number).
Conclusion: For overflow to happen, two positive numbers are added and the result is negative OR two negative numbers are added and the result is positive.
PS: 2nd result can be derived from 1st result by replacing the value of $C_n$ with $A_nB_n + C_{n-1}(A_n \oplus B_n)$ and 3rd result can be derived from the 2nd result by thinking a bit. Let me know if you find it difficult to derive it.