Handicap betting in Dota 2 works the same as handicap betting in traditional sports. When it comes to Dota 2, rather than score as it would be when betting on football matches, the handicap represents the game score in a best of series. So take a best-of-three match in Dota 2, you bet on team A with +1.5 handicap bet. Team B beats Team A 2-1 and since you bet +1.5 on Team A, 1.5+1=2.5 which is higher than the 2 of Team B so you win your bet. However, if Team A failed to win a single game and lost 2-0 then you would lose your bet because 0+1.5=1.5 and that is less than 2.
This works the same for negative handicap betting, if you bet on Team A with a -1.5 handicap bet and they beat Team B 2-1 then you would lose the bet as 2-1.5=0.5 which is lower than 1. But if they managed to win 2-0 then you would win the bet as 2-1.5=0.5 which is higher than 0. Handicap betting can be used for best-of-two matches with the +/- decimal 0.5 and best-of-five matches with a +/- 2.5 decimal.