The simplest answer is that there are just not a lot of values of k' that are allowed as finite states. So the total probability of scattering is small because there are very few final states to land in.
In addition to this, there is another factor. In Fermi's golden rule there is an energy conserving delta function
delta(E_i - E_f)
Now suppose that we write E = c k then (I set hbar=1) we have
delta(c (k - k') ) = delta(k - k) /c
note that c is downstairs -- hence the result is small.