I have tested Linux 3.10 LTS series and 3.13 so far
I have to say that 3.10 runs much smoother than 3.13...
Especially in some games you can easily see the difference between the two kernels...
with 3.13 you can observe lag spikes etc but this is not happening with 3.10
Anyway my opinion about the kernels is
Keep up maintain the latest Kernel available
Maintain the latest LTS Kernel
Maintain the latest kernel SES series (maintained by Canonical)
So in short you just maintain 3 Kernels and have everyone happy
