How much do comments slow down the compiler?

Is it really worth it to force the compiler to have to sort out which statements to skip and which to attend to? Seems like a non-zero slice of time is being wasted by the lexer just to make shit easier for the programmer.

It is non-zero but in practise, it's negligible and easy to ignore. Try stripping out the comments in the GCC source code and then measure the time it takes to compile GCC.