If you're talking about the loose swarf you get from a water stone, leave it on there. I splash a little water on the stone if it starts feeling too dry and muddy, but as the swarf breaks down it starts polishing the blade. If you use multiple grits it makes the sharpening process much faster as some of the finer work was done by the previous grit.

You might want to use a coarse grit diamond stone to flatten the water stones with, but a floor tile (or any other decently flat, hard surface) and sandpaper will flatten it. The diamond stone is just more convenient for me. Good diamond stones will last a long time, but cheap ones can be worn smooth because the diamonds aren't attached as firmly (or don't seem to be) but that is true when you sharpen knives as well.