Gregory's ServerI clean most of my peripherals quite regularly. Most often with a very slightly wet and soapy sponge to remove oils. My iPhone gets a proper wash with water as it is waterproof, and my earbuds get dry cleaned with a toothbrush regularly. The toothbrush itself then goes to the dishwasher.
It doesn't take a lot of time, and I just feel much better having clean stuff.