One of the urban myths that persists among patients taking antidepressants is that it's fine to take a little drug holiday, especially if the meds are decreasing interest in sex. Most doctors, however, don't agree that this technique works. "The libido is very complicated," says David Baron, a psychiatrist at Temple's School of Medicine. He doubts it's capable of bouncing back after a mere 24- or 48-hour absence of the drug. What may bounce back, however, is your depression.