Men have hormones too. Elevated estrogen levels can eclipse testosterone, zapping sex drive. A man's best defense against an estrogen invasion is to lose weight and build muscle. "Fat converts your testosterone to estrogen," says Jack Mydlo, chairman of the Department of Urology at Temple's School of Medicine. Dropping pounds will improve your testosterone-to-estrogen ratio, which improves sex drive.