Swimming with an ear infection?

Okay so I had a bad ear infection, with pain, discharge, and swelling in the ear canal. The pain is gone, i'm still on the antibiotics. The discharge/fluid is gone, the only thing left is the swelling in the ear canal, which is causeing me to not be able to hear. The doc told me i could swim this weekend. I had/have a hole in my eardrum (it slightly bursted, its a tiny hole he said). Should I swim?