The real danger is using things like scented candles, air freshners, aromatic essential oils etc in the house, but I have always avoided using highly scented plants in the same room as pigeons "just in case".
I would try removing them, but I don't think they would be causing diarrhea unless the dove is coming into contact with them. However, I could be wrong!
Cynthia