If its just the poo being green that's making you worry I'd try and stop!
My boys often had oddly coloured poo and excluding dairy can lead to intolerances later in life so id only exclude any food from your diet on the advise of a health professional x