Taking a break

I've been on the diet for five weeks now and it was actually going pretty well, but I'm considering taking a break and doing this again later for a variety of reasons. The main one, I guess, is that I've only been back at work from mat leave for two months and the climate is less secure than I'd thought. I've been on term contracts for the past three years and I think it would be better to be back longer before getting pregnant again. Secondary reasons are feeling totally overwhelmed lately with the two sons I already have, and the fact that I don't have post-partum AF back and wonder if I'm just wasting my efforts on LE diet if I might not be able to ttc anytime soon.

Will it hurt my odds in any way to give up the diet and exercise, as long as I go back to it 10-12 weeks before my first attempt? I feel conflicted about losing all that good effort I've put in so far, but I also think I was overly optimistic about how soon I could reasonably handle another baby...