Organising a conference well takes time: more than most people expect. For a conference of one hundred people or more, six months is a realistic minimum. Popular venues and keynote speakers are booked up a year in advance. Communication and registrations need to start at least three months before the event.
For smaller conferences of fifty to eighty people, four months can be enough. That works if you can secure the venue quickly and don't need an external keynote speaker. But the safe rule of thumb remains: start earlier than you think you need to. Every week you start too late costs you choices, and choices made under time pressure are rarely the best.