I usually build my starters several days in advance of the brews. Depending upon the yeast being grown, I will allow the starter to stay on the stir plate for 24 to 36 hours then chill it in my refrigerator to around 42F for about 24 to 48 hours. On brew day, I take the starter out of the refrigerator once I start the boil and decant as much of the starter wort as possible. Once the boil is done and wort is chilling, I take my chilled sample that I used for measuring gravity and post boil pH and add it to the flask. I swirl up the yeast to aerate it and then allow it to sit until it starts to develop a krausen (usually from 2 to 6 hours, again depending upon the yeast). By then the wort is chilled, in the carboy, and at my starting fermentation temperature. I pitch the yeast at that point.
Further points I have picked up which may also help:
I always pitch at or below my fermentation temperature. Since most of my fermentations start at the low end of the temperature range for the yeast I am using, I do not want to pitch warm and then chill, as it may cause the yeast to prematurely start flocculating. Warming the wort up a bit at the beginning of the fermentation is most suitable to their activity, since the fermentation produces heat.
It is better to pitch cold yeast into warmer wort. The yeast will wake up more quickly with the more hospitable environment.
Most of the time these days, I am over building my starters and doing cell counts to divide up my yeast cake by volume for the next brew. That way I know just what I pitched for cell count and can section off the remainder to grow up for the next brew.
Don't be afraid of a slight over pitch or under pitch, you will still get beer and the range for cell counts is pretty broad in my opinion. Too large of a gap between your cell count and what is projected for the wort you are fermenting, especially on the low side, and you may have more issues with off favors, stalling, poor clean up of byproducts of fermentation or lack of yeast character.
This is just my process and I am sure others have theirs which work just as well.