You can speed up loading of symbols by doing the following:
1. Verify that you haven't specified any network shares under Tools->Options->Debugging->Symbols especially to server that do not exist anymore. This will require timeouts.
2. Verify and check that _NT_SYMBOL_PATH or _NT_ALT_SYMBOL_PATH is set to network shares as cg60611 pointed out above.
3. Specify a local cache for symbols in Tools->Options->Debugging->Symbols. After you have downloaded symbols once from network shares you can disable those locations. Symbol loading will go a lot faster after the first attempt.
4. Do not specify any symbol lookup paths at all eithe in environment variable and specify paths in the options page but specify that they be used for manual loading. You can then manually load symbols for modules you care about either via the context menu in callstack or the modules window.
5. VS 2008 SP1 has made a few improvements in this area. It allows for cancelling loading of symbols as it is happening. This will allow you to get to your debug session much faster. Note that this is currently in Beta.