Elder Scrolls IV: Oblivion uses the Xbox 360 hard drive extensively to cache (copy and reuse) game data. This optimizes loading sequences. It's been found, in rare cases, that cached data can get fragmented. This exhibits itself by the game taking a long time to load menus, dialogue, sound, objects, levels, and more.
To solve this problem, reset your 360 and hold down the A button as Oblivion is booting up. If any button is held while loading, it will clear its hard drive cache, and create a new one.
Once you see the 'Bethesda Softworks' logo, you should be good to go!
NOTE: You will not lose any saved data.
If you're using a wireless controller, the best way to ensure it's been cleared correctly is to turn on your 360 by pressing the Eject button on the front of the console. This will bring you to the Dashboard. Then connect your wireless controller by holding the Xbox Guide button (silver/green X in the middle of the controller).
Now that the controller is connected move down to the Close Tray option and press A, immediately press A again and keep it held until the Oblivion logo appears. Now let go of A, press Start and select Continue to reload the game.