It’s the difference between connecting DTE and DCE devices and the way that the pins are named.
A DTE (Data Termination Equipment) device (such as a PC) has TxD as an OUTPUT (i.e. Data is Transmitted), and RxD as an INPUT (i.e. Data is Received)
A DCE (Data Communications Equipment) device (such as a Modem) has TxD as an INPUT (i.e Data is Received), and RxD as an OUTPUT (i.e. Data is Transmitted). This is so that a straight through cable between a DTE and DCE is wired TxD to TxD, RxD to RxD etc.
Now, the PC is a DTE device, and so is the PIC; the Modem is wired as a DCE device. Can you see now why your cable needs to be swapped around (crossed over) when connecting the PC to the PIC, then the PC to the Modem?
This issues are RS232 comms 101. You need to get your head around this.
Sounds like you are getting a buffer overrun in the PIC USART because you are not servicing the RxD interrupt quickly enough. When you are connected to the PIC using hyperterm, you are manually typing the responses which is giving the PIC enough time to read the USART buffer. The Modem is sending responses at full speed (i.e. 9600 baud), and your PIC gets the first character, moves it to the RX Buffer, then the second character is received before your PIC program has read the first one. When the PIC USART tries to copy the second received character into the RX buffer, it notes that the first character hasn’t yet been read - so it sets the Overrun Error bit and shuts down the USART RX.
Does this help?