Hi Mike,
Both console and custom linux application use HSL0, could you please double check? HS0 should not be enabled.
The Custom Linux application is a mode that keeps the port open but does not provide the linux console. It allows your custom application to send data to the serial port. You cannot send any linux commands in this mode.
Linux console means that you can execute command line Linux commands, run scripts, etc.
BR,
Chris