There is a temperature sensor in the inner workings of 90% of head units. When the unit gets hot, the sensor turns the fan on to cool it and exhaust hear from the top and back. When cold the sensor intakes heat by reversing the fan to make sure all is well. NOW, this is not the case with all stereos, BUT it sounds like your temp sensor or fan may be faulty. The sensor might be recognising that the system is cold and then trys to bring in heat, but the fan may be cooling instead making the unit "non-responsive" as the circuits are freezing and not working properly.
Best bet, Replace. It's too hard to determine the problem without pulling it apart and giving it to an electronics expert. But that's my guess.