An alternate mode (commonly shortened to "altmode" or "alt mode") is a term used to describe a Transformer's non-humanoid form, whether it be vehicle, animal, or otherwise.
Some Transformers don't have a humanoid form, so in those special cases, the altmode is perceived to be the mode which is more geared towards utility or disguise while their robot mode is the one geared more towards interaction with other Transformers. For example, for the G1 characters Laserbeak and Ravage (Although, when arriving on Earth, Laserbeak's bird mode changed, Not his cassette), their animal forms are their primary mode, and the cassettes their alternate modes. Even Grand Slam and Raindance, who transform from cassettes to vehicles, would be thought of as having a cassette alternate mode, as their vehicle forms -- while not humanoid or animalistic -- are at least self-mobile.
In the cartoon episode "The Killing Jar," the Quintesson scientist refers to the Quintesson guard's transformation as his alternate mode, the first such use of this term in the official fiction.
The earliest recorded use in the fandom of the shortened term "alt mode" is an alt.toys.transformers post from March 1994 . From the context, it appears that the term was already understood by fans at that time, as the post's author does not explain what he means.