Odor molecules are detected by the olfactory receptors (hereafter OR) in the olfactory epithelium of the nasal cavity. Each receptor type is expressed within a subset of neurons, from which they directly connect to the olfactory bulb in the brain. Olfaction is essential for survival in most vertebrates; however, the degree to which an animal depends on smell is highly varied. Great variation exists in the number of OR genes among vertebrate species, as shown through bioinformatic analyses. This diversity exists by virtue of the wide-ranging environments that they inhabit. For instance, dolphins that are secondarily adapted to an aquatic niche possess a considerably smaller subset of genes than most mammals. OR gene repertoires have also evolved in relation to other senses, as higher primates with well-developed vision systems tend to have a smaller number of OR genes. As such, investigating the evolutionary changes of OR genes can provide useful information on how genomes respond to environmental changes. Differences in smell sensitivity are also dependent on the anatomy of the olfactory apparatus, such as the size of the olfactory bulb and epithelium.
Nonetheless, the general features of the olfactory system are highly conserved among vertebrates, and, similarly to other sensory systems, olfaction has undergone fairly modest changes throughout the evolution of vertebrates. Phylogenetic analyses reveal that at least three distinct olfactory subsystems are broadly consistent in vertebrates, and a fourth accessory system (vomeronasal) solely arose in tetrapods.
Molecular evolution
Mutations affecting OR genes on the chromosome are primarily responsible for the evolution of smell. OR genes are grouped in clusters along multiple chromosomes and are responsible for coding respective OR proteins. These proteins contain seven transmembrane domains that are responsible for detecting specific sets of odor molecules. OR genes are located on error-prone regions of the chromosome, and consequently, the DNA of the OR gene is periodically duplicated during crossover. After this duplication event, one of the two genes may mutate and disable its function, rendering it as a pseudogene. Alternatively, the duplicated copy may mutate without dysfunctionality, and will continue making the same olfactory receptor but with altered structural changes. This protein adjustment can induce a subtle shift in the range of smells an animal can sense. The diversity of smelling genes present in humans today are attributed to multiple rounds of mutations that have occurred throughout vertebrate evolution.
In particular, repeated rounds of gene duplication, deletion, and pseudogene evolution contribute to the variety of OR gene number. Formally known as "birth-and-death evolution", these dynamics are measured by the number of gains and losses from genes in each branch of the phylogenetic tree in question. Statistical methods can be used to estimate the total number of gains and losses, which can be as large as several hundred per branch of the tree. Moreover, the number of gains and losses can be enormous even if two extant species possess the same gene number (for example, humans and macaques).
Both adaptation and random events can cause birth-and-death evolution. The probability of gene duplication is dictated by chance events and primarily occurs through unequal crossover; this excludes the rare event of whole genome duplication. Alternatively, the Fixation of duplicate genes can be influenced by natural selection or can occur randomly. Within the mammalian phylogenetic tree, a large number of gene gains and losses are observed for almost all branches, suggesting that a significant fraction of gene number changes were caused by inactivation events and random gene duplication. This process is known as genomic drift, or "random genetic drift of gene frequencies" in population genetics. Pseudogenes are also subject to genomic drift, since they are rendered as non-functional and are believed to evolve in a neutral manner.
Lineage
At least nine groups of vertebrate OR genes have been identified (α,β,γ,δ,ε,ζ,η, and θ), each of which derived from ancestral genes in the most common ancestor of tetrapods and fish. Specifically, vertebrate OR genes convey an evolutionary pattern of three separate lineages: fish, amphibians, and mammals.