Short answer
The term vertebrata stems from grouping animals according to obvious gross anatomical distinguishing features as typically used in the early days of taxonomy. However, from a phylogenetic point of view, the presence of the neural crest during embryonic development, which is unique to this subset of animals in the phylum chordata, is a much more important distinguishing feature than the presence of vertebrae.
Background
Vertebrates belong to the phylum chordata, but share some characters that make them unique. Chordates, with a few exceptions, are active animals with bilaterally symmetric bodies that are longitudinally differentiated into head, trunk and tail. The most distinctive morphological features of chordates are the notochord, nerve cord, and visceral clefts and arches.
Vertebrates feature neural crest cells, which sets them apart from any other group. These cells appear early in development, and only vertebrates have them. Neural crest cells form the skull and jaw bones.
The neural crest has sometimes been called the fourth germ layer because of its importance (Fig. 1). The neural crest cells produce a variety of cell types, including (1) the neurons and glial cells of the sensory, sympathetic, and parasympathetic nervous systems, (2) the epinephrine-producing (medulla) cells of the adrenal gland, (3) the pigment-containing cells of the epidermis, and (4) many of the skeletal and connective tissue components of the head (Gilbert, 2000).

Fig. 1. Regions of the neural crest. The cranial neural formt the bones and cartilage of the face and neck. It also produces pigment and cranial nerves. The vagal neural crest (near somites 1–7) and the sacral neural crest (posterior to somite 28) form the parasympathetic nerves of the gut. The cardiac neural crest cells arise from the neural crest by somites 1–3. Neural crest cells of the trunk (about somite 6 through the tail) make the sympathetic neurons, and a subset of these (at the level of somites 18–24) form the medulla portion of the adrenal gland. Source: Gilbert (2000)
In all, the term vertebrates is just a reflection of an apparent and obvious marker of this group of animals, namely the presence of vertebrae, but their distinguishing most important feature is the presence of the neural crest during development.
Reference
- Gilbert. Developmental Biology, 6th ed. Sunderland (MA): Sinauer Associates (2000)