I have experience related to this question from two countries: Slovakia and the United Kingdom.
In Slovakia, where the vegetarian and vegan products are scarce, it turned out to be quite significantly cheaper to go vegetarian. Although most of my shopping consisted of regular supermarket groceries, I believe this would still be true if I was regularly buying organic products. Meat, along with cheese, were the most expensive items on my shopping list. After the transition, the only thing comparably expensive are nuts, but I did not buy them as often as meat or cheese.
In the United Kingdom this is (for me) somewhat less true. It all depends on where you buy things and what are your eating habits, of course, but with the plethora of substitutes to choose from and their current "specialty" status, the difference can be much less visible. While living here, I got used to eating a lot of Quorn products, as well as some of the Tesco's own veggie brand. Since living here, I also became vegan and while Alpro milks can be found for reasonable prices, they are still 2-3 times more expensive per litre than regular milk. Vegetables and fruits are also more expensive here (again, it depends where you shop, Aldi and Lidl use to have some really good offers). Another thing that levels these two diets are the ubiquitous "junky" foods like burgers-to-go etc. and these are being sold rather cheaply. Considering my style of living, I would say it is actually slightly more expensive to be vegetarian/vegan here, but the benefits outweigh perceived costs.
To conclude - and I believe this will be true regardless of the country you live in - it all breaks down to what your eating habits are and what requirements you have regarding the freshness, origin and nutritional value of your food.