Comparison sites (gocompare, confused, moneysupermarket etc) are meant for the general masses. Remember that like every other company out there they are after making the easiest profit they can. Their business model is to try to sell to as many people as possible in an easy way - so they target the mainstream, and they use the internet.
The other model they have is to be as cheap as possible. On average the site will get £40 for every policy you sell through their website, and so because of this they are really only aiming to do 2 things - get as cheap a quote as possible to beat their competitors, and to get as many people as possible using their site.
Because most people are price sensitive (that is they are looking for a low cost, rather than a strong product) then the comparison sites are also price sensitive, because remember they want to sell to as many people as possible so they target the largest market group. Now it may surprise many people here to think of insurance as a 'product' and thus things like quality be mentioned, but anyone who has had a particular bad or good experience dealing with a claim, or found out that something they thought was covered is not covered then they will know exactly what I am talking about.
In order to be as low priced as possible comparison sites will rip their product to shreds (high excess, very little covered etc) in order to get dead low prices. Because they know people don't know much about what they are buying they know that it'll be a case of too little too late for many people that don't realise what they are getting. The insurance policies will therefore give a much lower price and come top of the listing.
The problem is that other insurers (such as Direct Line as in your example) don't offer a low cost low quality product (ignoring tesco simple that was introduced after they pulled out of the comparison game). These companies will feature on the list of insurers, but will hardly get any sales because their price is so much higher than the low quality products. Of course the comparison sites don't mind this at all, because although they are not getting much commission they can use that insurer in their adverts to attract more people onto their website. If a comparison site didn't have a single insurer you have heard of on it are you going to be less likely to go there? Chances are most people will answer yes.
So Direct Line a couple years ago made the (somewhat brave at the time, but I think correct) decision to pull out. They were getting nothing from it, and being used to drive profits of other companies that would be attracted onto a website that would take business away from them. Furthermore it actually damages the brand name of companies such as this because people who use that site will see their much higher price (often for a much better product) and just think "oh, they are expensive, I won't try them again".
Norwich Union has recently followed this trend (a good decision for them as they also offer a higher quality than average product), by pulling NU Direct. However the ironic thing is that many high street non-insurance names (Barclays, Asda, Post Office, AA, Marks and Spencers) are either underwritten by a big insurer or are on a panel of insurers. So even though NU are no longer 'on' the comparison sites, there is still a chance (at the moment) that you can get an NU underwritten policy.
Of course this doesn't make comparison sites evil, they are just doing what all the other big companies out there are doing. If you know what you are looking for then they can be a fantastic way of getting a very good price for the insurance you require. Both my car and
household insurance were purchased through comparison sites, and I would always suggest that people take the time to go to one or two to get an idea of a price range. However be warned that the lowest policy is not always the best policy for you - so be careful in what you sign up for!
insurance comparison sites 
