This one drove me mad until I worked it out.
The Favourites Bar icons had been duplicated but when you deleted the duplicated icons after a few seconds they’d all pop back again.
This turned out to be our friends at Apple being over zealous in their iCloud application. Basically every time they were deleted iCloud would read that they were gone and put them back again.
You need to turn off Bookmark synchronisation in your iCloud settings to stop this happening.
Once you’ve done this you can go back into Internet explorer and delete the duplicate or unwanted entries.