Grease a tin or oven-proof dish and set aside.
Add the sugar, water, vinegar, extra virgin olive oil and rosemary into a pot and stir to combine. Heat on a medium temperature until the sugar has dissolved. Stirring occasionally, heat until the mixture is boiling and boil until it reaches a hard crack stage. Remove the rosemary sprig and pour into the greased dish. Set aside to cool.
NOTE: An easy way to test the toffee as you are cooking is to have a saucer with ice cold water in it. Drop a few drops of the toffee into it and if the toffee is ready, it will set hard.