You can use js dblclick event on the annotation finding it by previously added className. Simply remove the annotation using chart.removeAnnotation with the annotation id as a parameter, see demo:
Thanks for reply, is there a way we can use 'Delete' Key on selected annotation to delete it ? this is because we are using stock tools. and when user clicks for the first time it selects the annotation.. double click would select once also.. so thinking of keyboard shortcut might be better..