In a notification without buttons: Click the notification, then reply in Calendar.
If your declined events disappear from Calendar, and you want to see them again, choose View > Show Declined Events.