Warm the olive oil in a sauté pan over medium heat. Add the garlic cloves, rosemary, fennel seeds, and orange zest. Heat for 2–3 minutes, or until the garlic is mostly cooked.
Add the olives. Stir to coat the olives with the olive oil mixture. Cover and let sit in the refrigerator overnight (or at least for 4 hours) for the flavors to marry.
To serve, pour the entire mixture in a sauté pan and stir gently until the olive oil is liquid again (it solidifies in the refrigerator). The olives can be served warm or at room temperature.