Preheat grill for high heat. Squeeze juice from lime wedges over fish fillets, then season with salt and pepper. Grill fish fillets for 5 minutes on each side until browned and fish can be flaked easily with a fork. Remove to a warm serving plate.
Butter Sauce: Heat oil in skillet over medium heat. Add garlic; cook and stir just until fragrant, about 2 minutes. Stir in butter, remaining lime juice, lavender, and cilantro. Serve fish with butter sauce.