Another vote here for 2 part epoxy for antler work, or just about anything else except certain plastics. Gorilla Glue itself is an excellent adhesive (friend of mine says you could use it to glue slime to snot), but it expands during the cure and can leave some problematic clean up.
When I use antler for sgian dubh handles, I use molten pewter (lead free) to secure the blade.