Thanks for all of your tips.
I guess I'm leaning towards just politely bowing out.
Up to now I've been leaving work for my lunch anyways.
Maybe I'm just looking for the polite way of handling it.