To add to the structured generation recommendations, github.com/dottxt-ai/ou... is designed for locally run models, can run on top of HF Transformers or Llama.cpp and provides a "generate one of these choices" function

GitHub - dottxt-ai/outlines: Structured Text Generation

Structured Text Generation. Contribute to dottxt-ai/outlines development by creating an account on GitHub.

This looks super interesting & relevant to our problem—thanks for the recommendation!