You can't prompt your way to a better prompt.

It’s WAY too easy to fall into this trap: your first prompt gets mediocre results; you ask your LLM to analyse why that happened and to give you a better prompt for next time. Bingo problem solved.

Only it isn’t. That new prompt MIGHT be better but the LLM has no way of knowing that.

LLMs are next-token predictors. When you asked it to analyse what went wrong, there’s a good chance that its answer was useful. Not because it understands how prompts affect the model but because hundreds of billions of parameters indicated that this particular output was the most probable string of words for your input.

And when you asked it for a better prompt, it just strung together the most probable stream of words given the earlier conversation. It had no way of verifying whether what it said was true.

LLMs can’t access the truth. For an LLM to be consistently useful, it needs some way of verifying that its outputs are useful. And this verification needs to come from outside of the model.