Deductive Verification with Natural Programs: Case Studies

In Tab. 18, we demonstrate that the language model (ChatGPT) not only successfully identifies ungrounded information, but also identifies logical errors within the given solutions.

In Tab. 19, we illustrate a case where the language model fails to detect ungrounded premise numbers, mistakenly assuming that these numbers can be derived from grounded ones.

Lastly, in Tab. 20, we illustrate a case where the language model is sometimes unable to correctly identify grounded numbers.

Table 12: Two-shot prompt for direct reasoning chain verification without Natural Program format.

Table 13: One-shot Natural Program prompt for reasoning chain generation on math word problems.

Table 14: One-shot Natural Program prompt for reasoning chain generation on math word problems with multiple choice.

Table 15: Two-shot Natural Program prompt for reasoning chain generation on the Date dataset.

Table 16: One-shot Natural Program prompt for reasoning chain generation on the Last Letters dataset.

Table 17: One-shot prompt for deductive verification of a single reasoning step, following our Natural Program format and step-by-step reasoning chain decomposition.

Table 18: Successful case: our deductive verification approach successfully discovers ungrounded information and reasoning mistakes.

Table 19: Failure case: our deductive verification process fails to find out ungrounded information in the reasoning step. The number 240 in the reasoning step is ungrounded, but the model states that it can be calculated from grounded numbers.

Table 20: Failure case: our deductive verification process sometimes treats grounded information as if they were ungrounded. The number 120 is provided in the given information, but the model states that it is ungrounded.

文章来源: https://hackernoon.com/deductive-verification-with-natural-programs-case-studies?source=rss
如有侵权请联系:admin#unsafe.sh