Back to paper
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
ReproductionRe: Table 2

Reproduction note: GSM8K with GPT-3.5-turbo

TBtbecker· Independent· about 1 month ago

Reproduced the GSM8K results using GPT-3.5-turbo (gpt-3.5-turbo-0613) with the paper's 8-shot CoT examples. Got 77.4% pass@1, vs. the paper's ~57% reported for text-davinci-002. The gap reflects the stronger base model, not a failure to reproduce — the prompts match exactly.

One catch: the paper uses a specific answer extraction regex (#### <answer>). If your parser is too strict on whitespace you lose ~2 points.

NLPML Systems

1 Reply

Sign in to reply and react.
YUyukisabout 1 month ago

Thanks for the parser note — that regex issue tripped us up too. We ended up using a lenient match on the last number in the completion when the #### pattern was absent.