Computer Science › Software Engineering

[Submitted on 3 Nov 2026]

Zero-Shot vs. Chain-of-Thought Prompting for Python Code Generation: Separating Execution, Functional Correctness, and Judged Quality

Sean Lester S. Miranda

View PDF

Abstract:Zero-shot and chain-of-thought (CoT) prompting can produce different program structures, validation logic, and explanations, but those differences need not translate into improved correctness. We present a small case study using ChainZCode, a code-evaluation toolkit with a retained collection of 180 Python implementations: 30 algorithmic tasks, three models (GPT-4o, Claude 3.5 Sonnet, and Gemini 2.0 Flash), and two styles. The evaluation separates retrospective external callable checks under explicit task contracts, whole-script completion, and five-criterion model-assisted quality scores. Of the 180 implementations, 178 pass the callable suites and 177 complete as whole scripts. GPT and Claude pass all 30 callable suites in each style; Gemini passes 29 in each, with different failed tasks. The failures distinguish a maximum-subarray convention, a syntax error, and defects in generated self-tests. Paired CoT-minus-zero-shot judge-score differences are −0.124, +0.180, and −0.497 points out of 20 for GPT, Claude, and Gemini, respectively. Gemini’s difference becomes +0.171 when the comparison requires two completed assessments, a sign reversal showing why failure policy and assessment coverage must accompany quality averages. Process completion, checked functional behavior, and judged quality capture different outcomes in this case study. The study contains one saved implementation per condition; its conclusions concern these programs rather than variation across newly sampled generations.

@misc{miranda2026zeroshot,
  title     = {Zero-Shot vs. Chain-of-Thought Prompting for Python Code Generation: Separating Execution, Functional Correctness, and Judged Quality},
  author    = {Miranda, {Sean Lester}},
  year      = {2026},
  month     = nov,
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23894899},
  note      = {Preprint},
  url       = {https://seanmiranda.com/papers/zero-shot-vs-cot.html}
}