Perhaps LLMs are not as effective at reasoning as appears on the surface due to contamination of evaluation benchmarks (sort of like getting test questions and answers prior to the exam): HERE