A Systematic Evaluation of Large Language Models of Code

The authors state that the current state-of-the-art code Large Language Models (LMs) (e.g., Codex (Chen et al., 2021)) are not publicly available, leaving many questions about their model and data design decisions. Hence, they have presented a systematic evaluation of the largest existing models such as Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot, across various programming languages. Furthermore, they have observed that existing open-source models do achieve close results in some programming languages, although targeted mainly for natural language modelling. They have released a new model, PolyCoder, based on the GPT-2 architecture, that was trained on 12 programming languages on a single machine.