From Newsgroup: comp.compilers

Four people at Huawei use a lightweight LLM to segment C code for
analysis.

Abstract
Semantic segmentation of code written in a C-family language remains a challenging problem, due to the language's complex syntax, macro
expansion, and irregular structural patterns. Existing chunking methods,
such as fixed-sized windows, heuristic splitting, and syntax-based tools,
often fail to capture meaningful functional units, limiting the efficacy
of retrieval and other downstream LLM driven tasks.
In this paper, we address the problem of chunking in C-related languages. First, we define a set of code chunk categories. Second, we train an
LLM-based classifier to a) identify chunk boundaries, and b) assign each
chunk a descriptive functional attribute (a category), which can be useful
for downstream tasks. By leveraging the LLM's ability to capture semantic context within the code, we assume flexible chunk boundaries, allowing to
adapt to the specific structure and context of each instance. Third, we introduce SemChunk-C, a family of lightweight language models for semantic chunking of C-related files (.c, .cpp, .h, .cs, etc.). These models are
based on the first four Ettin encoders [1] with 17M, 32M, 68M, and 150M parameters. Despite their relatively small size, they are capable of identifying cohesive code units, such as data structures, interface
blocks, and other components. Furthermore, we demonstrate the robustness
of our approach on real-world code, including challenging constructs such
as nested definitions and macros.
We test our approach on various datasets, and show that it achieves high boundary accuracy and semantic coherence, matching or outperforming
chunkers that are based on much larger code-oriented LLMs. We also
validate the improved performance of the downstream tasks on a few curated benchmarks.

https://arxiv.org/abs/2606.23697

Regards,
John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly
--- Synchronet 3.22a-Linux NewsLink 1.2