META Is Doing The Most Interesting AI Research Led By Its LCM Effort
META is doing the most interesting work in the Deep Learning/Neural Network/Gen AI space in my view.
With its Large Concept Models (“LCM”) effort, META seeks to more closely mimic how the human brain thinks/analyzes/learns versus the more deliberate, token-level approach of LLMs.
The LCM approach is innovative in that it models reasoning at the semantic level, which is independent from the syntax and even the language and modality of the content. The challenge for LCMs is that because it is operating at a higher semantic level, the number of possibilities when modeling the next sentence is infinite as compared to modeling at the token level, where the possible outcomes are typically limited to 100,000 and fall across a normal distribution.
The LCM approach tested in this paper yielded superior results to similar-sized LLMs (the LCM scaled from 1.6 billion parameters to 7 billion, a range of what may be considered a tiny model at 1.6 billion and a small model at 7 billion parameters). The jury is out. Let’s see how effectively the LCM approach scales to models with hundreds of billions of parameters. LCMs may represent a breakthrough in machine-based reasoning and would require less training data (i.e., less compute/expense), than LLMs.
