In this issue:

1. Google following the Meta strategy
2. Automatic prompt engineering
3. Graph reading for long-form documents

Subscribe now

1. Gemma 2: Improving Open Language Models at a Practical Size

Watching: Gemma 2 (paper /models )

What problem does it solve? The Gemma family of models aims to provide lightweight, state-of-the-art open models that deliver competitive performance while being more accessible and efficient than larger models. By offering models ranging from 2 billion to 27 billion parameters, Gemma 2 addresses the need for high-quality language models that can be deployed in various applications without the computational overhead of massive models.

How does it solve the problem? Gemma 2 introduces several technical modifications to enhance performance and efficiency. The architecture incorporates interleaved local-global attentions, which allow the model to capture both local and global dependencies effectively. Additionally, group-query attention is employed to reduce computational complexity. For the smaller models (2B and 9B), knowledge distillation is used instead of next token prediction during training, enabling them to learn from larger, more powerful models while maintaining a compact size.

What's next? The release of Gemma 2 models to the community opens up exciting possibilities for researchers and developers. With models ranging from 2 billion to 27 billion parameters, there is flexibility in choosing the appropriate model size for specific applications. The competitive performance of these models, even compared to larger counterparts, suggests that they could be widely adopted for various natural language processing tasks.

2. TextGrad: Automatic "Differentiation" via Text

Watching: TextGrad (paper /code )

What problem does it solve? As AI systems become more complex, often involving multiple large language models (LLMs) and other components working together, optimizing these compound systems becomes a significant challenge. Just as the development of backpropagation and automatic differentiation transformed the field of neural networks by making optimization more straightforward, there is a need for a similar principled and automated optimization method for compound AI systems.

How does it solve the problem? TextGrad introduces a framework that performs automatic "differentiation" via text, allowing for the optimization of individual components within a compound AI system. It leverages the power of LLMs to provide rich, general, and natural language suggestions for optimizing variables in computation graphs, ranging from code snippets to molecular structures. TextGrad follows a syntax and abstraction similar to PyTorch, making it flexible and easy to use. Users only need to provide the objective function, without the need to tune components or prompts of the framework.

What's next? TextGrad has demonstrated its effectiveness and generality across a diverse range of applications, including question answering, molecule optimization, and radiotherapy treatment planning. The excitement around DSPy - another framework for algorithmically optimizing LLM prompts and weights - has shown that there’s high demand for more systematic solutions to prompting. If TextGrad will be similarly received, we will hear a lot more about this framework in the near future.

3. GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Models

Watching: GraphReader (paper )

What problem does it solve? While LLMs have shown remarkable performance on a wide range of tasks, they still struggle with long-context inputs. As the input length increases, the computational complexity and memory requirements grow quadratically, making it challenging to process and reason over long texts efficiently. This limitation hinders the application of LLMs in domains such as document understanding, multi-hop question answering, and complex reasoning tasks that require processing and integrating information from lengthy sources.

How does it solve the problem? GraphReader addresses the long-context challenge by structuring the input text into a graph representation. Instead of processing the entire input at once, an agent autonomously explores the graph, focusing on relevant nodes and their neighbors. The agent follows a rational plan, invoking predefined functions to read node content and navigate the graph in a coarse-to-fine manner. By continuously recording insights and reflecting on its progress, the agent optimizes the exploration process until sufficient information is gathered to generate an answer. This graph-based approach allows for efficient processing of long texts while maintaining the ability to capture and reason over complex relationships and dependencies.

What's next? One direction is to explore the integration of GraphReader with other LLMs to enhance their long-context capabilities. Furthermore, improving the efficiency and scalability of GraphReader, possibly through techniques like graph compression and parallel processing, would make it more practical for real-world deployments. As the demand for processing and understanding long-form content continues to grow, GraphReader and similar graph-based approaches are likely to play a significant role in enabling new applications.

Papers of the Week:

• D2LLM: Decomposed and Distilled Large Language Models for Semantic Search
• Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
• MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool
• A Survey of Large Language Models for Graphs
• OAG-Bench: A Human-Curated Benchmark for Academic Graph Mining
• Grokfast: Accelerated Grokking by Amplifying Slow Gradients
• Confabulation: The Surprising Value of Large Language Model Hallucinations
• Poisoned LangChain: Jailbreak LLMs by LangChain