Mohammed Hamdy's picture

Mohammed Hamdy

mmhamdy

·

AI & ML interests

TechBio | AI4Sci | NLP | Reinforcement Learning

Recent Activity

liked a model 1 day ago

microsoft/wham

upvoted a collection 1 day ago

Reasoning Datasets

posted an update 2 days ago

🎉 We're excited to introduce MemoryCode, a novel synthetic dataset designed to rigorously evaluate LLMs' ability to track and execute coding instructions across multiple sessions. MemoryCode simulates realistic workplace scenarios where a mentee (the LLM) receives coding instructions from a mentor amidst a stream of both relevant and irrelevant information. 💡 But what makes MemoryCode unique?! The combination of the following: ✅ Multi-Session Dialogue Histories: MemoryCode consists of chronological sequences of dialogues between a mentor and a mentee, mirroring real-world interactions between coworkers. ✅ Interspersed Irrelevant Information: Critical instructions are deliberately interspersed with unrelated content, replicating the information overload common in office environments. ✅ Instruction Updates: Coding rules and conventions can be updated multiple times throughout the dialogue history, requiring LLMs to track and apply the most recent information. ✅ Prospective Memory: Unlike previous datasets that cue information retrieval, MemoryCode requires LLMs to spontaneously recall and apply relevant instructions without explicit prompts. ✅ Practical Task Execution: LLMs are evaluated on their ability to use the retrieved information to perform practical coding tasks, bridging the gap between information recall and real-world application. 📌 Our Findings 1️⃣ While even small models can handle isolated coding instructions, the performance of top-tier models like GPT-4o dramatically deteriorates when instructions are spread across multiple sessions. 2️⃣ This performance drop isn't simply due to the length of the context. Our analysis indicates that LLMs struggle to reason compositionally over sequences of instructions and updates. They have difficulty keeping track of which instructions are current and how to apply them. 🔗 Paper: https://huggingface.co/papers/2502.13791 📦 Code: https://github.com/for-ai/MemoryCode

View all activity

Organizations

mmhamdy's activity

upvoted a collection 1 day ago

Reasoning Datasets

18 items • Updated about 22 hours ago • 2

upvoted 2 papers 3 days ago

MMTEB: Massive Multilingual Text Embedding Benchmark

Paper • 2502.13595 • Published 4 days ago • 26

From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions

Paper • 2502.13791 • Published 4 days ago • 5

upvoted a paper 25 days ago

SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

Paper • 2501.17161 • Published 26 days ago • 106

upvoted a paper about 1 month ago

The Lessons of Developing Process Reward Models in Mathematical Reasoning

Paper • 2501.07301 • Published Jan 13 • 91

upvoted 5 papers about 2 months ago

METAGENE-1: Metagenomic Foundation Model for Pandemic Monitoring

Paper • 2501.02045 • Published Jan 3 • 21

EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation

Paper • 2501.01895 • Published Jan 3 • 51

LiveBench: A Challenging, Contamination-Free LLM Benchmark

Paper • 2406.19314 • Published Jun 27, 2024 • 23

ProcessBench: Identifying Process Errors in Mathematical Reasoning

Paper • 2412.06559 • Published Dec 9, 2024 • 80

PepTune: De Novo Generation of Therapeutic Peptides with Multi-Objective-Guided Discrete Diffusion

Paper • 2412.17780 • Published Dec 23, 2024 • 4

upvoted a paper 2 months ago

Bridging the Data Provenance Gap Across Text, Speech and Video

Paper • 2412.17847 • Published Dec 19, 2024 • 9

upvoted a collection 2 months ago

Falcon3

Falcon3 family of Open Foundation Models is a set of pretrained and instruct LLMs ranging from 1B to 10B parameters. • 40 items • Updated 10 days ago • 81

upvoted a paper 2 months ago

The Future of Open Human Feedback

Paper • 2408.16961 • Published Aug 15, 2024 • 21

upvoted an article 3 months ago

Article

Comparing Open-source and Proprietary LLMs in Medical AI

By

•

Oct 3, 2024

• 16

upvoted a paper 3 months ago

Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models

Paper • 2412.02980 • Published Dec 4, 2024 • 13

upvoted 5 collections 3 months ago

Insight-V

Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models • 5 items • Updated Nov 22, 2024 • 9

🧠 Reasoning Models

9 items • Updated Jan 20 • 37

MobileLLM

Optimizing Sub-billion Parameter Language Models for On-Device Use Cases (ICML 2024) https://arxiv.org/abs/2402.14905 • 9 items • Updated Nov 27, 2024 • 109

Models for dataset curation

9 items • Updated Dec 5, 2024 • 17

Florence

9 items • Updated Jan 8 • 164