17 54 37

Mohammed Hamdy

mmhamdy

AI & ML interests

TechBio | AI4Sci | NLP | Reinforcement Learning

Recent Activity

liked a model 1 day ago

microsoft/wham

upvoted a collection 1 day ago

Reasoning Datasets

posted an update 1 day ago

🎉 We're excited to introduce MemoryCode, a novel synthetic dataset designed to rigorously evaluate LLMs' ability to track and execute coding instructions across multiple sessions. MemoryCode simulates realistic workplace scenarios where a mentee (the LLM) receives coding instructions from a mentor amidst a stream of both relevant and irrelevant information. 💡 But what makes MemoryCode unique?! The combination of the following: ✅ Multi-Session Dialogue Histories: MemoryCode consists of chronological sequences of dialogues between a mentor and a mentee, mirroring real-world interactions between coworkers. ✅ Interspersed Irrelevant Information: Critical instructions are deliberately interspersed with unrelated content, replicating the information overload common in office environments. ✅ Instruction Updates: Coding rules and conventions can be updated multiple times throughout the dialogue history, requiring LLMs to track and apply the most recent information. ✅ Prospective Memory: Unlike previous datasets that cue information retrieval, MemoryCode requires LLMs to spontaneously recall and apply relevant instructions without explicit prompts. ✅ Practical Task Execution: LLMs are evaluated on their ability to use the retrieved information to perform practical coding tasks, bridging the gap between information recall and real-world application. 📌 Our Findings 1️⃣ While even small models can handle isolated coding instructions, the performance of top-tier models like GPT-4o dramatically deteriorates when instructions are spread across multiple sessions. 2️⃣ This performance drop isn't simply due to the length of the context. Our analysis indicates that LLMs struggle to reason compositionally over sequences of instructions and updates. They have difficulty keeping track of which instructions are current and how to apply them. 🔗 Paper: https://huggingface.co/papers/2502.13791 📦 Code: https://github.com/for-ai/MemoryCode

View all activity

Organizations

mmhamdy's activity

commented 2 papers 3 days ago

From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions

Paper • 2502.13791 • Published 4 days ago • 5 •

From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions

Paper • 2502.13791 • Published 4 days ago • 5 •

commented a paper 5 days ago

Linking In-context Learning in Transformers to Human Episodic Memory

Paper • 2405.14992 • Published May 23, 2024 • 1 •

commented a paper about 2 months ago

Bridging the Data Provenance Gap Across Text, Speech and Video

Paper • 2412.17847 • Published Dec 19, 2024 • 9 •

commented a paper 3 months ago

Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models

Paper • 2412.02980 • Published Dec 4, 2024 • 13 •

New activity in huggingface/HuggingDiscussions 3 months ago

[FEEDBACK] Collections

#12 opened over 1 year ago by

victor

commented a paper 6 months ago

$\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens

Paper • 2402.13718 • Published Feb 21, 2024 • 1 •

commented 4 papers 7 months ago

Consent in Crisis: The Rapid Decline of the AI Data Commons

Paper • 2407.14933 • Published Jul 20, 2024 • 12 •

The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI

Paper • 2310.16787 • Published Oct 25, 2023 • 5 •

Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?

Paper • 2404.12691 • Published Apr 19, 2024 • 2 •

Consent in Crisis: The Rapid Decline of the AI Data Commons

Paper • 2407.14933 • Published Jul 20, 2024 • 12 •

commented 3 papers 10 months ago

Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models

Paper • 2404.18796 • Published Apr 29, 2024 • 69 •

Time Machine GPT

Paper • 2404.18543 • Published Apr 29, 2024 • 2 •

Enabling Natural Zero-Shot Prompting on Encoder Models via Statement-Tuning

Paper • 2404.12897 • Published Apr 19, 2024 • 1 •

commented a paper 11 months ago

Augmenting Language Models with Long-Term Memory

Paper • 2306.07174 • Published Jun 12, 2023 • 18 •

commented 3 papers 12 months ago

New activity in hf-vision/course-assets about 1 year ago

Unit-13 images

#69 opened about 1 year ago by

mmhamdy