Hugging Face's logo

MMIE
community
Activity Feed
\n \n

MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models

\n

\n[📖 Project]\n[📄 Paper]\n[💻 Code]\n[📝 Dataset]\n[🤖 Evaluation Model]\n[🏆 Leaderboard]\n

\n
\n\nWe introduce **MMIE**, a robust, knowledge-intensive benchmark to evaluate interleaved multimodal comprehension and generation in LVLMs. With **20K+ examples** covering **12 fields** and **102 subfields**, **MMIE** is definitely setting new standards for testing the depths of multimodal understanding.\n\n
\n\n
\n\nMMIE is curated to evaluate models' comprehensive abilities in interleaved multimodal comprehension and generation. The dataset features diverse examples, categorized and distributed across different fields as illustrated above. This ensures balanced coverage across various domains of interleaved input/output tasks, supporting accurate and detailed model evaluations.\n -->","html":"
\n\n\n \n

MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models

\n

\n[📖 Project]\n[📄 Paper]\n[💻 Code]\n[📝 Dataset]\n[🤖 Evaluation Model]\n[🏆 Leaderboard]\n

\n
\n\n

We introduce MMIE, a robust, knowledge-intensive benchmark to evaluate interleaved multimodal comprehension and generation in LVLMs. With 20K+ examples covering 12 fields and 102 subfields, MMIE is definitely setting new standards for testing the depths of multimodal understanding.

\n
\n\n
\n\n

🔑 Key Features:

\n\n
\n

🔍 Key Insights:

\n
    \n
  1. 🧠 In-depth Evaluation: Covering 12 major fields (mathematics, coding, literature, and more) with 102 subfields for a comprehensive test across competencies.
    2. \n
  2. 📈 Challenging the Best: Even top models like GPT-4o + SDXL peak at 65.47%, highlighting room for growth in LVLMs.
    3. \n
  3. 🌐 Designed for Interleaved Tasks: The benchmark supports evaluation across both text and image comprehension with both multiple-choice and open-ended formats.
    4. \n
\n","classNames":"hf-sanitized hf-sanitized-YypifJ_RD7hgcA7LQKBKB"},"users":[{"_id":"643e9ee6f6bb3c31a26e7bc4","avatarUrl":"/avatars/acfaa7d6a23dada24c86b954c3be116a.svg","isPro":false,"fullname":"Peng Xia","user":"richardxp888","type":"user"},{"_id":"6644bb2c9bdbd85493074411","avatarUrl":"https://aifasthub.com/avatars/v1/production/uploads/6644bb2c9bdbd85493074411/iug8dkP1zjID-kXzNxDjD.jpeg","isPro":false,"fullname":"SHI QIU","user":"StarThomas1002","type":"user"}],"userCount":2,"collections":[],"datasets":[{"author":"MMIE","downloads":27,"gated":"auto","id":"MMIE/MMIE","lastModified":"2024-10-15T01:59:30.000Z","datasetsServerInfo":{"viewer":"viewer","numRows":20122,"libraries":["datasets","dask","mlcroissant"],"formats":["json"],"modalities":["text"]},"private":false,"repoType":"dataset","likes":12,"isLikedByUser":false}],"models":[{"author":"MMIE","authorData":{"_id":"67000c998dd86e94096e0218","avatarUrl":"https://aifasthub.com/avatars/v1/production/uploads/65941852f0152a21fc860f79/LzQmOAfmO3U5C5oGfLg4X.png","fullname":"MMIE","name":"MMIE","type":"org","isHf":false,"isHfAdmin":false,"isMod":false,"isEnterprise":false,"followerCount":4},"downloads":3,"gated":"auto","id":"MMIE/MMIE-Score","availableInferenceProviders":[],"lastModified":"2024-10-28T14:04:33.000Z","likes":1,"pipeline_tag":"image-text-to-text","private":false,"repoType":"model","isLikedByUser":false,"widgetOutputUrls":[],"numParameters":4146855936}],"spaces":[{"author":"MMIE","authorData":{"_id":"67000c998dd86e94096e0218","avatarUrl":"https://aifasthub.com/avatars/v1/production/uploads/65941852f0152a21fc860f79/LzQmOAfmO3U5C5oGfLg4X.png","fullname":"MMIE","name":"MMIE","type":"org","isHf":false,"isHfAdmin":false,"isMod":false,"isEnterprise":false,"followerCount":4},"colorFrom":"green","colorTo":"indigo","createdAt":"2024-10-04T23:57:11.000Z","emoji":"🥇","id":"MMIE/Leaderboard","lastModified":"2024-10-15T03:13:21.000Z","likes":2,"pinned":true,"private":false,"sdk":"gradio","repoType":"space","runtime":{"stage":"RUNNING","hardware":{"current":"cpu-basic","requested":"cpu-basic"},"storage":null,"gcTimeout":172800,"errorMessage":null,"replicas":{"current":1,"requested":1},"devMode":false,"domains":[{"domain":"mmie-leaderboard.hf.space","stage":"READY"}],"sha":"828686a3dbd36ed1a9254061c5906b1c974bf625"},"title":"MMIE Leaderboard","isLikedByUser":false,"originSpace":{"name":"MJ-Bench/MJ-Bench-Leaderboard","author":{"_id":"666bb425066ceea60c0d59b8","avatarUrl":"https://aifasthub.com/avatars/v1/production/uploads/64d448d3d095077728f7c740/eiPzzUiEPaUmUeJRB7oRy.jpeg","fullname":"MJ-Bench-Team","name":"MJ-Bench","type":"org","isHf":false,"isHfAdmin":false,"isMod":false,"isEnterprise":false,"followerCount":12}},"ai_short_description":"Display and search MMIE Leaderboard data","ai_category":"Data Visualization","trendingScore":0,"tags":["gradio","region:us"]}],"numDatasets":1,"numModels":1,"numSpaces":2,"lastOrgActivities":[{"time":"2025-08-13T07:12:38.456Z","user":"richardxp888","userAvatarUrl":"/avatars/acfaa7d6a23dada24c86b954c3be116a.svg","type":"paper","paper":{"id":"2508.05748","title":"WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent","publishedAt":"2025-08-07T18:03:50.000Z","upvotes":122,"isUpvotedByUser":true}},{"time":"2025-07-09T08:50:44.156Z","user":"richardxp888","userAvatarUrl":"/avatars/acfaa7d6a23dada24c86b954c3be116a.svg","type":"paper","paper":{"id":"2507.06229","title":"Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solving","publishedAt":"2025-07-08T17:59:22.000Z","upvotes":73,"isUpvotedByUser":true}},{"time":"2025-07-07T11:25:42.089Z","user":"richardxp888","userAvatarUrl":"/avatars/acfaa7d6a23dada24c86b954c3be116a.svg","type":"paper","paper":{"id":"2506.00555","title":"MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal\n Medical Reasoning","publishedAt":"2025-05-31T13:22:55.000Z","upvotes":1,"isUpvotedByUser":true}}],"acceptLanguages":["*"],"canReadRepos":false,"canReadSpaces":false,"blogPosts":[],"currentRepoPage":0,"filters":{}}">

AI & ML interests

None defined yet.

Recent Activity

richardxp888  authored a paper 25 days ago
WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent
richardxp888  authored a paper about 2 months ago
Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solving
richardxp888  authored a paper 2 months ago
MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoning
View all activity

Organization Card
Community About org cards

MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models

[📖 Project] [📄 Paper] [💻 Code] [📝 Dataset] [🤖 Evaluation Model] [🏆 Leaderboard]

We introduce MMIE, a robust, knowledge-intensive benchmark to evaluate interleaved multimodal comprehension and generation in LVLMs. With 20K+ examples covering 12 fields and 102 subfields, MMIE is definitely setting new standards for testing the depths of multimodal understanding.

🔑 Key Features:

  • 🗂 Comprehensive Dataset: With 20,103 interleaved multimodal questions, MMIE provides a rich foundation for evaluating models across diverse domains.
  • 🔍 Ground Truth Reference: Each query includes a reliable reference, ensuring model outputs are measured accurately.
  • ⚙ Automated Scoring with MMIE-Score: Our scoring model achieves high human-score correlation, surpassing previous metrics like GPT-4o for multimodal tasks.
  • 🔎 Bias Mitigation: Fine-tuned for fair assessments, enabling more objective model evaluations.

🔍 Key Insights:

  1. 🧠 In-depth Evaluation: Covering 12 major fields (mathematics, coding, literature, and more) with 102 subfields for a comprehensive test across competencies.
  2. 📈 Challenging the Best: Even top models like GPT-4o + SDXL peak at 65.47%, highlighting room for growth in LVLMs.
  3. 🌐 Designed for Interleaved Tasks: The benchmark supports evaluation across both text and image comprehension with both multiple-choice and open-ended formats.