Long Form Question Answering (LFQA) in Haystack
Question-Answering (QA) has exploded as a subdomain of Natural Language Processing (NLP) in the last few years. QA is a widely applicable use case in NLP yet was out of reach until the introduction of [transformer models](/learn/transformers/) in 2017.

Without transformer models, the level of language comprehension required to make something as complex as QA work simply was not possible.

Although QA is a complex topic, it comes from a simple idea. The automatic retrieval of information via a more human-like interaction. The task of information retrieval (IR) is performed by almost every organization in the world. Without other options, organizations rely on person-to-person IR and rigid keyword search tools. This haphazard approach to IR generates a lot of friction, particularly for larger organizations.

QA offers a solution to this problem. Rather than these documents being lost in an abyss, they can be stored within a space where an intelligent QA agent can access them. Unlike humans, our QA agent can scan millions of documents in seconds and return answers from these documents almost instantly.

With QA tools, employees can stop wasting time searching for snippets of information and focus on their *real*, value-adding tasks.

A small investment in QA is, for most organizations, a no-brainer.

🌲 Pinecone article:
https://www.pinecone.io/learn/haystack-lfqa

🔗 Code notebooks:
https://github.com/pinecone-io/examples/blob/master/integrations/haystack/haystack_lfqa.ipynb

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
04:20 Approaches to Question Answering
05:43 Components of QA Pipeline
08:58 LFQA Generator
09:40 Haystack Setup
10:32 Initialize Document Store
13:02 Getting Data
17:53 Indexing Embeddings
21:51 Initialize Generator
24:10 Asking Questions
26:12 Common Problems
29:32 Generator Memory
31:30 Few More Questions
34:54 Outro