title: How do I read in a large flat file
body: <p>I have a flat file that has 339276 line of text in it for a size of 62.1 MB. I am attempting to read in all the lines, parse them based on some conditions I have and then insert them into a database. </p>
  
  <p>I originally attempted to use a bufio.Scan() loop and bufio.Text() to get the line but I was running out of buffer space. I switched to using bufio.ReadLine/ReadString/ReadByte (I tried each) and had the same problem with each. I didn't have enough buffer space. </p>
  
  <p>I tried using read and setting the buffer size but as the document says it actually a const that can be made smaller but never bigger that 64*1024 bytes. I then tried to use File.ReadAt where I set the starting postilion and moved it along as I brought in each section to no avail. I have looked at the following examples and explanations (not an exhaustive list): </p>
  
  <p><a href="https://stackoverflow.com/questions/5884154/golang-read-text-file-into-string-array-and-write">Read text file into string array (and write)</a>
  <a href="https://stackoverflow.com/questions/17863821/how-to-read-last-lines-from-a-big-file-with-go-every-10-secs">How to Read last lines from a big file with Go every 10 secs</a>
  <a href="https://stackoverflow.com/questions/8757389/reading-file-line-by-line-in-go">reading file line by line in go</a></p>
  
  <p>How do I read in an entire file (either line by line or the whole thing at once) into a slice so I can then go do things to the lines?</p>
  
  <p>Here is some code that I have tried:   </p>
  
  <pre><code>                 file, err := os.Open(feedFolder + value)
                   handleError(err)
                   defer file.Close()
                   //              fileInfo, _ := file.Stat()
                   var linesInFile []string
  
               r := bufio.NewReader(file)
               for {
                       path, err := r.ReadLine("\n") // 0x0A separator = newline
  
                       linesInFile = append(linesInFile, path)
                       if err == io.EOF {
                               fmt.Printf("End Of File: %s", err)
                               break
                       } else if err != nil {
                               handleError(err) // if you return error
                       }
               }
               fmt.Println("Last Line: ", linesInFile[len(linesInFile)-1])
  </code></pre>
  
  <p>Here is something else I tried:  </p>
  
  <pre><code>var fileSize int64 = fileInfo.Size()
      fmt.Printf("File Size: %d\t", fileSize)
      var bufferSize int64 = 1024 * 60
      bytes := make([]byte, bufferSize)
      var fullFile []byte
      var start int64 = 0
      var interationCounter int64 = 1
      var currentErr error = nil
           for currentErr != io.EOF {
              _, currentErr = file.ReadAt(bytes, st)
              fullFile = append(fullFile, bytes...)
              start = (bufferSize * interationCounter) + 1
              interationCounter++
            }
       fmt.Printf("Err: %s\n", currentErr)
       fmt.Printf("fullFile Size: %s\n", len(fullFile))
       fmt.Printf("Start: %d", start)
  
       var currentLine []string
  
  
     for _, value := range fullFile {
        if string(value) != "\n" {
            currentLine = append(currentLine, string(value))
        } else {
           singleLine := strings.Join(currentLine, "")
           linesInFile = append(linesInFile, singleLine)
           currentLine = nil
                }   
        }
  </code></pre>
  
  <p>I am at a loss. Either I don't understand exactly how the buffer works or I don't understand something else. Thanks for reading. </p>