title: What is the best way to shift data by X bits into and out of a file?
body: <p>I have a dataset, made up of a mask file, and a data file.
  the mask file tells the decoder whether there are 8 bits per field present, or 4, for the current offset of the datafile.
  I need to shift the data out according to the mask, and write the decoded file, with all 8 bits per field.
  I'm trying to accomplish this in C.</p>
  
  <pre><code>void shift_4bits_left(unsigned char* array, unsigned short size)
  {
  int i;
  unsigned char shifted = 0x00;    
  unsigned char overflow = (0xF0 &amp; array[0]) &gt;&gt; 4;
  
      for (i = (size - 1); i &gt;= 0; i--)
      {
      shifted = (array[i] &lt;&lt; 4) | overflow;
      overflow = (0xF0 &amp; array[i]) &gt;&gt; 4;
      array[i] = shifted;
      }
  }
  </code></pre>
  
  <p>in function:</p>
  
  <pre><code>while(len&gt;count){
          //count=0;
          if(bit==0)yy=fread(blockchar,1,len,fp1);
          if(bit==0)memcpy(blockchar2,blockchar,len);
          count+=yy;
          memset(outputbuf,0,64);
          for(x=0;x&lt;len;x++){
          count2=0;
          //count2=1;
  
          if(sometests(blockchar[x])==1)
          {
            shift_4bits_left(&amp;blockchar[x],(yy-(bittest/2))+1);
            count2=1;
            total++;
            if(total%2==0)len--;
          }
  
  //set bit for current position in mask file's buffer
          if((x)%8 == 0)outputbuf[x]+=(count2&lt;&lt;7);
          if((x)%8 == 1)outputbuf[x]+=(count2&lt;&lt;6);
          if((x)%8 == 2)outputbuf[x]+=(count2&lt;&lt;5);
          if((x)%8 == 3)outputbuf[x]+=(count2&lt;&lt;4);
          if((x)%8 == 4)outputbuf[x]+=(count2&lt;&lt;3);
          if((x)%8 == 5)outputbuf[x]+=(count2&lt;&lt;2);
          if((x)%8 == 6)outputbuf[x]+=(count2&lt;&lt;1);
          if((x)%8 == 7)outputbuf[x]+=count2;
          }
  </code></pre>
  
  <p>The problem is this is very slow when working with large datasets, I would prefer to make this iterative over the file, without storing the files in arrays.</p>