title: Optimizing Memory Allocations of Pandas Code to Process Rows Using Explicit Loops with Numba Optimization
body: <p>Assume I have data in the form (As a Pandas' Data Frame):</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Index</th>
  <th>ID</th>
  <th>Value</th>
  <th>Div Factor</th>
  <th>Weighted Sum</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>1</td>
  <td>2</td>
  <td>1</td>
  <td></td>
  </tr>
  <tr>
  <td>2</td>
  <td>1</td>
  <td>3</td>
  <td>2</td>
  <td></td>
  </tr>
  <tr>
  <td>3</td>
  <td>2</td>
  <td>6</td>
  <td>1</td>
  <td></td>
  </tr>
  <tr>
  <td>4</td>
  <td>1</td>
  <td>1</td>
  <td>3</td>
  <td></td>
  </tr>
  <tr>
  <td>5</td>
  <td>2</td>
  <td>3</td>
  <td>2</td>
  <td></td>
  </tr>
  <tr>
  <td>6</td>
  <td>2</td>
  <td>9</td>
  <td>3</td>
  <td></td>
  </tr>
  <tr>
  <td>7</td>
  <td>2</td>
  <td>8</td>
  <td>4</td>
  <td></td>
  </tr>
  <tr>
  <td>8</td>
  <td>3</td>
  <td>5</td>
  <td>1</td>
  <td></td>
  </tr>
  <tr>
  <td>9</td>
  <td>3</td>
  <td>6</td>
  <td>2</td>
  <td></td>
  </tr>
  <tr>
  <td>10</td>
  <td>1</td>
  <td>8</td>
  <td>4</td>
  <td></td>
  </tr>
  <tr>
  <td>11</td>
  <td>3</td>
  <td>2</td>
  <td>3</td>
  <td></td>
  </tr>
  <tr>
  <td>12</td>
  <td>3</td>
  <td>7</td>
  <td>4</td>
  <td></td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>I want to calculate the column <code>Weighted Sum</code> as following (For the $i$ -th row):</p>
  <ol>
  <li>Look at all values from row 1 to <code>i</code>.</li>
  <li>Sum values by groups according to the <code>ID</code> value of each row. So we have <code>k</code> sum values where <code>k</code> is the number of unique <code>ID</code> value from the row 1 to <code>i</code>.</li>
  <li>Divide each sum (There are <code>k</code> sum values) by the number of elements in the group.</li>
  <li>Sum those <code>k</code> values and divide by <code>k</code> (The average of the averages).</li>
  </ol>
  <p>For example, let's do rows 1, 7 and 12:</p>
  <h3>Row 1</h3>
  <p>For <code>i = 1</code> we have a single value hence the sum is 2 and the average of the single group is 2 and average over all groups is 2.</p>
  <h3>Row 7</h3>
  <p>For <code>i = 7</code> we have only 2 unique values of <code>ID</code> above it: 1 and 2.<br />
  For the group of <code>ID = 1</code> we have: (1 + 3 + 2) / 3 = 2.<br />
  For the group of <code>ID = 2</code> we have: (8 + 9 + 3 + 6) / 4 = 6.5.<br />
  Then the average of averages is (2 + 6.5) / 2 = 4.25.</p>
  <h3>Row 12</h3>
  <p>For <code>i = 12</code> we have 3 unique <code>ID</code> values on the rows <code>1:12</code>.<br />
  For the group of <code>ID = 1</code> we have: (8 + 1 + 3 + 2) / 4 = 3.5.<br />
  For the group of <code>ID = 2</code> we have: (8 + 9 + 3 + 6) / 4 = 6.5.<br />
  For the group of <code>ID = 3</code> we have: (7 + 2 + 6 + 5) / 4 = 5.<br />
  Then the average of averages is (3.5 + 6.5 + 5) / 3 = 5.</p>
  <p><strong>Remark</strong>: The method should be feasible for the case of ~<code>1e7</code> rows and ~<code>1e6</code> unique ID's.</p>
  <p>As a follow up to <a href="https://stackoverflow.com/questions/72776330/apply-a-function-per-row-of-sub-groups-of-the-data-above-the-current-row">Apply a Function per Row of Sub Groups of the Data **Above** the Current Row</a>, I got a good answer yet it allocates a lot of memory in case the number of rows and the number of unique <code>ID</code> are high.</p>
  <p>I was wondering if there is a way to create a much smaller auxiliary data using explicit loops while accelerating them using <code>Numba</code>.</p>
  <p>The idea is to have similar or better performance while reducing the memory footprint considerably.</p>