{'fraction_non_alphanumeric': 0.03281780460203697, 'fraction_numerical': 0.01464856029171382, 'mean_word_length': 4.511781011781012, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Demand Analysis with a Thin Price Sample

For about 125 items of food, the Consumer Expenditure Survey (CES) schedule of the Indian National Sample Survey asks the interviewer to obtain both quantity and value of household consumption during the reference period from the respondent. This would appear to put a great burden on the respondent. But it is likely that the price usually paid is almost the same within each first stage unit (fsu). The present work proposes a new sampling scheme to estimate demand elasticities of essential food items. While the conventional sampling method used in practice (e.g. in NSS consumer expenditure survey) involves seeking price information from many households sampled from a fsu, the proposed procedure involves only one household chosen randomly from every fsu for price data collection and thus requires much less interview burden. Using unit records for vegetable items in the NSS's 2011-12 CES, our results show that in spite of requiring much less data, the new scheme captures the household food consumption behavior as precisely as before.


Introduction
India is the world's second most populous country. An analysis of household food consumption pattern and its response to changes in income and prices in one of the largest producing and consuming countries is essential to estimate the future demand of agricultural products in the country and also particularly important for agricultural products exporting countries ( [8], [1]).

An important Household consumption expenditure survey is the Consumer Expenditure
Survey conducted by the Indian National Sample Survey Office (NSSO). The NSS consumer expenditure schedule obtains, through face-to-face interview of a large random sample of households spread over rural and urban areas of every district in India, information on more than 130 food items. For most food items, both quantity and value of consumption during the reference period are recorded. In the last two surveys for which results are available, the reference period, for a large number of food items including all vegetables, was "last week" for one half of the sample households and "last month" for the other half. In recent years there has been growing concern over the time needed to canvass the NSS CES and the time that the average household in India is willing to provide for the interview. While research continues on the feasibility of shortening the schedule in terms of number of items of consumption listed (without loss to the survey objectives), surprisingly little attention has been focussed on the double effort of obtaining both quantity and value of consumption, which in the last survey was done for 125 of the elementary items of food listed, 14 items of tobacco, intoxicants, etc., 12 items of fuel, and 30 items of clothing, bedding and footwear. However, abridged or partially abridged schedules have been tried out experimentally. Abridging the schedule usually takes the form of compressing groups of items into single items.
In this work, we investigate that if we want to study the consumer expenditure pattern of a homogeneous society and ask one random household about the price of goods available there (thus using a much thinner sample of price data) and assume that this price holds for the whole society, then is it enough to predict the demand pattern of the society? Or do we have to collect this information from each of the households available there?
We propose a new sampling scheme to estimate demand elasticities of essential food items in this paper. While the conventional sampling method used in practice (e.g. in NSS consumer expenditure survey) involves seeking price information from all households sampled from a fsu, our proposed procedure involves only one household chosen randomly from every fsu for price data collection and thus requires much less interview burden. Using unit records for vegetable items in the NSS's 2011-12 CES, we aim to test whether the new scheme captures the household food consumption behavior as precisely as before.
In the next section we describe the data used for analysis. Some preliminary analysis of data and relevant theoretical results are developed across several subsections in Section 3. The main analysis is presented in section 4 along with some concluding observations.

The Data
We use the data from household consumer expenditure survey 68th round [9] to analyse the distribution of the demand for several food items. Conducted by National Sample Survey Organisation(NSSO) this survey covered, during 2011-12, 101651 households in 7469 villages and 5268 urban blocks spread over the entire country.
The household consumer expenditure survey provides estimates of average household monthly per capita consumer expenditure (MPCE henceforth), the distribution of households and persons over the MPCE range, and the break-up of average MPCE by commodity group, separately for the rural and urban sectors of the country, across different socio-economic groups and across various states and union territories [9]. The indicators of monthly per capita consumption spending play a pivotal part in assessing standard of living, shifting priorities in terms of baskets of goods and services across different strata of the population by providing the budget shares of different commodity groups for the rural and urban population separately.
Moreover, the estimated budget shares of a commodity at different MPCE levels make possible the study of consumption elasticity or responsiveness of demand for the commodity to change in purchasing power. A better understanding of demand elasticities facilitates prediction of future demand of food items under different scenarios of expenditure (considered as a proxy for income) and prices and may be useful to the policy makers on policy formulations [8].
The NSS consumer expenditure schedule is organized in blocks in which consumption of food and non-food items, including services, is recorded. For all items listed in the schedule, consumption expenditure is recorded. For some items, including nearly all food items, quantity consumed is also recorded. This study was confined to those food items for which the NSS consumer expenditure survey collects information on both quantity and value of consumption.
Since the NSS schedule of enquiry uses around 150 items for food, individual food items have very small shares in the total consumer expenditure. It was decided to limit the analysis to those food items which had a share in household consumer expenditure (estimated by the survey) of at least 0.1% at all-India level, separately for rural India and urban India. This gave a set of 29 food items as in Table 1 For each of these items we are given the data about several factors like: • Sector i.e, that is the data is collected from which kind of region which is indicated by (i) rural area and (ii) urban area.
• State i.e, which state the data are collected from. Here each state have been associated with a two digit code.
• Household Id i.e, identification of the sample household.
• Size of the household i.e, the total number of persons in the household.
• Value of the Item Actually, the value of an item is what someone is freely willing to give for it and someone is willing to take for it. So here to measure the valuation we have the total monthly expenditure for a particular item for a particular household.
• Quantity of the Item i.e, the total amount of the item consumed by the household each month.
Once we have the value and quantity of an item then we can easily obtain the price by dividing value by quantity.  The items "other vegetables" (Item 217) and "papad, bhujia, namkeen" (Item 292) also represented a heterogeneous collection of items which would clearly have a very wide price variation. Hence, we have excluded the three items "fish, prawn", "other vegetables" and "papad, bhujia, namkeen" from our analysis. The histograms for the price-ratios of other items are markedly concentrated in a region close to 1. The price-ratio histograms for the items "Rice(PDS)" (Item 101) and "Sugar" (Item 172) also are given in Figure 1.
Finally, "gram, split" was excluded as there were 4 other pulses (items with numbers 140, 143, 144, 145) in the list, and a fifth did not seem necessary, as the four pulses arhar, moong, masur and urd in 2011-12 together made up about 67% of consumption of pulses and pulse products in urban India and 63% in rural India [9].

Sampling Scheme
The We are interested in P(X N q) (prevalence probability henceforth) for a suitable choice of q ∈ (0, 1), which indicates the probability that the item is consumed in at least one household in 100q% of the N fsu's.
As X is a sum of independent Bernoulli random variables with possibly different success probabilities, it follows that X follows a Poisson Binomial distribution [4]. Since working with the exact p.m.f of a Poisson Binomial distribution is mathematically inconvenient and N is very large, we approximate the desired probability with the help of the following theorem.
Proof. We have, . Also, This gives Summing over i we obtain, Hence, when 0 < lim inf p i ≤ lim sup p i < 1, using Lyapunov Central Limit Theorem we obtain (1).
Hence, for sufficiently large N , the suitably standardized version of X approximately follows the standard normal distribution. So (1) implies, for any specific q ∈ (0, 1), where Φ denotes the c.d.f. of the standard normal distribution. Thus, we can approximate the prevalence probability of the item by the quantity in r.h.s of (2). Table 2

The Demand Model
Several demand models are available to estimate the price and income elasticities of demand for various food commodities. Linear Expenditure System (by Stone [14]), and Almost Ideal Demand System (AIDS) (by Deaton and Muellbauer [6]) are two widely studied demand models.
where α is a factor variable denoting the sector, β is a factor variable denoting the state, S is the size of the household, E is the monthly per capita expenditure of a household, P is the price and Q is demand of the item.
Then we fit the same model considering the randomly chosen price P * in place of actual price P i.e, logQ = α i + β j + δ 1 S + δ 2 logP * + δ 3 logE = α i + β j + δ 1 S + δ 3 logE + δ 4 logP + δ 5 log P * P (4) Here log P * P is the elasticity. To achieve our goal we have to check if the predicted distribution of the share depending upon the response variable in both the models are same or not. Another way is to check is whether γ 5 , i.e. the coefficient to the elasticity term, is zero or not.

Bias correction
Standard regression models assume that the predictor variables involved in the model are measured exactly, or observed without any error. In contrast, we have here a measurement error model ( [13], [12], [5]), i.e, we have independent variables measured with error, since we are randomly choosing one value from each fsu. To account for these errors, it is expected that a bias would be introduced in the regression coefficients.
A measurement error in a predictor variable is called "classical" if it is independent of the latent (unobserved) true variable; otherwise it is called "non-classical". Various methods are used to treat classical and non-classical measurement errors. In this article, we shall work with classical measurement errors.
We consider the following linear regression model: where instead of observing the true predictor X * , we observe X given by where V is a matrix of measurement errors. Here we assume the following: is a consistent estimator of β.
Proof. We at first consider the usual estimator of β: where the last step above use the facts that X * V P → 0, X * P → 0 and V P → 0 which hold because of our first three assumptions mentioned earlier. Hence,β is an inconsistent estimator of β. A consistent estimator based onβ may be obtained aŝ completing the proof.
In our case, measurement error is present only in the predictor indicating price. So, here each column of V (except the second column) is a zero column. Consequently, V V has a non-zero entry in second diagonal position and zero everywhere else. Now, Thisβ * is a consistent estimator of β, as mentioned earlier.

Results and Observations
Suppose F and G are the distribution functions of share based on the actual price and the randomly chosen price respectively. We wish to see if these two distributions are same, i.e we want to test using the usual Kolmogorov-Smirnof test.
As mentioned in Section 3.2, we are creating the randomized response variable by just picking one household randomly from every fsu. This algorithm may result in a biased acceptance or rejection of the test i.e, suppose for a fsu one of the extreme values happens to be chosen, then it will it will certainly affect the outcome of the test. So to avoid this bias, we have repeated this test 1000 times with each test considering newly chosen household sample with replacement so that all households are adequately represented.
where p i is the p-value of the test performed with the i th sample. Note that under H 0 , p i follows Uniform(0, 1) for i = 1, . . . , 1000. Hence, under H 0 , The 1000 samples are drawn from the full data with replacement and are therefore independent given the full data. Here by full data we mean the complete data available i.e, the data accumulated by NSS from all the possible households for each of the items. This implies that the corresponding p-values (p i ) are independent. Hence Z i 's are independent. Consequently, Our objective is to find a c ∈ N such that P H 0 (p (c) 0.05) 0.05. One also observes that Hence it is enough to find a c such that Towards this, we claim that if we find a c such that, P H 0 (Z > c|full data) 0.05 then the same value of c will satisfy (6). This is justified in the following chain of equations.
Hence the test given by (5) controls Type 1 error at 0.05 level.
We adopt the rejection criterion (5). The results are presented in Table 3. It is seen that for cereals, pulses, edible oils, and items coming from milk-and-milk-products the distribution In Table 3, LCB(δ 5 ) and UCB(δ 5 ) denote the 25'th and 975'th ordered values of δ 5 (given in (4)) obtained from 1000 repetitions, respectively. Then an empirical 95% confidence interval for δ 5 can be obtained as [LCB(δ 5 ), UCB(δ 5 )]. When the confidence interval for a particular item contains the value 0, one expects the corresponding null to be true. We observe from Table   3 that, these confidence intervals contain the value 0 for 12 items and among them, only for banana (item 220) the null hypothesis is rejected. We also note that this confidence interval procedure empirically tests the hypothesis whether the mean of the 1000 δ 5 values is zero for a particular item.
Similarly, an empirical 95% confidence interval for δ 4 (given in (4)) is given as [LCB(δ 4 ), where LCB(δ 4 ) and UCB(δ 4 ) denote the 25'th and 975'th ordered values of δ 4 obtained from 1000 repetitions. We observe that these confidence intervals contain the value of γ 2 (obtained from fitting the model (3)) for 12 items and among them, only for banana (item 220) the null hypothesis is rejected. In fact interestingly, it is the same set of 12 items mentioned above with respect to the discussion on δ 5 .

Item
No.  These results seem quite favourable for our thin sampling strategy. The departure of the precision of results from the usual situation turns out to be minimal and hence, given the substantial reduction in interview time (and hence cost), seems to lead to a more efficient sampling strategy. As this is not restricted to a particular geographical context, it would be interesting to see whether our strategy works also for other countries' consumer expenditure surveys. We hope that our results will encourage analogous studies with other consumer expenditure data sets.