{"push_id":2201405286,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"6ba6a33b80f848a9fe426161f802e15dad8e7c94","before":"a02efb0b242dac0257dee4c6275ecddce32e24f7","commits":[{"sha":"6ba6a33b80f848a9fe426161f802e15dad8e7c94","author":{"name":"JinmingLi","email":"d58811f2270dd0b33370b6c28cefdd9604a9beb9@users.noreply.github.com"},"message":"Data Error Checking in Python\n\nIn Harvard Yenching Library, we have roughly 3,500 Chinese county books which record all the agriculture, demography, education data and there are over 1,000 variables in total that are related to the historical changes of these 3,500 counties in China from 1949 to 1990. Everyday, our project would receive 50 excel files that contain all the agriculture, demography and education data from 50 county books. As the chief data analyst, my job is to check data errors and then correct them before inputting the data into database because during the digitization process many errors would pop up that result in unusable data. \r\n\r\nIn this Python code, I encapsulate the entire code into a module, so every time we want to use a specific function, we can simply call it. In the process of data error checking, (1) I first input the excel file, (2) I check the data type error by calling the check_type_print function. This function would output the error such as \"6..50\"  so I can correct the data error, (3) I  check missing unit error by calling the check_missing_unit. This function can help us track down which variables have no units, (4) I plot all the data with year in x-axis and variables in y-axis to detect outliers by calling plot_all function. If I find outlier, I show all the data in the county that has outliers and this helps me see which piece of data is abnormal, (5) in the end, I correct the outliers and output the excel file.","distinct":true,"url":"https://api.github.com/repos/JinmingLi/China-Gazetteer-Project-at-Harvard-University/commits/6ba6a33b80f848a9fe426161f802e15dad8e7c94"}]}
{"actor":{"display_login":"JinmingLi"}}