{"push_id":1565100074,"size":56,"distinct_size":1,"ref":"refs/heads/dev","head":"02fdae5a4ada2f5104b80f927d880e5e041fd878","before":"eeb4d13ad2101a2bd8072948f49fc40b068de6fd","commits":[{"sha":"07622f1fa8711e339cb50bb01c5c44fe206ae57d","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"Merge pull request #1 from schasins/dev\n\nDev","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/07622f1fa8711e339cb50bb01c5c44fe206ae57d"},{"sha":"b4d6acc85995e6879f2bced4856992234518c9e2","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"Handling the fact that on Mac Chrome, many messages seem to fail if they're sent early in the life of the recipient tab.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/b4d6acc85995e6879f2bced4856992234518c9e2"},{"sha":"8e76ec77b0b2b01965b9e675d96d773728c05491","author":{"name":"S. Chasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@D-10-18-218-188.dhcp4.washington.edu"},"message":"A couple of important improvements to our message sending utility functions.  First, avoiding having messages sent to a specific frame also triggering the normal handler for the same message type on the target frame, if the message type is used for both frame-specific and normal messages.  Second, avoiding having message listeners clobbering each other if they're added through different means (as through key-indifferent and key-specific/listen-once listener introducers.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/8e76ec77b0b2b01965b9e675d96d773728c05491"},{"sha":"077982455ec4024be07252d3aabc78cdb43fec05","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"A couple of important improvements to our message sending utility functions.  First, avoiding having messages sent to a specific frame also triggering the normal handler for the same message type on the target frame, if the message type is used for both frame-specific and normal messages.  Second, avoiding having message listeners clobbering each other if they're added through different means (as through key-indifferent and key-specific/listen-once listener introducers.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/077982455ec4024be07252d3aabc78cdb43fec05"},{"sha":"01b57bc684a9ee9a55693662fbf4a9774ce25ab7","author":{"name":"S. Chasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@D-10-18-218-188.dhcp4.washington.edu"},"message":"Merge branch 'master' of https://github.com/schasins/visual-pbd-scraping","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/01b57bc684a9ee9a55693662fbf4a9774ce25ab7"},{"sha":"2fd7ea4b39cc2a5322107d1c59f060ecc6aa7f28","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"Can now save programs that use text relations (although note that the text relations are not currently saved to the server independently; they are included in the stringified representation of the program, but not kept in a structured way in the db).","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/2fd7ea4b39cc2a5322107d1c59f060ecc6aa7f28"},{"sha":"6be95af5a682886e2e1a98e6f7d1e5ce1a00b177","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"When user has a relation preview displayed, adding the option for the user to indicate in the table (using checkboxes) whether to scrape each column.  Thus user has a convenient way to add and remove scrape statements without having to redemonstrate, and having a pretty good idea of what will be included, based on seeing the relation preview.  This is particularlly convenient for cases when an attribute is missing from the first item in a relation, and thus user can't demonstrate on it, but still wants to be able to scrape it when it is present.  Since we still require the user's demonstration to be on the first row, previously no way to add attributes missing from the first row.  Now, user can make sure it's present in the relation, and can add it this way.  Doesn't help for attributes that should be scraped with node-addressing rather than relation-finding, but it's a start.  In the process of implementing this, ending up putting in some of the infrastructure that we'll need for more general program editing.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/6be95af5a682886e2e1a98e6f7d1e5ce1a00b177"},{"sha":"0f6a8e24db2d0eda38fcaa548209ad0192a0bfea","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"Can now load in programs from the server that include uses of text relations.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/0f6a8e24db2d0eda38fcaa548209ad0192a0bfea"},{"sha":"a6a547d124ba598ceafeef27a4faf2c88a108b19","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"Refactoring so that it's easy to turn off (or more generally, adjust) logging.  Still more to do, but it's a start.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/a6a547d124ba598ceafeef27a4faf2c88a108b19"},{"sha":"349f60ab75522e51293ebad85e79e0ba7cf409a5","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"Several important but largely unrelated improvements.  1) More logging reduction.  2) We now do some reasonable backoff for some cases where we need to repeatedly request a message response in order to ensure we receive one.  In particular, when we ask a content script for fresh relation items, we now use some backoff.  3) Now have a nice little UI element that tells us how many rows we've scraped so far, which is convenient in practice and also rather satisfying.  1) and 2) are focused in particular on helping us operate on large, complicated webpages, especially if we interact with a single webpage for a long time.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/349f60ab75522e51293ebad85e79e0ba7cf409a5"},{"sha":"6f6b1e824ab4d0db9baeacd481ea4455604d461b","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@cs.berkeley.edu"},"message":"Was sorting the list of potential relations the wrong way around!","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/6f6b1e824ab4d0db9baeacd481ea4455604d461b"},{"sha":"9a485f41afce2cb82a5b0e9b0d5b475047eef36d","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"Playing around with how we decide whether to continue to request new relation items.  So still reducing the number of messages we send for that task.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/9a485f41afce2cb82a5b0e9b0d5b475047eef36d"},{"sha":"67d9fb64f9a96993778bc439fe7f37fa177ab0f0","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"Big improvements to how we request new relation items.  Now rather than counting how many responses we get from a page in order to decide when to give up, which relies on a page to actually respond, we have a nice human-understandable time cutoff.  Also, now handling keeping track of which frames have already responded properly, so we don't get the idea we're overwriting things when we're not.  Also, now a better way of handling big backlogs of messages that the content script can get.  Rather than store up the backlog, then respond to all of them, just respond to the most recent one in the given frame.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/67d9fb64f9a96993778bc439fe7f37fa177ab0f0"},{"sha":"c2f519fa2cef1b21381ea7885433d4e6b6fcb06d","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"Better logging.  Now can have named segments of logs, only show the labeled messages.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/c2f519fa2cef1b21381ea7885433d4e6b6fcb06d"},{"sha":"e522d4f954acf06d55e16aa66a0da818390a8b23","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"Trying to deal with cases where we accidentally get two batches of data at once.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/e522d4f954acf06d55e16aa66a0da818390a8b23"},{"sha":"b2947af2c6fa3b6cba17ad2f168594a76e171d06","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"In cases where running the next interaction doesn't actually load fresh relation items -- sometimes even if it's because the server is having trouble -- we try running the next interaction again a couple times.  This handles at least some observed cases.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/b2947af2c6fa3b6cba17ad2f168594a76e171d06"},{"sha":"b3717bd6da43296c0abdaa761c466c7ed4443a25","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"A couple fixes.  When we have to repeat the next interaction again, we need to make sure the state in prinfo tells us to actually do that.  Also, if all frames indicate that the list is just done, don't waste time by waiting, just move on.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/b3717bd6da43296c0abdaa761c466c7ed4443a25"},{"sha":"e8d0291e0f6c9fec7dcb08cb3d72a76cbd7b9982","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"Just a smaller time limit for waiting for relation items from a page.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/e8d0291e0f6c9fec7dcb08cb3d72a76cbd7b9982"},{"sha":"9127b119d0638dc4ba8dc2630450e6328c299b2a","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"A number of changes.  (1) No longer removing events based on believing they're the keypresses for turning scraping on and off.  The current approach to that appears to put some pages in a weird state, preventing us from doing important interactions like clicking to the next page.  (2) Improving how we decide on a preferred xpath, making sure we only do it if we actually managed to find an event that aligns.  (3) Clearing out the data associated with a scrape statement when there's no data collected for the most recent run.  Very important.  (4) We cache the results of node addressing on a given page, which is good for optimizing, but until now we hadn't been ensuring that the result nodes were still present on the current page when we retrieve them.  This is necessary for example when the 'next' items of a relation are loaded into the same page, but an old relation item is stored in the cache.  (5) Getting some speed back with the scraping stuff by reducing wait times associated with events that are associated with a scrape statement.","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/9127b119d0638dc4ba8dc2630450e6328c299b2a"},{"sha":"7d063daf8ddee56ba2a082cf4e44eb324c1a1e63","author":{"name":"schasins","email":"47115583b602ed6aa85b06c06f43b67c372588ed@gmail.com"},"message":"Merge branch 'master' of https://github.com/schasins/visual-pbd-scraping","distinct":false,"url":"https://api.github.com/repos/schasins/visual-pbd-scraping/commits/7d063daf8ddee56ba2a082cf4e44eb324c1a1e63"}]}
{"actor":{"display_login":"schasins"}}