{"push_id":1781372605,"size":4,"distinct_size":0,"ref":"refs/heads/shs-ng/M7","head":"c60b9ccfb79b7a4529ef9dad1c7c779435ef1541","before":"d7f4c2600b76b02449fe67d80145aca8bc3d8c9b","commits":[{"sha":"4c0bea98b27874f258fc1175cb01ed77c005833f","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M4.6: Remove JobProgressListener.\n\nThe only remaining use of this class was the SparkStatusTracker, which\nwas modified to use the new state store. The test code to wait for\nexecutors was moved to TestUtils and now uses the SparkStatusTracker API.\n\nIndirectly, ConsoleProgressBar also uses this data. Because it has\nsome lower latency requirements, a shortcut to efficiently get the\nactive stages from the active listener was added to the AppStateStore.\n\nAs part of this change I also modified the streaming UI to read the needed\ndata from the store, which was missed in the previous patch that made\nJobProgressListener redundant.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/4c0bea98b27874f258fc1175cb01ed77c005833f"},{"sha":"8aeee5e923e024cf4221b55f4efb49062e81f42b","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M5: Store SQL UI data in the app state store.\n\nThis change replaces the SQLListener with a new implementation that\nsaves the data to the same store used by the SparkContext's state\nstore. For that, the types used by the old SQLListener had to be\nupdated a bit so that they're more serialization-friendly.\n\nThe interface for getting data from the store was abstracted into\na new class, SQLAppStateStore (following the convention used in\ncore). The session's shared state now contains a pointer to this\nstore instead of the listener itself.\n\nAnother change is the way that the SQL UI hooks up into the core\nUI or the SHS. The old \"SparkHistoryListenerFactory\" was replaced\nwith a new \"AppStatePlugin\" that more explicitly differentiates\nbetween the two use cases: parsing event logs and showing the UI.\nThe SHS hooks into that new API to talk to the SQL module; the live\nUI still needs to be installed explicitly, but the SQL code now\nuses the same code to do it (in the SQLAppStatePlugin implementation).\n\nThe metrics gathering code was re-worked a bit so that the types\nused are less memory hungry and more serialization-friendly. This\nshould help with the caching layer used during writes; while the\ndata is cached in memory it should use less of it, and when it needs\nto be loaded from disk, it shouldn't be terribly expensive.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/8aeee5e923e024cf4221b55f4efb49062e81f42b"},{"sha":"c1f4e8155402bbc166e1432ed797029862bdfb71","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M6: Add cleaning of old elements from the store.\n\nThis change restores the functionality that keeps a limited number of\ndifferent types (jobs, stages, etc) depending on configuration, to avoid\nthe store growing indefinitely over time.\n\nThe feature is implemented by creating a new type (ElementTrackingStore)\nthat wraps a KVStore and allows triggers to be set up for when elements\nof a certain type meet a certain threshold. Triggers don't need to\nnecessarily only delete elements, but the current API is set up in a way\nthat makes that use case easier.\n\nThe new store also has a trigger for the \"close\" call, which makes it\neasier for listeners to register code that should be run before the\nstore is closed.\n\nThe old configurations for cleaning up the stored elements from the core\nand SQL UIs are now active again, and the old unit tests are re-enabled.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/c1f4e8155402bbc166e1432ed797029862bdfb71"},{"sha":"c60b9ccfb79b7a4529ef9dad1c7c779435ef1541","author":{"name":"Marcelo Vanzin","email":"31a02c4d82e3479e776883a3423bc52579c6a1cf@cloudera.com"},"message":"SHS-NG M7: Limit disk usage of the history server.\n\nThis change adds a new configuration option and support code that limits\nhow much disk space the SHS will use. The default value is pretty generous\nso that applications will, hopefully, only rarely need to be replayed.\n\nThis works by keeping track of how much data each application is using;\nbecause it's not possible to know, before replaying, how much space will\nbe needed, it's possible that usage will exceed the configured limit\ntemporarily. The code uses the concept of a \"lease\" to try to limit how\nmuch the SHS will exceed the limit.\n\nActive UIs are also tracked, so they're never deleted. This works in\ntandem with the existing option of how many active UIs are loaded; because\nunused UIs will be unloaded, their disk stores will also become candidates\nfor deletion. If the data is not deleted, though, re-loading the UI is\npretty quick.","distinct":false,"url":"https://api.github.com/repos/vanzin/spark/commits/c60b9ccfb79b7a4529ef9dad1c7c779435ef1541"}]}
{"actor":{"display_login":"vanzin"}}