{"action":"created","comment":{"url":"https://api.github.com/repos/sagemathinc/smc/pulls/comments/102048089","pull_request_review_id":22792199,"id":102048089,"diff_hunk":"@@ -0,0 +1,961 @@\n+SMC's README_ gives a very high level overview of its architecture.  However,\n+it doesn't really explain much about the implementation details--what\n+technologies are used, how, and why.  This is in part because a lot of it has\n+been developed by the seat of their pants (understandably; this is not meant as\n+a criticism) and a lot of the details have been in flux, and not worth\n+documenting.  After all, the details should only really be of interest of the\n+SMC developers themsevles.  The details even of setting up a personal SMC\n+server have mostly been squirreled away into install scripts and Dockerfiles\n+such that it doesn't require one to have a deep understanding of how it all\n+works.\n+\n+\n+Technologies used in SMC\n+========================\n+\n+This is a partial list of some of the major components that are used to make\n+SMC--especially the parts that make up SMC's own source code (as opposed to\n+external components such as the database and HAproxy, though those will be\n+mentioned as well).\n+\n+This will also attempt to give a brief overview of what those technologies\n+are used for in general, and how they are used by SMC.  The finer details of\n+the architecture will be discussed further in later sections.\n+\n+You can either read all these words first, or skip to the diagram where I've\n+tried to show how everything fits together.  But you won't understand the\n+diagram unless you know what most of these terms are.\n+\n+CoffeeScript\n+------------\n+\n+CoffeeScript is one of many languages that transcompile to JavaScript (a.k.a.\n+compile-to-JavaScript), and one of the most popular.  Wikipedia gives an easy\n+to digest `overview <https://en.wikipedia.org/wiki/CoffeeScript>`_.  It's\n+basically just syntactic sugar for JavaScript with inspiration from other\n+languages, like list comprehensions (my personal favorite).  It's somewhat more\n+Python-like (no braces or semicolons) with some syntax borrowed from functional\n+languages like Haskell; some of it is also resembles YAML.  Code written in\n+CoffeeScript can be translated directly into equivalent, and not too hard to\n+read JavaScript.  Its use is not essential to SMC's design, but William prefers\n+it over plain JS (and is in good company) so it's used virtually everywhere\n+(along with a variant known as coffee-react or CJSX to be explained later).\n+\n+Because of a number of factors, such as the prevalence of Node.js (see\n+below), and also the differences across web browsers in what variants of\n+JavaScript they support, as well as browser-specific quirks in their\n+JavaScript implementations, it has become quite common in the web\n+development community to use projects like `Babel`_ to\n+write JavaScript in some meta-language, and then compile it to JavaScript\n+that will actually work on different browsers (including\n+\"polyfills\"--essentially backports--of features that are not available yet\n+on all browser).\n+\n+In the bad old days we used to clutter our JavaScript with browser and/or\n+feature checks. But the movement in the web community has been toward writing\n+JavaScript to some standard, and then using build tools (such as `webpack`_\n+discussed later) to run Babel and other tools to prepare the \"real\" JavaScript\n+that will run in specific browsers.  It's basically just a repeat of high level\n+languages like C superseding the need to write assembly language for different\n+hardware platforms.  But in this case the \"assembly languages\" are\n+browser-specific JavaScript implementations.  This has led to an explosion in\n+\"high-level\" languages over native JavaScript, as well as associated compilers\n+and build tools.\n+\n+Node.js\n+-------\n+\n+Although SMC's source tree contains a smörgåsbord of (mostly small) Python\n+scripts for various management tasks, most / all of SMC's backend is built\n+not with Python, but with `Node.js`_.\n+\n+Most software developers are, at this point, at least passingly familiar\n+Node.js (often referred to as just \"Node\"). But in short, it is a\n+stand-alone JavaScript runtime that can run on a server.  It includes an\n+HTTP server and various I/O primitives that have been designed from the\n+groud up for even-driven asynchronous I/O.  Because it uses JavaScript it\n+has been meteorically popular in the web development world because it allows\n+developers to write both their front-end and back-end code in JavaScript,\n+and even share library code between the front- and back-ends.  This also\n+made it easier for developers who otherwise specialize in web front-ends to\n+cross into \"full-stack\" development.\n+\n+None of the technologies SMC uses to drive its front-end necessarily require\n+Node.js on the back-end--naturally; any server providing the same interfaces\n+will work.  But using Node.js makes sense due in part to the ecosystem\n+around it--server and client technologies that were designed to work\n+together--and because being able to write back- and front-end code in the\n+same language reduces cognitive overhead as well.\n+\n+Express\n+-------\n+\n+While Node.js has a decent standard library--mostly focused, as is its\n+purpose, on network I/O--it is mostly relatively low-level and doesn't make\n+too many impositions on how to build an application on top of it.  So for\n+example, while it includes a web server, it does not include a web\n+*framework* for handling the kinds of things web frameworks typically do\n+like URL routing and request handling.\n+\n+`Express`_ is one such framework, built on top of Node.js.  It is a fairly\n+minimalistic and flexible framework, similar in concept to the popular `Flask`_\n+framework for Python.\n+\n+SMC uses Express as the basic HTTP request handling framework for the\n+majority of its web server components (particularly the Hub).  That said,\n+much of the communication that goes on between the SMC client side and the\n+server happens (where available, which is most places these days) over\n+WebSockets. But more on that later.\n+\n+The main entry point to Express is an 'app'--an instance of its `Application\n+<http://expressjs.com/en/4x/api.html#app>`_ class.  The app serves as a request\n+handler for Node's `HTTP server\n+<https://nodejs.org/api/http.html#http_http_createserver_requestlistener>`_.\n+Every time the server begins handling a request, it fires off a ``'request'``\n+event for which the app is a listener, and the app handles the request and\n+returns a response to the client.  The Express documentation on `routing\n+<http://expressjs.com/en/guide/routing.html>`_ should give a basic idea of how\n+it works.  If you've used any web framework before it should be immediately\n+familiar.  If web development is new to you it might take a little more\n+thinking, but the idea is basically simple: The app receives a request, looks\n+at the URL and/or HTTP method of the request, and maps that information to a\n+specific function registered to handle that request.  The handler function is\n+passed a request (represending the incoming request), and a response object\n+that can be used for building up and sending an HTTP response to the client.\n+Typical handlers will be the end-point for a request, and so will end with a\n+call like ``res.send(...)`` which will send the response to the client and end\n+the connection.  However, there may also be middleware handlers that do\n+something with the request (such as check authentication) before handing it off\n+to other handlers.\n+\n+Primus\n+------\n+\n+Another important feature of modern web applications is real-time,\n+bidirectional communication between the client (or clients) and the server.\n+Because of the importance of this, and the many different ways to implement\n+it all with different advantages and disadvantages, there has been an\n+explosion of different real-time communication frameworks.  As with other\n+aspects of web development, the techniques to implment this functionality\n+(\"ajax\", long-polling (\"comet\"), WebSockets, etc.) have changed over the\n+years and have varying degrees of support on different clients.  There are\n+also higher-level connection features one needs such as multiplexing,\n+heartbeats, automatic disconnects/reconnects, etc. and these are the kinds\n+of features provided by these real-time frameworks.\n+\n+As is typical, the plethora of frameworks has led to a meta-framework, and\n+that's where `Primus`_ comes in.  It abstracts out an API common to many / most\n+real-time frameworks, and allows swapping out the underlying implementation.\n+It's fair to say that different real-time frameworks have different advantages\n+and disadvantages, and some are better than others for specific applications\n+depending on their implementation details.  Primus allows you to write your\n+application once in its common API and then swap out the lower-level core\n+implementation.\n+\n+Engine.IO\n+---------\n+\n+`Engine.IO`_ is one such real-time communication framework, designed to be\n+event-driven and asynchronous for integration into and compatability with\n+Node.js.  In fact, it attaches itself to Node.js's HTTP server, which it uses\n+initially to negotiate connections with the client via traditional HTTP methods\n+like long-polling.  It then works with the client (with help of a client-side\n+library) to discover support for other transport methods (like WebSockets) and\n+transparently switches to those channels as they become available.  The overall\n+design and advantage of using Engine.IO is best explained by `its documentation\n+<https://github.com/socketio/engine.io#goals>`_.\n+\n+Engine.IO is actually the core to another higher-level real-time framework you\n+will see reference to called `Socket.IO`_.  But SMC does not use Socket.IO\n+directly, opting instead to use Primus as its \"high-level\" real-time framework,\n+with Engine.IO being one of Primus's supported underlying transport layers\n+(whereas Socket.IO is designed to work only with Engine.IO).\n+\n+pug\n+---\n+\n+`pug`_ (formerly named \"Jade\" but recently renamed for trademark reasons) is a\n+template engine for templated HTML in particular, written for Node.js.\n+Although it has its own particular syntax, the concept should be familiar to\n+anyone who's written a web template before.  Pug/Jade is the *default* template\n+engine used by Express, though one can easily substitute it for any other\n+template engine (after all, at the end of the day all a template engine is\n+doing is returning an HTML string to be sent in the HTTP response).  If you've\n+used Flask, this is just like how Flask uses Jinja2 by default, but by no means\n+enforces its use.\n+\n+We won't go much more into pug as SMC barely uses it.  In fact there is\n+currently only one pug template in SMC (``webapp-lib/index.jade``) for the\n+main index page to SMC.  Mostly all this page does is provide some metadata\n+and favicons, and display the big \"Loading\" banner you see when you first\n+load SMC.  All the rest of the front-end is loaded in via React which we'll\n+discuss next.\n+\n+React\n+-----\n+\n+`React`_, also often referred to as React.js, ReactJS, etc. is a powerful\n+toolkit for web UIs, developed by Facebook.  Although one still uses HTML+CSS\n+to specifiy the look and feel of a UI component, React allows one to manipulate\n+components of a UI in an object-oriented manner, not unlikely traditional\n+desktop GUI toolkits.\n+\n+The example on their front page gives a great introductory example of a little\n+\"TODO list\" widget.  It's implemented as a class, which has a ``render()``\n+method used to display the widget in its initial state, a few internal\n+attributes for managing its state (such as the list items), and some methods\n+for handling different events on the widget.  There's also a very nice\n+`tutorial <https://facebook.github.io/react/tutorial/tutorial.html>`_ for\n+building a tic-tac-toe game.  If you can grok that then you'll have the hang of\n+React.\n+\n+If you've ever used a GUI toolkit like wx or Swing it shouldn't be too hard\n+to pick up on what it's doing.\n+\n+Using React is quite a bit different from the old-fashioned way of making\n+reactive web UIs with JavaScript.  What I'm calling the \"old-fashioned\" way is\n+a couple things.  For one, the server might render serve up a bunch of HTML\n+containing all the elements in your page, many of which might by \"hidden\" using\n+CSS, and the JavaScript would hide and unhide elements on the page.  Or the\n+JavaScript might generate some elements and insert them directly into the DOM\n+and remove them as needed, either using the DOM API directly or, somewhat\n+later, tools like `jQuery`_ (note: jQuery still has a role to play even in\n+conjunction with React though).\n+\n+In other words, gone are the days of servers rendering and returning HTML to\n+the browser.  All the rendering is pushed entirely to the client, with the\n+client-server communication focused on as light-weight as possible message\n+passing.  This potentially frees up enormous resources for the server, while\n+pushing much more work to the client (which is why so many of your browser\n+tabs are using over 100 MB of memory, among other reasons).\n+\n+The way React works, in short, is this: It maintains its own \"virtual DOM\"\n+separate from the actual DOM of the browser document, with the same API as the\n+real DOM.  Whenever you show, hide, or otherwise update the contents of a UI\n+element in the application, it uses a copy of its virtual DOM to figure out\n+exactly what needs to change in order for that to happen, and generates (and\n+subsequently applies) a stream of operations to perform on the actual DOM in\n+order to enact those changes.  The result is that there's nothing in the real\n+DOM except for what's actually displayed on the page, which is convenient for\n+debugging and inspection via your browser's development tools.  There's a\n+simple `demonstration\n+<https://facebook.github.io/react/docs/rendering-elements.html#react-only-updates-whats-necessary>`_\n+of this aspect in the docs.\n+\n+Another nice aspect of React is its JSX domain-specific language which I'll\n+discuss more next.\n+\n+As mentioned in the section on pug, essentially all of SMC's web frontend is\n+built using React.  Almost no HTML is ever sent from the server.  Instead\n+the frontend is built up by React.  When user interactions with the UI need\n+to be persisted, those are sent as event messages (typically over\n+WebSockets) to the server, which may in turn respond with events that result\n+in updating the UI in some appropriate way (the event messages are usually a\n+JSON object of some kind).  This is still an over-simplification (see for\n+example the section on Redux later), but that's the basic idea.\n+\n+JSX\n+^^^\n+\n+`JSX`_ is a language that comes as part of React.  It's a superset of\n+JavaScript that allows embedding templated HTML.  In some ways this resembles\n+the bad-old-days of mixing code with HTML à la PHP.  But it does have some\n+advantages too, described in the linked docs.  It's actually a very convenient\n+way to use markup to describe how a UI element should be rendered.  It's also a\n+convenient way to nest UI components.  For example, one might define some UI\n+component as a class that extends ``React.Component``::\n+\n+    class MyWidget extends React.Component { ... }\n+\n+This now lets you use ``MyWidget`` in JSX as though it were any other HTML\n+element like::\n+\n+    <div id=\"widgets\">\n+      <MyWidget name=\"foo\" />\n+      <MyWidget name=\"bar\" />\n+      <MyWidget name=\"baz\" />\n+    </div>\n+\n+and so on.  [Note: Simple React components that are stateless can also be\n+implemented as functions, which server as their ``render()`` method).]\n+\n+React can be used without JSX, but it saves a lot of verbosity and is\n+probably a bit clearer, especially to anyone with HTML template experience.\n+\n+CJSX\n+^^^^\n+\n+If JSX is the preferred way to write React components, this presents a\n+challenge for integrating JSX with codebases that otherwise use\n+CoffeeScript.  One could write everything in CoffeeScript *except* for the\n+code for React components (which would have to go in separate JSX files),\n+but that introduces another difficult cognitive overhead.\n+\n+To solve that, the `CJSX`_ language is just a simple superset of CoffeeScript\n+to support JSX-like syntax.  In other words, CJSX is to CoffeeScript as JSX is\n+to vanilla JavaScript.\n+\n+So this is what all the ``.cjsx`` sources (something that might be new to\n+most readers) are in SMC.  If you see a file in SMC with the ``.cjsx``\n+extension you can bet there's probably a React component defined in there\n+somewhere.\n+\n+.. note:\n+    In the process of researching this we learned that the original\n+    developer of CJSX has abandoned the project and there isn't really\n+    anything to take its place yet.  William `insists\n+    <https://github.com/sagemathinc/smc/issues/1545>`_ that SMC will\n+    continue to use it, and with good reason!, but it leaves me not without\n+    doubts...\n+\n+Redux\n+-----\n+\n+It's a little tricky to explain exactly what Redux 'is' without specific\n+examples.  According to its `docs <http://redux.js.org/>`_:\n+\n+    Redux is a predictable state container for JavaScript apps.\n+\n+It's really little more than a simple protocol for application state\n+updates by way of immutable state containers and pure functions that return\n+an updated state based on some action performed on it (where the action\n+can be any abstract operation that results in an updated application\n+state). These functions are called reducers.\n+\n+There's very little else in Redux--it's mostly convenience functions for\n+managing a state object, and combining reducers to produce new states from\n+state changing actions.\n+\n+The purpose, all in all, is to provide a sane, predictable, reproducible\n+way to manage and track (using middleware that logs actions) the live\n+state of a complex application.  We'll come back to this later with some\n+specific examples.  SMC wraps most of its use of Redux into its own\n+abstractions that are implemented in ``smc-webapp/smc-react.coffee``.\n+\n+React-Redux\n+^^^^^^^^^^^\n+\n+SMC's ``smc-react.coffee`` modules also makes use of the `React Redux`_\n+JavaScript module to tie Redux state objects to React containers (i.e. update\n+displays when the state changes--abstracting the state itself from any given\n+view of the state).  This is just a package for making it convenient to\n+implement model / view separation in React components.  The web developers\n+describe this as \"container components\" but really they're just reinventing MVC\n+abstraction.  The idea is to design React components that are stateless and\n+just display a \"snapshot\" of some data that might be in the state, and then\n+wrap the stateless views in \"container components\" that handle updating the\n+view upon state changes.\n+\n+React Redux makes it easy to auto-generate these \"container components\" that\n+connecting a React container to a Redux state and its reducers, to re-render\n+the underlying view every time the state changes.  This includes defining\n+a function called ``mapStateToProps`` which, given any application state,\n+specifies which \"props\" (variable data) of the view are associated with the\n+given state.  So when the application state changes, it can check which\n+\"props\" in the view have changed, and determine whether or not the view\n+needs to be re-rendered.\n+\n+If this is unclear, probably the best way to understand quickly is to read\n+the `example in the Redux docs\n+<http://redux.js.org/docs/basics/UsageWithReact.html>`_.  We will go more\n+into exactly how SMC uses React-Redux later.\n+\n+webpack\n+-------\n+\n+Preparing a large, multiple-file web application consisting of specialized\n+JavaScript dialects like CoffeeScript and JSX with many interdependencies,\n+as well as external dependencies, and getting everything to load in the\n+correct order is tricky.\n+\n+For one, the modern ECMAScript supports features not supported by the\n+JavaScript on browsers, such as the ``import`` statement for loading\n+variables, classes, and functions into other files (without polluting the\n+global namespace, as was necessary to share between JavaScript files in the\n+bad old days).  Unfortunately, most (in fact no) browsers support this\n+feature.  One also needs ways to find static resources relative to\n+JavaScript modules, transform the development dialect into JavaScript that\n+can run in the browser, minify and uglify the code, and put it all together\n+in a big bundle that loads everything in the correct order.\n+\n+`webpack`_ is one of a number of popular build tools that serve this purpose.\n+The entry-point to a webpack project is a file called ``webpack.config.js`` (or\n+in SMC's case ``webpack.config.coffee`` since it uses CoffeeScript just about\n+everywhere).  You can think of ``webpack.config`` a little bit like the\n+``setup.py`` in a Python project, but don't take the analogy too far--it\n+doesn't work the same way (a larger part of this purpose is also served by the\n+|package.json|_ file that defines npm packages).\n+\n+The ``webpack`` CLI then reads in this ``webpack.config`` and outputs a single\n+file containing all your Javascript.  This is of course the most basic\n+usage--SMC currently actually generates three JS files (from three separate\n+\"entry points\" to the dependency graph webpack generates).  It also generates\n+the ``index.html`` file that is served at the root of the website (from the\n+aforementioned ``index.jade`` template) into which webpack inserts ``<script>``\n+tags that load its generated JS files.  It also does some other tricks, such as\n+appending a hash to the JS filenames so that they can replace cached versions\n+whenever the source changes.\n+\n+In practice it's less convenient to run ``webpack`` over and over\n+again--instead one can run ``webpack --watch`` which watches all files for\n+changes and rebuilds continuously.\n+\n+Conclusion\n+----------\n+\n+As previously stated, this is only a partial list of the tools that go into\n+building SMC--particularly the core backend and client code.  It doesn't\n+even discuss the many dependencies that go into its various features, such\n+as Jupyter, browser-based editors and terminals, chat clients, etc.  Later I\n+may include an update to list more of those.\n+\n+\n+How it all works\n+================\n+\n+High level view\n+---------------\n+\n+The high level architecture diagram from the Readme in SMC's source is\n+accurate::\n+\n+       Client    Client    Client   Client  ...\n+         /|\\\n+          |\n+       https://cloud.sagemath.com (primus)\n+          |\n+          |\n+         \\|/\n+     HAproxy (load balancing...)HAproxy                  Admin     (monitor and control system)\n+     /|\\       /|\\      /|\\      /|\\\n+      |         |        |        |\n+      |http1.1  |        |        |\n+      |         |        |        |\n+     \\|/       \\|/      \\|/      \\|/\n+     Hub<----> Hub<---->Hub<---> Hub  <-----------> RethinkDB <--> RethinkDB  <--> RethinkDB ...\n+               /|\\      /|\\      /|\\\n+                |        |        |\n+       ---------|        |        | (tcp)\n+       |                 |        |\n+       |                 |        |\n+      \\|/               \\|/      \\|/\n+     Compute<-------->Compute<-->Compute <--- rsync replication  to Storage Server, which has BTRFS snapshots\n+\n+(with the exception that William is in the process of replacing RethinkDB\n+with PostgreSQL).\n+\n+It may be helpful to explain some of the entities in this diagram a bit\n+more.\n+\n+Client\n+^^^^^^\n+\n+This is the SMC client interface, built primarily with React and bundled\n+together webpack as described previously.  When a user goes to the root of SMC\n+in their web browser, the HAproxy configuration serves it the ``index.html``\n+from its default backend, which happens to be a simple nginx server dedicated\n+to static files.  It also gets images, and the client JavaScript from the\n+static server.  Once the JavaScript takes over everything else happens in the\n+browser including setting up the appropriate view for the client (whether or\n+not they're logged in, etc.) and communicating with the hub using\n+Primus/Engine.IO (through the HAproxy--more on that next).  The majority of the\n+client is implemented in the code in ``smc-webapp`` and ``webapp-lib``, with\n+some bits from ``smc-util``.\n+\n+HAproxy\n+^^^^^^^\n+\n+HAproxy serves as the front line to all connections from clients to SMC.  It\n+routes all connections to different backends depending, primarily, on the\n+URL (and port).  The main frontend is of course HTTPS over port 443.  By\n+default requests are sent to the static file server (nginx) as mentioned\n+above.  Most other requests are sent to the 'hub' backend, which may be\n+running any number of the hub servers, one of which is selected using the\n+currently configured load balancing scheme (it also uses a session cookie to\n+keep individual clients connected to the same hub instance).\n+\n+Hub\n+^^^\n+\n+The \"Hub\" is the primary server backend for SMC, built on Node.js as described\n+previously.  It consists of an HTTP server with Primus + Engine.IO attached to\n+handle real-time bidirectional client/hub communication.  Most communication\n+between the Client and the backend happens through the Hub, whose HTTP server\n+uses Express to route requests to different services (account management,\n+project management, payment, etc.).  Each Hub also sets up a\n+``ComputeServerClient`` which gives it access to all the running compute\n+servers (discussed next).  The names and URLs of all the available compute\n+servers live in a system table in the database.\n+\n+It also uses `node-http-proxy`_ to create an HTTP proxy server associated\n+with each Hub (on port number one higher than the Hub's HTTP port).  If we\n+understand correctly, the proxy handles all requests that are to be\n+forwarded to individual compute nodes (such as requesting files, or\n+resources on web servers belonging to a specific project).  HAproxy doesn't\n+know anything about the compute nodes themseves--it just sees URLs that look\n+like they belong to a project (they begin with a project UUID) and forwards\n+those requests to the Hub's proxy, which in turn checks that the requester\n+is authenticated and has permissions to access that project's resources.\n+The proxy then forwards the request to the appropriate port on that\n+project's compute node::\n+\n+    Client <--> HAProxy <--> Hub Proxy <--> Compute\n+\n+Compute\n+^^^^^^^\n+\n+Compute servers are where the real work gets done in SMC projects.  Every\n+project is associated with a specific compute server where all their data\n+is stored (by way of storage servers mounted on the compute node) and where\n+all process and computation tasks done by the project are performed.  This\n+includes running Sage.  The Compute servers are Linux VMs with varying\n+degrees of hardware capacity, depending on how much you're willing to pay.\n+In most cases the servers are shared between projects (you don't have admin\n+on the servers) though in principle one could pay for one's own compute\n+server as well.\n+\n+Otherwise, one can do quite a bit of different things on their compute node,\n+including log in to the shell directly (you log in as a user named after\n+your project's UUID).  This can be done either through the web terminal in\n+SMC, or one can SSH in directly.\n+\n+Each compute node also runs a simple socket server\n+``smc-hub/compute-server.coffee`` that is used by the Hub to communicate with\n+the compute node (using simple JSON messages). For example, one can make status\n+inquiries on the node, or send commands to run a command in a project.\n+\n+Additionally, each *project* runs a couple per-project daemons when the\n+project is created and running.  These include the the console server\n+(``smc-project/console_server.coffee``) which provides the backend for the\n+web terminal, and the \"local hub\" (``smc-project/local_hub.coffee``).  We\n+haven't yet detailed everything that the \"local hub\" does, but whereas the\n+\"compute-server\" manages the entire compute node, the \"local hub\" runs\n+per-project (as that project's user) and helps coordinate connections\n+between software running in the project and the \"global hub\" (i.e. the Hub,\n+through which the client is communicating).\n+\n+As we understand it, whereas the compute server is used to issue commands to\n+the compute node on behalf of a project, such as starting the project's\n+local hub, the local hub then takes care of the rest.  This all makes sense,\n+but gets rather complicated as starting the actual local hub is buried under\n+a pile of Python scripts.  Some of this may still be legacy from earlier\n+versions of SMC that were written primarily in Python, and is need of\n+cleaning up.","path":"src/doc/design_overview/overview.rst","position":543,"original_position":543,"commit_id":"db388b79d91f17640fa9c4b0ccacc5ba538ed52f","original_commit_id":"db388b79d91f17640fa9c4b0ccacc5ba538ed52f","user":{"login":"embray","id":676149,"avatar_url":"https://avatars.githubusercontent.com/u/676149?v=3","gravatar_id":"","url":"https://api.github.com/users/embray","html_url":"https://github.com/embray","followers_url":"https://api.github.com/users/embray/followers","following_url":"https://api.github.com/users/embray/following{/other_user}","gists_url":"https://api.github.com/users/embray/gists{/gist_id}","starred_url":"https://api.github.com/users/embray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/embray/subscriptions","organizations_url":"https://api.github.com/users/embray/orgs","repos_url":"https://api.github.com/users/embray/repos","events_url":"https://api.github.com/users/embray/events{/privacy}","received_events_url":"https://api.github.com/users/embray/received_events","type":"User","site_admin":false},"body":"Am I correct though in saying that much of the complication stems from historical rewrites, and not so much out of necessity?  I'm not sure what you mean is being rewritten here either. I'd be interested to learn how the kubernetes stuff works (I have gone through their tutorial but have never actually used it).","created_at":"2017-02-20T16:01:04Z","updated_at":"2017-02-20T16:01:05Z","html_url":"https://github.com/sagemathinc/smc/pull/1696#discussion_r102048089","pull_request_url":"https://api.github.com/repos/sagemathinc/smc/pulls/1696","_links":{"self":{"href":"https://api.github.com/repos/sagemathinc/smc/pulls/comments/102048089"},"html":{"href":"https://github.com/sagemathinc/smc/pull/1696#discussion_r102048089"},"pull_request":{"href":"https://api.github.com/repos/sagemathinc/smc/pulls/1696"}}},"pull_request":{"url":"https://api.github.com/repos/sagemathinc/smc/pulls/1696","id":106785789,"html_url":"https://github.com/sagemathinc/smc/pull/1696","diff_url":"https://github.com/sagemathinc/smc/pull/1696.diff","patch_url":"https://github.com/sagemathinc/smc/pull/1696.patch","issue_url":"https://api.github.com/repos/sagemathinc/smc/issues/1696","number":1696,"state":"open","locked":false,"title":"Draft of partial notes on SMC design","user":{"login":"embray","id":676149,"avatar_url":"https://avatars.githubusercontent.com/u/676149?v=3","gravatar_id":"","url":"https://api.github.com/users/embray","html_url":"https://github.com/embray","followers_url":"https://api.github.com/users/embray/followers","following_url":"https://api.github.com/users/embray/following{/other_user}","gists_url":"https://api.github.com/users/embray/gists{/gist_id}","starred_url":"https://api.github.com/users/embray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/embray/subscriptions","organizations_url":"https://api.github.com/users/embray/orgs","repos_url":"https://api.github.com/users/embray/repos","events_url":"https://api.github.com/users/embray/events{/privacy}","received_events_url":"https://api.github.com/users/embray/received_events","type":"User","site_admin":false},"body":"These notes are based on my personal notes taken while exploring the SMC source code.  As is these notes are incomplete--I still have more notes on the server side to organize.\r\n\r\nThe separate sections on SMC's dependencies and the source code walk through might make more sense, ultimately, to break up into separate documents.  I just wrote everything together to start with.","created_at":"2017-02-17T18:22:41Z","updated_at":"2017-02-20T16:01:05Z","closed_at":null,"merged_at":null,"merge_commit_sha":"a95e7d1b7ec7e5108e28fd295816a9b2c9e58f23","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/sagemathinc/smc/pulls/1696/commits","review_comments_url":"https://api.github.com/repos/sagemathinc/smc/pulls/1696/comments","review_comment_url":"https://api.github.com/repos/sagemathinc/smc/pulls/comments{/number}","comments_url":"https://api.github.com/repos/sagemathinc/smc/issues/1696/comments","statuses_url":"https://api.github.com/repos/sagemathinc/smc/statuses/db388b79d91f17640fa9c4b0ccacc5ba538ed52f","head":{"label":"embray:design-overview","ref":"design-overview","sha":"db388b79d91f17640fa9c4b0ccacc5ba538ed52f","user":{"login":"embray","id":676149,"avatar_url":"https://avatars.githubusercontent.com/u/676149?v=3","gravatar_id":"","url":"https://api.github.com/users/embray","html_url":"https://github.com/embray","followers_url":"https://api.github.com/users/embray/followers","following_url":"https://api.github.com/users/embray/following{/other_user}","gists_url":"https://api.github.com/users/embray/gists{/gist_id}","starred_url":"https://api.github.com/users/embray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/embray/subscriptions","organizations_url":"https://api.github.com/users/embray/orgs","repos_url":"https://api.github.com/users/embray/repos","events_url":"https://api.github.com/users/embray/events{/privacy}","received_events_url":"https://api.github.com/users/embray/received_events","type":"User","site_admin":false},"repo":{"id":82324972,"name":"smc","full_name":"embray/smc","owner":{"login":"embray","id":676149,"avatar_url":"https://avatars.githubusercontent.com/u/676149?v=3","gravatar_id":"","url":"https://api.github.com/users/embray","html_url":"https://github.com/embray","followers_url":"https://api.github.com/users/embray/followers","following_url":"https://api.github.com/users/embray/following{/other_user}","gists_url":"https://api.github.com/users/embray/gists{/gist_id}","starred_url":"https://api.github.com/users/embray/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/embray/subscriptions","organizations_url":"https://api.github.com/users/embray/orgs","repos_url":"https://api.github.com/users/embray/repos","events_url":"https://api.github.com/users/embray/events{/privacy}","received_events_url":"https://api.github.com/users/embray/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/embray/smc","description":"SageMathCloud: collaborative LaTeX, R, Jupyter, Python, SageMath, Octave, Julia","fork":true,"url":"https://api.github.com/repos/embray/smc","forks_url":"https://api.github.com/repos/embray/smc/forks","keys_url":"https://api.github.com/repos/embray/smc/keys{/key_id}","collaborators_url":"https://api.github.com/repos/embray/smc/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/embray/smc/teams","hooks_url":"https://api.github.com/repos/embray/smc/hooks","issue_events_url":"https://api.github.com/repos/embray/smc/issues/events{/number}","events_url":"https://api.github.com/repos/embray/smc/events","assignees_url":"https://api.github.com/repos/embray/smc/assignees{/user}","branches_url":"https://api.github.com/repos/embray/smc/branches{/branch}","tags_url":"https://api.github.com/repos/embray/smc/tags","blobs_url":"https://api.github.com/repos/embray/smc/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/embray/smc/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/embray/smc/git/refs{/sha}","trees_url":"https://api.github.com/repos/embray/smc/git/trees{/sha}","statuses_url":"https://api.github.com/repos/embray/smc/statuses/{sha}","languages_url":"https://api.github.com/repos/embray/smc/languages","stargazers_url":"https://api.github.com/repos/embray/smc/stargazers","contributors_url":"https://api.github.com/repos/embray/smc/contributors","subscribers_url":"https://api.github.com/repos/embray/smc/subscribers","subscription_url":"https://api.github.com/repos/embray/smc/subscription","commits_url":"https://api.github.com/repos/embray/smc/commits{/sha}","git_commits_url":"https://api.github.com/repos/embray/smc/git/commits{/sha}","comments_url":"https://api.github.com/repos/embray/smc/comments{/number}","issue_comment_url":"https://api.github.com/repos/embray/smc/issues/comments{/number}","contents_url":"https://api.github.com/repos/embray/smc/contents/{+path}","compare_url":"https://api.github.com/repos/embray/smc/compare/{base}...{head}","merges_url":"https://api.github.com/repos/embray/smc/merges","archive_url":"https://api.github.com/repos/embray/smc/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/embray/smc/downloads","issues_url":"https://api.github.com/repos/embray/smc/issues{/number}","pulls_url":"https://api.github.com/repos/embray/smc/pulls{/number}","milestones_url":"https://api.github.com/repos/embray/smc/milestones{/number}","notifications_url":"https://api.github.com/repos/embray/smc/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/embray/smc/labels{/name}","releases_url":"https://api.github.com/repos/embray/smc/releases{/id}","deployments_url":"https://api.github.com/repos/embray/smc/deployments","created_at":"2017-02-17T18:13:15Z","updated_at":"2017-02-17T18:13:20Z","pushed_at":"2017-02-17T18:17:11Z","git_url":"git://github.com/embray/smc.git","ssh_url":"git@github.com:embray/smc.git","clone_url":"https://github.com/embray/smc.git","svn_url":"https://github.com/embray/smc","homepage":"https://cloud.sagemath.com","size":175749,"stargazers_count":0,"watchers_count":0,"language":"CoffeeScript","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"sagemathinc:master","ref":"master","sha":"4301efd8d2a2e70ba6a22b971a89b4e5a5554cb5","user":{"login":"sagemathinc","id":11307305,"avatar_url":"https://avatars.githubusercontent.com/u/11307305?v=3","gravatar_id":"","url":"https://api.github.com/users/sagemathinc","html_url":"https://github.com/sagemathinc","followers_url":"https://api.github.com/users/sagemathinc/followers","following_url":"https://api.github.com/users/sagemathinc/following{/other_user}","gists_url":"https://api.github.com/users/sagemathinc/gists{/gist_id}","starred_url":"https://api.github.com/users/sagemathinc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sagemathinc/subscriptions","organizations_url":"https://api.github.com/users/sagemathinc/orgs","repos_url":"https://api.github.com/users/sagemathinc/repos","events_url":"https://api.github.com/users/sagemathinc/events{/privacy}","received_events_url":"https://api.github.com/users/sagemathinc/received_events","type":"Organization","site_admin":false},"repo":{"id":32500369,"name":"smc","full_name":"sagemathinc/smc","owner":{"login":"sagemathinc","id":11307305,"avatar_url":"https://avatars.githubusercontent.com/u/11307305?v=3","gravatar_id":"","url":"https://api.github.com/users/sagemathinc","html_url":"https://github.com/sagemathinc","followers_url":"https://api.github.com/users/sagemathinc/followers","following_url":"https://api.github.com/users/sagemathinc/following{/other_user}","gists_url":"https://api.github.com/users/sagemathinc/gists{/gist_id}","starred_url":"https://api.github.com/users/sagemathinc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sagemathinc/subscriptions","organizations_url":"https://api.github.com/users/sagemathinc/orgs","repos_url":"https://api.github.com/users/sagemathinc/repos","events_url":"https://api.github.com/users/sagemathinc/events{/privacy}","received_events_url":"https://api.github.com/users/sagemathinc/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/sagemathinc/smc","description":"SageMathCloud: collaborative LaTeX, R, Jupyter, Python, SageMath, Octave, Julia","fork":false,"url":"https://api.github.com/repos/sagemathinc/smc","forks_url":"https://api.github.com/repos/sagemathinc/smc/forks","keys_url":"https://api.github.com/repos/sagemathinc/smc/keys{/key_id}","collaborators_url":"https://api.github.com/repos/sagemathinc/smc/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/sagemathinc/smc/teams","hooks_url":"https://api.github.com/repos/sagemathinc/smc/hooks","issue_events_url":"https://api.github.com/repos/sagemathinc/smc/issues/events{/number}","events_url":"https://api.github.com/repos/sagemathinc/smc/events","assignees_url":"https://api.github.com/repos/sagemathinc/smc/assignees{/user}","branches_url":"https://api.github.com/repos/sagemathinc/smc/branches{/branch}","tags_url":"https://api.github.com/repos/sagemathinc/smc/tags","blobs_url":"https://api.github.com/repos/sagemathinc/smc/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/sagemathinc/smc/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/sagemathinc/smc/git/refs{/sha}","trees_url":"https://api.github.com/repos/sagemathinc/smc/git/trees{/sha}","statuses_url":"https://api.github.com/repos/sagemathinc/smc/statuses/{sha}","languages_url":"https://api.github.com/repos/sagemathinc/smc/languages","stargazers_url":"https://api.github.com/repos/sagemathinc/smc/stargazers","contributors_url":"https://api.github.com/repos/sagemathinc/smc/contributors","subscribers_url":"https://api.github.com/repos/sagemathinc/smc/subscribers","subscription_url":"https://api.github.com/repos/sagemathinc/smc/subscription","commits_url":"https://api.github.com/repos/sagemathinc/smc/commits{/sha}","git_commits_url":"https://api.github.com/repos/sagemathinc/smc/git/commits{/sha}","comments_url":"https://api.github.com/repos/sagemathinc/smc/comments{/number}","issue_comment_url":"https://api.github.com/repos/sagemathinc/smc/issues/comments{/number}","contents_url":"https://api.github.com/repos/sagemathinc/smc/contents/{+path}","compare_url":"https://api.github.com/repos/sagemathinc/smc/compare/{base}...{head}","merges_url":"https://api.github.com/repos/sagemathinc/smc/merges","archive_url":"https://api.github.com/repos/sagemathinc/smc/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/sagemathinc/smc/downloads","issues_url":"https://api.github.com/repos/sagemathinc/smc/issues{/number}","pulls_url":"https://api.github.com/repos/sagemathinc/smc/pulls{/number}","milestones_url":"https://api.github.com/repos/sagemathinc/smc/milestones{/number}","notifications_url":"https://api.github.com/repos/sagemathinc/smc/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/sagemathinc/smc/labels{/name}","releases_url":"https://api.github.com/repos/sagemathinc/smc/releases{/id}","deployments_url":"https://api.github.com/repos/sagemathinc/smc/deployments","created_at":"2015-03-19T04:16:53Z","updated_at":"2017-02-20T04:12:57Z","pushed_at":"2017-02-20T03:51:45Z","git_url":"git://github.com/sagemathinc/smc.git","ssh_url":"git@github.com:sagemathinc/smc.git","clone_url":"https://github.com/sagemathinc/smc.git","svn_url":"https://github.com/sagemathinc/smc","homepage":"https://cloud.sagemath.com","size":176012,"stargazers_count":326,"watchers_count":326,"language":"CoffeeScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":60,"mirror_url":null,"open_issues_count":499,"forks":60,"open_issues":499,"watchers":326,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/sagemathinc/smc/pulls/1696"},"html":{"href":"https://github.com/sagemathinc/smc/pull/1696"},"issue":{"href":"https://api.github.com/repos/sagemathinc/smc/issues/1696"},"comments":{"href":"https://api.github.com/repos/sagemathinc/smc/issues/1696/comments"},"review_comments":{"href":"https://api.github.com/repos/sagemathinc/smc/pulls/1696/comments"},"review_comment":{"href":"https://api.github.com/repos/sagemathinc/smc/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/sagemathinc/smc/pulls/1696/commits"},"statuses":{"href":"https://api.github.com/repos/sagemathinc/smc/statuses/db388b79d91f17640fa9c4b0ccacc5ba538ed52f"}}}}
{"actor":{"display_login":"embray"}}