text:
When O'Reilly launched Safari back in July, 2000 (it was definitely what would now be called an MVP), our founder and CEO Tim O'Reilly said:
We've always followed our own path, rather than conform to the conventional practices of the publishing industry. Safari is a logical extension of that approach. In designing the service, we've exploded the notion of 'book' and built a front end for the content of our books that truly harnesses the power of the Web.
Safari brought the information in our books–and soon after, those of many other excellent publishers–to the Web, but 16 years later, Safari is about so much more than books. From concise webcasts to comprehensive training courses to full conferences, Safari also has nearly 4000 videos. If you haven't yet, check out the new Learning Paths on important tech and business topics, and the popular Best of 2015 collections.
One more thing–if you want a quick way to keep track of what matters now (and why!) in the technology world, sign up for one or more of O'Reilly's email newsletters for your weekly dose of news and insight every week. In the words of one reader, "I'd have to spend hours combing the net and filtering junk to get half of what you guys present us every week."]]>
First and foremost, it must be understood that metadata drives your online publishing platform functionality. Without it nothing really works and Todd put it very well when he said during his presentation that "without metadata, users will 'walk by'". Whenever publishers ask us if our PubFactory platform has been developed to maximize discoverability or to support the latest cool, new curation features, my typical response is to ask "Yes…and how will you provide the metadata needed to enable the requested feature(s)?". Surprisingly often, the answer I get is "I have no idea".
The kind of metadata that is needed is driven by what you are trying to accomplish. Most metadata discussions center around discovery but for this discussion I would like to examine the metadata needs for a publisher created platform. The use case that I have in mind is of a publisher supplying content and metadata for an online delivery and reading system for their books which will primarily serve an institutional/academic market.
Just as search is the starting point for the typical user journey when accessing scholarly content, so to it is a critical component of any online platform and has many and varied effects on the metadata that needs to be supplied along with the content. One of the first considerations is how search should function for a collection of books. In some cases having search return whole books results makes sense, but in many others the need to provide very granular results means that returning chapters or even sections within chapters is critical.
If chapter-level results are returned you must provide title and author metadata at the chapter-level. The platform may be able to automatically generate this metadata for monographs but for edited collections it is likely that this metadata will need to be supplied by the publisher. This is a case where internal publisher workflows may need to be updated if chapter-level metadata is not currently being created and managed so that it can be supplied to the platform.
The next thing to consider is what metadata fields should be searchable for a standard search. This will vary depending on the types of books in the collection. Also important when returning chapter-level results is recognizing that not all chapters should be presented in the search results. For example, returning front matter sections such as the title or copyright pages or the index from the back of the book doesn't generally address the users' needs. And as a result, it is necessary to provide data for each section of the book indicating whether or not it should be included in search and whether or not it should be accessible on the platform.
It can also be helpful to provide abstracts in search results. Publishers usually can supply abstracts for books but may not have chapter-level abstracts readily at hand. If the data is not available are there alternatives? For instance, in this case, one possibility is to extract the first page of content in lieu of an authored abstract. In general, the key is to think through and consciously decide on all the metadata issues that come up.
How does content format impact metadata?
The second major determinant of the type of metadata required will be the format of the content itself. There are a wide range of choices for book content ranging from the ever popular PDF, to ePubs and various forms of XML. The most common industry standards for book XML format include NLM (which has evolved into BITS), DocBooks, TEI, and DITA. The XML formats typically include support for most of the metadata that is needed in the format itself, whereas ePub includes some metadata but will likely need to be supplemented and PDFs generally have almost no metadata associated with them.
If a separate metadata format is necessary, then ONIX is one obvious industry standard to use. Despite the fact that ONIX 3.X has been around for a long time a surprising number of publishers are still using ONIX 2.X. ONIX is most typically thought of as a metadata format for title level metadata but it can support chapter-level metadata as well.
There are a number of important details to address when dealing with ONIX metadata, including –
Although it can be quite complex and require significant effort to properly create and maintain not just basic metadata, but rich metadata for your content – the investment is more than worth it. During the conference session, Chuck Koscher, Director of Technology at Crossref, said the following which is worth repeating here (and elsewhere) and that is "Metadata is its own entity. It goes places that your content never goes to". The corollary that I'm emphasizing here is that metadata is critical to facilitating the functioning of your platform.
Watch out for my next post in which I will delve into taxonomies, business models and pricing as core components of your metadata management.
"Follow-up is the key to sales success."
If you're in sales, you've probably heard those words so many times they've lost their meaning. Let's change that: it's time to put power back in those words.
Forget everything you think you know about the follow-up. We're going to look at it from a fresh perspective. It's time the follow-up stopped being a "good idea" and became a measurable action.
If your prospect doesn't respond, is following up really going to make a difference?
In my experience, hell yeah. I once had to follow up with an investor 48 times before I was even able to set up a meeting. That's a lot of rejection. But I persisted, and he ended up investing.
Don't take my word for it. Here's what Brandon Gracey of Handshake has to say:
What if I'd given up after 47 attempts? What if Brandon had only followed up 40 times? Neither of us would have closed those life-changing deals.
The results speak for themselves: If you want to become a priority to someone, the price is tireless follow-up.
If you're pursuing a warm lead (prospects you've had some form of positive communication with), then follow up until you get a clear "yes" or "no".
For the first month, check in once a week. After the first 30 days, check in monthly.
If you're willing to consistently follow up on your warm leads, I guarantee that you'll get a response. That response may be a "no", but a clear "no" is much better than a half-assed "maybe".
Cold leads (prospects you've never had any interaction with) are a little different. Don't be annoying, but don't give up too soon either.
At Close.io, we follow up a total of three times over the span of two weeks. Here's our schedule:
The follow-up is only as effective as the foundation it's built on. Make sure that your initial outreach is both informative and valuable for the prospect. If you're new to sales, check out our guides to leaving effective voicemails and writing compelling cold emails.
If your prospect doesn't respond to your initial outreach, it doesn't mean they're not interested. They might have been too busy or distracted to respond. You can counter that by following up at a different time of day.
For example: If you sent your initial email at 9:00am and didn't get a response, follow up with them around 3:00pm. They might have busy mornings but take a relaxed lunch break.
This follow-up message should be a restructured version of your initial message with no new information.
Like before, make sure you follow up at a different time of day.
This message should restate your call-to-action. For example, "Hey, this is Steli from Close.io again. Any chance you're free for a five-minute call on Thursday at 2:00pm?"
If two weeks have passed and you haven't received any response, it's time to break up with your prospect.
The break-up email is the most powerful part of any follow-up campaign. It counts on the fact that people want something more when it's taken away from them.
It starts with a powerful subject line like:
And clearly communicates the following points:
Although the break-up email is a powerful tool, there's no guarantee that your prospects are going to respond. If they don't, then it's time to move on to more promising opportunities.
This doesn't mean you close the account forever; just for the time being. If it's a substantial account, consider revisiting it next year.
Our book, "The Ultimate Startup Guide to Outbound Sales" is available in full on Safari. It includes tips and best practices for topics such as B2B lead generation, utilizing cold sales emails, and successful sales calls blueprints. You can also learn more about our inside sales CRM on our website.
Sales success is about relentless follow-up.
Close.io is a sales platform with built-in calling and automatic email tracking designed to reduce data entry so clients can focus on selling, instead.]]>
As part of a continuing feature here at the Safari blog, we'd like to shine a spotlight on the imprints of Elsevier that provide content to Safari.
CISSP Study Guide by Joshua Feldman, Seth Misenar, Eric Conrad
Learn everything you need to know to pass your CISSP certification with this proven and effective study guide which is now in it's third edition. It now covers all exam topics through January 2015 and provides two separate practice exams.
Google Hacking for Penetration Testers by Johnny Long
Learn how to take advantage of the powerful capabilities of Google to uncover sensitive and hard-to-find information with this second volume of techniques and methods from Syngress. Topics in this book include database digging, finding exploits, and how to track down web servers, login portals, and network hardware.
Building a Scalable Data Warehouse with Data Vault 2.0 by Michael Olschimke & Dan Linstedt
Author Dan Linstedt invented the standard Data Vault which has been applied to data warehousing projects across the globe. This book covers everything you need to know to create scalable data warehouses including dimensional modeling, data extraction, and implementing data quality.
Learning Processing: A Beginner's Guide to Programming Images, Animation, and Interaction by Daniel Shiffman
Learn the building blocks of programming which are needed to build graphics applications with this lab-manual style book which is now in its second edition from Morgan Kaufmann. Spanning from algorithmic design to data visualization, you will learn everything you need to know including libraries, event probability, and sound synthesis.
Negotiating in the Leadership Zone by Ken Sylvester
From Academic Press, an imprint of Elsevier, comes this new book which draws up on 50+ years of tech leader experience to teach readers how to become a better negotiator in any kind of leadership role. The case studies, examples, and well-researched insights will provide any leader with the tools they need to improve their negotiation skills in all facets.]]>
Did you know that NASA has debugged a spacecraft 100M miles away with LISP REPL?
Would you like to have that same power in your Java app today? (As an aside, some might contend this powerful enough to create a risk, so you should consider if this fits your needs. Don't give access to this to anyone who shouldn't have root access your server.)
Today we're going to build a web REPL in Clojure. Then we'll integrate it into a Java application. Then we'll use it to read a value inside the Java application that otherwise we'd have no visibility of.
lein new app repl-jar
lein install
lein run
Clojure> (inc 41)
You should get the result:
We've got the first part of our Clojure Web REPL working. Now let's integrate it into a Java Application.
mvn archetype:create -DgroupId=demo -DartifactId=app -DarchetypeArtifactId=maven-archetype-quickstart
mvn compile
mvn exec:java -Dexec.mainClass="demo.App"
Clojure> (import demo.Model)
You should get the result
Clojure> (def appInstance (new Model))
You should get the result
Clojure> (.getLaunchCode appInstance)
You should get the result
You'll notice we've added the following web-related library dependencies:
[compojure "1.4.0"] [ring/ring-defaults "0.1.5"] [ring/ring-core "1.4.0"] [clj-http "2.0.0"] [[ring/ring-devel "1.4.0"]]
These are for routing and handing HTTP requests.
This library dependency:
[ring/ring-jetty-adapter "1.4.0"]
gives us the Jetty web server.
In addition this json library:
[org.clojure/data.json "0.2.6"]]
Is for handing data embedded in json from the AJAX requests.
This change:
:main repl.ApplicationREPL :aot :all
Is for changing the class that is finally created. By default, Leiningen wouldn't create a Java class when we run lein install, it would simply create a jar with the uncompiled Java source files inside. The :aot step points it to compile the class files, and the :main tells the execution and packaging steps in Leiningen that we've remapped the resulting class. By default, our Clojure class repl-jar.core would compile down to a class named repl_jar.core, which isn't very friendly to those with a Java background. In the Clojure file, we include some code:
(:gen-class :name repl.ApplicationREPL
to tell the Clojure compiler to compile to a Java class named repl.ApplicationREPL instead.
The first thing to note is the :gen-class statement. Here we set the name of our compiled class to a Java style class name, to be friendly for integration. We also export the start method as a static method.
The next thing to notice are the libraries pulled in by :require. The Ring and Compojure libraries are for handling web requests. The Hiccup libraries are for generating html from Clojure data. The final Clojure json library is for unwrapping the data embedded in the GET statement as the expression is sent down from the browser.
The repl-sessions definition holds the environment for each of the REPLS in every browser session opened. The defonce tells the Compojure handlers not to reinitialize this value each time a new browser sends a request.
The current-bindings function is a wrapper around the base current Clojure environment to provide a framework for controls around what can and cannot be run.
The functions bindings-for, store-bindings-for and with-session are about linking the current session key to the current environment for that browser. Given a new session, create a new environment and store it associated with that session. Given a new request for an existing browser session, retrieve that environment and evaluate the expression in that context.
The do-eval function is the meat of the whole project. The main part of it is the expression (eval form). Outside that we wrap the stdout, handle errors and provide the environment in which to evaluate the expression.
The handle-request function pulls the session key from the request and passes it down to the evaluation function so that the corresponding environment can be retrieved.
The html-page function is where we get to use the Hiccup libraries to generate HTML. We see CSS and JavaScript being loaded, and we provide a console div that our jQuery console will hang off later.
The app-routes definition is a library call to a Compojure macro to define our routes. Similar to the Ruby Sinatra library, we can map an HTTP action and path to a function.
The ring-app definition wraps our Compojure routes with handlers for parameters, CSS, sessions and error handling. These libraries are all Ring middleware.
The start-jetty function makes a call to the embedded jetty library. The library function run-jetty takes a Compojure route definition, and a port.
The –main function is the Clojure equivalent of the public static void main(String args[]) you see in Java. This is our default entry point into the application (we'll use another to make the wrapper simple). We're using it to kick off the start-jetty function.
The –start function is our entry point for integration into the application. We define it as static in the gen-class definition. This simply starts the –main function. We call this method from the Java class using .start().
We keep this fairly simple, with the jQuery Console tied to the #console div. Into this we feed a welcome message, a validation method and a handler method. The validation message just does a simple check to exclude empty strings. The handler method passes the expression down to the server using a synchronous GET, and returns the result.
Here we just add one dependency for our repl jar. Everything else is boilerplate.
Here we just add one line to start the web REPL running in the background. This is how easy it is to integrate into your application.
This is a plain Java class. We include a hard-coded instance-level value, and a getter, to keep our testing simple, but more realistic values would be an unpredictable value, sourced from another system. (See if you can guess the cultural reference. )
We've built and tested a Clojure Web REPL. We've integrated it into a Java Web application. We've used it to determine a value inside the Java application, that otherwise we'd have no visibility of. (We've used a fixed value, but imagine if it were a random number – or a value from an external source.)
You can find the code listing for this blog post on GitHub. If you need sandboxing, Anthony Raynes has done some great work on the clojail library.
Any Clojure WebREPL is indebted to the pioneering work of Anthony Raynes and Jay Fields.
Chris Done's JQuery Console project has earned its status as a foundation of many web REPL projects across languages.
My book, "Clojure Recipes" is available in Safari now. Focusing on Clojure 1.7 and higher, topics include creating a REST Server in Compojure, loading a data file into Cascalog, and getting started with Storm.
Julian Gamble is a software engineer who has worked in the financial services industry for more than a decade. When he's not enabling billions of dollars to orbit the globe, he writes and presents on all things software-related at juliangamble.com/blog.]]>
On a daily basis we are called on to help others. We do this as parents, spouses, and friends; and we do this in the workplace, as employees, colleagues, and managers. But how often do we consciously and skillfully pull from our toolbox the right strategy for guiding, motivating, and inspiring another person? And, if our first efforts fail, how creative are we in finding another strategy? Collaboration, coaching, and mentoring are certainly rewarding when done well, but they can be both painful and ineffective when done poorly.
Imagine that you are preparing to talk with a colleague, employee, or a family member. Your goal is to help them improve their current situation, performance, or life. What is your approach? How do you decide what strategy to use? Or, do you think in terms of strategy at all? If one approach isn't working, do you have a plan B or C ready? When I interview executives who are required to give feedback to employees, I ask them to list their strategies for this challenging task. They often say, hesitantly, "I just tell it like it is…" or "I give it to them straight." I then inquire, "So, how well is that working?" The answer, usually accompanied by an uncomfortable smile, is "sometimes." And sometimes, direct feedback – no matter how painful it is to give or receive – is the right response. But not always.
To achieve strong collaboration, different situations call for varying types of feedback and support. For those situations where a more strategic approach is called for, allow me to provide you with a "gourmet guide" of support that will help you to provide assistance and motivate others at work and at home. For those times when you are the one in need of support, you can also use this menu to help identify what you might need from others.
There are seven strategies that are especially useful and are easily recalled using the mnemonic of INSPIRE. The word is a fitting, since inspire means "to take in." When you are providing feedback or support, you want the other person to take in and be inspired by what you have to say.
So what are the INSPIRE strategies?
Our new book, "Mastering Fear," which was published in January of 2016 by Career Press is now available in Safari. You will learn how to deal with the stress of fear in all facets of your life including your career, relationships, and during the creative process.
Bob Maurer is a clinical psychologist on the faculty of the UCLA and the University of Washington School of Medicine. He is also the Director of Behavioral Science for the Family Medicine Residency in Spokane, Washington.
Michelle Gifford, MA, CCC-SLP, is a clinical speech language pathologist in private practice and a consultant-educator for families and teams, developing programs of excellence on behalf of children with special needs.]]>
Vision statements on a wall, on a website, or on a piece of paper can serve a purpose for businesses, but they don't lead to success. The purpose of a vision is to have something not just to move towards, but to achieve. We need to challenge ourselves to move beyond the just the emotional connection to a vision and focus on mapping out the hard work it takes to get somewhere you want to go. A big mistake leaders make is that they think simply having a vision is enough, it's not.
Research and reality tells us how it is. Psychologist Lien Pham and Shelley Tailor from University of California (Pham & Taylor, 1999) put students into two groups. They asked one group (Group A) to visualize how great it would be to get a high grade. The other group (Group B) wasn't asked to visualize the positive feelings associated with the end state of a high grade. Both groups kept track of the hours they spent studying. Even though the group that visualized the positive feelings associated with a good grade only did this for a few minutes it had a significant impact on the amount of time they studied as well as their grades. The group of students (Group A) that visualized the positive feelings ended up studying less and ended up with lower grades. The visualization may have made them feel good, but it did not prepare them for success and therefore set them up for the failure. My view is that Group A, that associated the positive feelings with the outcome, became overconfident and were not aware and/or not realistic about how to get to what they wanted.
What makes the point even more clearly is that the researchers had a third group of (Group C). This group was asked to visualize the "how" or process of getting an A for a few moments a day. They were asked to get into more detail and visualize how and what they would do to get an A. Compared to the group (Group A) that just visualized the outcome of getting an A, and to the group that wasn't asked to do anything (Group B), the group that was asked to visualize the "how" (Group C) ended up studying more hours and earned higher exam grades than both groups. The researchers concluded that visualizing the steps to prepare them for success put them in a more realistic and practical vision to succeed.
I have seen the same story play out again and again. Countless executives I work with tell stories of motivational speakers that get them fired up by seeing success. Seeing success is not enough, you also have to see the steps.
So what do you do? Put in the precision with three essential steps: Vision, Envision, and Revision. Vision is where you want to go. Envision is where you are, what you are getting, and how you get to where you want to go. Finally, Revision is what you need to change to get to where you want to be. In the Revision stage you can either change your vision or change your envision (steps).
Now if you want to inspire people toward a vision, think of your senses: See, Feel, Think, and Act. Tap into as many senses as possible. Connect with people multiple ways so you provide them with an emotional hook. The goal is to create initiative. A clear vision with the following components not only brings you closer to where you want to go, but it gets people motivated to work with you rather than against you. Another simple way to remember what you need in a vision is to just think of it as using your eyes, heart, head, and hands.
See (Eyes) Where are we going? People want to have something to move toward together. When you help people "See" you are answering one main question, where are we going? Think of this as painting a picture for people.
Feel (Heart) Why are we going there? Nothing is more powerful than emotion. Purpose taps into emotion. The main questions you are answering when you create the "Feel" is why are we doing this and what is the purpose? People get behind a purpose. The purpose can be as simple as something clearly telling people what the value is in achieving the vision.
Think (Head) How will get there and Who is responsible? Think is all about the plan. It's the how and who aspect of the strategy. This is where some of the details of a plan get communicated. In this step in the vision process you let people know who will be playing what part. It's the roadmap. The more specific you can be the better. Set up milestones and metrics. Include how you will measure progress, success, and failure.
Act (Hands) What is the first step and action items? Act is where you communicate what the steps are and where to start. This is different than the "Think" aspect of the vision in that it is more detailed and specific. The focus is on doing rather than just planning.
I remember hearing about a 2007 McKinsey & Company study (Dye and Sibony, 2007) that highlighted about half of executives who participate in a strategic planning process are not satisfied. Furthermore, less than a quarter of the executives indicated that major strategic decisions were made during the progress. The best visions are the ones that become a reality. It takes work, but now you know how. By reading this you just another step toward success. Make your next step count and enjoy the journey.
My new book titled "Simple is the New Smart" is available in its entirety in Safari now. Topics include using your mind muscle memory, cultivating a personal board of advisors, and understanding and implementing the Persuasion Pyramid.
Dr. Rob Fazio is the Managing Partner of OnPoint Advising, Inc. Rob partners with executives, athletes, and businesses internationally to guide them toward success. He can be reached through email or on Twitter.]]>
Only a few years ago there did not exist a job with the title Chief data scientist. But that was then. Small and large corporations, and increasingly government agencies are putting together teams of data scientists and analysts under the leadership of Chief data scientists. Even White House has a Chief data scientist position, currently held by Dr. DJ Patel.
The traditional role for those who analyzed data was that of a computer programmer or a statistician. In the past, firms collected large amounts of data to archive rather than to subject it to analytics to assist with smart decision-making. Companies did not see value in turning data into insights and instead relied on the gut feeling of managers and anecdotal evidence to make decisions.
Big data and analytics have alerted businesses and governments to the latent potential of turning bits and bytes into profits. To enable this transformation, hundreds of thousands of data scientists and analysts are needed. Recent reports suggest that the shortage of such professionals will be in millions. No wonder we see hundreds of postings for data scientists on LinkedIn.
As businesses increasingly depend upon analytics driven decision making, data scientists and analysts are simultaneously becoming front-office superstars, which is quite a change from them being the back office workers in the past.
Sooner than later, workers will find their managers asking them to assume additional responsibilities that would involve dealing with data, and either generating or consuming analytics. Smart professionals, who are uninitiated in data science, would therefore proactively address this shortcoming in their portfolio by acquiring skills in data science and analytics. Fortunately, in the world awash with data, the opportunities to acquire analytic skills are also ubiquitous.
For starters, professionals should consider enrolling in open online courses offered by the likes of Coursera and BigDataUniversity.com. These platforms offer a wide variety of training opportunities for beginners and advanced users of data and analytics. At the same time, most of these offerings are free.
For those professionals who would like to pursue a more structured approach, I suggest that they consider continuing education programs offered by the local universities focusing on data and analytics. While working full-time, the professionals can take part-time courses in data science to fill the gap in their learning and be ready to embrace impending change in their roles.
Computer programming skills are a definite plus for data scientists, but they are certainly not a limiting factor that would prevent those trained in other disciplines from joining the world of data scientists. In my book, Getting Started with Data Science, I mentioned examples of individuals who took short courses in data science and programming after graduating from non-empirical disciplines, and subsequently were hired in data scientist roles that paid lucrative salaries.
The choice of analytics tools depends largely upon the discipline and the type of organization you are currently working for or intend to work for in the future. If you intend to work for corporations that generate real big data, such as telecom and Internet-based establishments, you need to be proficient in big data tools, such as Spark and Hadoop. If you would like to be employed in the industry that tracks social media, you would require skills in natural language programming and proficiency in languages such as Python. If you happen to be interested in a traditional market research firm, you need proficiency in analytics software, such as SPSS and R.
If your focus is on small and medium size enterprises, proficiency in Excel could be a great asset, which would allow you to deploy its analytics capabilities, such as Pivot Tables, to work with small sized data.
A successful data scientist is one who knows some programming, basic understanding of statistical principles, possesses a curious mind, and is capable of telling great stories. I argue that without the story telling capabilities, a data scientist will be limited in his or her abilities to become a leader in the field.
Schools, colleges, universities and others involved in education and learning are putting big data and analytics to good use. Universities are crunching large amounts of data to determine what gaps in learning at the high school level act as impediments to success in the future. Schools are improving not just curriculum, but also other strategies to improve learning outcomes. For instance, research in India using large amounts of data showed that when children in low-income communities were offered free meals at school, their dropout rates declined and their academic achievements improved.
Big data and analytics provide instructors and administrators the opportunity to test their hypothesis about what works and what doesn't in learning, and replace anecdotes with hard evidence to improve pedagogy and learning. Learning has taken a new shape and form with open online courses in all disciplines. These transformative changes in learning have been enabled by advances in information and communication technologies, and the ability to store massive amounts of data.
Change is inevitable. Despite what modern governments and societies like, they would have to embrace change. Fortunately, smart governments and societies have already embraced data-driven decision-making and evidence-based planning. Governments in developing countries are already using data and analytics to devise effective poverty reducing strategies. Municipal governments in developed economies are using data and advanced analytics to find solutions to traffic congestion. Research in health and well-being is leveraging big data to discover new medicines and cures for illnesses that challenge us all.
As societies embrace data and analytics as tools to engineer prosperity and well-being, our collective abilities to achieve a better tomorrow will be further enhanced.]]>
I cringe each time I hear the oft repeated declarations that "every company will be compromised" and that "it isn't a matter of if, but when". These statements are the basis of the FUD (fear, uncertainty and doubt) driven cyber sales machine. What is closer to the truth is that Internet connected systems have a high probability of being subject to a targeted or opportunistic attack, inadvertent exposure, or malicious subversion. However, it is (and I stress) not inevitable that the attacker will be successful. Motivation, work factor, evasion capabilities, resiliency, and sometimes, luck all play a part. Threat modeling can be used to understand these factors and influence the outcome.
Threat modeling is used to identify and categorize potential threats. Conventional cybersecurity threat modeling uses one of three approach; attacker-centric, architecture-centric, or asset-centric.
Many organizations find this task daunting. Do not despair! Threat modeling does not have to be overwhelming. A simplified approach to threat modeling is to answer four essential questions that identify threat adversary motivation, attack workfactor, organizational threat intelligence and detection capability, and resiliency.
Answer these four questions to your satisfaction and you will be well on your way to being a threat-modeling guru.
Let's start with motivation. We need to ask ourselves, what property, information, or power does the organization have that is so valuable to the attacker that they are willing to risk prosecution and/or retaliation? Once we have identified the asset(s), the natural follow-on questions are – where is it located (physically and logically), why do we have it, and do we really need it?
The last question perhaps seems redundant – why would be have it, if we didn't need it? Every organization should honestly evaluate the data sets they collect, aggregate and/or mine in terms of both security and privacy. Truth be told, organizations collect, store, configure a multitude of assets that are of less value to them then to an attacker. Simple rule – if you don't need it; securely dispose of it. What remains, should be your focus.
Workfactor is the time, effort, and talent needed for an attacker to successfully achieve their objective. In other words, how much time they need to invest, how hard they have to work, and what type of skills and expertise are needed to overcome protective barriers. The intensity of the workfactor should match the criticality and/or sensitivity of the asset you are protecting.
It is vital that organizations classify their assets to insure that funding and resources are being properly allocated. Workfactor is a powerful weapon and can be used to dissuade all but the most motivated of adversaries. However, since attack tools and techniques are constantly evolving, assessing workfactor should be a iterative process.
Evasion is a means of escaping or advoiding detection. What are your organization's detection capabilities? Would you know if your organization was being attacked or would the attack evade detection for hours, days, weeks, or even months? Incident detection capabilities include monitoring, alerting, incident reporting, and analysis as well as participation in threat intelligence and information sharing activities.
Threat intelligence is verified information about threats, and related vulnerabilities (weaknesses) and exploits. Gartner defines threat intelligence as "evidence based knowledge including context, mechanisms, indicators, implications and actionable advice about an existing or emerging menace or hazard to assets that can be used to inform decisions regarding the subject's response to that menace or hazard". Threat intelligence is available from a variety internal and external government, industry, open-source, and commercial sources including
The true value of threat intelligence is in its application. Good threat intelligence can change your security posture from reactive to proactive. If you understand your adversaries, you can develop tactics to combat current attacks and to prepare for the future.
Resilience is the capacity to withstand attack. A defense-in-depth approach requires that organizations implement layered controls while simultaneously preparing for their failure. The criticality of a practiced rapid response cannot be overstated. Exercises can vary from simple and short to very complex.
Frequent exercises coupled with a rigorous methodology will result in great confidence that your organization will not become a statistic.
Sometimes the hardest part of a task is getting started. My advice to you is to start small. Focus on a single category of assets. Brainstorm threat adversary motivation, related workfactor, organizational threat intelligence and detection capability, and resiliency. Invite others to join in the discussion and challenge each other's assumptions. Keep your eye on the ball and make threat modeling a winning team sport.
You can learn more from my "CISSP Complete Video Course" which is available in Safari.
Sari Greene is an information security practitioner and entrepreneur. She is also the author of Security Program and Policies: Principles and Practices, 2nd Edition.]]>
They're becoming more complex and coming at us so fast that we can either get run over by them or strive to identify, understand, and conquer them.
In Building the Learning Organization, author Michael Marquardt talks about action learning, a powerful tool that has helped some of the world's most successful companies solve problems and seize opportunities. This compelled me to read his book dedicated to the topic, Optimizing the Power of Action Learning.
Action learning is a process where a small group of people focus on a complex business problem with the goal of solving it while developing themselves and the organization. It consists of six components that, when operating together, deliver powerful results ranging from innovative new products to drastic reductions in production and delivery time to improved customer service. The benefits grow exponentially when the process is replicated and expanded within the organization.
"Since Reg Revans introduced action learning in the 1940s, there have been multiple variations of the concept, but all forms of action learning share the elements of real people resolving and taking action on real problems in real time and learning while doing so. The great attraction of action learning is its unique power to simultaneously solve difficult challenges and develop people and organizations at minimal costs to the institutions." – Chapter 1, Optimizing the Power of Action Learning
The problem or opportunity is at the heart of action learning and should be of high importance to the organization and urgent in nature. It can be anything from reducing turnover to orchestrating a reorg. Marquardt goes into detail about choosing a problem, the feasibility of solving it, the learning opportunities it should provide, as well as the authority the team has to solve the problem.
The action learning team is made up of four to eight people, ideally from different departments and functions within the company. There are a number of considerations for team members including their familiarity with the problem, their power to implement tasks, and their commitment to organizational growth the and building of networks and future support systems. Many teams include an action learning coach who focuses on the learning and development of members and the organization.
What Marquardt calls a 'working process of insightful questioning and listening' is one focused on finding the right questions rather than the right answers. In doing this, the group can enter into dialog, clarify the problem, avoid mistaken assumptions, learn about each contributor's perspective and capabilities, and discover the beginnings of a solution. The author explains the benefits of questioning, why asking the right questions is so important (and difficult), and shares examples of the types questions that are the most and least effective.
Actions include problem reframing, establishing goals, developing strategies, and executing those strategies with testing, garnering support and resources, gathering information, pilot testing, and ultimately, implementing strategies. Marquardt explores systems thinking vs. linear thinking, outlines the stages of action learning, and shares an action framework.
While solving the problem at hand is critical, learning and the knowledge gained from it on an individual, team, and organizational level adds exponential value as the process can be expanded and replicated to create quantum leaps for an organization. The author delves into the the following areas:
A dedicated coach helps the team stay focused and thinking about what they are learning and how they are solving problems. For a more effective action learning program, a business can appoint an employee to become a Certified Action Learning coach (CALC) to help team members improve how they ask questions, listen, process, plan, and work.
Besides a thorough explanation of the process and a how-to chapter on introducing, implementing, and sustaining action learning in the organization, Optimizing the Power of Action Learning is sprinkled with real life examples, ideas, and success stories from organizations of all kinds. If you're a Safari member, you already have unlimited access to this title and thousands more. If you don't use Safari and would like to try it for free or explore options for your organization, click here.
A data model is a precise representation of an information landscape, in much the same way as a map is a precise representation of a geographic landscape. "Precise" is the key characteristic of a data model, which means that there is a clear, unambiguous way of reading every symbol and term on the model. A team of business analysts for example, will all read the data model the same way and understand exactly what is being communicated, and then afterwards can debate whether the data model reflects their understanding of the business or whether the measures on the data model reflect what is required in the business requirements. An application development team will all read the data model the same way and understand exactly what is being communicated, and afterwards can discuss the ideal way these structures should be implemented.
Without the data model, we rely more on conversations and requirements documents, both of which are traditionally ambiguous. Conversations and requirements documents are often essential inputs to the data model, however.
For example, "A Customer has Accounts" is typical of an ambiguous statement that might be made verbally or appear in print. Business analysts discussing this statement would need to invest valuable time first understanding exactly what is being communicated. Developers discussing this statement might make erroneous assumptions as to what a customer or account is, leading to a poor design choice.
"A Customer has Accounts" cannot appear on a data model until this statement is made precise, which involves answering many questions first, such as these ten:
Once these ten questions are answered, the results can be shown visually on a data model, leading to our precise communication tool. Here is an example of a data model which captures the answers to a number of these questions:
This model is read as follows:
Once this model is created, business analysts, developers, and other important roles to application development do not need to invest valuable time asking the same questions again such as those that appear above. Instead they can jump right in and read the model which conveys the answers to these questions, and then discuss and debate parts of the model, continuously improving it to reflect reality and the necessary business requirements.
For example, everyone can read the model above and know that an Account can only be owned by one Customer, not two and not zero. Because everyone reads this statement the same way, it can be discussed to determine if everyone agrees with this statement. For example, if someone can prove that we can have joint accounts, that is an Account may be owned by two Customers, than this model will need to change.
Since all application development roles read the model syntax the same way, they can jump right in and discuss the semantics. A simple formula:
data model = less analysis time + more successful development decisions
Steve's Data Modeling Master Class is recognized as the most comprehensive data modeling course in the industry. Steve is the author of nine books on data modeling, including the bestseller "Data Modeling Made Simple." He is also the author of "Data Modeling Made Simple with ER/Studio Data Architect," "Data Modeling Made Simple with CA ERwin Data Modeler r8," and "Data Model Scorecard: Applying the Industry Standard on Data Model Quality."
One of Steve's frequent data modeling consulting assignments is to review data models using his Data Model Scorecard® technique. He is the founder of the Design Challenges group, Conference Chair of the Data Modeling Zone conference, and recipient of the 2012 Data Administration Management Association (DAMA) International Professional Achievement Award.]]>
For C++ programmers, maintaining legacy code has always been important. However, the last few upgrades to the specification have created what some experts have called "a whole new language." So how do you start using it?
For your own projects, you may want to start using new features which improve your programs without breaking your old code. There are some excellent features that can be used this way.
1) Thousandth-Place Separator
Programs using big constants have always caused readability problems. For example, what number is this?
long my_int = 10000000;
Can you tell at a quick glance what this is? A million? Ten million? A hundred million?
The C++14 specification supports a new place-separator (') that makes the identity of the large constant much more obvious:
// Set to ten million. long my_int = 10'000'000;
Of course, anyone reading this in English would like to see commas used here instead, but the resulting syntax would be impossible. Consider this function call:
If a comma (,) were used as internal-place separator, would this statement call a function taking three arguments—10, 100, and 200—or would it take one argument equal to 10,100,200?
So the apostrophe (') works best. The nice thing is that you can start using it right away, even in old programs… old-fashioned literals such as "10000000" are still accepted as always, but for readability's sake, should be revised whenever you can.
2) Range-Based "for"
If there's a new compiler feature designed to potentially eliminate hours of debugging work, it's ranged-based "for." A grizzled old C programmer may prefer to use the old-fashioned "for" statement, and of course it's still supported. For example, it's typical to process array elements this way:
for (int i = 0; i < ARRAY_SIZE; ++i) { array_of_flt[i] = 0.0; }
But anyone who's been programming for a while can tell you that many bugs are created by setting border conditions incorrectly. It's therefore safer—as well as easier—to process an array with the new "for" statement:
for (double &x : array_of_flt ) { x = 0.0; }
This just says, "For each element in array_of_flt, set that element to 0.0."
This code is easier to type in. More importantly, it's more error proof. It says to the compiler, "You figure out where the beginning and end of the array are, then go ahead and do the right thing." There's almost no reason not to go ahead and use such code on existing arrays. (Note that the reference operator, &, is necessary if you want the loop to be able to alter element values.)
One limitation is that the compiler must be able to determine the size of the array… and it isn't very clever. Using range-based "for" is fine with arrays declared as global variables, as well as arrays processed in the same function in which they're declared. But what you'd really like to do is pass an array from one function to another, while still being able to use ranged-based "for."
Unfortunately, there's no way to do that. I've tried. But with the STL collection classes, such as vectors, sets, and maps, you can. You can pass a vector from one function to another and the compiler can always determine the beginning and end.
#include <iostream> #include <vector>
std::vector iVec<int> = {1, 2, 3};
for (int i : iVec) { std::cout << i << std::endl; // Print every element. }
This ranged-base "for" loop works even if the vector iVec is declared in one function and passed to another.
3) The auto Keyword
As long as you're using range-based "for," you ought to get in the habit of using the "auto" keyword as well. This keyword no longer has the meaning it once had in C… in which "auto" modified a variable declaration to say that such a variable was allocated on the stack. Such usage was, in practice, never necessary or useful.
Nowadays, the "auto" keyword declares a type determined by context. This is not a variable-type declaration; your loop variable will only hold one kind of data.
In the case of ranged-based "for," it's convenient to use "auto" instead of some complex type (assuming that someday you are using a complex type). But more importantly, there is no way "auto" can ever be wrong in this context. It will automatically be set to the base type of the container.
for (auto &x : array_of_flt ) { x = 0.0; // Set each element to 0.0. }
In this article, I've barely scratched the surface of the new features of C++14. But I've mentioned some that you can start to use right away without breaking legacy code.
Brian Overland is the author of many books on C, C++, and Visual Basic. His latest book, "C++Without Fear" is available in Safari. He can be contacted at his personal site.]]>
And looking ahead to 2016, there is much more to come! In addition to enhanced reporting, we are working to enable A/B testing and survey tools through PubFactory to allow publishers to refine and improve the user experience across their various products. A recent Scholarly Kitchen article highlights the tangible benefits of A/B testing for publishers in differentiating their products and heightening user engagement. Safari uses industry-leading A/B testing tools and a data-driven approach to product improvements and growing our subscription customer base which increased by over 600% in the last 12 months.
Another area of focus for us in 2016 will be to provide the tools for publishers to build a variety of content curation systems within the framework of the PubFactory platform, which will include allowing users to upload content; extending our current personalization capabilities to support public list building, including user generated content; development of publisher-created and user-created tutorials; and enrichment of content with learning paths that dynamically guide users through content.
The consistent and healthy growth of Safari as a whole continues to propel our publishers, our technology, and our services forward in 2016. Off we go!]]>
A Safari member, VM also charmed us with one of our favorite tweets ever.
Years ago, soon after I first became a manager, a colleague told me, "Just pick up the damn phone." It was in reference to providing good customer service, but it stuck with me. Digital tools like email and IRC are all amazing, but often the best thing we can do to help foster better relationships and understanding is just to pick up the phone and listen.
I've found that the best sources for advice can come from my own teams, but it's not easy to get to the point where people feel comfortable providing that advice (or managers feel comfortable asking for it).
Communication of that sort requires trust, and trust is only effective if it goes both ways. If either side harbors or fosters distrust, dishonesty, or suspicions then trust isn't possible. However, once trust starts to build, asking for advice and feedback, listening to it, and then taking actions is not only a great way to develop as a manager but also to reinforce and increase that level of trust on both sides.
You go from being a "good" manager to a "great" manager when you stop being a manager at all and start being a leader.
It's a like a video game: There are quests you need to learn about and fulfill. The quests can include setting up the right processes and policies (for the right reasons), determining staffing needs then interviewing and hiring the right people, onboarding new team members, coaching so your team members can continue to grow, communicating and setting expectations, balancing the needs of the company, team, and individuals, and gaining the trust of your team and learning to trust them in return.
If you can do these things then you'll find you're no longer managing. You don't have to. Your team doesn't really need oversight anymore. You've leveled up: You're a leader.
The majority of your time is no longer spent keeping an eye on things. Instead, now you're able to work more strategically and light the way for your team and your company.
I try very hard not to dictate or proscribe tools. Every company, team, project has different needs and no tool is good for all of them.
Rather than suggest tools, I prefer to suggest methods for selecting tools. Take the time to determine the needs of your team now and to think ahead to the probable needs of your team in the next year or two. Write down a set of criteria, requirements, and use cases. This is the sort of stuff you're doing when you start working on a new feature or product (or at least I hope you do). That's really all tool selection is: another project, and a very important one at that.
Changing tools takes an immense amount of work and selecting the wrong one can be disastrous for productivity and morale. So I urge you to give tool selection the time and attention it deserves rather than jumping to grab the shiniest, newest, most trendy thing available.
As for the tools for my personal productivity, I'd be utterly lost without Trello, RememberTheMilk, Evernote, Dropbox, and IRC. These are my exobrains and I love them dearly.
They communicate it to me. While I keep up to date on the latest developments in the tech world and focus my attention on news related to our stack, I can no longer call myself a technologist as such.
Therefore, I usually rely upon my developers/ops folks to come to me when there's a need to add, change, or upgrade our technology. They're closer to the problem and much more familiar with what needs to be done.
Whatever the problem is, changing technologies is like changing any other tool: Difficult, expensive, and not to be undertaken either lightly or by the faint of heart.
Before I approve any technology upgrade, addition, or change I would prefer to see a well-considered discussion on the following questions:
The process doesn't have to be formal, and the level of detail required when answering will, of course, vary by the relative importance of the technology to the project. What looks like a small technology change can have big impact that is amplified if you don't have a process for thinking through it before jumping into the deep end of the adoption pool.
Anything that Bob Sutton posts is good, and you can usually find a lot of interesting stuff in Harvard Business Review (ed. note: It's all on Safari!). Beyond that, I don't really follow any specific blogs. Most of my managerial blog article reading ends up being one-offs which I find by way of Twitter and then save to my Evernote.
For books, there are three which I recommend to everyone, manager or not:
There's one podcast I recommend: Manager Tools. The presenters approach things from a more corporate big-enterprise sort of way, but their advice can typically be applied to a lot of teams and companies. I've been managing for a long time now, and I still find plenty to think about each time I listen to an episode. Find the topics and advice which match your needs and give those episodes a listen.
I'm also excited about some of the topics I'm working on for talks this year. The first one with be a three hour technical conference speaking tutorial which Josh Berkus and I are co-presenting at SCALE14x. We've made all of the mistakes and, thanks to this tutorial, you won't have to. Even if you don't want to see this presentation, you should definitely check out SCALE14x. It's definitely a front-runner for best community-run FLOSS conference.
You have to practice, and to practice in the right way. It's not simply a matter of, "Lead, lead, lead, lead, lead, leadership mastered!"
For example, if you were trying to learn calligraphy and all you ever did was write without stopping to see what you'd done properly and what needed work, you'd never become a calligrapher. So you need to lead, stop, think, analyze, correct, adjust, repeat. It's an iterative process.
There are countless opportunities to practice your leadership outside of the office as well. For instance, have you ever gotten into the, "Where do you want to eat tonight?" discussion with your partner or friends? It goes back and forth undecided until it's too late to do anything at all. Instead, you can practice your leadership skills by taking the lead. "Would you like to go to dinner tonight? Here are three places which I think could be fun…" You'd be surprised how much these seemingly mundane little leadership exercises can help you learn and improve.
Check out more of VM's work at her website. Interview condensed and edited for clarity by the Safari editors.]]>
In the age of rapid globalization, mind-blowing technology advancements, and growing consumer power, one thing is becoming increasingly clear: for a business to survive and succeed, it must become a learning organization first and foremost. Regardless of industry, size, age, or business model, every organization must forge ahead in learning better and faster in order to stay competitive.
"As more companies realize that the key resource of business is not capital, personnel, or facilities, but rather knowledge, information, and ideas, many new views of the organization are beginning to emerge." – Building the Learning Organization, Chapter 1
The big challenge, of course, is how to create a learning organization.
In Building the Learning Organization, author Michael Marquardt explains how a business can adapt and survive in a rapidly changing landscape and build a foundation for becoming a learning organization. Based on his experience with hundreds of learning organizations, Marquardt discovered that this can only be done when a company understands and develops five related subsystems. He created the Systems Learning Organization model and in this book, guides leaders through each subsystem with in-depth information, real world examples from successful learning organizations, and actionable ideas.
Learning is the core subsystem of the learning organization. It takes place at the individual, group, and organizational levels. Marquardt drills down into the levels and types of learning that are crucial for organizational learning and skill development.
The organization itself, the setting and body in which the process occurs, is a subsystem of a learning organization. The author explains the four key components of the organization subsystem — vision, culture, strategy, and structure — and how they serve the learning organization.
The people subsystem includes managers, leaders, employees, customers, business and alliance partners, suppliers, vendors, and the surrounding community. Marquardt shows how each group is an important part of the learning organization and can be empowered and enabled to learn.
The knowledge subsystem is where the acquired and generated knowledge of the organization is managed. It includes the acquisition, creation, storage, analysis/data mining, transfer and dissemination, and application and validation of knowledge.
The technology subsystem encompasses supporting, integrated technological networks and information tools that allow access to and exchange of information and learning.
Action learning is an approach every organization can use regardless of size or budget to tackle specific problems such as reducing turnover, orchestrating a re-org, or improving information systems. Marquardt goes as far as to say it could be the most effective tool in building a learning organization, and he dedicates a chapter in the book to describing what it is and how a company can harness action learning to adapt to constant change and create a culture of continuous learning.
Action learning is a process that involves a small group people working on critical organizational problems with the goal of learning how to better solve problems, work as a group, and explore how its learnings can benefit the organization. Its power and benefits come from these six interdependent elements:
"We learn most when faced with a real problem which we are obliged to solve." – Lord Weinstock
Action learning was pioneered by professor Reg Revans in the mid-1900's, and has been successfully adopted by hundreds of organizations around the world. It is being continually advanced by business management experts including Marquardt, and through organizations such as the World Institute for Action Learning (WIAL) and the International Foundation for Action Learning (IFAL).
While there is no single way to grow a learning organization, there are a number of common strategies and actions the world's most successful learning organizations have adopted and developed. Here are a handful of the 16 steps in building a learning organization that Marquardt details in the last chapter of his book:
Looking to build or evolve your learning organization? Below are a few more great reads, and a compelling video. If you're a Safari member, you already have unlimited access to these titles. If you don't use Safari and would like to try it for free or explore options for your organization, click here:
As part of a continuing feature here at the Safari blog, we'd like to shine a spotlight on Manning Publications and their catalog of content for application developers, system administrators, software engineers, architects, and anyone else involved with the computer business.
Learn how to build large-scale data systems to help you capture and and analyze the vital web data you need to make crucial business decisions. The authors walk you through the process step-by-step from the batch layer to the serving layer to the speed layer.
A brand new title which was recently added to Safari, this book is a comprehensive and cohesive guide to the Go language. Proficient experience with other languages is assumed by the authors as it is written for the intermediate developer. Topics covered include packaging, libraries, and testing and benchmarking.
Expected to release commercially in 2016, the Oculus Rift headset will bring a cutting-edge virtual reality experience to the market. In this book, you will learn how to create 3D games and other programs exclusively for the Oculus Rift. Experience with C++, C#, or another OO language is required as you read about how to use the Oculus C API, the VR user experience, and advanced integrations with the Oculus Rift.
One of the most popular Manning titles within Safari, this classic book is the career guide that technically-minded professionals need to read. Author John Sonmez explains the importance of the 'soft skills' which are important in today's workplace such as how to work remotely, the value of learning, and developing positive habits.
You can also check out this list in Safari, which includes an offering of the newest and most popular Manning Publication titles.]]>
Edgar Papke is the author of the recently-published "Elephant in the Boardroom" from Career Press. He is an executive coach and an internationally award-winning speaker.
The need for innovation has long been a permanent and increasingly valued pursuit of business. Over the last two decades it has found its way to the top of the list of what we admire and value most in organizations and those who lead them. So much so, that it is hard to find any noteworthy business publication without at least one article featuring something, or someone, that defines innovation.
We now have a host of annual conferences, gatherings, and innovation festivals to choose from. All dedicated to the constant pursuit of the creation of change, the display of quickly emerging new ideas, and celebration the innovative thinkers and leaders behind them. This includes the recognition of the winners in the race to get their new idea to market.
It's no wonder that in every virtually every workshop of executives I lead, one of the topics we inevitably lean into is innovation. The key challenge they face is figuring out how they can get their people to be more innovative. Eventually, this conversation finds its way to exploring what motivates people to want to be more innovative and how they be applied as key traits and values in the cultures of the organizations and teams they lead.
In exploring the outcomes of these conversations further, I came to the conclusion that while there are several contributors to the manifestation of innovation, there are five elements of human motivation and behavior that, when brought together, create the ideal conditions. This alchemy of elements is also the source of influence that leaders can apply in motivating the creativity and risk-taking they expect of the people in their organizations.
Element #1: Curiosity – As human beings, it is natural for us to be curious. We all have the desire to explore, discover and learn. The more we succeed in applying our learning, the more we are naturally motivated to learn more. Typically, we also become more sophisticated in pursuing our curiosities, asking better questions and becoming increasingly more effective at investigating new possibilities. It is also the human trait of curiosity that engages us to be analytical and look at existing data in new and unique ways, and makes us hungry for more information.
How: Encourage people to ask questions… plenty of them and without boundaries. This takes hard work. It's easy to state that no question is a stupid question. It's much more difficult and challenging to respond in a manner that gets others to believe it. And give everyone access to the information and knowledge they are looking for.
Element #2: Courage – A fear of not being competent is a great barrier to innovation. Being creative takes courage – an ability to take risks and act without inhibition. Being open and expressive is what creativity is all about, requiring one to overcome the fear of being humiliated, seen as incompetent, or feeling stupid or shamed. It is courage that manifests as the resilience we admire in those that pursue new paths and ideas and feeds innovation.
How: Ask others to see every idea as a possibility and potential opportunity. You'll only succeed in getting others to do so by doing it yourself.
Element #3: Collaboration – Most great ideas and innovations are not the result of someone working singlehandedly. Nor are they the result of one single thought. They are the result of building one thought and idea upon another, requiring people to collaborate with one another. And no great team or organization succeeds in being innovative without members that are able to collaborate with one another. People naturally want to be inclusive and be a part of something bigger than any one person. At the heart of collaboration is the ability for us to be open to the ideas and thoughts of others. Innovation requires us to listen to one another.
How: It's important to bring people together and create environments that encourage the participation that leads to collaboration and teaming. In some instances, it becomes necessary to teach them the meaning of collaboration and train them in the communication and interpersonal skills they need to succeed. Listening is a necessary skill that must often be trained. At the very least it must be role modeled and reinforced.
Element #4: Competency – There is always a level of knowledge and skills required to be successful. One must know how to how to tune a guitar and have the basic skills of strumming and fingering the fret board to turn out a tune. Let alone write a song. The more we endeavor toward levels of excellence and mastery of competency and know-how, the more we begin to see new levels of possibility in our performance.
How: Give people the opportunity to learn and build the competencies that provide the foundation of the skills and knowledge to be successful and competently take the risks you're asking them to engage in. And, don't be afraid to ask people what competencies they need to be successful.
Element #5: Conflict – In the end, conflict is the most important of the five elements. For conflict is the spark and source of the power that drives innovation. At its most primal level, conflict is the natural tension and creative energy that exists between what is and what we want… the current state and a desired future. As much as necessity is the mother of invention, conflict is the engine of innovation. Embracing and using conflict as the natural source of innovation it is, allows us to leverage it constructively and not fall prey to our fears.
Conflict is the real magic in the alchemy of innovation. While once in while you can get away with not having one of the other four elements. The same cannot be said about conflict. Avoid conflict, remove the tension, and you deaden the creative spirit. You kill innovation.
How: Take the time to engage others in defining the nature of conflict. It is the natural tension that acts as the engine that drives our human desire to explore and discover new ideas, and is the powerful force behind the innovation we crave. Needless to say, at times this belief is not easily conveyed to others. As natural as the relationship of conflict and innovation is, it doesn't make it any easier for people to let go of their fears and embrace it. Getting people to embrace this thinking is the job of the leader.
You can read my new book "The Elephant in the Boardroom" in Safari which includes topics such as creating cultures of innovation, how to deal with conflict, and leading with intention.]]>
Steve Hoberman is the author of the recently released "Data Model Scorecard: Applying the Industry Standard on Data Model Quality" which is published by Technics Publications. He also teaches the "Data Modeling Master Class" which is a leading data modeling course in the industry. He is the founder of the Design Challenges group, Conference Chair of the Data Modeling Zone conference, and recipient of the 2012 Data Administration Management Association (DAMA) International Professional Achievement Award.
For many years I have been reviewing data models. A data model is a diagram (along with supporting documentation) that describes business terms (such as Order and Product) and the relationships between these terms (such as a Product may appear on many Orders). It is so important to get the data model right because the data model is the foundation for the database and development work. Skip or misrepresent a requirement on the data model and be prepared to watch the building crumble shortly after the ribbon cutting ceremony for the shiny new application. Correctly model all of the requirements on the other hand, and be prepared for application success and longevity.
I created the Data Model Scorecard to proactively determine how "good" a data model, and therefore improve the structures before the database and code work have begun. Here is the template I complete for each model review:
Total score
Model score
How well does the model capture the requirements?
How complete is the model?
How well does the model match its scheme?
How structurally sound is the model?
How well does the model leverage generic structures?
How well does the model follow naming standards?
How well has the model been arranged for readability?
How good are the definitions?
How consistent is the model with the enterprise?
How well does the metadata match the data?
Each of the ten categories has a total score that relates to the value your organization places on the question. Because we want to express the result as a percentage, the total must equal 100. The model score column contains the results of a particular model review. For example, if a model received 10 on Category 1 ("How well does the model capture the requirements?"), then that is what would go in this column. The % column stores the model score in category divided by the total score in category. For example, receiving 10 out of 15 would lead to 66%. The comments column contains any pertinent information to explain the score in more details or to capture the action items required to fix the model. The last row contains the total score, tallied up for each of the columns to arrive at an overall score for a particular model review.
Here is a summary of each of these ten categories:
How well does the model capture the requirements? Here we ensure that the data model represents the requirements. If there is a requirement to capture order information, in this category we check the model to make sure it captures order information. If there is a requirement to view Student Count by Semester and Major, in this category we make sure the data model supports this query.
How complete is the model? Here completeness means two things: completeness of requirements and completeness of metadata. Completeness of requirements means that each requirement that has been requested appears on the model (note we don't check for whether the requirement has been modeled correctly, which is the first category), as well as that the data model only contains what is being asked for and nothing extra. Completeness of metadata means that all of the descriptive information surrounding the model is present as well; for example, if we are reviewing a physical data model, we would expect formatting and nullability to appear on the data model.
How well does the model match its scheme? Here we ensure that the model type of the model being reviewed matches the definition for this type of model. The model type includes the detail of the model (conceptual, logical, or physical) as well as the perspective of the model (relational, dimensional, or NoSQL). The conceptual data model defines the scope and captures the business need, the logical is technology-independent and represents a business solution, and the physical is technology-dependent and tuned for performance, security, and development tool constraints, capturing the technical solution. The relational perspective captures business rules, the dimensional perspective captures business questions, and the NoSQL perspective captures how non-RDBMS technologies store data such as in a document or graph.
How structurally sound is the model? Here we validate the design practices employed to build the model. For example, a null primary key would be corrected to be required instead of optional.
How well does the model leverage generic structures? Here we confirm an appropriate use of abstraction. Going from Customer Location to a more generic Location, for example, allows the design to more easily handle other types of locations such as warehouses and distribution centers. However, it comes with the high price of obscurity and difficulty of enforcing rules. I would expect applications which receive data from many sources and send data to many targets to be good candidates for generic structures, such as a data warehouse or integration hub.
How well does the model follow naming standards? Here we ensure correct and consistent naming standards have been applied to the data model. We focus on naming standard structure, term, and style.
How well has the model been arranged for readability? Here we ensure the data model is easy to read. Placing parent entities above their child entities, displaying related entities together, and minimizing relationship line length all improve model readability.
How good are the definitions? Here we ensure the definitions are clear, complete, and correct.
How consistent is the model with the enterprise? Here we ensure the structures on the data model are represented in a broad and consistent context, so that one set of terminology and rules can be spoken in the organization.
How well does the metadata match the data? Here we confirm the model and the actual data that will be stored within the resulting structures are consistent with each other. Does the column Customer_Last_Name really contain the customer's last name, for example?
You can learn more about these methods in my brand new book called "Data Model Scorecard: Applying the Industry Standard on Data Model Quality." Topics include how to measure model consistency, how to validate your models with the scorecard, and conceptual data model adherence.]]>
In this post, we'll cover the mechanics of parsing more parts of a real log file followed by the process of reading log files then applying our parser to the input. Finally we'll discuss some ways we can improve our parser's performance. The ultimate aim here is to show readers how to build a parsing utility that can work on NCSA Common and NCSA Combined log formats. Although this post will include code snippets, we are also providing readers with a fully-functional reference implementation.
Covering every single aspect of building a fully-functional parser could take far more than two blog posts, so this code is provided to help readers understand all the things that go into making a parser. This code is provided under the BSD3 open source license with the hope that readers will feel comfortable forking and modifying this code to meet their needs.
When we last left off, we were able to parse log files that looked like this:
This simple format included just an HTTP method and an HTTP status code. Let's compare that to a full NCSA common log line:
126.96.36.199 - - [12/Jan/2015:20:37:55 +0000] "GET index.htm HTTP/1.0" 200 215
The space-delimited list of tokens corresponds to these values:
Here is a NCSA extended log line:
188.8.131.52 - - [12/Jan/2015:20:37:55 +0000] "GET index.htm HTTP/1.0" 200 215 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)"
This is the same format, with two added values at the end:
We've already shown how to parse the request method and status code in part one; let's work on filling in some of the gaps, starting with IP address.
For the sake of brevity, we'll assume we are only dealing with IPv4 addresses. These addresses are represented in the log by four decimal numbers, each ranging from 0 to 255, separated by periods. Each part represents a group of 8 bits (an octet) of the address. In Haskell, a useful strategy is to think about the problem in terms of types and reason back from there. How might we represent an IP address using Haskell's type system? We could just store it as a string, but if we wanted to answer a question like "how many users in this log came from a particular subnet?" we'd have to resort to string searching to answer the question. Here's a more flexible way to represent IP addresses:
data IP = IP Word8 Word8 Word8 Word8 deriving (Eq)
Here an IP address is represented as a sequence of four
Word8 values. A Word is an 8-bit unsigned integer type. How can we parse a string like "188.8.131.52" into this IP type? Here's one approach:
-- |Parse a period character. period :: Parser Char period = char '.' parseIP :: Parser IP parseIP = fmap IP (decimal <* period) <*> (decimal <* period) <*> (decimal <* period) <*> decimal
These functions use two attoparsec parsers,
decimal and
char, to handle parsing an IP address. We begin by defining a utility function that we will end up using in a number of places. Sometimes we need to create a parser that will consume a single predefined character that we expect to encounter in a log line. Let's say we wanted one parser for a dash and one for a period. Attoparsec provides a useful function,
char, that will match a single specific character and fail on any other input:
period = char '.' dash = char '-'
In an IP address we expect to encounter four numbers separated by three periods. If we were to represent this as a sequence of parsers, it would be:
decimal -> period -> decimal -> period -> decimal -> period -> decimal
However simply applying these parsers in sequence isn't enough. We want to keep the value parsed by
decimal and discard the "." parsed by
period. The
parseIP function shows how we can use attoparsec's parser combinators to achieve this in an elegant way. The
<* combinator sequences two actions, discarding the value of the second argument. So the expression
decimal <* period produces a single function that will expect to parse a digit and then a period from the input, but will throw out the result of the
period action (which is just the value
".") while returning the decimal value. Thinking in these terms, we need to have a sequence of three
decimal <* period and a single
decimal. The sequential application function
<*> allows us to apply these 4 parsers one after another, giving us 4 values of the type
Parser Word8. We want to construct an IP address from these 4
Word8 values, but since they're inside a
Parser monad, we use
fmap to apply the IP constructor to the values inside. This same function could also be written using do-notation, which might be easier to read for Haskell newcomers:
parseIP :: Parser IP parseIP = do a <- decimal <* period b <- decimal <* period c <- decimal <* period d <- decimal return IP a b c d
We'll leave parsing of the user identifier and name as an exercise for the reader. If you're curious about one possible solution, you can look in the reference implementation.
In this case we already have a Haskell type that we should use to represent this value:
ZonedTime, which stores a local time together with a
TimeZone. The question is, how do we go from a string like
[12/Jan/2015:20:37:55 +0000] to a
ZonedTime value?
It's relatively straightforward to tease out the individual pieces of the date we want:
parseTimeAndDate :: Parser ZonedTime parseTimeAndDate = do _ <- char '[' day <- parseDigits 2 _ <- slash month <- take 3 _ <- slash year <- parseDigits 4 _ <- char ':' hour <- parseDigits 2 _ <- char ':' minute <- parseDigits 2 _ <- char ':' second <- parseDigits 2 _ <- char ' ' timezone <- count 3 letter_ascii _ <- char ']'
However, there are two complications. For the month, we have a value like "Apr" when what we really need is the month number,
4. Similarly, we need to parse the ISO-8601 timezone string (e.g. -0700, +05, 09) into a
TimeZone object. Let's start by tackling the month problem. We have a head start in the
System.Locale library: it includes a
TimeLocale type that includes a listing of full and abbreviated months in calendar order:
import System.Locale show (months defaultTimeLocale)
Which prints:
[("January","Jan"),("February","Feb"),("March","Mar"),("April","Apr"),("May","May"), ("June","Jun"),("July","Jul"),("August","Aug"),("September","Sep"),("October","Oct"), ("November","Nov"),("December","Dec")]
We don't care about the full month names, we only want the second item in the tuple, the abbreviation:
map snd (months defaultTimeLocale) -- returns: ["Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"]
Let's modify this a bit to lower-case the month names so our parser can be case-insensitive. The
Data.Char library has a function
toLower that converts a single character to lower-case. Since a string is just a list of characters,
map toLower effectively converts an entire string to lowercase by mapping that function over every character in the
String. We can compose this function with our
snd function when we map it over the list of month tuples:
map (map toLower . snd) (months defaultTimeLocale) -- returns: ["jan","feb","mar","apr","may","jun","jul","aug","sep","oct","nov","dec"]
How might we use this to convert from a month name to a number (e.g. from "feb" to 2)? This seems like a good place for a map, a data structure that goes from key (month abbreviation) to value (month number). The
Data.Map library includes just such a type. It also includes a convenience function called
fromList that takes a list of key/value tuple pairs. So if we can give it a list that looks like this:
[("jan", 1), "feb", 2),...]
…then we can build a mapping from month abbreviation to number. We already have the list of month abbreviations, we just need to
zip it up with a list of numbers:
let monthNames = map ((map toLower) . snd) (months defaultTimeLocale) let monthMappingList = zip monthNames [1..]
[1..] list is just an infinite sequence starting at one. One of the advantages of Haskell's lazy iteration is that this won't cause any problems. The zip function will only evaluate the infinite list as far as
12, since that's how long the list of month abbreviations is. The
monthMappingList looks like
Out final
monthMap function is this:
-- | A map from three-letter month names (case insensitive) to their numeric representation monthMap :: Map.Map String Int monthMap = let monthNames = map ((map toLower) . snd) (months defaultTimeLocale) in Map.fromList $ zip monthNames [1..]
We can then wrap this with a
monthNameToNumber function which takes a
String representing the month abbreviation and returns a
Maybe Int:
-- | Attempt to convert a three-letter month name (case-insensitive) to its numeric representation. monthNameToNumber :: String -> Maybe Int monthNameToNumber monthName = Map.lookup normalizedMonthName monthMap where normalizedMonthName = map toLower monthName
This expression—
monthNameToNumber "Jun"—evaluates to
Just 6, whereas
monthNameToNumber "Baz" evaluates to
Nothing. Finally we can wrap this function into a parser,
-- |Parse a three-letter month name into its numeric representation. parseMonthName :: Parser Int parseMonthName = monthNameToNumber <$> count 3 letter_ascii >>= checkMonth where checkMonth m = maybe (fail "Invalid month name.") return m
We count three ASCII characters from the input, apply
monthNameToNumber to it and then pass that
Maybe to a function that returns a
fail if it got
Nothing and
Int if it got
Just Int.
Notice how we keep the pure part of the operation—the transformation of
"Oct" to
10—in its own function and the monadic parts—parsing the string and failing on an invalid month value—in their own function context. This makes testing much easier.
Fortunately time zone parsing is simpler. A timezone should be prepended by an optional "-" or "+" and have either 2 or 4 characters. Here are some examples:
"07": 7 hours ahead of GMT
"-07": 7 hours behind GMT
"+0700": 7 hours ahead of GMT
"-0730": 7.5 hours behind GMT
Here is a function which can parse a timezone string following those rules:
parseTZ :: Parser TimeZone parseTZ = minutesToTimeZone <$> signed parseTZDigits where parseTZDigits = liftA2 (+) ((* 60) <$> parseDigits 2) (parseDigits 2 <|> return 0)
We parse 2 digits, which represents our hours offset, then attempt to parse 2 more digits. If the parser encounters something other than a number for the second set, we assume that we're dealing with a two-digit and not a four-digit offset, and represent our minutes as
0. We multiply our hours value by
60 to convert to minutes, then add that to the minutes position. This gives us the total offset in minutes, and we can apply
minutesToTimeZone to that to get a
TimeZone object.
parseDigits function referenced above is a useful utility that can be used in several places in our log parser. It looks like this:
-- | Parse a given number of numeric characters into a Numeric representation (i.e. Int or Integer). -- `n` is the number of sequential numeric characters to parse. parseDigits :: (Read a, Num a, Eq a) => Int -> Parser a parseDigits n = (fst . head . readDec) <$> count n digit
Our final timestamp parser looks like this:
-- |Parse a date/time string into a LocalTime object from a -- string like "[10/Oct/2000:13:55:36 -0700]" parseTimeAndDate :: Parser ZonedTime parseTimeAndDate = do _ <- char '[' d <- parseDigits 2 _ <- slash mm <- parseMonthName _ <- slash y <- parseDigits 4 _ <- char ':' h <- parseDigits 2 _ <- char ':' m <- parseDigits 2 _ <- char ':' s <- parseDigits 2 _ <- char ' ' tz <- parseTZ _ <- char ']' let localTime = LocalTime { localDay = fromGregorian y mm d, localTimeOfDay = TimeOfDay h m s } return ZonedTime { zonedTimeToLocalTime = localTime, zonedTimeZone = tz }
From a parsing standpoint, the timestamp is our most difficult challenge. Therefore we'll defer to the reference implementation for ways to implement the rest of the log parsing.
Now that we have walked through parsing four types of data from a log file (ranging from simple to complex) and provided an implementation of the remainder, we can assemble this into a program that parses a whole log file.
Let's begin by defining a data type using record syntax to represent a parsed log line.
-- |A single log line from an NCSA Common or Extended-formatted log data LogEntry = LogEntry { ip :: IP, identity :: Maybe ByteString, userid :: Maybe ByteString, timestamp :: ZonedTime, method :: Maybe HTTPMethod, url :: URL, proto :: Maybe Protocol, protoVer :: ProtocolVersion, status :: Maybe Int, byteSize :: Int, referrer :: Maybe URL, userAgent :: Maybe ByteString, browser :: Maybe UAResult, platform :: Maybe OSResult } deriving (Show)
This type is a composite of types like
Int and
ByteString as well as some domain-specific types we've defined such as an IP address:
data IP = IP Word8 Word8 Word8 Word8 deriving (Eq)
…and HTTP method, protocol, and version:
data HTTPMethod = Get | Post | Put | Delete | Options | Head | Trace | Connect deriving (Show, Eq) data Protocol = HTTP | HTTPS | FTP deriving (Show, Eq) data ProtocolVersion = ProtocolVersion { majorVersion :: Int, minorVersion :: Int } deriving (Eq)
…and a URL, which is just a type-synonym for a bytestring:
URL = BC.ByteString
For parsing the user agent string, we've turned to an existing Haskell library called ua-parser which includes a pair domain-specific types of its own that model the family and major, minor, and patch versions of both the browser and operating system.
Based on the specifications for an NCSA log file, some information is not guaranteed to be present, including the user identity, referrer, and user agent. In cases like these, we use the
Maybe type to encapsulate an optional value.
Finally, we define a type to represent the result of a parsed logfile, which is nothing more than a list of
LogEntry objects:
Log = [LogEntry]
At this point we need to build a pipeline that goes from a file path for an existing log file all the way to a value of the type
Log. The reference implementation builds a command-line tool that takes a file path as its input; however, command-line argument parsing is beyond the scope of this post. Let's assume a hard-coded file path for simplicity's sake:
So what we want is a function that satisfies this type signature:
import Data.ByteString readLog :: FilePath -> [ByteString]
Note here that we're using
ByteString here rather than Haskell's default
String type. This is very important for speed and memory usage. In Haskell a
String is represented as a linked list of
Char values. For large volumes of string data, this is a very inefficient representation. By contrast a
ByteString is a list of chunks and the data is stored in continuous blocks of memory. You can read more about this distinction in Chapter 8 of the book Real World Haskell: "Efficient file processing, regular expressions, and file name matching"
Here's an implementation of the
readLog function:
import qualified Data.ByteString as B import qualified Data.ByteString.Char8 as BC readLog :: FilePath -> IO [B.ByteString] readLog path = B.readFile path >>= return . BC.lines
We read the file into a
ByteString then use the
lines function to split it into individual lines.
readLog ~/logs/access.log gives us a list of lines in that file. Now we need to parse them.
The following parser sequences all the smaller parsers we wrote for individual, space-separated pieces of the log line and returns a LogEntry value.
parseAsCommonLogLine :: Parser LogEntry parseAsCommonLogLine = fmap LogEntry parseIP <*> (space *> parseUserident) <*> (space *> parseUserident) <*> (space *> parseTimeAndDate) <*> (space *> quote *> parseHTTPMethod) <*> (space *> parseURL) <*> (space *> parseProtocol) <*> (slash *> parseProtocolVersion <* quote) <*> (space *> parseHTTPStatus) <*> (space *> parseByteSize) <*> return Nothing <*> return Nothing <*> return Nothing <*> return Nothing
This parser references some functions we've walked through building (like
parseTimeAndDate ,
parseHTTPMethod, and
parseHTTPStatus) and some that are defined in the reference implementation. As the name of the function suggests, this parses an NCSA common log-formatted line. Therefore the four referrer and user-agent components of our LogEntry
type are represented as
Nothing since these are not available in that log format.
Now we need to define a function that takes a
ByteString and returns either a
LogEntry if the parse succeeded, or an error string if the parse failed.
import Data.Attoparsec.ByteString (parseOnly) parseFileLine :: B.ByteString -> Either String LogEntry parseFileLine logLine = parseOnly parseAsCommonLogLine logLine
Attoparsec actually provides a function called
parseOnly that does exactly that. As we've done in several cases with our parser examples, we can rewrite this in point-free style:
parseFileLine :: B.ByteString -> Either String LogEntry parseFileLine = parseOnly parseAsCommonLogLine
parseOnly takes two arguments: a parser and a
ByteString. Haskell is perfectly comfortable with partial application. We can apply a function to one of its arguments and that will give us a new function that just takes the remaining arguments.
Now let's write a function that takes a list of lines from the log file and returns a fully parsed
import Data.Either (rights) parseFileLines :: [B.ByteString] -> Log parseFileLines logLines = rights (map parseFileLineStrict logLines)
We just
map the
parseFileLine function over a list of log lines, which gives a list of
[Either String LogEntry] It's up to you what you do with the errors in your list (which will be on the left side of each
Either). For simplicity's sake, we're just going to assume the log line was malformed and we'll toss those out. The
rights function takes a list of
Either and extracts all the Right elements in order. We can easily rewrite this in point-free style too:
parseFileLines :: [B.ByteString] -> Log parseFileLines = rights . map parseFileLineStrict
Now that we have our list of log entries, let's do something with it. The reference implementation outputs log data as a CSV file. However, walking through the implementation of that is beyond the scope of this blog post. We will implement something less useful but easier: let's say we just want to know the total number of bytes transmitted by our server in the period the log file covers:
sumBytes :: Log -> Int sumBytes log = sum (map byteSize log)
We just need to extract the bytes from each
LogEntry (which the
byteSize accessor does). Mapping this over all the
LogEntry types gives us a list of
Int values which we then sum. We can print it out at the end:
main :: IO () main = do logLines <- readLog "~/logs/access.log" let parsedLog = parseFileLines logLines let totalBytes = sumBytes parsedLog putStrLn ((show totalBytes) ++ " bytes returned.") return ()
If we compile this code and run it we notice two things.
Firstly, it's pretty fast. For an NCSA common log file of 1,400,800 lines, it takes around 1.9 seconds to finish. Let's compare this to a simple bash one-liner using awk:
awk 'BEGIN { FS = " " } ; { sum += $10; } END { print sum; }' "$@" < ~/logs/access.log
That takes around 3.9 seconds, so we've managed to be twice as fast as awk, which is pretty good for a first try. (Of course if all you're doing is summing up the number of bytes in a log file, you're still better-off sticking with awk in the first place.)
Secondly, the parser uses a little over 81MB of memory to parse the log file. This seems high, and when we look at the size of the input log file (79MB), we might begin to suspect that our program is reading the whole file into memory before processing it. Fortunately, we don't have to go on a hunch; the GHC compiler has great tools for profiling. Some compilation flags will allow us to generate a graph of memory allocation on the heap over the life of the program. In this case the flags
-with-rtsopts=-N -p -s -hy -i0.1 will break down the graph by type. When we do this we see a graph like this:
The good news is that memory usage stays flat over time. The bad news is that pretty much all of our memory is allocated to the
ARR_WORDS type. This is the compiler's term for a
ByteArray, which is used to implement the
ByteString type. The memory consumption of our parser has a linear relationship with the size of the input file currently and we need to fix that. The obvious answer is to read the input file in small chunks and keep track of the total number of bytes seen as we go. In this case memory usage would be constant, and roughly equal to the size of the chunks we read in (plus a small amount of overhead for the rest of the program).
So far we've used the ByteString type to read in the file and feed it to the parser line-by-line. If we read the documentation for the ByteString library, we can see the problem right away:
Byte vectors are encoded as strict Word8 arrays of bytes, held in a ForeignPtr, and can be passed between C and Haskell with little effort.
The strictness of that type has caused our memory problems. We could try writing a function where we read in a little of the input at a time and feed it to the parser, but we actually don't have to because the ByteString library also has a lazy bytestring variant:
A key feature of lazy ByteStrings is the means to manipulate large or unbounded streams of data without requiring the entire sequence to be resident in memory.
That sounds exactly like what we're dealing with. The attoparsec library is (mostly) strict, which means it requires the
ByteString it parses to be fully read into memory. But we are only feeding it one line at a time. We can store our whole file as a list of lazy chunks and only read enough of the string into memory at a time to obtain a line, which we then feed to the parser. This means we really just have to change two functions and let Haskell do the rest:
readLog, which will now return a list of lazy bytestrings representing our file lines.
parseFileLine, which needs to convert the lazy bytestring into a strict one before attempting to Parse it.
readLog function is very easy to fix. We just swap strict versions of our functions with lazy ones (the
Data.ByteString.Lazy API basically parallels the strict
Data.ByteString API)
import qualified Data.ByteString.Lazy as BL import qualified Data.ByteString.Lazy.Char8 as BLC readLog :: FilePath -> IO [BL.ByteString] readLog path = BL.readFile path >>= return . BLC.lines
parseFileLine function requires just a little more work, since we have to convert from a lazy to a strict bytestring before we attempt to parse the line:
parseFileLine :: BL.ByteString -> Either String LogEntry parseFileLine logFileLine = parseOnly parseAsCommonLogLine strictLogFileLine where strictLogFileLine = (B.concat . BL.toChunks) logFileLine
If we recompile this and run it again, we see the new version is just as fast, but never uses more then 4MB of memory, no matter how much data we give it.
At this point, we have a parser with good performance and low memory usage. If we wanted to really to push harder on the "high performance" angle, there are still more things we could try:
These are beyond the introductory scope of this article, but will be of interest to those wanting to dive more deeply into Haskell.
Update: December 16, 2015. Execution times for informal Haskell vs. Awk comparison updated to reflect the performance of the 0.1.1 ncsa-logparse release.]]>
Already, over 80 pieces of content are available in Safari including two e-book compilations called "The Executive Guide to Strategy" and "Don't Blame Your Culture." You can also read from "The Thought Leader Interview" series including discussions with Eric Ries, the author of "The Lean Startup" and Robert Sutton, the author of "Hard Facts, Dangerous Half-Truths, and Total Nonsense" (which is also available in Safari).
Additionally, this list in Safari shows the array of content that s+b is bringing to Safari. In the coming weeks, we will be uploading many videos from s+b to our platform, as well, so stay tuned for more.]]>
Frankie Bagnardi is a senior front-end developer creating user experiences for various clients. In his free time, he answers questions on StackOverflow (FakeRainBrigand) and IRC (GreenJello), and enjoys small projects. He is a co-author of the brand new second edition of Developing a React Edge by Bleeding Edge Press.
React and the tools around it move fast. Each version comes with new features and drops old ones. You can read the official release notes that cover what has been released in React 0.14, but here in this post we're going to cover some changes in the ecosystem.
First, react-tools has been officially deprecated. This includes reactify, react-loader, gulp-react, the jsx command line tool, and many more. These were all built on a custom esprima build with jsx and some es6/es7 features added on.
The new recommendation is to use babel. Babel supports JSX, and full ES6 with much of ES7 preemptively. The main developer, Sebastian McKenzie, now works full time at Facebook on babel, and the community is creating plugins for it to do amazing things. For all of these variants mentioned above, replace 'react' with 'babel' in the name. You then create a file called <code>.babelrc</code> and put a configuration like this in it:
{ "stage":"1", "optional": [ "es7.classProperties", "es7.decorators" ] }
classProperties is the
class C { static defaultProps = {...} }, and decorators is the
@something. These are pretty stable and unlikely to change, but they aren't final so keep an eye on the spec changes if you use them.
There are several options to base your builds on (
require.js), but
webpack has emerged as the top choice for most people. It's incredibly powerful and uses the dependency tree for everything, instead of the traditional glob approach found in gulp/grunt.
It happens to have possibly the worst docs out of any project with its popularity, so I strongly recommend finding a boilerplate and using their webpack config as a base.
One such boilerplate is react-transform-boilerplate, which brings us to the next change.
Hot module replacement
React Hot Loader is deprecated in favor of react-transform-hmr. It's a babel plugin, which allows it to leverage the information that babel exposes, making it more reliable and flexible. It opens up a ton of possibilities for developer tooling, including these existing examples (quoted from babel-plugin-react-transform):
It sometimes feels like we spend too much time on tooling, but the tooling is what allows us to write modular, maintainable, well tested code. Plus, there are nice things like react-transform that entirely exist to reduce our frustration, which is a valuable resource that should be saved for the interesting problems.
I hope you find the React 0.14 changes that I've discussed in this post useful as you create your next app using this great framework.
My new book, "Developing a React Edge, Second Edition" is available as an Early Release title in Safari. The book covers React 0.14 and includes topics such as data flow, composing components, and animations.
"In an industry on the forefront of big data technologies, access to the right learning tools is mission-critical," says Acxiom's Head of Learning, Wes Hinesley. "We need continuous improvement to stay competitive, and Safari has demonstrated success in expedient product development, and a long history of bringing in publisher powerhouses that will help our teams innovate and grow."
Today's 21st century talent spotting practices require hyper-growth industries to reevaluate how they are looking at hiring, retaining, promoting and retaining employees. Learning potential, emotional intelligence, soft skills and leadership abilities all play critical roles in developing top-tier talent that accelerates a company's ability to remain competitive.
"We know that great workplaces invest in giving their people the opportunity to learn and improve the skills they need to do their best work, and promote a culture of sharing that knowledge," said Safari CEO, Andrew Savikas. "And we're proud to have been chosen by Acxiom for such a critical role in fueling that kind of learning culture."
Acxiom's technology teams have utilized Safari's technology rich continuous learning content since 2011, and now the organization plans to integrate the growing leadership and business resources into existing training and curriculum programs.
Leadership development teams at Acxiom are expecting to integrate Safari's growing business and leadership content into training programs in 2016, and will be working with Safari to provide insight into how future-forward leadership programs can best take advantage of Safari's robust library. Safari is expected to launch enterprise-wide January 2016.
Acxiom is an enterprise data, analytics and software-as-a-service Company that uniquely fuses trust, experience and scale to fuel data-driven results. For over 40 years, Acxiom has been an innovator in harnessing the most important sources and uses of data to strengthen connections between people, businesses and their partners. Utilizing a channel and media neutral approach, we leverage cutting-edge, data-oriented products and services to maximize customer value. Every week, Acxiom powers more than a trillion transactions that enable better living for people and better results for our 7,000+ global clients. For more information about Acxiom, visit Acxiom.com. Acxiom is a registered trademark of Acxiom Corporation.
Safari is the leading content delivery platform for technology and business learning resources with more than 40,000 books, video courses and articles from top industry experts and publishers. It delivers a personalized, mobile-friendly learning experience to help professionals advance their personal and professional development. Established in 2001, Safari is headquartered in Sebastopol, California, and is a wholly owned subsidiary of O'Reilly Media.
1. What was the genesis of the book? Why did you choose to write it? I've been an educator for pretty much all of my professional life while also working in industry and recently I had been teaching a number of corporate training courses on data science and R through UC Irvine Extension. I developed some educational content based on my experiences teaching and through feedback from my students at large companies like Toyota and Southern California Edison. Over time, I sensed some common threads in terms of how people were able to embrace the subject matter – like recurring questions and pain points. I decided to formalize the content in the form of a book since there were no suitable books out there that addressed the needs of professionals trying to transition into the field from other disciplines. There were a number of good books, but they were too advanced; they weren't a good launching pad.
2. Who will this book help to get into data science and machine learning? I found that many professionals from fields like finance, sales, marketing, product development and others, needed a way to jump-start their entry into data science using methodologies like machine learning. I would get questions through places like LinkedIn, Quora, Twitter, etc. from people from all over the world, and the central theme was the same, they needed a guide to the Data Science Process. I wrote the book to help these people
3. Why did you choose the R language for the book? R is the statistical environment I use in my consulting practice for my own data science projects so that's the language I know best. Plus, R is getting a lot of attention worldwide as the number one choice for data scientists. I'm a member of the R Meetup group in my home town of Los Angeles, and we have transplants from many countries – many have used R for years. R now has over 7,500 packages that extend the language which is one reason why R is so popular.
4. How did you decide on the organization of the book? Over the years, I've refined my own "Data Science Process" so I organized the book to align with the process I use to approach machine learning projects. It's worked well for me over the years. The book allows the reader to build up the beginnings of their own "data science toolbox" with techniques they can draw upon to solve specific needs. After reading the book, they'll want to add many techniques to their toolbox to match their needs for the specific problem domains in which they work. Over time, I hope they'll have a very broad toolbox that can address most needs.
5. Can the book be used independently or would it help to take a class alongside? Yes, I think the book can be a standalone guide to getting started in the field, that's how I wrote it. All you need is the book, the R code found in the book that's available from the publisher's website, and some open source software – R and RStudio. But I think using the book in conjunction with a course also is a good idea. In fact, I'm using the book as the required text for a new online course I'm teaching for UC Davis Extension in January 2016.
You can read all of Daniel's book inside of Safari. In it, you will learn about topics such as data munging, how to evaluate model performance, and clustering.]]>
As part of a continuing feature here at the Safari blog, we'd like to shine a spotlight on Seven Dimensions and their wide array of video titles.
Build Your Brand and Your Career by Eve Ash, Sadhana Smiles, Ben Walkenhorst, Peter Wallbridge
Learn practical and applicable advice on how to build and leverage your personal and professional brand including tips on public speaking and career resiliency.
Emotional Intelligence, Arrogance, and Humility by Eve Ash
These videos are set inside of the fictional "Cutting Edge Communication Corporation" and feature comedienne Erin Brown. Watch as the employees learn how to boost their emotional IQ and how to embrace humility in the workplace.
Be a Rock Star Leader and Drive Success by Eve Ash, Peter Wallbridge, Pete Williams
Entrepreneur Pete Williams and senior executive Peter Wallbridge talk to Eve Ash about what it takes to be an effective leader of a productive team – even in virtual settings.
Efficiency Champion: Planning and Scheduling by Eve Ash, Jan Durrans
In this 14-part video series, Eve Ash interviews senior vice president and chief of operations of BOFI Federal Bank, Jan Durrans, about strategies for optimal organization and achievement. Topics in this series include how to chunk projects, self-discipline, and task prioritization.
Overcome Fear, Unlock Potential by Eve Ash
In this two part series, you will learn how to overcome your fears and insecurities by learning how to face them and manage them. These videos are set inside of the fictionarl "Cutting Edge Communication Corporation" and feature comedienne Erin Brown.
You can also check out this list in Safari, which includes an offering of the most popular Seven Dimensions titles covering personal and professional development skills.
In this book, Bergman provides a framework for evaluating and considering the endless parade of emergent technological trends and paradigm shifts. Learn how to lead through change while identifying the potential disruptions likely to have the largest impact in your industry or field.
Critical thinking has never been more imprtant for a professional than in today's ever-evolving work environment. This new release from O'Reilly Media helps readers to think more deeply about causality including techniques on how to question claims, make better decisions, and measuring causality.
Collecting wisdom from interviews with over 100 prominent marketing leaders, author Drew Neisser has synthesized this collective wisdom into a valuable reference table for leaders and marketers. Ideas and concepts that have been elementalized include research, grassroots marketing, and using data.]]>
As part of a new regular feature here at the Safari blog, we'd like to shine a spotlight on Packt and their deep offering of content.
If you want to take your data to the next level then implementing machine learning into your workflow is essential. In this book, author Sebastian Raschka covers topics such as compressing data, model evaluation, and a roadmap for building machine learning systems.
Learn how to use the fastest web user interface library to build, maintain, and deploy user interfaces with React.js. With a hands-on and practical approach which includes worked examples and code samples for you to use, React.js Essentials shows you how to create your project's structure, how to build elements and components, and how to test applications.
If you want to learn Docker but don't know where to start, this brand new video series is the ideal course for you. With hands-on guidance, you will learn to master Docker commands, share a container, and build docker files.
In this recently released second edition, author Ben Frain covers new responsive hot topics such as Flexbox, CSS Level 4 selectors, and SVGs. He also provides a modern take on responsive standards such as media queries, responsive images, and typography.
With the growth in full-stack JavaScript applications, it's more important than ever to properly understand Javascript to allow you to build applications that can scale up and down to meet future demands. Mastering JavaScript Design Patterns shows you how to build a fully worked application that incorporates the design patterns you need to build more maintainable and functional applications with JavaScript.
You can also check out this new list in Safari which includes a selection of Packt's latest and more popular titles covering a wide-range of topics. The Safari blog also recently published "Exploring Python's New Asynchronous Statements and Expressions" by Daniel Arbuckle, another Packt author.]]>
Of the multitudes of enlightening sessions, Nuance's Director of Technical & Professional Learning, Cristin Crain, delivered one that illustrates the three steps Nuance's learning team has taken — and yours can, too — to drive business results. Nuance is a leading provider of voice and language solutions that transform the way people interact with devices and systems. To stay current and relevant, Nuance has to be innovative and always evolving. Its learning team supports this through:
The learning team must make decisions that serve the entire organization and move it forward, so choosing which priorities to tackle can be daunting. Here's how Nuance does it:
In order to stretch your team's ability to meet your organization's diverse and growing needs, design your learning team to function as an internal consulting and networking firm. Here are some guidelines Crain gave around creating a successful in-house consultancy:
When it comes to working with those outside your team:
If you're operating in a fiscally restrictive environment, finding ways to minimize costs is crucial. At the same time, you've got to provide the right resources – not only so employees can get their work done, but also for their personal development and satisfaction. Having opportunities and resources to develop their skills to stay ahead in a competitive marketplace is a valuable benefit for employees — and of course it helps the organization immensely. Here are some crucial questions for which you need answers:
With its highly technical, geographically diverse workforce Nuance's approach has to be very non-traditional. There are no face-to-face training classrooms for employees. In fact, aside from compliance training, less than five percent of its employee development offering includes internally generated asynchronous elective online learning courses.
As a hybrid approach to meeting some specific company needs, Nuance's learning team partners with subject matter experts to create custom tutorials using essential content on Safari. So while employees have on-demand access to the 33,000 books and videos on Safari's business and technology learning platform, they can also take advantage of tutorials developed for their specific learning needs covering topics such as Process Improvement, Managing Virtually, and Systems Thinking.
In wrapping up her session, Crain admitted that forging ahead is hard for any learning team — it requires intelligence, business acumen, and perhaps most important, learning to be comfortable outside our comfort zones. She reminded us to expect struggles and mistakes, and expect to learn… after all, we are all striving to build a learning organization.]]>
We reached out to Arthur with a few questions to hear his thoughts on how mathematics can impact your career and personal development.
What kind of reaction do you typically get from someone when you talk to them about your love of all things mathematical? How do you convince them to embrace these principles?
In my experience, you can motivate someone to learn mathematics through either relevance or elegance. Either they need to see that the math they are learning will be useful for solving problems that they care about now (not three math classes from now) or the pattern or logic is so beautiful that the applications don't matter. For example, music may have applications, but most people just enjoy it for its own beauty.
What are the benefits of mathematical fluency in today's world? What advantages does it offer a young professional or someone perhaps trying to switch careers later in life?
The world needs independent thinkers, and I think nothing does a better job of training you to think logically, critically, or creatively than mathematics. Certainly in this world when we are surrounded by data, we need people who can understand what all of these numbers are saying and not saying.
How should a professional or student include mathematics and mathematical thinking into their daily learning habits and goals? What are some of the more efficient ways to study and learn mathematics?
Early in the book, we focus on mental math strategies that will improve your general number sense. Then we move to algebra to improve your ability to think abstractly. The chapters on Proofs and Geometry help you think more logically, and the chapters on counting, Fibonacci numbers, infinity and the number 9 allow you to have more fun with the subject. The book is written for both math newcomers and more experienced mathematicians with indicators when it's perfectly fine to skip ahead. When you get stuck, sometimes the best thing to do is to keep going and then come back to the trickier material when you have more perspective and experience.
Recently, there has been an explosion in interest on the topics of data and data mining. Do you think that the role of mathematics will continue to grow in importance for professionals and students? Will industries, careers, and hobbies that did not involve much math before will absorb some of these principles moving forward?
We have been teaching pretty much the same math topics for the last 200 years and I agree that this should change. First of all, we have so much computing power at our disposal, that it is not as important as it used to be to be able to perform certain calculations by hand anymore. This gives us the chance to think more about concepts, ideas, and allow more time for data science in the curriculum.]]>
Starting from scratch allowed us to think about the overall structure and components of the application. We use Backbone in most of our javaScript so a Backbone + Marionette + Backbone.Radio application made sense. Marionette allowed us to simplify our application in a number of ways including; better binding between our models and views using Marionette Layout, Composite, Item and Collection Views, more reusable components using Marionette Behaviors, easier view rendering and cleanup with Marionette Regions, templateHelpers for teasing logic in our JavaScript templates.
We started by breaking the app into modules. This included a module for 'facets', which are the publishers, topics and author options you can check off in the UI to narrow your search. 'Suggest' for the autocomplete module. The main part of the app handled pagination and the templates, collection and views for rendering search results.
Each of these modules included their own collections, models and views. We kept these mostly isolated components that used local event messaging. We used 'controllers' for app-wide event messaging and managing their module components.
We broke our views down using Collection and Item Views. In Marionette, a CollectionView takes a Backbone Collection and renders the individual models data inside it's child ItemView's. This reduced a lot of the code involved in just rendering the UI and let us focus on the collection and model logic which is fairly involved for us. It also let us keep a lot of data outside of views and in models where they are easier to manage.
We put reusable components like analytics into Marionette Behaviors for mixing into our views. Passing an options hash into the initialize method allowed for a lot of versatility.
While Marionette helped the application become more modular and component-based, Backbone.Radio became the spine of communication for the app. Radio simplifies the Backbone.Events messaging system by allowing you to segment your messages into Channels and organize between Events (which are like notifications with no intention) and Requests (which ask for a very specific thing to occur). Our messaging bus became more focused and easier to trace through and understand complex behavior.
Using Backbone.Radio we created a single point in the application for handling http request and response for direct communication between our app and the microservice. Some partial code for that handler looks like this,
radio.channel('search').reply('doSearch', function (searchHash, appOptions) { var model = new QueryModel(searchHash); appOptions = appOptions || {}; model.save(model.toJSON(), _.extend(appOptions, { success: function success(model, response, options) { radio.channel('search').trigger('sync:success', model, response, _.extend(options, appOptions)); }, error: function error(model, response, xhr) { radio.channel('search').trigger('sync:error', model, response, xhr); } })); });
Another part of the app will issue a request to do a search while passing in a hash Object with search data.
radio.channel('search').request('doSearch', searchData, options);
The Backbone.Radio reply handler passes this search data to a Model that handles it and POST's it to the microservice. The success or error callbacks issue radio events for other parts of the application to listen to and build search results from.
Similarly, there is a central handler for rendering search results. A 'Controller' Object handles some of the core logic in the Views and also listens to 'sync:success' events in the 'search' Channel. On such an event, it collects all the data from the collections of facets and then issues a 'buildResults' request. The handler for that event builds and renders the UI from the response the microservice sends back, it looks like this,
radio.channel('search').reply('buildResults', function (model, response, options) { var resultsView = SearchResultsView.get(new DocumentCollection()), paginateView = new Pagination.View({ collection: new Pagination.Collection() }), totalsView = new ResultTotals.View({ model: new ResultTotals.Model({ 'total': response.total, 'page': response.page, 'no_results': _.isEmpty(response.results) }) }), queryUrl = model.getQueryUrl(); regions.addRegions({ "results_total": "#js-results-count-region", "results": "#js-results-region", "pagination": "#js-pagination-region" }, {'triggerAttach': false}); resultsView.collection.reset(response.results); regions.get("results_total").show(totalsView); regions.get("results").show(resultsView).$el.fadeTo("fast", 100); if (response.results.length) { paginateView.collection.reset(Pagination.digestHits(model, response, options)); if (paginateView.collection.length) { regions.get("pagination").reset(); regions.get("pagination").show(paginateView).$el.fadeTo("fast", 100); } } if (!options.router) { Backbone.history.navigate(queryUrl); } model.destroy(); });
A 'buildResults' request is issued from inside a handler listening to http success events,
radio.channel('search').request('buildResults', model, response, options);
Setting up the messaging bus like this allowed us to decouple our application following a 'mediator' type design pattern. Using Backbone.Radio, our modules didn't have to talk directly to one another and avoided tightly coupled bindings that are difficult to test and debug.
test("Using keyboard 'down' arrow should select the next item down from inside a dropdown that isn't hidden.", function () { var self = this; this.$list.find('li:eq(3)').addClass(FILTER_HIDE_CLASS); stop(); setTimeout(function () { // test moving to the first child self.$input.trigger($.Event('keyup', { keyCode: c.keys.DOWN_KEY, which: c.keys.DOWN_KEY })); ok(self.downSelectionSpy.calledOnce); ok(self.$list.find('li:eq(1)').hasClass(SELECTED_CLASS)); start(); }, 50); stop(); setTimeout(function () { // test moving to the second child self.$input.trigger($.Event('keyup', { keyCode: c.keys.DOWN_KEY, which: c.keys.DOWN_KEY })); ok(self.downSelectionSpy.calledTwice); ok(self.$list.find('li:eq(2)').hasClass(SELECTED_CLASS)); start(); }, 50); stop(); setTimeout(function () { // test moving to the fourth child because the third child is hidden self.$input.trigger($.Event('keyup', { keyCode: c.keys.DOWN_KEY, which: c.keys.DOWN_KEY })); ok(self.downSelectionSpy.calledThrice); ok(self.$list.find('li:eq(4)').hasClass(SELECTED_CLASS)); start(); }, 50); });
This was a big project and if I have any advice to offer for building a large JavaScript application in Backbone + Marionette its this;
One very specific way to think about this user-driven approach to development and deployment is in the context of the "Minimum Viable Product" (MVP) borrowed from The Lean Product Playbook. The flexibility that the MVP approach provides publishers with is invaluable. MVP allows fleshing out of your various roadmaps along the way rather than many months or years in advance which also allows for discarding of incorrect assumptions and rapid responses to sometimes surprising user feedback. In the context of platform development, this means that we identify the minimum set of features and/or functionality that will constitute the platform or solve a problem and we deliver that to publishers and their users as quickly as possible. We then pay close attention to usage data and direct feedback, and iterate quickly to provide an optimal solution.
Publishers are increasingly sensitive to the necessity of looking outside of the industry for trends that are shaping the global marketplace, not just the publishing segment. And this brings to mind a talk given by John T. Chambers, Executive Chairman of the Board of Cisco Systems, at CES 2015 this past January in which he said, "This is not a technology discussion…Even the most powerful company in the world knows they have to move dramatically or risk getting left behind."
We've all come to grips with the fact that publishers are no longer just content providers, but have become technologists, content consultants, and publishing service providers. This multidimensional role requires that decisions be driven not by perception of market need, but by supporting data that validates stated market needs. For this reason, users need to be integrated as a core and active part of business planning…now. And this brings us back to the concept of building a MVP which creates the framework for including users in building new platforms, sales and marketing initiatives, educational programs, editorial products, and much more.
Today, we find that publishers still, quite understandably, are focused on partnering with vendors to develop products with a complete feature set. Although it may not be possible to have a world where publishers and vendors can agree to start on a project(s) without a fixed budget or sense of the full feature set, there are some things that we can do to help create a more MVP-like collaboration, specifically:
Clearly there are some constraints when working with vendor partners, but if you keep these ideas in mind, you will find that you are able to be more nimble and can better focus your efforts.
For more on the ideas behind lean development and producing MVPs see The Lean Product Playbook.
My first contact with open-source was the same as it is for most people: using open-source projects. I'm too curious to stop at that. I wanted to know how these projects I've been using were written and which architecture and patterns they use. I wanted to debug its code so I could both use them better and learn to write my own software better. Open-source software allows me to legally look under the hood and read its source. In fact, it actively encourages doing so. It encourages people to contribute, implementing new features which matter to them, filing bugs, fixing bugs, writing documentation, helping other users, and so on.
That's exactly what I did almost ten years ago. I started learning and using a Java Web framework, Apache Tapestry, in my free time, and then pushing its use at the company I was working at that time. I fell in love. I routinely read different parts of its source code and learned an awful lot from it. At the same time, I started participating in the project's users mailing list. It's part of my nature to try to help by sharing what know with others (at one point I was a part-time university professor), so soon I was not just posting questions, I was also answering questions and joining discussions about the project. To date I've posted 6624 times.
I started being a known name in the Tapestry world (small though it is). It was noticed by the Tapestry team. In 2009, they invited me to be a committer despite having contributed no code yet. As defined by the ASF's internal process, there was a vote done by e-mail and I was approved. In the same year, I was also invited and approved to Tapestry's project management commitee. In 2015, I was nominated and approved to be an ASF member.
Participating in open-source projects, whether the project is yours or not, is a very good way of advertising your own work and skills. Participation can include code, documentation, discussions and help for other people in mailing lists, forums and question-and-answer sites like StackOverflow. It helps growing your professional network to beyond the people you know in person and expands it to the whole planet, creating more opportunities. I call my own open-source projects "runnable CVs".
This all led to . . .
In the beginning of 2010 I was without a job and almost broke after a failed startup attempt. Looking at the Tapestry's user mailing list, as usual, I noticed a job announcement post looking for a software engineer to work with Tapestry in Boston. I live in Belo Horizonte, Minas Gerais state, Brazil, with no plans to relocate, but I'd lose nothing but some minutes applying to work remotely, so that's what I did.
The job was posted by Dan Adams, who's also an Apache Tapestry committer and then one of the lead architects of PubFactory, at that time owned by iFactory and later acquired by Safari Books Online in 2013. This is not a coincidence: PubFactory uses Tapestry heavily in both its Web and business logic layers. I'd learn later that the only code contribution I had done to Tapestry for a long time, the URL rewriting support, is used by PubFactory itself. He answered my application asking for code samples. I provided him a link to my open-source projects in SourceForge (later moved to GitHub).
The interview was giving me access to PubFactory's code repository and issue tracker and giving me a small but real bug to fix. I had to set up my environment, including checking out the source, creating an SSH tunnel for the database, fixing an XSLT snippet and writing an automated test with Selenium. I had almost no previous experience with many of those technologies but they're all important for the work. In a field in which hiring is frequently done based on puzzles and other brain-teasers, the decision to hire me was based on work on a real project using the skills needed by it. It cannot get any more hands-on than that. I was approved and I've been part of the PubFactory team since then, having worked on projects for almost all PubFactory customers so far.
This also shows how much trust Dan had in me for knowing me from the Tapestry community. Some of my teammates got really surprised when they learned how my job interview was carried out. This trust was also crucial for . . .
I've been working as a remote contractor in the PubFactory team since March 2010. Any contractor should know that the amount of work and earnings aren't constant and you should be prepared for that. In 2012, I was informed that I'd have a very light load of work to the end of that year.
I always wanted to contribute more to the Tapestry code, but I could only do that in my scarce free time. I had just learned my free time wouldn't be scarce for some time, so I had the idea of creating a crowdfunding campaign so I could work on Tapestry for one month (160 hours) instead of trying to find paid work to pay my bills. After asking the ASF community to check whether it wouldn't break any Foundation bylaws and principles and receiving lots of very useful feedback, the A Month of Apache Tapestry 5 IndieGogo campaign was born.
To be honest, I didn't have much hope to get the funding goal met. The deadline was at around 1AM. Before having dinner, few hours before the deadline, I checked the campaign funding progress and it was still far from the goal. I told Cristilene, my wife, the campaign wouldn't be a success, but it was worth the effort because of all the positive feedback I got from the Tapestry community. A couple hours later, when I again checked the campaign status, I got a happy surprise: It had reached its goal! In the meantime, people in the mailing list donated and rallied others to do the same.
I believe the funding campaign success was the consequence of a couple factors. One was a very clear description of what I'd do, in the form of a list of tasks, each one being a bug, improvement, or new feature, so people could find one or more wishes to be satisfied. The other one, the most important, was a sense and opportunity of contribution, of giving back to someone who helped a project and its community. The campaign had perks including free hours, but few were redeemed. I'm very flattered and honored by the outcomes of this crowdsourcing campaign. I'm very grateful to everyone who helped in a way or another.
In the end, I worked 182 hours instead of 160 because an old pet peeve of mine, TAP5-2029 (Copy annotations from service implementation to proxy) took 8 attempts to get it working and the numerous corner cases covered. But I don't mind. I had a lot of fun. I hope I have another chance to spend a month working on the open-source project I love.
I've just made a summary of my life with open-source, so now I'd like to see a summary of your open-source experience in the comments. If you don't have it yet, I urge you to experience it. Find the open-source projects that cover subjects in which you're interested, learn them, help with documentation (which is an overlooked but very important aspect of any software), discuss them with other people around the world, read the source, find bugs, create patches, be a committer, and someday create your own projects. In other words, get involved. The ASF pages about this are a very good starting point, even for non-ASF projects. And, of course, Safari Books Online has an abundance of books and videos about open-source software.]]>
Transitioning to HTML as our canonical content source format immediately allowed us to realize many benefits, including Web-based authoring and a digital-first approach to next-gen ebook development. Building print templates in CSS also proved to be surprisingly straightforward, once we got up to speed on Paged Media, and the particular dialect of it spoken by our PDF formatter software. We were able to quickly implement CSS templates for a wide variety of book series interior designs:
However, as we completed a critical mass of templates and moved from the proof-of-concept phase to production, CSS maintenance considerations rose to the fore. In our initial rollout of Paged Media designs, we built one set of core CSS files that contained a complete implementation of the styling for all standard book elements (headers, footers, chapter openers, tables, figures, etc.) for one of our simplest and most popular interior designs. CSS for all other interiors designs then
@imported this core.css and performed selective overrides to adjust styling as appropriate.
This approach served us well for a while, but as we added more and more templates, the monolithic nature of our CSS architecture became a liability. We had amassed a mountain of customizations that were piled on top of—and somewhat precariously balanced on—the style base. For any given template, it required effort and care to distinguish the aspects of the design that had dependencies on the base versus those that were overrides of the base. Changes to core.css were thus especially fraught, as they could have unanticipated ripple effects on numerous designs.
With each new template, the compromise here felt increasingly unpleasant, so we swung the pendulum in the other direction and rearchitected our CSS to take the opposite approach: eliminate core.css entirely, and implement a distinct, independent set of CSS stylesheets for each interior design.
Now, happily, each design was completely siloed. There was zero chance of code changes to one template having side effects on another. But sadly, each design was completely siloed. The common CSS that designs formerly shared was now replicated in multiple independent codebases, which made global updates a tedious, error-prone chore.
It was clear we needed a more nuanced, middle-ground approach to CSS development. Stepping back a bit to look at the big picture, these were the goals we wanted to accomplish:
While our overarching aim was to make our print-design workflows more efficient, none of the above challenges was at all print-specific, which led us toward exploring CSS maintenance solutions adopted by the greater web design community. Inspired by Lea Verou, who designed the print stylesheets for CSS Secrets using Sass, we decided to explore the Sass language to see if we could leverage CSS preprocessing to tackle the above challenges.
Over the course of several months of experimentation, we found that we were indeed able to apply many Sass language features within a print-design context to achieve our goals of streamlining template development and maintenance. Here's a look at some key principles we put into practice when refactoring our stylesheets using Sass. (Note: all examples in the following sections use Sass's SCSS syntax, which is a superset of CSS syntax.)
Our first objective in refactoring our stylesheets with Sass was to eliminate duplicate code across our templates. Sass's
@mixin functionality was a great solution to this problem. Mixins allow stylesheet developers to define modular styling blocks, which can contain a set of properties and/or full rules, e.g.:
@mixin heading-style { font-family: Arial, sans-serif; font-weight: 600; text-transform: capitalize; }
that can then be inserted via
@include statements:
h1, h2, h3 { @include heading-style; }
When the above Sass is compiled to CSS, this is the result:
h1, h2, h3 { font-family: Arial, sans-serif; font-weight: 600; text-transform: capitalize; }
Mixins are far more flexible than standard CSS
@imports, because they can be interpolated anywhere within a stylesheet, as many times as needed. This makes it possible to create a library of frequently used styling modules that can be dropped in whenever and wherever appropriate.
We did an audit of our stylesheet corpus and identified many such modules that were being replicated verbatim in every template. These styling blocks were clearly globally applicable to a wide variety of designs, and encompassed handling ranging from hyphenation rules and PDF bookmark configuration to syntax highlighting for code listings and MathML equation handling. We removed all this CSS from individual template files and placed it into a new mixins library that was accessible to every theme:
@mixin hyphenation-rules { hyphens: auto; hyphenate-before: 3; hyphenate-after: 3; -ah-hyphenate-hyphenated-word: false; -ah-hyphenate-lines: 2; } @mixin pdf-bookmarks { figure[data-type="cover"] { bookmark-level: 1; bookmark-state: closed; bookmark-label: "Cover"; } section[data-type="copyright-page"] { bookmark-level: 1; bookmark-state: closed; bookmark-label: "Copyright"; } } @mixin syntax-highlighting { /* Redacted for space; you get the idea... */ }
This solved our duplicate-code problem, while simultaneously modularizing shared code into manageable, clearly labeled blocks. We invoked mixins only when needed, at the exact locations they were needed, making it much easier to pinpoint dependencies in any given template and foresee the implications of changes to these shared resources.
Someday native CSS syntax for variables will be broadly implemented, but that day is not today. Until then, one of Sass's most compelling features is its support for variables. In Sass, variable names start with a
$ and are defined using the same syntax as properties:
$my-favorite-color: #008700;
Somewhat ironically, we found Sass variables immensely beneficial when employed as constants, allowing us to attach human-friendly names to otherwise opaque Unicode and color values.
For example, take the following snippet:
blockquote p.attribution::before { content: "\2014"; }
Unless you have the Unicode codepoint memorized, it's likely not readily apparent that the character being inserted before
p.attribution elements (U+2014) is an em dash.
By contrast, if the following variable is defined in the Sass stylesheet:
$em-dash: "\2014";
The above rule can now be rewritten as:
blockquote p.attribution::before { content: $em-dash; }
Which is much easier to grok at a glance, relieving much cognitive overhead for the beleaguered designer.
In addition to augmenting readability, Sass variables-as-constants also enhance stylesheet maintainability. Take the following example, where a variable for a company's logo background color is defined:
$logo-background-color: $trademarked-shade-of-crimson;
And then invoked potentially dozens of times throughout the stylesheet. Should the company change the logo background color from
$trademarked-shade-of-crimson to
$trademarked-shade-of-chartreuse, a single variable declaration can be updated, obviating the need for global find-and-replace.
Our Production team often needs to make template customizations on a book-by-book basis to perform minor design adjustments. So when we first rolled out our new CSS book templates, we knew that we'd need to build in the capability to override styling for individual books. Our original solution was to add support for a oneoff.css file that could be added to any book project. The oneoff.css
@imported the designated template CSS, and then overrode styling as appropriate.
This served us reasonably well, but the main shortcoming of this approach was that it required the Production team to be conversant with the base template CSS to successfully implement overrides. We did our best to lower overhead by including boilerplate snippets in the oneoff.css that could be toggled on/off as needed, e.g.:
/*---- Uncomment to turn on automatic code wrapping pre { white-space: pre-wrap; word-wrap: break-word; } ----*/
But we wanted to see if we could leverage Sass to implement a more elegant solution that eliminated the need to maintain that extra level of CSS. Perhaps instead of modifying styling via oneoff CSS override files, we could define parameters for frequently used design customizations (e.g., code-wrap=true) and pass them into our Sass stylesheets as variables?
Unlike Less, which offers a
--global-var option for passing variable values into stylesheets, the Sass preprocessor does not directly accept custom parameters. So we hacked the same functionality by replacing our oneoff.css with a _variables.scss partial file that defined only the custom design parameters (no styling) needed for a given book:
$trim-width: 6in; $trim-height: 9in; $code-wrapping: false; /* And so on... */
Then the template Sass stylesheets
@imported _variables.scss, incorporating its parameter values.
Here's an example showing this approach in action. The following
@page declaration uses the
$trim-width and
$trim-height values defined in _variables.scss to dynamically set PDF trim size:
@page { size: $trim-width $trim-height; }
And then the following SCSS employs Sass math operations to calculate
$figure-width based on
$trim-width, and set image
max-width accordingly:
$figure-width: $trim-width - 2.2in; img { max-width: $figure-width; }
As we built out our custom-parameter architecture, we realized that requiring users to configure a full set of parameters in _variables.scss for each individual book would likely be a real nuisance. To make our lives easier, we defined default values for all parameters in the template CSS using Sass's
!default flag:
$trim-width: 7in !default; $trim-height: 9.1875in !default;
Then we only needed to configure a parameter in _variables.scss when we wanted to override one of the defaults. Not only was this architecture much more efficient, but more extensible, too. If we needed to add new custom parameters in the future, we could populate them with default values in the template stylesheets, making it unnecessary to retroactively update all the _variables.scss files for legacy books.
When developing responsive stylesheets for the Web, the key tool in the designer's arsenal is the media query, which makes it possible to adapt styling based on features of the browser environment—width and height, orientation (portrait or landscape), resolution, and so on:
@media (min-width: 900px) { /* CSS HERE IS APPLIED ONLY IF VIEWPORT WIDTH EXCEEDS 900px */ }
Ever since our early days of print typesetting with CSS, we desperately wanted the ability to craft our own custom media queries for different "print book environments." For example, if a book were to be printed with a one-color (black-and-white) interior, we might want table header rows to be shaded gray, but if it were printed with a four-color (CMYK) interior, we might want table header rows shaded
Ideally, we wanted to be able to write queries like the following:
@media print and (print-book-color-count: 4) { th { background-color: $trademarked-shade-of-chartreuse } }
Sadly, the
print-book-color-count media feature did not exist in the W3C Media Queries spec or in our PDF formatter's documentation; it was viable only in our hearts.
But when we started exploring SassScript, we realized we had an equivalent solution at our fingertips. We could create the media features we wanted by defining custom parameters (see previous section), and then use
@else directives to craft our own queries.
To implement the above example, we defined the interior color count in our _variables.scss:
$print-book-color-count: 4;
And then added an
@else block to specify styling for each possible condition:
@if $print-book-color-count == 4 { th { background-color: $trademarked-shade-of-chartreuse; } /* More 4-color styling here */ } @else { th { background-color: gray; } /* More 1- or 2-color styling here */ }
With this DIY "media query" approach in place, our stylesheets could easily adapt to any combination of print-book specs. This allowed us to take what had formerly been multiple distinct templates — "trade book," "trade book four-color," "trade book small trim," "trade book four-color small trim"—and collapse them into one "smart" parameterized template ("trade book"). Huge boon for maintainability!
By opening up our stylesheets to accept user-supplied values for parameters like
$print-book-color-count, we knew we were also introducing the possibility of unexpected inputs, e.g.:
$print-book-color-count: "pizza!";
To address this, we wanted to add error handling to our stylesheets to validate parameter values and raise exceptions in cases where inputs did not meet template requirements.
One area where validation was essential was for our parameters defining the trim size for book pages:
$trim-width and
$trim-height. Many of our templates were intended to be compatible with only a handful of whitelisted trims. For example, one template was designed to support a trim of either 6in×9in or 7in×9.1875in, and we wanted a stylesheet error to be thrown if
$trim-width and
$trim-height did not conform to one of these specifications.
SassScript provides two directives that can be used for exception handling:
@warn, which can be used to output warning messages to stderr, and
@error, which both outputs an error messages to stderr and terminates the Sass preprocessor.
To ensure preprocessing failed on bad trim inputs, we added the following SCSS to our template stylesheet:
@if not (($trim-width == 6in and $trim-height == 9in) or ($trim-width == 7in and $trim-height == 9.1875in)) { @error("Invalid trim size. Trims supported are 6in×9in or 7in×9.1875in") }
In other circumstances, bad input values did not necessitate terminating the preprocessor, as we could gracefully recover by using appropriate defaults. In these cases, we used
@warn to alert users to the fallback behavior:
@if $color-count > 4 { @warn("Invalid color count of #{$color-count}. Defaulting to maximum color count of 4"); $color-count: 4; }
@error and
@warn handling in place protected the integrity of our designs and provided salient feedback to users configuring them in inappropriate ways.
At time of publication, we have officially deployed and launched Sass support in our publishing toolchain, and are releasing our first set of Sass stylesheets into production. In the coming months, we'll be continuing to add new designs to our Sass stylesheet corpus, as well as iteratively improve existing templates. We're looking forward to sharing more best practices we learn along the way!]]>
By Luke Williams
Luke Williams is the executive director of entrepreneurship and founder of the W.R. Berkley Innovation Lab at New York University's Stern School of Business. He is the international bestselling author of the second edition of "Disrupt: Think the Unthinkable to Spark Transformation in Your Business."
Different organizations have different opinions about what kind of leadership behaviors will work best for them, and these opinions are often inherited in prepackaged form. In business, we call them best practices: borrowing a successful way of doing something from another company or industry. (If you've ever participated in a brainstorming session, you've seen first-hand how the concept of best practices keeps everyone in the room thinking the same way about a particular business or industry.)
At their core, best practices are no more than a proven way of allocating resources within the industry in which they operate. They're useful, but when you buy prepackaged best practices, you're buying into a zero-sum game: Putting things together in a certain way for one practice makes it difficult to put them together in a different way for a different practice. For much of the 20th century, the best practice many industries used was some variation of the "razor and blades" concept: If you hook your customers with a free or cheap product below cost (the razor handle), you can charge a lot more over time for the necessary add-ons to that product (the blades). The concept was set in stone, and a lot of leaders still accept it. "We make our money by making desktop computers as inexpensive as possible but getting users to spend big bucks on software licenses." Or "We make our money by selling our video game consoles for less than what they cost to manufacture, but we get $50 a piece for the games."
The problem is that many companies become so fixated on maintaining today's best practices that they don't see that those practices are actually decaying. There's no rational reason to look for a better way of doing something if what you're already doing works just fine. And things get worse with time. The longer we hold a set of assumptions about what success looks like, the more value we attribute to those assumptions and the less likely we are to give them up or change them.
The fast-food industry, which is explicitly designed to increase volume and reduce costs, is a great example. For most of the last quarter of the twentieth century, fast-food chains served enormous amounts of food quickly and cheaply by sourcing their ingredients from industrial suppliers, freezing the goods, and utilizing assembly-line production methods. The trade-off was that choice and freshness were quite limited. Then came the inevitable change in consumer tastes and preferences. The generation of Millennials and self-proclaimed foodies wanted fresh, organic, locally sourced ingredients and were willing to pay extra. Traditional fast-food chains were still doing a great business, but their conceptual competencies (i.e., concepts and practices they think are critical for success) of cheapness, convenience, and mass production were pretty much incompatible with the new emphasis on taste, freshness, and customization.
This led to the rise of restaurants known in the trade as "fast-casual" (think Panera Bread, Shake Shack, Nando's chicken restaurants, and Chipotle Mexican Grill) that bridge the gap between fast-food outlets and full-service restaurants. The average check is more than 40 percent higher than that of traditional fast-food, and the "sustainable-food" approach is enjoying success across the world. In 2014, America alone, combined sales of fast-casual outlets rose by 10 percent, compared with six percent for fast-food chains. And Chipotle, previously owned by McDonalds, is now opening a new restaurant almost every other day and enjoying 20 percent annual growth.
As this example shows, it's a mistake for any organization to believe that being successful at one point in time and staying on the same path will guarantee continued success. It's nearly impossible to capitalize on the opportunities brought about by changes in technology and consumer behavior by simply repeating what you've done before, no matter how successful you were. Although there's nothing wrong with maintaining the status quo when things are good, it's critical to understand that clinging to outdated concepts and practices makes it hard to change course when things (inevitably) go bad.
The second edition of my book, "Disrupt: Think the Unthinkable to Spark Transformation in Your Business" is now available for reading on Safari. Topics include pitching innovation, how to be a disruptive leader, and generating disruptive ideas.]]>
Docker Swarm is native clustering for Docker. It allows you create and access to a pool of Docker hosts using the full suite of Docker tools.
With the increased attention of containerization and microservices, Docker is an obvious choice for development and perhaps production. How can an Infrastructure team leverage shared machine resources and build something self-service for their awesome Engineering teams?
The answer may be Docker Swarm, which will allow you to build a cluster of Docker hosts that can each run many Docker containers, and can scale with your needs. In this post I will walk you through setting up a test environment for you to play with Docker Swarm.
If you have not already, then please download and install the Docker Toolbox.
A good run-through is the Get started with Docker Swarm page; I will mostly be following that, but with the addition of showing you how to set up service discovery at the same time.
Service discovery just means a central location for programs to set and get information about other programs in your environment. For example, if my program provides a service called foo on port 1337, then I can announce that to my discovery system. Any other program looking for the service foo will learn that it is located at my IP on port 1337.
For our purposes we will use consul.io for the service discovery part. From your command prompt issue the following commands:
docker-machine create --driver virtualbox consul1 eval $(docker-machine env consul1) docker pull progrium/consul docker run -d --name consul -p 8400:8400 -p 8500:8500 -p 8600:53/udp -h node1 progrium/consul -server -bootstrap -ui-dir /ui docker-machine ip consul1 192.168.99.101
With the first command we create a new virtual machine named consul1 on VirtualBox. This machine runs a docker daemon. The second command allows all the future docker commands to use our new consul1 machine. Next we pull a Docker image of the consul program from the Docker Hub, and run it. The options to our run command name the container, do some port mapping, container host naming, and pass options to the consul program, one of which is to start the web UI. (For more details on this image see the Docker Hub page for it: https://hub.docker.com/r/progrium/consul/.)
The last command tells us what IP address our new consul service will be listening on. We will need this for the rest of the setup.
You now should be able to point your browser here and see the web interface for consul: http://<Your Consul IP>:8500/ui/#/dc1/services
Next let's create our swarm, which will consist of one swarm-master and two swarm-agents. Each member of the swarm will use the consul service we setup to find each other.
docker-machine create --driver virtualbox --swarm --swarm-master --swarm-discovery consul://192.168.99.101:8500/ swarm-master docker-machine create --driver virtualbox --swarm --swarm-discovery consul://192.168.99.101:8500/ swarm1 docker-machine create --driver virtualbox --swarm --swarm-discovery consul://192.168.99.101:8500/ swarm2
Now you should have at least four docker machines. Note when you installed Docker Toolbox you probably created some other machines as well, so you should have something like this now:
➜ ~ docker-machine ls NAME ACTIVE DRIVER STATE URL SWARM consul1 * virtualbox Running tcp://192.168.99.101:2376 default - virtualbox Stopped dev - virtualbox Stopped swarm1 - virtualbox Running tcp://192.168.99.103:2376 swarm-master swarm2 - virtualbox Running tcp://192.168.99.104:2376 swarm-master swarm-master - virtualbox Running tcp://192.168.99.102:2376 swarm-master (master)
You can now see what consul has stored for these nodes by pointing your browser here: http://192.168.99.101:8500/ui/#/dc1/kv/docker/swarm/nodes/
or via curl on the command line like so:
➜ ~ curl http://192.168.99.101:8500/v1/kv/docker/swarm/nodes/\?recurse [{"CreateIndex":17,"ModifyIndex":75,"LockIndex":1,"Key":"docker/swarm/nodes/192.168.99.102:2376","Flags":0,"Value":"MTkyLjE2OC45OS4xMDI6MjM3Ng==","Session":"fef49de0-36cd-70b7-ef77-41a7207adb09"},{"CreateIndex":31,"ModifyIndex":74,"LockIndex":1,"Key":"docker/swarm/nodes/192.168.99.103:2376","Flags":0,"Value":"MTkyLjE2OC45OS4xMDM6MjM3Ng==","Session":"93b3c670-7908-d24f-5c1a-a61595213d06"},{"CreateIndex":48,"ModifyIndex":76,"LockIndex":1,"Key":"docker/swarm/nodes/192.168.99.104:2376","Flags":0,"Value":"MTkyLjE2OC45OS4xMDQ6MjM3Ng==","Session":"bff09510-0a3d-b5e8-53c2-4ef22a81d845"}]%
You can see we received JSON back. Consul also has a DNS interface for doing the same; see the Consul Getting Started documentation for more on how you can store and retrieve values from consul.
Let's check our Swarm by communicating with the Swarm Master:
eval "$(docker-machine env --swarm swarm-master)"
This command sets our terminal environment so our docker commands are issued against the swarm-master.To see what your Swarm looks like try the following:
➜ ~ docker info Containers: 4 Images: 3 Role: primary Strategy: spread Filters: health, port, dependency, affinity, constraint Nodes: 3 swarm1: 192.168.99.103:2376 └ Containers: 1 └ Reserved CPUs: 0 / 1 └ Reserved Memory: 0 B / 1.021 GiB └ Labels: executiondriver=native-0.2, kernelversion=4.1.12-boot2docker, operatingsystem=Boot2Docker 1.9.0 (TCL 6.4); master : 16e4a2a - Tue Nov 3 19:49:22 UTC 2015, provider=virtualbox, storagedriver=aufs swarm2: 192.168.99.104:2376 └ Containers: 1 └ Reserved CPUs: 0 / 1 └ Reserved Memory: 0 B / 1.021 GiB └ Labels: executiondriver=native-0.2, kernelversion=4.1.12-boot2docker, operatingsystem=Boot2Docker 1.9.0 (TCL 6.4); master : 16e4a2a - Tue Nov 3 19:49:22 UTC 2015, provider=virtualbox, storagedriver=aufs swarm-master: 192.168.99.102:2376 └ Containers: 2 └ Reserved CPUs: 0 / 1 └ Reserved Memory: 0 B / 1.021 GiB └ Labels: executiondriver=native-0.2, kernelversion=4.1.12-boot2docker, operatingsystem=Boot2Docker 1.9.0 (TCL 6.4); master : 16e4a2a - Tue Nov 3 19:49:22 UTC 2015, provider=virtualbox, storagedriver=aufs CPUs: 3 Total Memory: 3.064 GiB Name: 317580416dc0
You can see we have three members in our Swarm: swarm1, swarm2, and the swarm-master.
To see the docker images running in your swarm already:
➜ ~ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b5ea7a9847d0 swarm:latest "/swarm join --advert" 8 minutes ago Up 8 minutes 2375/tcp swarm2/swarm-agent 4245df61bc25 swarm:latest "/swarm join --advert" 10 minutes ago Up 10 minutes 2375/tcp swarm1/swarm-agent 3da358427d4b swarm:latest "/swarm join --advert" 13 minutes ago Up 13 minutes 2375/tcp swarm-master/swarm-agent 317580416dc0 swarm:latest "/swarm manage --tlsv" 13 minutes ago Up 13 minutes 2375/tcp, 192.168.99.102:3376->3376/tcp swarm-master/swarm-agent-master
To deploy a container to the Swarm all we do is issue a docker run:
➜ ~ docker run -d nginx 369d42cb00fd7b7e965f0516d78d6a10752b2300d99abf3d5edd546216747858 ➜ ~ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 369d42cb00fd nginx "nginx -g 'daemon off" 6 seconds ago Up 6 seconds 80/tcp, 443/tcp swarm1/kickass_bartik
You can see our nginx container ended up on swarm1.If we do this a few times then you will see your containers spread around:
➜ ~ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0c1331011f15 nginx "nginx -g 'daemon off" 23 seconds ago Up 22 seconds 80/tcp, 443/tcp swarm-master/gigantic_meitner c6a40a290abd nginx "nginx -g 'daemon off" About a minute ago Up About a minute 80/tcp, 443/tcp swarm2/sad_bardeen 369d42cb00fd nginx "nginx -g 'daemon off" 2 minutes ago Up 2 minutes 80/tcp, 443/tcp swarm1/kickass_bartik
You can use constraint filters to assign containers to certain swarm nodes. Try running the following three times:
docker run -d -e constraint:node==swarm1 nginx
As you can see all three containers ended up on the same machine:
➜ ~ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e66d2e1af3eb nginx "nginx -g 'daemon off" 2 seconds ago Up 1 seconds 80/tcp, 443/tcp swarm1/mad_lovelace baa1b65d2dcf nginx "nginx -g 'daemon off" 18 seconds ago Up 17 seconds 80/tcp, 443/tcp swarm1/agitated_gates 0032ba6a7b09 nginx "nginx -g 'daemon off" 25 seconds ago Up 24 seconds 80/tcp, 443/tcp swarm1/big_feynman
Registrator automatically registers and deregisters services for any Docker container by inspecting containers as they come online.
Registrator works by running on our Swarm nodes, listening to the Docker daemon socket, and reporting back to consul with the service information of containers that are running in the swarm. This will allow us to have a central location where we can view all the services available from our Swarm. We will start by running the registrator container on each of the members in the Swarm:
➜ ~ docker run -d -e constraint:node==swarm1 --net=host --volume=/var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest consul://192.168.99.101:8500/ 2fd784958ddfc92738fdacb73a2c540a7cfb8737524bb819dd2cc58a5a2ae820 ➜ ~ docker run -d -e constraint:node==swarm2 --net=host --volume=/var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest consul://192.168.99.101:8500/ 42f31106a6c94c1eaa6f2860632b2251158444e99f835d470803869a71c6f05e ➜ ~ docker run -d -e constraint:node==swarm-master --net=host --volume=/var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator:latest consul://192.168.99.101:8500/ 41614f616715f73a5b0db8ea6f4a187b1e19eba2df4b9026d373e369af05ba03 ➜ ~ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 41614f616715 gliderlabs/registrator:latest "/bin/registrator con" 3 seconds ago Up 3 seconds swarm-master/stupefied_fermi 42f31106a6c9 gliderlabs/registrator:latest "/bin/registrator con" 15 seconds ago Up 14 seconds swarm2/suspicious_hugle 2fd784958ddf gliderlabs/registrator:latest "/bin/registrator con" 31 seconds ago Up 30 seconds swarm1/jovial_carson
Now if we deploy our nginx container again and expose an external port, we will see the service appear in consul:
➜ ~ docker run -d -p 8080:80 nginx 01f1003542cad73359a5972eaf134330fbccfdb671e0f506feb7c0e3ff59f722 ➜ ~ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 01f1003542ca nginx "nginx -g 'daemon off" 2 seconds ago Up 1 seconds 443/tcp, 192.168.99.104:8080->80/tcp swarm2/serene_ramanujan
Using curl we can check for the catalog of services in consul:
➜ ~ curl 192.168.99.101:8500/v1/catalog/services {"consul":[],"nginx-80":[],"swarm-3376":[]}
If we remove the container, then we see the service is de-registered in consul:
➜ ~ docker rm -f swarm2/serene_ramanujan swarm2/serene_ramanujan ➜ ~ curl 192.168.99.101:8500/v1/catalog/services {"consul":[],"swarm-3376":[]}%
We can use Docker Swarm and service discover tools to build our own Infrastructure services, and getting started is as easy as launching a few virtual machines on your laptop. In addition to the links in this post here are some good reads to expand your knowledge about Docker:]]>
"Today, 34 percent of web users have installed some sort of ad blocker. Eighteen percent of tablet users have installed one, and almost a quarter of all mobile phone users—24 percent—have installed an ad blocker."
VentureBeat, November 2015
34% is a large percentage, and it's growing by the month!
Whatever your feelings on the ethics of ad blocking, widespread adoption of them is a game changer for everyone on the web. Ad networks and sites that rely on ad revenue (Safari does not) are getting the most attention, but site usability is a concern for web developers and customers alike. Even if you don't serve ads, ad blockers are throwing the practice of web analytics into a period of uncertainty and change.
Looking for some good news? I think those of us in the analytics space have gotten pretty lazy when it comes to how and what we measure. A little crisis is just what we need to rethink our practices and refocus on what matters. Tools like Google Analytics and Adobe Analytics have become so commonplace that the competitive advantage of relying on standard metrics has virtually disappeared. Sure, we all have to track visits, page views, conversions and so on, but doing so is a basic need rather than a way to get ahead.
There are two major challenges that ad blockers (and some forms of "private browsing") throw at traditional web analytics:
It's not hard to imagine a day in the near future where half of your customers will have an ad blocker installed. How many are going to whitelist GA tracking pixels on your site? Blocking third-party cookies has been on the rise for a number of years, and so most analytics packages are equipped to operate with a combination of 1st party cookies and tracking pixels. If those pixels are blocked, you'll be in the dark.
Most commercial websites want to track users from entry to exit in a few high-level scenarios:
In simpler terms:
That's not a pretty picture! While there has never been a perfect view of what's happening on your site, the value of third-party tracking is diminishing at an alarming rate. If you're in the business of making money on the web, the data about behavior on your web properties is a competitive advantage. You need to own it.
The first step is to collect the data generated by user events on your site. Common events you'll want to track include:
Start with your web server logs. For high-volume sites like Safari, those logs come from your Content Delivery Network (CDN). Most CDNs give you the capability to get your logs streamed to Amazon S3 or a local file share. If you don't use a CDN, you can pull the logs right from your web host(s).
Web server logs are a blessing and a curse. They typically give you lots of data, but not everything you need and plenty of information you don't. You will certainly get page loads, but may need to explicitly include first-party cookies in your configuration (we did). We also realized that we had a lot of client-side events that didn't show up in the web logs. That's a harder problem to solve, but thankfully we could still infer most actions with a combination of page view records plus API requests. As for data we don't always want? Let's just say that not every analyst wants to be bogged down with a record of each image request made on our site.
Raw log files don't do the business much good, so the next step is to get that useful data into a place where you can process it. We took a two-step approach. We knew the volume of data in our web logs would eventually make ingesting and processing it in a traditional relational database unrealistic, but in order to get off the ground quickly we began with a locally hosted Postgres database. My colleague, James Stevenson, wrote extensively about the process of ingesting the log data using Haskell, which has proven to be quite scalable and flexible, and critically, independent of the final host datastore.
The concept of a clickstream database is nothing new. The goal is to tie each record in the web log (the "click") to a visitor, and then be able analyze what that visitor does. It really gets interesting when you have thousands, or millions, of visitors! This is essentially what Google Analytics or Adobe Analytics do behind the scenes, but now you own the data.
Once the logs are parsed and living in a place where they can be queried, it's time to make sense of the information. This is where the volume of data can become problematic, and why we migrated our clickstream database from Postgres to Amazon Redshift. The main clickstream table contains a row for each log event and is both deep and wide. To make any sense of the data, you must run some pretty strenuous queries to do the following:
The first one is the hardest but it makes the other three possible. It's also the one that throws third party analytics tools for the biggest loop. Since we're collecting the data ourselves on the server side, we don't care if the visitor is blocking third party cookies and tracking pixels. We'd love to have them accept first party cookies, but it's not a total loss if they don't. Log events for visitors who never identify themselves by signing up and/or logging can be strung together either by first party cookies (if they are set) or a combination of cookies, IP address, user agent, timestamps and other log data. If a customer is accepting first party cookies, or even better is logged in, the task is far easier.
Once that's done, it's up to you to decide what matters to you. Visit attribution is a discipline of its own but now you've got the raw materials to get it right, even if your visitors are using an ad blocker. Because clickstream databases often live in something like Redshift or Hadoop, it's necessary to build tables that summarize key metrics and events and send those back to your traditional data warehouse. A common use case is to create a summary table of the first and last visit prior to a transaction for each visitor and use that much smaller table in your BI tools for reporting purposes.
One thing to remember is that you'll never attribute all of your clickstream records to visitors, and even fewer to actual customers. That's OK! It's not a perfect science, but if you do even a decent job you unlock the real power of your clickstream database: to empower your team to go beyond counting visits and basic visit attribution.
Ad blockers or not, having a clickstream database provides tremendous value for data scientists, analysts and engineers in your organization. Statistical and machine learning models are hungry for data, and what better place to get it than your clickstream database? Once you done the hard work of attributing "clicks" to visitors, you can segment the data in a seemingly infinite number of ways. That's great for your data science team as well as more traditional analytics in BI who can now write SQL queries to answer questions like, "How many visitors from Canada came to the site yesterday?", or "Which of my customers are heavy readers of my blog posts?" In addition, you now have the raw materials to build more advanced customer segments, perform path analysis, and more.
Engineers and IT often use raw web logs to monitor issues with the web site or to understand how heavy traffic is at certain times of the day. Giving them access to a place where they can query the raw data or predefined aggregate tables in SQL opens up a new world of possibilities for them.
I know, I just wrote all about the reasons you shouldn't trust third party tools like Google Analytics. Well guess what, they still provide a ton of value! Mostly likely you'll still be collecting enough data in those tools to get an idea of trends and an aggregate view of your business. Keep in mind that certain segments of your audience might be more likely to use ad blockers, thus skewing your data, but the value of quick access to numerous reports and dashboards that come form something like GA is still immense. Just keep the limitations in mind.
Those of us in Analytics can get caught up in the excitement of building something like a clickstream database, and the challenges of understanding who's doing what on our web sites. It's important to remember that there's a reason for the popularity of ad blocking. It's not just about blocking ads, but also driven by concerns about privacy. Tracking customers across domains for the purpose of advertising to them is not what we're interested in here. We care about OUR customers and how to optimize our product for them.
With that in mind, be thoughtful about what data you store in your clickstream database and what you do with it. We don't store any personal information about our customers in our clickstream database, not just because it would be creepy to do so but also because we don't need it. An analyst might want to know about how a segment of similar customers uses the site/product, but doesn't need to know who they are. Of course, if we did a good job in attributing clickstream records to customers who end up signing into the site, we link some key findings from the clickstream database to the limited customer information we store in our data warehouse. Just like anything else in Analytics and Marketing, think about the impact on the humans in that database before acting. Just because it's legal doesn't mean it's within the ethical bounds set by your employer, or even more importantly those you've set for yourself.
Kathryn Zonghetti is an empowerment strategist at Purposeful Challenges: Discover Your True Capabilities. You can subscribe to her YouTube channel here.
We all have a story that needs to be shared with the world. We are all fighting our own battle. The best way to face our own challenges and adversity is by being vulnerable and authentic with others.
"Your story inspired me to be myself." These are the powerful words that were said to me when I shared my vulnerabilities and story with the world. The fear of sharing my true self with others kept me from inspiring others to do the same for many years. I chose to share my message through public speaking and social media.
Public Speaking
Several years ago I had a fear of public speaking. I decided to join Toastmasters, a club for practicing public speaking. But I was terrified to set foot in it for the first time. I was motivated to enhance my public speaking skills because I believed that I had a message to share with the world. I believed that my story would resonate with others. But, I didn't want to be judged by others. What if I stumbled on my words? What if I turned bright red? What if I tripped? Many thoughts ran through my mind before joining this public speaking club. I took a deep breath and made it happen. This offered me the ability to learn from my mistakes in a safe and judgment free environment.
Social Media
Social media is the perfect way to share your insights and stories with the public. From LinkedIn, Facebook, Twitter, to Tumblr, Pinterest, and Instagram. We have the opportunity to express ourselves and share our message. The goal is to focus on what motivates YOU, not what you think would motivate everyone else. This way, if the external praise subsides during some points (which it will), you will continue to keep going and not give up.
After years of experience in the personal and professional development industry, it always seems to come back to 3 things that hold us back:
1) Self Doubt
To overcome our own self-doubt is true success. In today's world, it's easy to compare ourselves to others and to quite frankly, never feel 'good enough'. The goal is to fall in love with ourselves. Not in a conceited way, that would be counterproductive. We are with ourselves all of the time. However, the majority of the time, we allow other people's perception of us take over what we truly know in our hearts to be true.
2) Fear of failure
Perfection. Perfection. Perfection. The fear of making mistakes stunts our growth and development. In order to learn and grow, we NEED to fail. Yes, it's uncomfortable. But, that means we are learning. We fear putting ourselves out there to protect ourselves from making mistakes. It would be a shame knowing that you never failed in this life because it means that you didn't allow yourself to be stretched enough.
3) Being judged by others
We allow people to gain power over and dictate our own internal dialogue! Ironically enough, that very person who you feel is judging you is also worried about being judged by others. Not to mention, a person's view of you may take only a few minutes of their day but a large percentage of YOUR daily emotional energy. It simply doesn't make sense.
Change "what if" from a negative to a positive!
The common dialogue when it comes to sharing your story with the world through speaking and social media:
… the list goes on.
Answer this:
What if your story saves lives?
Sam Phippen has been writing software for more than seven years. He routinely gives conference talks about software design and testing and currently serves the Ruby community as a member of the RSpec core team. Sam regularly contributes to Open Source Ruby applications and works as a consultant at Fun and Plausible Solutions. He is the presenter of the recently released video course called "Effective Ruby" from Addison-Wesley Professional.
Hash class is one of the most widely used in Ruby. We use it to represent everything from parameters objects, to database rows, and even domain specific data structures. In this post, we'll explore a number of specific ways that you can improve your use of hash objects to make better rails applications.
Hash#fetch over
The most common way to get values from a hash is to use the square brackets, or subscript, method. This method directly looks up the value under the passed key in the hash and returns the value stored under the key in the hash. If the key is not found,
[] instance method returns
So let's talk about
fetch. On first inspection, the
fetch method is similar to the
[] method:
(master)$ irb >> cat = {:say => :meow} => {:say=>:meow} >> cat.fetch(:say) => :meow >>
Unlike the
[] method, the
fetch method raises an exception if the key being looked up is not found.
This behavior of fetch is very useful. Primarily, it means that you can find key places where data values are missing in your system with ease. It may be the case that you sometimes want to provide a default value, even when the hash does not contain a value stored under that key. With
[] you may be used to
||ing the value in, using the
nil as the missing value behavior.
fetch makes this easy and explicit.
Instead of having a separate (
||) syntax for providing a default the
fetch method provides us with an explicit way of doing this. Fetch actually has three signatures:
fetch(key) which raises an exception if the provided key is not found
fetch(key, default) which returns the provided default value if the provided key is not found
fetch(key) { ... } which evalutes the provided block and returns its return value if the key is not found.
It's quite common to create a hash literal using the empty curly brackets syntax
{} or to provide some values in the hash with the hashrocket
{:a => :b} or symbol key
{a: :b} syntax. These are useful when you don't need a default value in your hash, but there are other forms.
Consider the following example:
value = "hello world I am Sam"
letter_counts = {}
value.each_char do |char|
letter_counts[char] ||= 0
letter_counts[char] += 1
We're counting the occurences of each character in the String, but to do so we're first initialising the value stored under that key in the hash to zero. We're doing extra work that we don't need to. The Hash class has two additional constructor forms we can use to make this easier:
Hash.new(value) which constructs an empty hash that will store the provided value to the constructor under any missing key when it is accessed.
Hash.new { |hash, key| ... } which constructs an empty hash that will invoke the block provided for any missing key which is accessed. The return value of the block will then be stored in that missing key.
These additional forms are useful for when we know we have some default value. We can eliminate the
||= memoization trick from our code. This puts the default value and the construction of the hash in the same place, reducing the cognitive overhead, and performance cost, of our collection of values.
each_with_object to transform collections
In the previous example I showed counting the number of times individual characters occur in a string, which is a simple transformation of that string collection. We can, however, achieve this transform with fewer lines of code using a standard protocol on
Enumerable in Ruby called
each_with_object. Consider:
value = "hello world I am Sam"
letter_counts = value.each_char.each_with_object(Hash.new(0)) do |char, hash|
hash[char] += 1
This code achieves the same thing, but instead of leaking the letter counts local variable before it is complete, we return it from the
each_with_object call. We build the hash inside this call, incrementing the values inside the block provided to
These are some simple, but powerful, uses of Hash objects in Ruby. Finding opportunities to use them within your Rails or Ruby applications will almost certainly yield more readable and usable code. The overarching point here is that the Ruby standard library is very powerful. Being aware of how it works, what's inside it, and what is possible with it will make you a better Ruby developer. Hash is one of the most fundamental collections, and these and other applications of it will change the way you work with the Ruby language.
You can watch my entire video series "Effective Ruby" in Safari — which is over four hours long and is based on the book of the same name by Peter Jones.]]>
As a high school student I developed an eccentric sense of style that culled from Army Navy stores, flea markets, thrift stores, and my grandmother, which earned me the dubious title of "best dressed" from my choir. (Long live art kids.) Fast forward almost ten years and I still have a penchant for ripped tights, messy hair, weird fitting cardigans, and massive scarves, so dressing appropriately for work is a continuous challenge for me, even in a casual environment like Safari.
Below, some things I've learned:
1. Choose a color palette and develop a uniform
I chose black as my color palette because black is versatile, dynamic, and simple. While black clothing can be difficult to match together, (never assume that two black pieces are going to match!) they are also easy to accessorize, look more polished, wash better, and generally wrinkle less than light colored clothing. Also, if you are like me, all your white dresses will have coffee stains, which are fortunately easily covered with large scarves. (Just kidding!)
NYMag ran a series of excellent posts on stylish women and uniforms, and I largely echo those sentiments. Wearing a uniform has made getting dressed in the morning easier and infinitely more chic.
2. Purge often
I had bedbugs in college, which meant I had to get rid of a fairly large collection of hideous patterned 1980s sweaters collected from years of thrifting.
At the time I was devastated, but now I see that experience as a blessing because it taught me the importance of purging my clothes often. Are your favorite pair of pants pilled beyond recognition? Do you have dresses that look wrinkled no matter how many times you iron them? What's sitting in the back of your drawer that you never wear? Give these clothes to charity, sell them to a consignment shop, or have a clothes swap with your friends: there's no reason to hold onto things you don't wear, and getting rid of clothing means you have more room for the things you actually feel good in.
3. Accessorize
Accessories are an easy way to make every outfit feel more complete, from scarves to statement necklaces to big vintage rings. I inherited a large collection of scarves from my grandmothers, and I scour vintage stores, flea markets, and consignment sales for costume jewelry. I also try to buy simple, inexpensive pieces from local designers at small boutiques in the area.
I rarely leave the house without jewelry and a subtle perfume. A distinctive scent can help you feel more polished. (I wear Jo Malone 154.) If you're scent sensitive, I highly recommend finding an organic essential oil you enjoy. A few drops can make the whole day brighter.
4. Quality always trumps quantity
While we live in an age of fast fashion, you can find well-made clothes for less if you take some time to think about your buying values. While I occasionally buy for convenience, I also am extremely aware of the quality of my clothes due to years of trawling consignment stores for top designers like Marc Jacobs and Chloe.
Good quality clothing doesn't have to be expensive.. Brass, a local Boston startup that works directly with factories to provide high quality garments for less, ran a piece last year about how to judge quality in clothing, and I've found it helpful when shopping for work clothes.
5. Never buy retail
Like one of my fashion role models, Fran Fine, I would never buy retail. (Particularly if my "cousin" was Todd Oldham!)
For better or worse, most retailers exist in a constant sales model where it's easy to patiently wait for a piece to go slightly out of season in order to get a deal. My focus on sales has helped me find some amazing pieces from high-end designers for much less than their original price.
6. Don't forget your shoes
I am from a "shoe family" (I'm the first in three generations not to work in the women's shoe business!) so shoes are clearly important to me. With shoes, price does reflect quality and comfort, so it's worth investing in a few good pairs that you can wear for a long time.
Also, meet your local cobbler and make friends! If you love a pair of shoes and wear them out they can usually be fixed.
Ethical shopping for the fashion-conscious
Before I buy, I try to think about the ethical and environmental consequences of buying an article of clothing, which is why you're more likely to find me at a local boutique or consignment store than a mall. I also have a strong belief in "less is more." I buy clothes infrequently, though I often wander boutiques and stores or browse sites for fashion ideas that I can incorporate into my wardrobe.
Local consignment stores
Depending on your neighborhood and geographic area, consignment can be a lifesaver. (Pro tip: the Beacon Hill Second Time Around is the best consignment in the Boston area!) Consigning is usually easier if you wear a very common or very small size and are willing to spend a good deal of time sifting through ill-fitting, pilled, or otherwise weird clothing to find something special, but I know lots of women of every size who love consignment stores.
If you are looking for work clothes, I wouldn't bother with places like Goodwill unless you have a lot of time and patience to sort through clothing. (That girl you met at a party wearing an Alexander Wang from Goodwill was probably lying.)
Boutiques and local designers
Most of my favorite clothes and accessories come from local boutiques and traveling. Boutiques tend to be more expensive, but in addition to supporting local business, I find my most unique pieces through boutique shopping. In Boston, my favorite boutiques include Oona's, Tess (when I can afford it!), Mint Julep, Holiday, and December Thieves and Magpie for accessories. Go at the end of the season to find the best deals.
I find that it's usually it's worth splurging for local design as well. In San Francisco, I discovered the designer Curator, and in Portland I discovered the incredible textile designer Sara Bergman. These pieces have stayed with me because they're unique, special, and remind me of my time traveling.
Everlane and Brass
A friend of mine describes Everlane as "Eileen Fisher for 20-somethings," and their casual, inexpensive, comfortable clothes are my absolute favorites, though they still have a slightly limited selection. Brass currently sells only dresses and I have yet to get off the waitlist, but I think their online store is gorgeous. I like these more "ethical" producers, who work directly with factories to ensure fair treatment for workers, and have been pleased with the quality of their clothing.
My favorite fashionable places on the Internet for inspiration:
Last year, I wrote a program that generated a nonsense book inspired by the mysterious 16th century Voynich Manuscript. This year I took inspiration from one of the earliest known works of computer-generated fiction, the little-known SAGA II.
SAGA II was written in 1960 by Douglas T. Ross and Harrison R. Morse at MIT, and was featured in The Thinking Machine, a delightful 1961 CBS television special. In the special (which can be viewed in its entirety online), men in suits sit in comfortable chairs smoking pipes and debating whether computers can be truly creative. What sets the show apart from other examples of breathless futurism is the emphasis on contemporary research—we see actual engineers manipulating room-size computers, light pens, and teletypes.
The SAGA II project was commissioned by the show's CBS producer, who hypothesized that TV westerns were so stereotyped that a computer could plausibly write a screenplay. In response, the MIT engineers coded a simulation that could generate an infinite number of variations on a classic Western scene: a robber, pursued by a sheriff, breaks into a house, has a few drinks, and engages in a deadly shoot-out. The survivor takes the loot and leaves the scene. CBS filmed two of the computer-generated scripts and, charmingly, one buggy one in which the robber gets stuck in an endless loop, perhaps the earliest form of glitch art.
Here's what an original script looked like:
(The gun is in the right hand; the money is in the left hand; the glass is on the table; the bottle is on the table; the holster is on the robber; the sheriff's gun is in the sheriff's right hand; the sheriff's holster is on the sheriff.)
ROBBER: (The robber is at the window.) Go to door; open door; go thru door; close door; go to corner; put money down at corner; go to window; put gun down at window; lean on window and look; lean on window and look; go to corner; count money; go to table; pick up glass with right hand (empty); take glass from right hand with left hand; pick up bottle with right hand; pour; put bottle on table; take glass from left hand with right hand; take a drink from glass; put glass on table; pick up bottle with right hand; pour; go to corner; put bottle down at corner; go to window; pick up gun with right hand; check gun; put gun in holster; go to table; pick up glass with right hand; take a drink from glass; go to window; put glass down at window.
SHERIFF: (The sheriff is at the window.) See robber; (robber sees sheriff); go to door.
ROBBER: Take gun from holster with right hand; check gun; go to door; check gun; put gun down at door.
SHERIFF: Open door; see robber; (robber sees sheriff); go thru door; go to window.
ROBBER: Pick up gun with right hand.
SHERIFF: Go to table.
ROBBER: Aim; fire; MISSED; aim; fire; SHERIFF HIT; blow out barrel; put gun in holster.
SHERIFF: Drop gun; sheriff dies.
ROBBER: Go to corner; pick up money with right hand; go to door; go thru door; close door. CURTAIN.
The source code for SAGA II is long gone, but enough artifacts remained that I felt confident I could write a reasonable fascimile. I had three primary sources:
I chose to write my implementation in modern Python rather than imitating the original's global variables and weighted probabilities. That said, I still had to write a lot of code; there's a surprising amount of hidden logic in the original, including objects which can contain or support other objects, a model of handedness, and the quantity of whisky and bullets remaining in the actors' inventory. In the end I arrived at a weird hybrid of interactive fiction, role-playing games (each 'actor' rolls for initiative based on their current state), an event queue, and the kind of object-oriented code you normally only write in computer science class ("a Robber is a type of Person that is a type of Thing"). And since I had to satisfy the one rule of NaNoGenMo, I made the program generate 350 unique scenes for a total of 53,392 words.
The most important code in the simulation: shooting.
def shoot(self, target, aimed=False): """Shoot first, ask questions never""" gun = self.get_if_held(Gun) if gun: # Usually we'll aim and then fire, sometimes we'll just fire if not aimed: if random.randint(0, 5) > 1: print("aim") self.queue.append((self.shoot, target, True)) return False print("fire") log.debug("%s is trying to shoot %s", self.name, target.name) # Establish any weighting factors based on the type of person that affects # their accuracy (e.g. the Robber is more accurate when he's drunk) hit_weight = self.starting_hit_weight() # As in the original, the actors get inexplicably more accurate if they're on their # last bullet if gun.num_bullets == 1: hit_weight += 1 # Also defying common sense, they get more accurate if they're injured if self.health < DEFAULT_HEALTH: hit_weight += 1 weighted_hit_or_miss = [('miss', 3), ('nick', 3 * hit_weight), ('hit', 1 * hit_weight)] hit_or_nick = random.choice([val for val, cnt in weighted_hit_or_miss for i in range(cnt)]) # Print the relevant 'HIT' or 'MISS' message based on the outcome... print(GUN_DAMAGE[hit_or_nick]['message'].format(target.name)) # ...and update the target's health based on that outcome target.health += GUN_DAMAGE[hit_or_nick]['health'] gun.num_bullets -= 1 return True
(I asked my husband to proofread this post and he noted that the code never checks if there are bullets remaining. PATCHES WELCOME.)
The gun is in the robber's right hand. The money is in the robber's
left hand. The holster is on the robber. The sheriff's gun is in the
sheriff's right hand. The sheriff's holster is on the sheriff. The
glass is on the table. The bottle is on the table.
ROBBER: (The robber is at the window.) Open door; go through door;
close door; go to corner; put money on corner; go to table; go to
window; check gun; go to corner; go to table; pick up the glass with
the robber's left hand; go to window; go to corner; count money
SHERIFF: Go to window; open door; go through door
ROBBER: Fire; sheriff NICKED
SHERIFF: Close door; aim
ROBBER: Fire; MISSED
SHERIFF: Fire; MISSED
ROBBER: Fire; sheriff NICKED
SHERIFF: Aim; fire; robber NICKED
ROBBER: Aim; fire; sheriff HIT; aim; fire; sheriff NICKED
SHERIFF: Sheriff dies.
ROBBER: Blow out barrel; put gun in holster; pick up the money with the
robber's right hand; go to table; open door; go through door; close
Source code and complete NaNoGenMo "novel" are on Github.
For fun I also tried a variant where I spawned dozens of robbers and sheriffs who all wanted to kill each other:
GREEN ROBBER: close door; go to corner
NAVY ROBBER: aim
OCHRE SHERIFF: close door
NAVY ROBBER: fire; MISSED; fire; navy sheriff NICKED; aim
NAVY SHERIFF: close door; aim
GREEN SHERIFF: close door
RED ROBBER: aim
GRAY ROBBER: aim
NAVY ROBBER: fire; navy sheriff NICKED; navy sheriff HIT
NAVY SHERIFF: navy sheriff dies.
…but I think to make more compelling I'd need to introduce some interactions that weren't present in the original.
National Novel Generation Month runs through the end of November (but since it's not a real contest anyway, anyone can participate any time). Inspiration can be found in the list of projects that are completed or in-progress for 2015.]]>
We will discuss different ways to surface the most relevant and the most desired content to the user(s) seeking it. From the point of view of a platform provider, this is not a challenge that can be answered strictly through technological means, but rather one that requires a combined effort by platform providers and by publishers. The publisher effort is specifically needed to provide metadata and additional content in order to fully leverage the technology tools available within online publishing platforms.
Getting users to content involves two basic capabilities, both of which require both the full text content and a certain amount of metadata to work properly:
Taking linking further
There has been a great deal of interest in using semantic enrichment technologies to provide related links. One of the main draws of this technology is the ability to provide high quality links without the need for a significant expenditure on extra metadata. However, in many cases, the reality is that simple lists of related content can be underwhelming for users. This is in large part because it is difficult for systems to provide more than just the title of the related content item, which often does not convey just how the content is related.
So how do we provide more context to help the user quickly understand which links are most likely to provide the results they are looking for? One idea would be to provide context on why the link was selected. For example, make it clear to the user that one link will lead them to related research data, while another will provide access to a breaking news item in the research area. At this point, we are just starting to see some attempts at creating this kind of contextual guidance between linked content. SpringerLink is a very recent example of a publisher who is working in this area and for a subset of their biotech content extracts key concepts from the article and provides related links based on each of the concepts. This is intriguing technology, but it is too early to determine whether or not it is scaleable and can support content across all, or even most disciplines.
Learning Paths
Another approach to creating contextual guides that we have used at Safari is to have subject matter experts create lists and place each piece of content in context to provide a learning path through the list. We refer to these learning paths as lists Of course this requires as much or more editorial effort than creating a taxonomy. One possible way of reducing that load is to rely on your users to provide the data. We have built products that do this, specifically both Sage Research Methods Online and the Loeb Classical Library allow users to build and share lists.
A common lament about the online world, especially as it involves publishing, is that we have lost the serendipity that we had in the real world. The nostalgic assertion that when walking into a good bookstore or a good library we were liable to stumble upon things of interest that we didn't even realize that we were looking for. To the degree that this is true and, not just nostalgia for the past, it happened because of a conscious curation effort on the part of the store clerks and librarians. As we improve our online publishing products the question becomes how can we bring this serendipity back?
As you can see from the discussion above we are just at the beginning of these efforts. There is hope (and a level of observable success) for technological solutions to the problem of bringing most needed content to the user in need/want at the right time, but most of the solutions require substantial effort on the part of the publishers involved to create the metadata necessary to provide a shape to the content collection and allow users to navigate through it. The as yet unachieved "trick" to-date is marrying human intervention with technology tools to create the most precise content connections in the most efficient manner possible.
"Digital Content Curation: How to Lead Users to Where They Need to Go and Make Sense of What They Find When They Get There" takes place at 3:30pm on Friday November 6, 2015 at the Charleston Conference.]]>
This post explains the technical choices we made while refactoring our search feature from a component within a monolithic Django project into a microservice. It explores our efforts to break dependence on a shared database, use central authentication and plan for redundancy, among other things.
I hope that by sharing in detail some of these choices, including our mistakes, I will help you on your own refactoring project. I know that I can't shield you completely, but perhaps in the course of your project someone will mention renaming a few curiously-named fields or rewriting the JavaScript front-end, and you'll get a twinkle in your eye and think of me.
If the hype hasn't already deafened you, a "microservice" is basically a narrowly-focused web service. In theory, this narrowness should make them easier to understand than more complex projects. When they split a software product like Safari across well-defined boundaries, they are supposed to allow teams to change pieces of the product independently.
At Safari we want to go fast, and in our experience speed tends to decrease as teams and codebases grow. So, like a lot of engineering teams, we're trying out microservices to see if they help us go faster.
If you want to learn more about microservices, check out Sam Newman's book Building Microservices or, if you favor video, The Principles of Microservices by the same author.
Some areas of our monolithic Django application suffered more from long cycles of development and deployment than others. Search was one of the slow-boaters. If you had to tackle even a medium-sized search feature — or, heck, fix a search bug, since they often required reloading the search index (AKA, "reindexing") — you might lose an entire sprint. Coordinating the release of a search feature could be difficult, as it required close work with our QA team to get new code onto a QA machine and reindex, a process that often blocked other QA work.
We wanted to make search tickets fun again. (I think we succeeded, but I'm biased.)
Engineering estimated four to six weeks to get an initial service up and integrate it with a major part of our system, either on the web site or in one of our mobile applications. After some deliberation, we chose the web site as our first integration point, thinking, in part, that doing so would let us iterate faster than in a mobile app. Then we would work through the other integrations piece-by-piece.
We deployed an initial search microservice to a production server within a few weeks. And then…
Well, we started building in March of 2015, and I'm writing the teardown in October. I can't say we totally beefed it, but we did not deliver a major integration until several months after we started building the microservice. We did open the service to use by partners over the summer. We did build a central authentication mechanism so that we could rate-limit external requests to the now publicly-available API. We did make every piece redundant. And then finally, sometime in late summer, the mobile apps team completed the first major integration with our new microservice when they used it for the Android app beta's search feature. We shipped the next integration soon after that by finally turning on a new search form that had been hidden behind a flag for months on the site. Other integrations remain.
Of course we succeeded. We built a thing! It's running right now, serving search results! I wish we had succeeded faster, but on the flip side, we learned a ton.
We also failed — I did, anyway. My original estimates were wrong, in part due to my inexperience with our search engine (Solr) and microservices. I had been brought in to build the microservice so that I could, in my ignorance, soak up all the knowledge of a departing search master, which bit me a little. I wish I had been able to say, "Wow, refactor all this code that's tightly coupled to the database into a microservice, then refactor something else to use it, huh? That could take months. Might be a bloodbath." Then someone could have said, oh, no, we'd rather do anything with that money other than build a search microservice, that sounds like setting cash on fire!
Still, we built it, and I believe that our decisions were sound. We grew our knowledge and tooling around microservices, from both of which we continue to get value.
Having established that we succeeded in some definition of the word, the remainder of this post is dedicated to a teardown of the technical choices we made throughout the project, especially the ones that caused us to, uhm, almost fail completely.
Our first task was to discover all of the logic hiding in the nooks and crannies of our existing search code (no judgment, but you know how it is — there's always some file that contains the critical thing that makes every other piece work correctly).
At a high level, there appeared to be two major zones of search code in the monolith: the first was indexing, and the second was handling user searches. Indexing seemed like the more complicated area, so we tackled it first.
The monolith used a library called Haystack to convert data from its database into XML documents that it sent to Solr.
SearchIndex classes handled these transformations. These index classes were a mix of declarative and imperative code that defined exactly how to transform a Django model instance to data ready to send to Solr as XML. The indexing code layered on top of the monolith's database models and referenced them explicitly. You can see how Haystack's
SearchIndex works by looking at this example from the library's documentation:
import datetime from haystack import indexes from myapp.models import Note class NoteIndex(indexes.SearchIndex, indexes.Indexable): text = indexes.CharField(document=True, use_template=True) author = indexes.CharField(model_attr='user') pub_date = indexes.DateTimeField(model_attr='pub_date') def get_model(self): return Note def index_queryset(self, using=None): """Used when the entire index for model is updated.""" return self.get_model().objects.filter(pub_date__lte=datetime.datetime.now())
We had a set of classes like this in our monolith that wrapped our content models (e.g., book, chapter, video and video clip). Except, of course, they were much larger than the example and performed many more operations to transform a model into a Solr document.
We had already split both the search features and the core platform models into separate Django "apps" or components. This meant that a developer could build search component features by checking out the search component, writing code, and running just that component's tests. However, the search component was tightly coupled to the data models it shared with the rest of the monolith, so, while it was nice that as a developer you could run tests independently of the monolith, at a certain point you had to integrate new versions of your code with the monolith, which meant installing your version of the component into a local build of the monolith and writing tests that proved that the integration worked.
This is a typical situation in large Django projects, and we do it all the time. Still, especially with search, we had noticed that all of these bare wires tended to entangle and electrocute hapless developers, so we hoped that microservices could lead us to a slightly better system.
Now, we could have pulled the index classes out of the monolith (specifically, the search component that it used) and copied them into a new repository. We would have needed the shared data models and a connection to the monolith's database as a dependency. We probably would have ended up with a Celery task that ran asynchronously to pull content directly from the production database and put it into a Solr instance.
Retaining the coupling between search and the database would have meant that when a developer working on a monolith feature migrated the shared database, the change could have broken the search microservice. We would need to coordinate migrations of the shared database to make sure that all search nodes were using the latest version of the schema through the shared data models. This path sounded like it led to the dark side. In fact, Chapter 4 of Building Microservices has a whole section dedicated to convincing people to avoid doing exactly this. We didn't want a new flavor of entanglement. We wanted to go fast. But what was the alternative?
We looked at whether we could continue to use our Haystack index classes, but point them at something other than a Django model. Could we build some kind of facade that pulled data from a REST API, but reused all of the code we had embedded in the index classes? After studying Haystack, we concluded that this was probably not possible.
The best alternative to retaining the database coupling that we came up with was to rebuild the indexing code, such that instead of pointing at the database to get search content, it pointed at a higher-level interface — the monolith's REST API. This API served content to our iOS and Android applications, and it included everything that we needed for indexing.
But if we ditched Haystack, what would we lose? It turned out that a significant amount of business logic lived in our Haystack index classes.
As we already discussed, the purpose of that code was to transform a Django model into an XML document (or many documents) that lived in Solr's index. We wondered if there was something else that we could use for this job that was a natural fit if we pulled content from a REST API for indexing. Since we had already planned on using Django Rest Framework (DRF), that library's "serializer" classes seemed like they could work — we could validate and deserialize API responses, perform transformations on the result, and then send the validated and transformed data to Solr using the
pysolr library directly, instead of Haystack.
We ended up taking this approach. It's not perfect: now instead of depending on a library meant to transform Django model instances into documents in a search engine, we're dependent on a REST API framework and a lower-level Solr library (
pysolr). However, we don't need a shared database. The input data can come from any source, whether that's a REST API now or an AMPQ message body later (who knows?).
In most cases, converting logic from a Haystack index class to a DRF serializer was straightforward. Instead of a model object, the input was a JSON document. In both cases, the output (with the help of
pysolr) was an XML document prepared for Solr, that matched the fields in its index definition.
In the monolith, we indexed new documents into Solr as one step of an asynchronous workflow that constantly pulled new content from an upstream API and broke it up for storage in the monolith's database. We wanted to continue to make new content available in search as soon as it was available in the monolith. Since we were already beginning to use Celery for asynchronous tasks in the microservice, and another team had recently added code that sent an AMPQ message when new content arrived in the monolith (not using Celery), we built a Celery
ConsumerStep to consume these messages. When the microservice received a message that the monolith had added a new title or removed a title, it spawned a new asynchronous task that eventually indexed that content from the monolith's REST API. The code for the "book was added" consumer looked like this:
import logging from celery import Celery from celery import bootsteps from django.conf import settings from kombu import Consumer, Exchange, Queue from indexer import tasks exchange = Exchange(settings.EVENT_EXCHANGE, type='topic') content_ingestion_queue = Queue(settings.CONTENT_INGESTION_QUEUE, exchange=exchange, routing_key=settings.CONTENT_ROUTING_KEY) app = Celery('consumers', broker=settings.EVENT_BROKER) log = logging.getLogger(__name__) class IngestionConsumerStep(bootsteps.ConsumerStep): def get_consumers(self, channel): return [Consumer(channel, queues=[content_ingestion_queue], callbacks=[self.handle_message], accept=['json'])] def handle_message(self, body, message): log.info("Received content update message: {0!r}".format(body)) if 'identifier' in body: tasks.index_book.delay(body['identifier']) else: log.warn("Ignored content update message because it lacked an identifier") message.ack() app.steps['consumer'].add(DeletionConsumerStep)
But because we (I) did not trust messages completely, we also built a "catchup" task that ran occasionally to reconcile the Solr index with the total set of titles in the monolith, pruning or adding any for which the microservice had missed an update or remove operation.
This approach has been good enough for now, but it takes way too long to reindex from an empty collection (i.e., a full reindex) — usually about two days. Because the reindexing process involves spawning thousands of tasks that make sometimes hundreds of API calls each, we think that the speed will improve with a threaded or
asyncio approach rather than the purely multi-process concurrency that we get now, running several Celery workers. I look forward to trying that optimization.
I carried over much of the Solr index schema from our existing search code (by which I mean the XML definition of the fields stored in the Solr index, not the Haystack index classes), which saved a lot of time and preserved much of our past search work. However, in my haste to stake a boundary around the new search microservice, I changed several fields that appeared to have inherited their names directly from fields in the production database, but for which there were clearer names. I also pluralized any field name that was singular but represented a multi-value field in Solr, to avoid confusion. I meticulously used find-and-replace to change all of these names across the Solr field configuration and schema, except where I didn't.
My error while executing the seemingly trivial job of replacing text across multiple files, for which I used the finest tools available, became a whole class of error conditions that our whole team dealt with on and off over the next few months. And then after we had finally nailed down the last missing rename operation and had done another reindex to get those values stored correctly, we received reports that our replacement search page mishandled some bookmarked searches. Searches that referenced the field names that I had changed directly in an "advanced search" capacity did not work. I had not known that people used the field names directly in search queries, or that someone wanted the new search form to preserve people's bookmarked searches. We finally added a layer of aliases to the API that would accept both the old and new field names as input.
Overall, it was a costly find-and-replace!
Users can search the latest version of Safari, which is what I work on, in a few places: the search box at the top of the web site, the web site's
/search page, the web site's "search inside the book" tool, the iOS app and the Android app.
In the monolith, we used "form" classes derived from Haystack's
SearchForm class to handle all of these cases (except the Android Queue app, which was in beta at the time that we built the microservice and did not yet have a search interface). Here is an example of a
SearchForm from the Haystack documentation:
from django import forms from haystack.forms import SearchForm class DateRangeSearchForm(SearchForm): start_date = forms.DateField(required=False) end_date = forms.DateField(required=False) def search(self): # First, store the SearchQuerySet received from other processing. sqs = super(DateRangeSearchForm, self).search() if not self.is_valid(): return self.no_query_found() # Check to see if a start_date was chosen. if self.cleaned_data['start_date']: sqs = sqs.filter(pub_date__gte=self.cleaned_data['start_date']) # Check to see if an end_date was chosen. if self.cleaned_data['end_date']: sqs = sqs.filter(pub_date__lte=self.cleaned_data['end_date']) return sqs
On the web site, we used a form class to render the search-specific parts of the search page and "search inside the book" tool. Users visiting the site could adjust the inputs on the rendered form to build query parameters, and when they submitted it, we used JavaScript to make an AJAX request to an HTTP endpoint that used our form class to validate the user's input and make the Solr query. Then we rendered the result of the query with a Django template and returned the rendered HTML. Our JavaScript replaced the search content element on the page with the new rendered response from the server.
The monolith's mobile search API also used the form classes. It would receive POST parameters that needed to map to the inputs on the search form. Then it used the form class to validate the POST and returned the results as JSON instead of rendering them as HTML.
After I talked to one of our front-end engineers about the problem, he felt excited about rewriting the search page so that it consumed JSON directly. So, we grew the scope of the project, not realizing how much time a front-end rewrite of the search page would add. We worked on the new form in piecemeal over several weeks when we could, and eventually we deployed it hidden behind a flag. QA would try it once a sprint or so, file a list of problems that blocked us from turning it on, and disable it. Once again, when we had time, we would work on those tickets and restart the process. We've done a lot of navel-gazing since then about how to avoid this particular wheel of suffering.
Next time, I hope we accept more risk that new code may have problems, and that we'll stick a couple of people on a feature until it's done. Or that we scale back our ambitions, knowing what we know now.
Anyway, back to the technical side of things. We looked at the job that the Haystack form classes were doing in the monolith and found that they had two primary roles: validating user input and building a Solr query based on the validated input.
The last responsibility seemed to produce most of the code in our repository, as the form classes built a query by passing validated user input through various branches, to arrive, finally, at a Solr query. The classes then performed the search using
pysolr behind the scenes, and returned the results inside a wrapper object.
With the new microservice, we decided to accept search parameters the same way, via GET or POST, and validate them against a DRF serializer. Pretty typical stuff. DRF serializers are great at expressing a schema for input validation. It then seemed straightforward to add a helper method to the serializer that built a query from the validated input, and simply returned the built query string.
We ended up with an API endpoint in Django that received HTTP requests, validated the input against a serializer, then asked the serializer for a Solr query based on the input. The API view then sent this query to Solr using
pysolr and returned the results as JSON.
As with our use of serializers to replace Haystack index classes, we mixed the responsibility for validation with something else (validation and transformation in the indexing case, validation and query building in the searching case). More than once I have looked askance at this code after writing it. I suspect that the sentiment is shared by other people. Maybe someday we'll compose these responsibilities into a new set of objects (suppose an
Index like Haystack's but without database dependence, and a
QueryBuilder for the search API). Or, we we might separate the responsibilities completely and build pipelines (validation, transformation, Solr update for indexing; validation, query building, Solr search for searching).
The most important goal that we had was to draw a boundary around search that excluded the main production database, rather than shared it. We accomplished that goal. Now we have room to make improvements.
Our first plan was to deploy the microservice without any authentication and see if we felt the need to add it later. However, we had to switch gears when the company decided to use the microservice to serve public requests made by partners.
Opening the search API up to public/external search requests meant that we would need to satisfy an internal set of requirements for all public APIs, of which this would be the first. That list included rate-limiting, which is based on the notion of identity, so we would need a way to authenticate clients after all.
Our monolithic search endpoints all relied on either session or oAuth authentication, both of which referred to user data stored in the monolith's database. But, the new microservice didn't have access to this anymore.
We thought that the fastest thing might be to add token-based authentication. That would have required us to build a separate repository of credential information inside the microservice, when we already had a system containing that information: the monolith. It also wouldn't have worked for mobile app users, whose oAuth tokens were tracked through the monolith.
Instead of doing using token auth, we made another hard decision that added complexity to the project — to build in central authentication. We worked with other engineers who had begun experimenting with central authentication middleware that could authenticate users of a microservice through network requests to the monolith.
The code for the central authentication middleware was pretty straightforward. It copied an inbound request (after slimming down headers) to the microservice as a new outbound request to the monolith, with any authentication headers intact, and built a user facade from data in the response. Smart caching reduced the number of redundant requests for subsequent searches by the same user. It looked good (and props to the engineers who worked on it), so we incorporated it into the microservice.
Meanwhile, we housed the microservice URL under our main site, using our top-level domain as an API gateway. We proxied requests to the path
/api/v2/search/ to a load balancer that distributed work among a pool of nodes running the microservice's search API.
We turned on rate-throttling with central authentication and it worked. So cool! Now we could authenticate partners with an auth token (stored in the monolith, associated with their account information), people using the JavaScript front-end, and people using our iOS and Android applications. Perfect.
A problem dogged us, though: we began seeing high latency on authentication requests. The latency was so high, in fact, that requests were timing out.
The timeouts were happening in our authentication calls to the monolith. New Relic showed clear, occasional spikes from that call, too.
But they were not always timing out, and we couldn't reproduce the problem with
curl. We worked with IT to switch the URL that we used to one that did not travel through our CDN, but the problem resurfaced. We had also gradually raised the timeout value from milliseconds until we were timing out after ten seconds. We had a serious problem, and yet New Relic had no record of these slow responses from the monolith's side.
After a few deep dives into the problem, we narrowed it down to POSTs only, which led us to consider the headers coming in on the requests. We captured and replayed a problematic request, rebuilding it by hand one header at a time and sending it to the microservice with
curl, until we found the issue, which in retrospect was obvious: our authentication middleware was copying over the
Content-Length header from an inbound POST request to an outbound GET request with no body. This caused a server on the other end to stall indefinitely, waiting for input that never arrived.
Computers! Why do they always do exactly what you tell them?
Previously we had one running Solr instance and two collections. The monolith pointed updates and searches at the current collection, and did nothing with the "off" collection. When we had to perform full reindexes of content, we reindexed into the "off" collection, and when the reindex finished, we changed the collection that the app nodes pointed at. This was reasonably simple, but there was one problem — we only ever had a single Solr instance primed. If that server ever failed, search on the site and in all mobile apps would go down until we could recover (point of order: I don't remember this ever happening).
In designing the microservice, we wanted to continue to have zero-downtime reindexes while also gaining more redundancy, so that a single search server could fail and search would continue to work.
SolrCloud looked appropriate for our use case, but at the same time, we were trying to move fast and it looked complicated (ZooKeeper…?). Solr supports replication, so with our eye on using SolrCloud in the future, we built out a topology that included a master "indexer" Solr instance, which received all index updates that we generated in response to messages that the monolith had added content, but served no search requests; and two Solr "searcher" nodes whose indexes constantly replicated from the indexer and which served all search requests. Our application nodes pointed at the searchers and were load-balanced such that both Solr instances handled search traffic.
We deployed the two nodes with different VM specifications, but on the first day that the microservice received heavy traffic, one search server strongly underperformed when compared to the other. We beefed up the underperformer, only to discover that even with specifications that matched its twin, it continued to underperform.
The only other difference we found was that the slower machine was chewing on disk IO constantly, whereas the fast node barely touched it. With some inspection of the processes running on the slower machine, we found that it had been consuming update messages bound for the indexer and updating its index with new content — constantly — while it also replicated from the indexer. We followed the trail back to Chef, where we found that we had left a "recipe" configured for one of the searcher nodes that set it up as Celery worker, pulling indexing tasks. It's amazing how much heartache a typo can cause.
Another post-launch issue that gave me the sweats but stabilized quickly was that with two Solr instances serving search requests there were, of course, two query caches. This meant that after we reindexed, the search API took longer than we expected to reach a sub-400 millisecond average response time.
Microservices are all fun and games until someone asks you to refactor part of a monolith. And yet, if you subscribe to the Monolith First strategy, that is how you'll spend most of your time building microservices!
Seriously, though, I learned a lot working on this project, and I'm glad that Safari asked me to work on it. I made more mistakes than I am happy with, so I hope that by describing all of them in painful detail, I will help someone (perhaps you?) avoid some small amount of pain in the future.]]>
In his recently released book, "Disrupting Digital Business: Create an Authentic Experience in the Peer-to-Peer Economy" (Harvard Business Review Press, May 2015), author R Ray Wang makes it clear that we are no longer in an economy of products and services. The digital transformation demands that we focus our attention on experiences and outcomes. Organizations must pivot with and ahead of social, organizational, and technological shifts or risk being left behind.
He makes the case that the the most successful companies moving forward are the ones that embrace a new way of thinking around these major trends:
We asked Wang a few questions a few questions about his book:
What are the main characteristics of the digital transformation that's taking place, and how is it affecting organizations?
The biggest shift is the urgency that's required to make the change in business models, not technologies. The business models have to align with the organization's brand promise. We see successful digital transformation when we humanize digital. Successful companies take a Board-driven, management-led, and employees-engaged approach. Culture beats tech in digital transformation.
Transformational innovation moves beyond just making a product faster, better, and cheaper. In fact, transformation is about changing the business models and thinking about how to not only defend from multiple industries, multiple business models, but also lead by attacking and commoditizing one's own models.
We see more than just a gig economy emerge. What we see are companies like Task Rabbit and Etsy helping connect professionals, creatives, and artists with potential types of work. They build trust networks to ensure authenticity. They allow for value creation and exchange in the network. P2P delivers scale and democratizes the ability to distribute work and value.
You can read this book on Safari now, and while you're at it, explore other top titles from Harvard Business Review Press.
Trisha Gee is the author of the new video series called "Building Java 8 Web Applications with Microservices" from Addison-Wesley Professional.
If you're a Java developer you'll have heard about lambda expressions making their way into Java SE 8. You might not be aware, however, that there's much more to Java 8 than a new arrow operator. In this article we're going to look at how lambda expressions and other new features in Java 8 will make your life as a developer easier.
The shift in mindset that lambda expressions encourages is the idea of passing behaviour around, not just objects. For example, you can give a method a piece of code to execute:
userService = new Service<>("ws://localhost:8081/tweets/", "/users/", 8083, message -> getTwitterHandleFrom(message));
The Service class contains all the boilerplate needed to instantiate a new web sockets service. It's a very simple service that connects to a server endpoint ("ws://localhost:8081/tweets/") that provides data to consume, and starts its own server endpoint ("/users/", 8083) where it publishes data. The final parameter is our lambda, which tells our micro-sized service what the business logic is – consume Twitter messages, extract the Twitter username and publish it.
Before lambdas, you might have had copy-and-paste coding as you copy all the service boilerplate code and replace the business logic section for each new service. Or you might have an abstract service class that you override for each individual service. With lambdas, you can be much clearer about separating the infrastructure-level
Service code and its business logic.
The Streams API provides a clean way to query your collections, which can decrease the amount of code you have to write. Let's say, for example, you've parsed a message into a
List containing all the words in that message. Given you have a
Map that maps a word to some mood (happy, sad, angry, playful…) you can find out which moods your message contains with something like:
Set moods = wordsInTweet.stream() .map(String::toLowerCase) .map(lowerCaseWord -> WORD_TO_MOOD.get(lowerCaseWord)) .filter(mood -> mood != null) .collect(toSet());
We turn our word into the lowercase version (here we're using a Method Reference, which is a shortcut that makes simple lambda expressions even shorter); get the mood associated with that word; filter out any nulls (words that don't have moods) and finally place them all into a
Set. We're doing a complex set of operations on a collection with no need to write a for-loop or keep track of intermediate state.
If you're planning to migrate code to Java 8, you could find all the places you currently use a for-loop to see if a series of Stream operations will do the job in fewer, more descriptive lines of code.
The Streams API is a way to execute a pipeline of operations on anything that implements the
Collection interface. But it doesn't stop there – individual collections have also had new helper methods added to them for common operations (thanks to Java 8's addition of default methods to interfaces).
Let's consider the case where you want to either get an existing value from a
Map, or create a new one and place it in the map. Before Java 8, you might write:
private final Map<String, TwitterUser> allTwitterUsers = new HashMap<>(); public void onMessage(String twitterHandle) { TwitterUser twitterUser = allTwitterUsers.get(twitterHandle); if (twitterUser == null) { twitterUser = new TwitterUser(twitterHandle); allTwitterUsers.put(twitterHandle, twitterUser); } // then do something... }
But now there's a
computeIfAbsent method that does a lot of this for us:
private final Map<String, TwitterUser> allTwitterUsers = new HashMap<>;(); public void onMessage(String twitterHandle) { TwitterUser twitterUser = allTwitterUsers.computeIfAbsent(twitterHandle, (handle) -> new TwitterUser(handle)); // then do something... }
That's just one example of a helpful new method, take a look at the API documentation, you'll find a lot more.
Comparators are where we often used Java's predecessor to lambdas, anonymous inner classes. With the introduction of Streams and the
sorted method, we'll use them more than ever. The introduction of helper methods on the
Comparator interface (made possible by the introduction of static methods on interfaces in Java 8) makes this easier. Previously, to sort alphabetically by Twitter handle, you might have written something like:
Comparator twitterUserComparator = new Comparator() { @Override public int compare(TwitterUser o1, TwitterUser o2) { return o1.getTwitterHandle().compareTo(o2.getTwitterHandle()); } };
Now, you can use
Comparator.comparing combined with Method References to declare which value you want to compare:
Comparator twitterUserComparator = Comparator.comparing(TwitterUser::getTwitterHandle);
If you want to sort in descending instead of ascending order, you don't have to swap the order of the comparison around, you can simply say:
Comparator twitterUserComparator = Comparator.comparing(TwitterUser::getTwitterHandle).reversed();
Instead of using
java.util.Date, where all the useful-looking methods are deprecated, we have a new Date and Time API. Now we have the idea of a date without needing a time, we have
Periods and
Durations, and we have an API where naming is intuitive and helpful. For example, Instead of having to know that a new
Date object represents the time right now, we say:
LocalDateTime now = LocalDateTime.now();
We can do logical things to the time, like get the time in ten minutes:
LocalDateTime timeInTenMinutes = now.plusMinutes(10);
and query for specific values:
int minute = now.getMinute(); DayOfWeek dayOfWeek = now.getDayOfWeek();
In summary, Java 8 has been designed with us, the programmers who use it every day, in mind. We have lambda expressions to let us pass behaviour around; we have the Streams API to make it easy to manipulate our data; we have new methods on collections that reduce the amount of boilerplate we have to write; we have improvements to Comparators to help us to use Streams features is in an easier and more readable way; and we finally have a Date and Time API which is readable and safe to use.
For more tips and techniques on how to use all of these features in a real world Java 8 application, watch my new video series "Building Java 8 Applications with Microservices."]]>
… all which must be accomplished in tandem with getting the job done and contending with the barrage of information, pings, and requests that beg for one's attention throughout each day.
Here's a great infographic depicting the modern learner. While the possibilities for professional and personal growth are exhilarating, it's clear to see they are also overwhelming. And a little bit mind blowing. And more than a little distracting.
So what is a modern learner to do?
Take control of your learning.
You can do this by developing a deliberate learning plan and then making the commitment to follow it. Here are the steps to get started:
Start by developing a running list of the critical skills and knowledge you need to be successful in your job (or dream job) today, as well as the emerging trends, technologies and tools you want to put on your radar. Include soft skills you'd like to develop such as communication, strategy, negotiation, or innovation that are key to your professional and personal growth.
Next prioritize the topics in order of importance and level of difficulty. If you create a one-year plan and break it down into 3-month segments, you can map out more challenging or time-consuming goals to stretch over multiple segments, keep track of smaller goals, and swap out topics as your priorities change. Approach your learning plan as a fluid, working document that will evolve and grow with you and your career. Each three-month segment can include a variety of 3-5 goals that might include:
For example my personal learning plan includes learning about marketing analytics and improving my writing skills, both of which are key to my success as a content strategist. Even though I'm an experienced writer, there are always advancements and trends to learn about and improvements to be made. Every field is evolving so quickly these days, that it's not easy to attain or maintain true expert status. Other goals in my learning plan include developing good time management habits and incorporating yoga into my daily routine. Both of these goals have the benefit of reducing stress, increasing productivity and boosting my overall well-being.
Once you've determined your learning goals, start looking into the resources you'll use to execute your plan, or in other words, learn your stuff. Find out what your employer offers in terms of training courses, a learning portal, and/or an education budget that covers learning materials, classes, conferences, or subscriptions. If you don't have access to company resources, there are plenty of cost-effective options in the form of subscriptions and memberships, and of course loads of free content on the internet. Change up the resources and formats in your plan from quarter to quarter to keep yourself engaged and interested:
books – audiobooks – white papers – guides – articles – blog posts – video courses – certification courses – in-house training – in-person or online classes -workshops – conferences – webinars – tutorials – online forums – professional meet-ups – lectures or talks – mentors
I'm lucky enough to work at Safari Books Online, the business and technology learning portal with 35,000+ of the best and most current books, audio books, video courses and tutorials. This means I have on-demand access to a huge variety of content from leading industry experts and top publishers like Harvard Business Review Press. But I'm also including internet guides, blog posts, classes, and good old fashioned books in my mix.
Once you've determined what you're committed to learning and how, it's time to set your schedule. Determine how much time you'll dedicate each day or week to learning, then block off time on your calendar as you would any other important meeting or commitment. Like physical exercise, it's best to make learning a daily habit, but if it isn't realistic, set aside at least a couple hours a week dedicated to learning. Here are some tips for making the most out of your learning time:
Here are some titles to help you in developing your professional learning plan:
Learn Your Way to Success – Shows you how to customize your professional learning plan to accelerate your career.
Be Your Own Career Coach – This practical and accessible guide includes assessments, assignments, planning exercises and fascinating case-studies.
Experiential Learning – Experiential learning is a powerful and proven approach to learning that is based on one incontrovertible reality: people learn best through experience.
Building the Learning Organization – advice for learning leaders on how to harness the collective genius of people in organizations to build, maintain and sustain the power of the Systems Learning Organization model.
Safari is an invaluable resource for those committed to their professional learning and development. If you already have a membership, you're all set. If you don't use Safari, and want access the dozens of resources referred to in this post — along with 35,000+ other books, videos, articles, audio books, and tutorials — try a 10-day free trial. If you want to learn about Safari for your team or organization, click here.
For years, in the world of JavaScript UI programming, it's been normal to divide our application state among several components, with each component owning its own local state. With the architectures of the past this was essentially unavoidable, but the rise of React and declarative rendering has opened the door for a more powerful paradigm to take hold: the single state atom.
Now that more and more JavaScript libraries are abandoning local component state in favor of the single state atom, certain questions have begun to arise. Can we control which portions of the single state atom nested components can access? Is it possible to make reusable components without giving them ownership over their own state? Could we implement an entire application like this using React 0.14's functional stateless components?
Perhaps most importantly: now that we have a strong ecosystem that can support a single state atom, has local component state gone from a best practice to an alluring trap?
How many databases does a typical Web App use to store its application state? Almost always the answer is "exactly one." Many employ special-purpose databases for queuing and caching (e.g. Redis, Memcache), but relatively few Web Apps grow to the point where it becomes appealing to divide application state across multiple databases.
There's no technical barrier to carving up a database, so what if we did? Let's suppose we put our user account information in one MySQL database, their roles and permissions in a second MySQL database, and recent activity in a third MySQL database.
Doing this would have some immediate upsides. We could be certain that the data for each concern was not mixing with the other concerns. When writing queries for each, they would necessarily be small and simple, because it would take extra coordination effort to loop in the others. There's a certain conceptual appeal to this strict separation.
However, there would also be drawbacks. For one, it would introduce synchronization bugs. What happens if we delete a user account right when something else is querying that user's recent activity from the other database? Someone will get back some crazy data unless we can make that update an atomic transaction—which would be trivial if those values lived in the same database, but which would be more difficult and error-prone with the state distributed like this.
Another drawback is that backups get way harder. Instead of backing up a single database and being able to restore it on a whim if things go wrong, we would instead have a pile of databases to back up and restore. What if we correctly restore almost all of them, but mess one up? Now our application is in a crazy state – and one we may not realize is crazy until it's hung around long enough for nonsensical data to propagate and create more broken data.
These problems stem from the fact that we've taken data that is innately coupled from a business perspective—user accounts, their permissions, and their recent activity—and decoupled it architecturally. Whether we like it or not, these pieces of data depend on one another, and dividing up their storage will make it harder to work with them in tandem.
We encounter similar challenges in front-end programming when each component has its own local state. We all know "Undo/Redo" makes for a nice UX, but just like "backup/restore," it's difficult and error-prone to implement when your state is spread across multiple sources of truth. It's also easier for components to incorrectly synchronize with one another, leading to crazy application states such as the infuriating message notification component that displays "1 unread message" even as the message list component insists that all messages have been read.
A single source of truth eliminates these synchronization problems at the source. The rise of immutable data and declarative rendering, popularized by React, has set the industry on a path to getting rid of these problems once and for all. More and more front-end architectures are embracing the notion of representing application state as an atomic single source of truth, and are seeing the corresponding benefits that back-end databases have enjoyed for years.
Mozilla's James Long has embraced the single state atom for Firefox's developer tools by using Dan Abramov's Redux architecture for React, which allows hot reloading with time travel. David Nolen's Om library for ClojureScript uses a single state atom to enable things like trivial undo/redo, which Sean Grove has reaped the benefits of in easily adding undo/redo functionality to Glint. At NoRedInk we have been happily using Elm with the Elm Architecture, around whose single state atom Laszlo Pandy built the original Time-Traveling Debugger.
Although the benefits of a single state atom are becoming clearer as more and more front-end architectures move in that direction, there are still questions in the community around how to accomplish familiar tasks in this new world. How can we maintain the conceptual benefit of local component state—the fine-grained control over which pieces of code can access which parts of state—when state has only one owner? And how do we build reusable components without giving each component ownership over its own state?
As with many new technologies, the answer turns out to be surprisingly simple.
Consider an accordion widget. When the user clicks a particular section, that section should expand if it was collapsed, and collapse if it was expanded.
Clearly there is state involved there. We need to record which sections are expanded and collapsed—in order to tell how to render the accordion, as well as what to do when a user clicks a given section. How do we do this without giving the accordion some local state to call its own?
On the back-end, we see reusable libraries for common state-based tasks like user authentication. However, these libraries do not come prepackaged with their own MySQL databases; instead, they have an API which allows the application to use them in conjunction with an existing database. The application can store the user data however it pleases, because the authentication library is designed to delegate state storage rather than prepackaging it.
This idea works just as well on the front-end.
Our accordion needs state for two things: rendering and handling user actions. It needs to know the current state (namely, which sections are expanded and collapsed) in order to render, and it needs to change that state (namely, changing which sections are expanded and collapsed) when the user clicks.
Historically, the common way to meet these needs was to give it a chunk of local state, have it read from that to determine how to render, and modify it when the user clicks.
With a single atomic state, instead we delegate: the accordion's API accepts as an argument the expanded/collapsed data necessary to render, as well as state-updating functions ("
expandSection()," "
collapseSection()," etc.) that it can invoke in response to user input. So when the user clicks an accordion section and the accordion wants it to be expanded, it invokes the
expandSection() function it was passed, which in turn updates the single state atom as appropriate.
Not only does this delegation approach scale, it provides a greater degree of control over which components can impact which parts of state. Parent components can provide children with "read access" (providing portions of current state as arguments) but not "write access" (providing functions to the child which can update specifically-chosen parts of the single state atom) or vice versa. You can even provide access selectively depending on the current state!
How would this architecture impact what we can do for our end users?
Suppose we want to implement a UX improvement: if the user refreshes the page, accordions are still expanded and collapsed as they were when the user left. We can easily persist the necessary information in
localStorage, but how hard is this to implement using local state versus a single state atom?
With local component state, every time the user clicks an accordion, we must remember to update
localStorage to persist it. If we forget to, even once, our states will be out of sync, and we could end up in a crazy state where things are displaying in a way that should not be possible through normal user interaction. Those bugs are not fun to track down.
With a single state atom, and an accordion that delegates state ownership to it, this is the easiest thing in the world. Every time our state changes, we save the whole atom to
localStorage. Done. It's never out of sync, and our application is architected such that all we have to do is restore that one atom and all of our UI components Just Work, reusable accordions and all.
This is exactly the approach we took at NoRedInk when we made a reusable accordion in Elm. Its API is designed to delegate state management to a single state atom rather than owning its own local state, giving us the best of both worlds: reusable components and a single state atom.
The release of React 0.14 introduced stateless functional components. From the release notes:
In idiomatic React code, most of the components you write will be stateless, simply composing other components…This pattern is designed to encourage the creation of these simple components that should comprise large portions of your apps.
Here is the example of such a component, from that post:
// A functional component using an ES2015 (ES6) arrow function: var Aquarium = (props) => { var fish = getFish(props.species); return {fish}; };
Let's run with that example. What if we want to let user flip a light switch on and off within our fish tank? We could implement that using local component state, with each tank owning whether its light was turned on or off, but let's try delegating to a single state atom instead.
Here's how that would look in jsFiddle.
Here we have one traditional React component at the root of the hierarchy, called
App. Its job is solely to hold our app's single state atom, and its children are responsible for updating that state as appropriate.
Note that both the
Aquarium and its child
Tank components are stateless; you can be sure of this, because they are using 0.14's stateless functional component style—which does not even support local component state! Nevertheless, the Tank class has no problem updating the application state as necessary, because its parent has provided it with a
setLightOn function which allows it to do so selectively.
Here we can see that the conceptual benefit of local state—fine-grained control over which parts of state components are permitted to access—can be achieved by simple function passing instead, without sacrificing the benefits of the single state atom.
Tank does not have access to the entire application state; it only knows about its one single tank, and all it can do to affect that tank is to switch its lights on or off.
In short, we have avoided mixing its concerns with the rest of our application state simply by being selective in what we pass it!
This may seem like an unusual way of doing things, but remember what this gets us: now if we ever want to serialize our entire application state in order to store it somewhere, or to roll back to a previous state, we can do so at the drop of a hat. Making transactional updates across different parts of the application state is now trivial, so race conditions are much easier to avoid.
For more on stateless components, the second edition of "Developing a React Edge" discusses them as well as other changes introduced in React 0.14. I also expect single state atoms to be a recurring theme at the Reactive2015 conference in Bratislava, given the number of talks on technologies based around single state atoms, such as Redux, Om, and Elm.
Despite the conceptual allure of local component state, the benefits of the single state atom are too strong to ignore:
● A single source of truth makes it easier to perform transactional updates and dodge race conditions.
● Serialization for quick persistence and retrieval of state no longer requires a massive coordination effort.
● Debuggers can reliably support time travel across past UI states.
● Undo/Redo becomes a feature whose implementation is easily within reach.
Considering we can use delegation and function passing to build reusable stateless components with even more fine-grained state permissions as what we had with stateful components, even the conceptual benefit of local component state has an easy replacement in the world of the single state atom.
So here's to the next evolution of front-end development, where we avoid the trap of local component state and start reaping the benefits back-end programmers have enjoyed for years.
Check out the first edition of "Developing a React Edge: The Javascript Library for User Interfaces" and stay tuned for the early access launch of the second edition in the near future. You can also read the other cutting edge books from Bleeding Edge Press on Safari on topics such as Docker, Swift, and AngularJS,]]>
Daniel Arbuckle is a computer scientist and software engineer who has worked with, studied, and built a career around Python starting with version 1.5. He is the author of multiple books and videos on Python for Packt Publishing.
Python's first steps toward an implementation of coroutines came in December 2001 with the release of Python 2.2. At that time, generators were added to the language and Python 2.5 made it easier to use generators as coroutines by changing the
yield statement into an expression which also made it easier to send information into a generator as the result of a
yield expression.
Now, Python 3.3 made it even easier to use generators as coroutines thanks to a new
yield from syntax.
As of Python 3.4, the
asyncio framework is part of the standard library and generator-based coroutines were part of the toolkit. Asynchronous execution in a single thread is great for I/O scheduling and lightweight threading. Now,
asyncio gives us all of that in one convenient package.
All of which brings us to the recent release of Python 3.5, which includes an explicit coroutine type, distinct from generators, new asynchronous
async def,
async for, and
async with statements, and an
await expression that takes the place of
yield from for coroutines.
Why does Python need explicit coroutines and new syntax if generator-based coroutines had become strong enough for inclusion in the standard library? The short answer is that generators are primarily for iteration, so using them for something else (no matter how well it works conceptually) introduces ambiguities. For example, if you hand a generator to Python's
for loop, it's not going to treat it as a coroutine but instead as an iterable.
There were ways to work around this, of course, but it required some odd idioms:
for future in (yield from returns_an_iterable(): value = yield from future yield from do_something(value)
That works, but it's a bit strange. Worse, that's a best case. The idiom gets even stranger when it's not known ahead of time how many objects will need to be iterated over.
It's much clearer now in Python 3.5:
async for value in async_iterable: await do_something(value)
There's another problem, related to Python's special protocol methods such as
__enter__ and
__exit__ which are called by the code in the Python interpreter, leaving the programmer with no opportunity to
yield from it. That meant that generator-based coroutines were not compatible with various important bits of Python syntax such as the
with statement. A coroutine couldn't be called from anything that was called by a special method, whether directly or indirectly, nor was it possible to wait on a future value in a function that was.
The new changes to Python are meant to address these problems. So, what exactly are these changes?
async def is used to define a coroutine. Apart from the
asynckeyword, the syntax is almost identical to a normal
def statement.
The big differences are, first, that coroutines can contain
async for, and
async with syntaxes, and, second, they are not generators, so they're not allowed to contain
yield or
yield from expressions. It's impossible for a single function to be both a generator and a coroutine.
await is used to pause the current coroutine until the requested coroutine returns. In other words, an
await expression is just like a function call, except that the called function can pause, allowing the scheduler to run some other thread for a while. If you try to use an
awaitexpression outside of an
async def, Python will raise a syntax error.
async with is used to interface with an asynchronous context manager, which is just like a normal context manager except that instead of
__enter__ and
__exit__ methods, it has
__aenter__ and
__aexit__ coroutine methods. Because they're coroutines, these methods can do things like wait for data to come in over the network, without blocking the whole program.
async for is used to get data from an asynchronous iterable.
Asynchronous iterables have an
__aiter__ coroutine method which functions like the normal
__iter__ method, but can participate in coroutine scheduling and asynchronous I/O.
__aiter__ should return an object with an
__anext__ coroutine method, which can participate in coroutine scheduling and asynchronous I/O before returning the next iterated value or raising
All of these new features and the convenience they represent are 100% compatible with the existing asyncio scheduler.
Further, as long as you used the @asyncio.coroutine decorator, your existing
asyncio code is also forward compatible with these features without any overhead.
My new video series called "Mastering Python" is available in full on Safari and I have written two books on Python testing called "Learning Python Testing" and "Python Testing Beginner's Guide."]]>
Neil Ford writes about all aspects of information security for IT Governance Ltd, the international provider of best-practice information security solutions.
According to IBM/Ponemon Institute's 2015 Cost of Data Breach Study: Global Analysis, the average global cost of a data breach is now US$3.79 million. Although high-profile cybersecurity incidents affecting large organizations like Ashley Madison and the OPM – to use two recent examples – garner great media interest, it's important not to overlook the fact that small and medium-sized companies also face critical cybersecurity risks every day. Cyber attacks are global, automated, and indiscriminate, targeting every organization that owns or uses electronic information.
Put simply: if you're connected to the Internet, you're vulnerable.
Small businesses are mistaken if they think they're too insignificant to attack. Even if you don't store financial information such as customer payment details, the data you do hold – such as username and login credentials, employee payroll details, proprietary data, or client information – is valuable to someone. For example, even though your website may not be obviously valuable in itself, the information it holds could be used as a means of attacking a larger organization in the supply chain.
Perhaps most alarmingly, it's statistically likely that you've already been successfully attacked but don't know it. The majority of intrusions aren't detected for months – more often when patterns are noticed in stolen data usage that link back to the breached organization. The 2015 Trustwave Global Security Report found that 81% of data breach victims did not detect the incident themselves, and Mandiant/FireEye's M-Trends 2015 report found that the median number of days that threat groups were present on a victim's network before detection was 205. For reference, it takes criminal hackers an average of seven days to exploit a vulnerability according to NopSec's 2015 State of Vulnerability Risk Management report.
So, what should you do?
The first thing to note is that cybersecurity is a business issue, not an IT issue. Many companies mistakenly believe that technology solves all their cybersecurity problems, but solutions like firewalls, and anti-malware and antivirus programs, though obviously important, only provide partial protection. A robust information security management system (ISMS) should address people and processes as well as technology.
The principle is the same as with physical security: there's no point spending thousands on monitored alarms, window shutters, and sturdy locks if no one knows to lock up when they leave the office at night.
Passwords are a common point of intrusion. Far too often, default passwords are left unchanged, or weak and easily cracked passwords are employed by lazy users.
If another website has been compromised and login details have been stolen, criminals will automate attacks using these username/password combinations to see what else they can gain access to. Password reuse is rife, so the statistical chances of criminals gaining access to multiple sites with a single set of stolen credentials are vast.
Likewise, default passwords for servers and software are frequently left unchanged, and can be exploited by criminals with little to no effort. Misconfigured installations are exposing petabytes of data according to recent analysis by BinaryEdge.
Direct cyber attacks are not your biggest problem, however: the biggest risk you must address is the threat posed by your own staff and suppliers who – whether maliciously or, more likely, inadvertently – expose your systems to cyber criminals.
The IBM X-Force Threat Intelligence Quarterly, 2Q 2015 report 2015 found that 95% of insider breaches were found to be the result of human error, such as clicking on malicious links in phishing emails. When every employee can endanger a company with a single mouse-click, staff training is essential.
This will ensure that your employees are fully aware of the information security threats they face and can act quickly and instinctively to phishing campaigns, spam emails, malicious websites, and the like.
Privilege management processes must also be followed to ensure that access to critical systems and information is limited strictly to those who need it. According to Verizon's 2015 Data Breach Information Report, privilege abuse accounted for 55% of insider threats.
Patch management is another major problem. Many organizations' websites use common, off-the-shelf content management system (CMS) platforms, software, applications, and plugins, which often contain vulnerabilities that can be exploited by hackers. Although patches for vulnerabilities are frequently released, few organizations manage to install them within the recommended time.
Verizon's 2015 Data Breach Investigations Report found that over 90% of attacks exploited known vulnerabilities for which patches were already available: "Many existing vulnerabilities remain open, primarily because security patches that have long been available were never implemented. In fact, many of the vulnerabilities are traced to 2007 — a gap of almost eight years." According to Trustwave's 2015 Global Security Report, 98% of tested web applications were found to be vulnerable.
For organizations that want to address information security threats properly, ISO 27001, the information security management standard, sets out the requirements of an ISMS (information security management system) – a holistic approach to information security that encompasses people, processes, and technology, and addresses all of the points illustrated above.
An ISO 27001-compliant ISMS enables organizations of all sizes, sectors, and locations to mitigate the risks they face with appropriate controls, limiting the inadvertent threats posed by untrained staff, inadequate procedures and out-of-date software solutions.
It requires staff to be adequately trained, their access rights to be suitably controlled, and a best-practice approach to information security to be adopted throughout the organization.
If you're concerned about your organization's susceptibility to insider security threats, you need to ensure that everyone in the organization behaves responsibly.
Cybersecurity isn't just for the IT department – it's for everyone.
IT Governance Ltd. provides a large selection of cyber security and IT-related titles to Safari. For more on this topic, check out "Information Security Breaches: Avoidance and Treatment Based on ISO27001, Second Edition" and "CyberWar, CyberTerror, CyberCrime and CyberActivism, 2nd Edition."
You can also dig through our new cyber security list in honor of National Cyber Security Awareness Month 2015.]]>
Introverts and extroverts often find their counterparts an indecipherable mystery, but it does not have to be that way according to author Jennifer B. Kahnweiler. With some careful nurturing and planning, introverts and extroverts can exploit their diversity to make truly creative and powerful partnerships.
Just in time for National Cyber Security Awareness Month, Basic Books added this book to Safari. The author explains how to think like an attacker to plan and implement your security strategies. The topics cover the best practices in areas such as private business and the military.
A selection from the "Sharpening Your Skills" series from Harvard Business Review Press – this volume guides managers through decision-making while avoiding making quick decisions from the gut.
Updated with new practical summaries of the key topics, this second edition helps anyone better understand the personality strength and weaknesses of the vital people in their life. Chapters cover culture-based values, which kinds of personalities to avoid, and gene expression's role in character traits.]]>
Sari Greene is an information security practitioner and entrepreneur. She is the author of Security Program and Policies: Principles and Practices, 2nd Edition and The CISSP Complete Video Course.
What keeps me up at night isn't the threat actors or the cybercriminals, the newly identified exploit, or even the prospect of every "thing" having an IP address. My worry is the frightening lack of qualified cybersecurity leadership. We need professionals with critical thinking skills and risk-based reasoning. Women and men who can educate, motivate, mentor, and inspire. Dedicated, committed, and passionate individuals who recognize that cybersecurity is about protecting our companies, our communities, and our countries from those who seek to do us harm as well as the mundane accidental or inadvertent consequence of every day actions.
A Rand Corporation study estimates there are around 1,000 top-level cybersecurity experts globally vs. a need for 10,000 to 30,000.
More than 209,000 cybersecurity jobs in the U.S.A. are unfilled and postings are up 74-percent over the past five years, according to a Peninsula Press analysis of numbers from the Bureau of Labor Statistics. The demand for information security professionals is expected to grow by 53 percent through 2018.
2015 PwC US State of Cybercrime Survey asked whether organizations have the expertise to address cyber-risks associated with implementation of new technologies and only 26-percent said they have capable personnel on staff.
The cybersecurity leadership shortage isn't just sheer numbers nor is it a technology issue. The challenge is to fill cybersecurity positions with people who can align cybersecurity and organizational objectives and more importantly, who can effectively communicate those objectives to a diverse population. In addition to information systems professionals, we should be encouraging people of all ages with experience in law, finance, compliance, audit, engineering, education, law enforcement, and the military to join the ranks of cybersecurity professionals. Once we've attracted the talent, we need to expediently transition professionals from other disciplines into the realm of cybersecurity. And we need to know they're ready to take on the task.
The (ISC)2 CISSP certification is the globally recognized gold standard for cybersecurity professionals. According to research published by BurningGlass, in 2014 there were nearly 50,000 postings for workers with a CISSP (Certified Information Systems Security Professional) credential. BurningGlass estimates that amounts to three-quarters of all the people who hold that certification in the United States—and presumably most of them already have jobs
Earning a CISSP certification requires documented applicable experience and demonstrated knowledge. The CISSP exam has long been derided as being a mile wide and an inch deep. The examination poses 250 questions that span across eight security domains ranging from governance and risk management to software development. In my opinion, that is exactly the reason it is the right exam and the right certification for the next generation of cybersecurity leadership.
This interdisciplinary approach requires that the CISSP candidate study topics that may be outside of their comfort zone or area of interest. Most candidates start their certification journey being intimately familiar with two or three security domains, moderately familiar with two or three, and unfamiliar with the rest. Studying for the CISSP exam forces candidates to learn and correlate concepts that may otherwise go unknown and unnoticed. The result is a holistic rather than myopic understanding of cybersecurity security objectives. And the learning process doesn't end with the exam. Maintaining a CISSP certification requires ongoing continuing education across domains.
There are a variety of study options available. Candidates should choose a study option that matches their learning style and is designed to truly enhance their cybersecurity knowledge and understanding; not just regurgitate exam answers.
Having a study plan and sticking to it is critical. But going it alone can be difficult. Fortunately, there are (ISC)2 Chapters around the world that provide an opportunity to build a local network of peers. Chapters welcome CISSP candidates and many offer mentoring programs.
Cybersecurity is a career with a purpose. It is so much more than a paycheck. It is a chance to really make a difference. Cybersecurity leaders influence decisions that impact individual privacy, critical infrastructure service availability, economic prosperity and global security. Whatever the interest – from a small nonprofit to a multinational corporation, jobs are waiting for the right leader.
The CISSP Complete Video Course contains 24+ hours of training with content divided into 9 lessons with 94 video sub-lessons. The videos consist of live trainer discussions, screencasts, animations, and live demos. The video lessons in this course review each exam objective so you can use this course as a complete study tool for taking the CISSP exam. It also includes plenty of end-of-lesson quizzes and a full practice exam.
You can also browse a new list of popular and recent cybersecurity titles and don't forget to tweet about Cyber Security Awareness on Twitter with the hashtag #NCSAM.]]>
As the first installment of a new regular feature here at the Safari blog, we'd like to shine a spotlight on Perseus and their spectrum of content.
Written by noted futurist James Canton, this book is designed to challenge the way you think and inspire you to become more predictive about future trends and changes that might affect your business, career, or life. Topics include reinventing education, climate engineering, and the future of work and jobs.
Through an investigation of sixteen landmark American moments dating back to colonial times, this fourth edition examines how the media (from print to electronic) has shaped the country since it's inception.
An illuminating look at how easily your privacy and personal information (no matter how seemingly mundane) can be compromised in today's digital age. Scheer describes the implications of surveillance and data collection on realms such as foreign policy and the Internet.
If someone you know is struggling with how to bring the online world into their life then this book from Seal Press might be the right starting point for them. Author Sarah Granger thoughtfully explains concepts such as friending/following, using the web for business, and even how and why to hashtag.
Don't settle for less or accept the first offer you receive. Implement the tips and techniques from this book into your life to improve your negotiation and influence skill with topics such as mapping out the negotiation, finding common ground, and multiparty negotiation tactics.
You can also check out this new list in Safari which includes a slew of Perseus titles covering a myriad of topics. We also recently posted a Q+A with "Rise of the Robots" author Martin Ford – which is also from Basic Books.]]>