Victoria Drake's Blog the latest posts from Victoria Drake.Hugo -- (Victoria Drake) (Victoria Drake)Tue, 27 Jul 2021 04:26:26 -0400Measuring productivity with GitHub issues, 30 Aug 2021 05:35:02 (Victoria Drake) a little Pandas help improve your development velocity? <img src="" width="640" height="427"/>How long does it take for a bug to get squashed, or for a pull request to be merged? What kind of issues take the longest to close?

Most organizations want to improve productivity and output, but few technical teams seem to take a data-driven approach to discovering productivity bottlenecks. If you’re looking to improve development velocity, a couple key metrics could help your team get unblocked. Here’s how you can apply a smidge of data science to visualize how your repository is doing, and where improvements can be made.

Getting quality data

The first and most difficult part, as any data scientist would likely tell you, is ensuring the quality of your data. It’s especially important to consider consistency: are dates throughout the dataset presented in a consistent format? Have tags or labels been applied under consistent rules? Does the dataset contain repeated values, empty values, or unmatched types?

If your repository has previously changed up processes or standards, consider the timeframe of the data you collect. If labeling issues is done arbitrarily, those may not be a useful feature. While cleaning data is outside the scope of this article, I can, at least, help you painlessly collect it.

I wrote a straightforward Python utility that uses the GitHub API to pull data for any repository. You can use this on the command line and output the data to a file. It uses the list repository issues endpoint (docs), which, perhaps confusingly, includes both issues and pull requests (PRs) for the repository. I get my data like this:

$ python -h
usage: [-h] [--token TOKEN] repository months
$ python OWASP/wstg 24 > data.json

Using the GitHub API means less worry about standardization, for example, all the dates are expressed as ISO 8601. Now that you have some data to process, it’s time to play with Pandas.

Plotting with Pandas

You can use a Jupyter Notebook to do some simple calculations and data visualization.

First, create the Notebook file:

touch stats.ipynb

Open the file in your favorite IDE, or in your browser by running jupyter notebook.

In the first code cell, import Pandas and load your data:

import pandas as pd

data = pd.read_json("data.json")

You can then run that cell to see a preview of the data you collected.

Pandas is a well-documented data analysis library. With a little imagination and a few keyword searches, you can begin to measure all kinds of repository metrics. For this walk-through, here’s how you can calculate and create a graph that shows the number of days an issue or PR remains open in your repository.

Create a new code cell and, for each item in your Series, subtract the date it was closed from the date it was created:

duration = pd.Series(data.closed_at - data.created_at)

Series.describe() will give you some summary statistics that look something like these (from mypy on GitHub):

count                           514
mean      5 days 08:04:17.239299610
std      14 days 12:04:22.979308668
min                 0 days 00:00:09
25%          0 days 00:47:46.250000
50%                 0 days 06:18:47
75%          2 days 20:22:49.250000
max               102 days 20:56:30

Series.plot() uses a specified plotting backend (matplotlib by default) to visualize your data. A histogram can be a helpful way to examine issue duration:

duration.apply(lambda x: x.days).plot(kind="hist")

This will plot a histogram that represents the frequency distribution of issues over days, which is one way you can tell how long most issues take to close. For example, mypy seems to handle the majority of issues and PRs within 10 days, with some outliers taking more than three months:

Histogram for mypy issues over the last six months

It would be interesting to visualize other repository data, such as its most frequent contributors, or most often used labels. Does a relationship exist between the author or reviewers of an issue and how quickly it is resolved? Does the presence of particular labels predict anything about the duration of the issue?

You aim for what you measure

Now that you have some data-driven superpowers, remember that it comes with great responsibility. Deciding what to measure is just as, if not more, important than measuring it.

Consider how to translate the numbers you gather into productivity improvements. For example, if your metric is closing issues and PRs faster, what actions can you take to encourage the right behavior in your teams? I’d suggest encouraging issues to be clearly defined, and pull requests to be small and have a well-contained scope, making them easier to understand and review.

To prepare to accurately take measurements for your repository, establish consistent standards for labels, tags, milestones, and other features you might want to examine. Remember that meaningful results are more easily gleaned from higher quality data.

Finally, have fun exercising your data science skills. Who knows what you can discover and improve upon next!

There are better options for a privacy-respecting phone, 11 Aug 2021 11:37:35 (Victoria Drake) the new, better Apple. Here's how to choose your phone and set it up. <img src="" width="640" height="480"/>Whether you think the news of Apple scanning your private devices was a big deal, run-of-the-mill, or something we all should have seen coming, you might be wondering, “What now?” We know full well that Google is looking at the stuff on your phone too (and Gmail, and… well, everywhere else) so it’s not like there are other options after Apple… right?

If a move towards privacy is what we’re after, we know a new off-the-shelf Google phone isn’t a better answer – but there are more options.

If you don’t want the details, jump straight to The TL;DR at the end.

Linux phones (sort of)

Unless you’re a rather tolerant tech-savvy tinkerer, a Linux phone isn’t one of these options… yet. I’ve personally been very excited about the bevy of emerging options in this space, from freedom-oriented hardware to fully open source, crowd-developed operating systems.

The current state of these efforts is that this magical mashup just isn’t ready yet. Most Linux phone OS such as Ubuntu Touch, Mobian, Pure OS, etc, are in a “mostly working” state, with the missing features ranging from “lack of reliable push notifications” to “intermittent Bluetooth connectivity” to “camera.”

If all you need is text messaging and a web browser, yes, you can probably go this route. For most users however, this isn’t going to make daily-driver status.

If a Linux phone would suit you, I recommend getting your hands on a PinePhone and running Arch Linux ARM (releases on GitHub) with Plasma Mobile.

De-googled Android

For a daily-driver, “de-googled” Android is your best bet. Android itself (specifically, the Android Open Source Project source code) is based on a modified Linux kernel and is free and open source software. When we typically think of “Android phones,” we refer to Android devices with Google’s proprietary software added to the mix, including Google Play Services. A “de-googled” Android phone is essentially the Android OS without Google’s spyware services included by default.

Keep in mind that this route still involves some DIY. You’ll need to install an OS on a device yourself. Don’t worry, there are step-by-step guides available – the most technical thing you’ll likely have to do is copy and paste some commands into your terminal.

Free and open source Android OS comes in multiple flavors, and the choice isn’t arbitrary. Your selection of a “de-googled” phone is going to be determined by a couple factors: the hardware device you have or that you want to use, and the apps (software) you want to run on it.


The phone you may already have (or the one you’re willing to purchase) will influence your choice of operating system (OS).


At the time I’m writing this, if you have an older Pixel or another model of Android phone, your best bet for a hassle-free OS with A-class support will be Lineage. Here’s a link to the LineageOS list of supported devices. Clicking on your device here will get you to some installation instructions for your phone.


If you have a newer Pixel (generation 3 up to the newer 5) then GrapheneOS could be the way to go. Here are the devices officially supported by GrapheneOS. They also have easy-to-follow installation instructions and help via chat. It is possible to run GrapheneOS on other phones, but not without substantial DIY for which technical knowledge would help.

Generally speaking, GrapheneOS is intended to be a security-hardened operating system targeted at individuals who won’t be miffed if there are tradeoffs for mitigating vulnerabilities. If you don’t have those requirements or intend to use Google Apps on your phone (see Software), then LineageOS will likely suit you better.

Speaking of security, yes, my favorite VPN runs just fine on GrapheneOS via the Aurora store (see below).

New phone, who dis?

If you’re looking to purchase a new phone, you have some flexibility. My general recommendation is to pick up last-season’s version of the model you want. Not only will this likely be cheaper (and often a great deal if you buy refurbished) but the open source community that develops these operating systems will have had more time to work with the device itself, which could help ensure better compatibility and a smoother set up.

Consider buying a refurbished phone (sometimes called “renewed”) locally when you can. This can help fund the small businesses that offer them.


What do you need to do on your phone? Privacy and convenience are typically at odds (a far larger topic I won’t dig into right now) so it can help to narrow down the functionality you need. If your needs look something like:

  • Calls and texts
  • Web browser
  • Web-based email via browser

Then you’re good to go, right out of the box, with either LineageOS or GrapheneOS. They’ll both include free and open source apps that let you do all these things.

If you want a particular application that doesn’t come pre-installed, here’s where we get into some nuance. Your choices depend on the level of privacy you’d like to maintain. Here are your avenues for installing apps, listed in order of preference.

1. Official APKs

Some particularly privacy-focused applications offer an Android Package Kit (APK) that you can download directly in order to install the app. You should only download these when you’ve navigated directly to a domain that the organization owns. Here are my favorites:

You can download and install APKs whether you choose LineageOS or GrapheneOS.

2. Use F-Droid

If you can’t find an APK for something you want, search for it on F-Droid.

The F-Droid software repository allows you to download and install apps in much the same way that the Google Play store does, with a couple notable differences. All the apps here are free and open source, and no account or profile is required to download them. The F-Droid APK itself can be downloaded and installed from directly on either LineageOS or GrapheneOS.

Just like any open source software, it’s up to the user (you) to ensure that you’re downloading and installing software you trust. If you want help or advice, F-Droid has a healthy community that you can interact with in lots of ways, including via IRC, Matrix, and the Fediverse.

You can find an app for pretty much anything here: from your general-store type functions such as to-do lists, music players, and maps; to specific niche security applications, and even a tea timer. Here are some well-known choices I can easily recommend:

3. Aurora Store

If you need an app that isn’t available on F-Droid, your next stop is the Aurora Store. This is an unofficial client for the Google Play Store that lets you download free applications anonymously, without signing into a Google account. Most applications found in the larger stores can be downloaded this way, without requiring Google’s proprietary stuff on your phone, including the aforementioned VPN app.

When loading Aurora Store for the first time, be sure to choose the “Anonymous” option instead of signing in.

The Aurora Store itself can be installed via F-Droid or It works on either LineageOS or GrapheneOS – however, apps that require less private permissions or access will probably work better on LineageOS.

Keep in mind that your phone OS in no way supports these apps directly, or knows what’s in them, or what sort of tracking and information exchange they may be up to. It’s a slight privacy downgrade, but still better than a fully Google-ified OS.

4. If you need Google Apps

If this will be your only phone and you simply must have Google Apps on it (think Google Play Store, Gmail, Calendar, Photos, etc) then go with LineageOS. You can choose to try emulating Google Play Services using LineageOS for microG, or install the Google Apps add-on when you install LineageOS.


Here’s the “Internet personality quiz” version of everything above. You are…

  1. Knowledgeable about Linux; mostly use a phone for text, calls, and web browser; and potentially want to help develop Linux phone software.
    • Try a Linux phone such as the PinePhone, but consider one of the other options as a back up for when you just need stuff to work.
  2. Security or privacy inclined, happy to use FOSS apps, or do most things via web browser anyway.
  3. Someone who needs Google Apps to work, or you want a phone that isn’t a Pixel, or you’re setting up a device for someone who’s fine using Android but needs it to look familiar.
    • Use LineageOS with any of its supported devices. Click on the device name for installation instructions.
    • If you must have Google Apps and need Google Play Services to work, install the add-on at the same time you install LineageOS.
    • Optionally, download the F-Droid or Aurora Store for installing apps.

Whichever route you choose, my advice is to treat this like a learning experiment. You’re sort of building your own phone, after all, and gaining all the technological independence that comes with that knowledge. If possible, don’t ditch your current phone until you try out one (two?) of these paths. The one you end up liking most could surprise you! It’s great to have options.

Building in context, 09 Aug 2021 03:17:49 (Victoria Drake) best laid plans are made better by staying close to context. <img src="" width="640" height="466"/>It’s a comedy classic – you’ve got a grand idea. Maybe you want to build a beautiful new dining room table. You spend hours researching woodcraft, learn about types of wood and varnish, explore different styles of construction, and now you have a solid plan. You buy the wood and other materials. You set up in the garage. For months you measure and saw, sand, hammer and paint. Finally, the effort is worthwhile. The table is finished, and it’s fantastic.

In a frenzy of accomplishment you drag it into the house – only to discover that your dining room doorway is several inches too small. It doesn’t fit.

Art only imitates life, so you may say that this comedic example is unrealistic. Of course an experienced DIY-er would have measured the doorway first. In real life, however, unforeseen hindrances rarely come in ones: once you get the table in the door, you discover the floor’s uneven. Perhaps the chairs you’ve chosen are a few inches too short… and so on.

Far from attempting to persuade you away from your next DIY project, I’d like to help make those and any other projects you take on go even smoother. The same patterns are found in furniture-building as in software development: it’s always better to build in context.

The planning fallacy

Few software developers are accurate when it comes to time and cost estimates. This isn’t a failing of software engineers, but a human tendency to towards optimism when it comes to predicting your own future. First proposed by Daniel Kahneman and Amos Tversky in 1979, the planning fallacy is no new concept.

In one study, students were asked to estimate how long they would take to finish their senior theses. The estimates, an average 27.4 days at the optimistic end and 48.6 days at the pessimistic end, came up predictably short. The average actual completion time was 55.5 days.

The study proposed two main hypotheses as to why this happens: first, that people tend to focus on their future plans rather than their past experiences; and second, people don’t tend to think that past experiences matter all that much to the future anyway.

You can probably find examples of this in your own life without trying too hard, perhaps couched in the infamous “only because” envelope. Sure, that last “weekend project” turned into a two-week affair, but that was only because you had to go run some unexpected errands. Or maybe you didn’t finish that blog post when you mean to, but that’s only because your siblings dropped in to visit. You’re absolutely, positively, definitely certain that next time would be different.

In reality, people are just plain poor at factoring in the unexpected daily demands of life. This makes sense from a certain perspective: if we were good at it, we’d probably have a lot more to fret about on a daily basis. Some measure of ignorance can make life a little more blissful.

That said, some measure of accurate planning is also necessary for success. One way we can improve accuracy is to work in context as much as possible.


Let’s consider the dining room table story again. Instead of spending months out in the garage, what would you do differently to build in context?

You might say, “Build it in the dining room!” While that would certainly be ideal for context, both in homes and in software development, it’s rarely possible (or palatable). Instead, you can do the next best thing: start building, and make frequent visits to context.

Having decided you’d like to build a table, one of the first questions is likely, “How big will it be?” You’ll undoubtedly have some requirements to fulfill (must seat six, must match other furniture, must hold the weight of your annual twenty-eight-course Christmas feast, etc.) that will lead you to a rough decision.

With a size in mind, you can then build a mock up. At this point, the specific materials, style, and color don’t matter – only its three dimensions. Once you have your mock table, you now have the ability to make your first trip to the context in which you hope it will ultimately live. Attempting to carry your foam/wood/cardboard/balloon animal mock up into the dining room is highly likely to reveal a number of issues, and possibly new opportunities as well. Perhaps, though you’d never have thought it, a modern abstractly-shaped dining table would better compliment the space and requirements. (It worked for the Jetsons.) You can then take this into account in your next higher-fidelity iteration.

This process translates directly to software development, minus the Christmas feast. You may have already recognized the MVP approach; however, even here, putting the MVP in context is a step that’s frequently omitted.

Where will your product ultimately live? How will it be accessed? Building your MVP and attempting to deploy it is sure to help uncover lots of little hiccups at an early stage.

Even when teams have prior experience with stacks or technologies, remember the planning fallacy. People have a natural tendency to discount past evidence to the point of forgetting (memory bias). It’s also highly unlikely that the same exact team is building the same exact product as the last time. The language, technology, framework, and infrastructure have likely changed in at least some small way – as have the capabilities and appetites of the engineers on your team. Frequent visits to context can help you run into any issues early on, adapt to them, and create a short feedback loop.

Go for good enough

The specific meaning of putting something in context is going to vary from one software project to another. It may mean deployment to cloud infrastructure, running a new bare metal server, or attempting to find out if your office across the ocean can access the same resources you use. In all cases, keep those short iterations going. Don’t wait and attempt to get a version to 100% before you find out if it works in context. Send it at 80%, see how close you got, then iterate again.

The concept of building in context can be applied at any stage – of course, the sooner the better! Try applying this idea to your project guidance today. I’d love to hear how it goes.

Leading with a cybersecurity mindset, 27 Jul 2021 04:26:26 (Victoria Drake) for considering software development from a security standpoint.Times and technologies change, but a few good ideas are still the same. With consistent application, a handful of wise practices can help deter a slew of cybersecurity attacks. While implementation differs across applications, learning to lead development teams with a cybersecurity mindset boils down to a few fundamental concepts:

  • Be a bad guy
  • Fail secure
  • Practice software minimalism

A slight change in thinking can create a sea change in security. Let’s examine how.

Let’s be bad guys

When it comes to cybersecurity, I take a pragmatic approach. There aren’t enough sheaves of NIST recommendations in the world to help you if you aren’t accustomed to thinking like the bad guy. To best lead your team to defend against hacking, first know how to hack yourself.

A perusal of the resources linked at the end of this article can help you with a starting point, as will general consideration of your application through the lens of an outsider. Are there potentially vulnerable forms or endpoints you might examine first? Is there someone at your company you could call on the phone and surreptitiously get helpful information from? Defense is a difficult position to hold in any battle. If you aren’t the first person to consider how your application might be attacked, you’ve already lost.

Develop your sense of how to be the bad guy. Every component of software, every interaction, every bit of data, can be useful to the bad guy. The more you hone your ability to consider how a thing can be used for ill, the better able you’ll be to protect it.

When looking at information, ask, “How can I use this information to gain access to more important information?” When considering a user story, ask, “What happens if I do something unexpected?”

In all things, channel your inner four-year-old. Push all the buttons.

Playing offense on your own application lets you fix vulnerabilities before they happen. That’s a luxury you won’t get from the real bad guys.

Fail secure

Every part of a system will fail with 100% certainty on a long enough timescale. Thinking a step ahead can help to ensure that when it does, the one failure doesn’t leave your application wide open to others.

To fail secure means that when a system or code fails to perform or does something unexpected, any follow-on effects are halted rather than permitted. This likely takes many forms in many areas of your application, so here are the more common ones I see.


When gating access, deny by default. This most often takes the form of whitelisting, or colloquially, “no one is allowed, except for the people on this list.” In terms of code flow, everything should be denied first. Only allow any particular action after proper credentials are verified.


For automated workflows such as deployments, ensure each step is dependent on the last. Don’t make the (rather common) mistake of connecting actions to triggers that can kick off a workflow before all the necessary pieces are in place. With the smorgasbord of cloud and CI tools available, failure events may not be obvious or noisy.

Be careful to avoid running flows on timed triggers unless they are completely self-contained. Workflows that unpredictably run faster or slower than expected can throw a whole series of events into disarray, leaving processes half-run and states insecure.

Exception handling

Errors are a frequent gold mine for attackers. Ensure your team’s code returns “pretty” errors with content that you can control. “Ugly” errors, returned by default by databases, frameworks, etc, try to be helpful by providing lots of debugging information that can be extremely helpful to a hacker.

Software minimalism

If your development team doesn’t currently have one central source of information when it comes to keeping track of all your application components, here’s a tip you really need. In software security, less is more (secure).

The more modular an application is, the better its various components can be isolated, protected, or changed out. With a central source of truth for what all those components are (and preferably one that doesn’t rely on manual updates), it’s easier to ensure that your application is appropriately minimalist. Dependency managers, such as Pipenv, are a great example.

Few industries besides technology seem to have produced as many acronyms. Philosophies like Don’t Repeat Yourself (DRY), Keep It Simple Stupid (KISS), You Aren’t Going to Need It (YAGNI), and countless other methodologies all build upon one very basic principle: minimalism. It’s a principle that warrants incorporation in every aspect of an application.

There’s a reason it takes little skill to shoot the broad side of a barn: barns are rather large, and there’s quite a lot of one to hit. Applications bloated by excessive third-party components, repeated code, and unnecessary assets make similarly large targets. The more there is to maintain and protect, the easier it is to hit.

Like Marie Kondo’s method for dispatching the inevitable creep of household clutter, you can reduce your application’s attack surface by considering each component and asking whether it brings you joy. Do all of this component’s functions benefit your application? Is there unnecessary redundancy here? Assess each component and decide how integral it is to the application. Every component is a risk; your job is to decide if it’s a worthwhile risk.

Bonus: your personal Yodas

With the basic principles of learning to think like the bad guy, failing securely, and practicing software minimalism, you’re now ready to steep in the specifics. Keeping the fundamentals in mind can help you lead your team to focus your cybersecurity efforts where it matters most.

No Jedi succeeds without a little help from friends. Whether you’re a beginner in the battle against the dark side or a twice-returned-home Jedi Master, these resources provide continuing training and guidance.

I hope you find these thought systems helpful! If you find your interest piqued as well, you can read more of what I’ve written about cybersecurity here.

A GitHub guide for non-technical leaders, 24 May 2021 00:00:00 (Victoria Drake) leaders can get to where the work happens and take part in business decisions on GitHub. <img src="" width="640" height="218"/>As I write this, the front page of declares in big bold letters that this is “Where the world builds software.” This is true. In technology companies today, the creation of your product is largely happening where your developers spend time. It’s where big and small product decisions are made every day – the kind of decisions that, wittingly or not, will decide the future of your company.

I’m writing this guide for a very specific person – possibly you, or someone you know. I’ll explain how a non-technical business leader can find information and take part in the decisions and questions that happen only on GitHub. You don’t need to know how to use Git. You just need a few minutes to follow along, and a desire to be a resource and servant leader for your teams. Let’s do it!

If you haven’t signed up yet, click below to read the very first steps. Once you’re logged in, read on to join in!

The very first steps

GitHub comes in two different flavors: GitHub Enterprise, or If your team uses, you can sign up here using your work email.

With GitHub Enterprise, signing up depends on your individual company’s configuration. For example, you may be set up to log in with SAML single sign-on (such as through your GSuite credentials). Get in touch with the folks administering Enterprise in order to get signed up or logged in.

For the rest of this guide, it doesn’t matter if you’re using GitHub Enterprise or – they’re largely the same. Just ensure you get connected to your company’s Organization or Team, if there is one. Someone with administrative privileges needs to invite you using the email you signed up with.

Where the magic happens

On GitHub, work is typically grouped by projects or products into what’s called repositories. Your team or company may have just one of these that they regularly use (they might call it a “monorepo”) or several repositories that represent different technical components of a single product.

Once you log in, you’ll be on the Recent activity page. You can search for the name of the repository you want to visit in the search bar at the top left. If your company’s repositories are private, you may need to be invited by an administrator in order to view it.

Example search

When you view a repository, it looks like this. I’ve pointed out some of the important bits.

Screenshot of the OWASP Web Security Testing Guide repository

In any repository, there are two main areas where decisions usually take place. These are in Issues and Pull Requests, and you’ll mainly focus your attention here. Click on the Issues tab to see these.

You’ll be presented with a list of Issues, which you can think of as individual topics. This format is essentially a discussion board. Clicking on any of the Issue titles will take you to its thread.

Issues page screenshot

Here’s where the magic happens! Folks on your team use Issues to discuss all kinds of topics. These may be a very technical and esoteric cost-benefit analysis, or a fundamental customer-facing design decision. A quick read of the first message in the thread is likely to reveal whether it’s a decision that could use your help.

Issues are a starting point for work. Here, team members make decisions about the type and scope of a change they plan to make.

When someone has their changes ready, they’ll open a Pull Request so that other team members can preview and give input on those changes before they become part of the repository. Click the Pull Request tab at the top to view these.

You’re presented with a very similar view on this page – yes, it’s another discussion board! You can click on any Pull Request to view its thread as well.

Pull Requests have some additional tabs that team members use for code reviews. All the conversation will show up in the Conversation tab.

Screenshot of Pull Request

Sorting out what’s relevant

A lot of discussion happens in Issues and Pull Request threads, and not all of it may be relevant for you to look at. Thankfully, GitHub has some excellent collaboration tools that can help your team direct your attention to where it’s most needed. These are @mentions and labels.

Ask to be @mentioned

These work the same way on GitHub as they do on Twitter. When someone @mentions you, you’ll receive a notification via email, or you’ll see it in your Notification Center when you’re logged in. This depends on your notification settings, which you should adjust to your liking. If you only want to be notified when someone @mentions you or replies to you, you should uncheck everything on the Notification Settings page except for your preferred options under Participating.

Notification Settings for @mentions

Now when someone references you in a discussion on GitHub, you’ll be notified and you’ll have the chance to respond!

Using labels

Another less direct way to see where you can effectively contribute is to ask your team to use labels. You may recall seeing these in the right sidebar of Issue and Pull Request threads:

Screenshot of labels in the right sidebar

You can create different labels to categorize a discussion, and you can apply as many labels to a discussion as you like. In order to have your team draw your attention to threads that might benefit from your input or guidance, ask folks to use a label to point these out. This could be the question label, or any new label of your choosing.

Clicking on a label in the sidebar will take you to a page that shows all the Issues or Pull Requests with that label. The URL will look something like:<organization name>/<repository name>/labels/question

You can also bookmark this page to easily check it on a regular schedule. This is a great, low-friction way for your team to indicate areas that could use your input.

Collaborating on files

Similar to tracking changes in Google Docs or Word documents, you can edit documents in GitHub in a way that lets your team see the changes you’ve made. This is a fantastic method for collaborating with your team where they work, and avoids the hassle of emailing attachments around.

Text files in repositories have extensions such as .md for Markdown, or .txt for plain text. The majority of documentation on GitHub is in Markdown format. Clicking on any file in the repository file list will open it on the GitHub website. At the top right of the document, look for these buttons:

Document actions

Clicking on the pencil icon will let you edit the file right there in your web browser. You may not see anything special as you’re typing, but once you commit (like saving) your file, all your changes are tracked with Git! For a step-by-step guide to editing, see Editing files in your repository.

Here are some helpful articles for formatting text with Markdown on GitHub.

Proactive participation

GitHub is well-structured as a collaboration platform. That’s why people of all professions use it not just for software development, but also for networking, getting jobs and sponsorships, and even for hosting simple no-code websites. My own company uses GitHub for everything from collaborating on company documentation to automated Change Control Board processes for FedRAMP.

At your leisure, I encourage you to chase your curiosity and explore. Don’t be shy about asking questions, or asking technical folks on your team to explain something if you think it will enable you to be a bigger help to them. With so much of the world building software on GitHub, there’s a lot you can contribute when you’re where the work happens.

Digital resilience: redundancy for websites and communications, 22 Feb 2021 04:00:43 (Victoria Drake) you can make your digital life more resilient when using services you don't own.When what seems like half the planet noped out of WhatsApp after its terms of service update, applications like Signal (which I highly recommend) saw an unprecedented increase in user traffic. Signal had so many new users sign up that it overwhelmed their existing infrastructure and lead to a 24-hour-ish outage. The small team responded impressively quickly, especially given that a 4,200% spike in new users was utterly implausible before it ocurred.

The downside of so many people moving onto this fantastic application is that it caused a brief outage. If you rely solely on a certain application for your communications, brief outages can be debilitating. Even when it seems implausible that your favorite chat, email, or website service could just – poof – vanish overnight, recent events have proved it isn’t impossible.

Have a backup plan. Have several. Here’s how you can improve your digital resiliency for things like websites, messaging, and email.


I recommend Signal because it is open source, end-to-end encrypted, cross-platform, and offers text, voice, video, and group chat. It’s usually very reliable; however, strange things can happen.

It’s important to set up a backup plan ahead of any service outages with the people you communicate with the most. Have an agreement for a secondary method of messaging – ideally another end-to-end encrypted service. Avoid falling back on insecure communications like SMS and social media messaging. Here’s a short list for you to explore:

If you’re particularly technically inclined, you can set up your own self-hosted chat service with Matrix.

Having a go-to plan B can help bring peace of mind and ensure you’re still able to communicate when strange things happen.

Cloud contacts

Do you know the phone numbers of your closest contacts? While memorizing them might not be practical, storing them solely online is an unnecessary risk. Most services allow you to export your contacts to vCard or CSV format.

I recommend keeping your contacts locally on your device whenever you can. This ensures you still know how to contact people if your cloud provider is unavailable, or if you don’t have Internet access.

Full analog redundancy is also possible here. Remember that paper stuff? Write down the phone numbers of your most important contacts so you can access them if your devices run out of battery or otherwise can’t turn on (drop your phone much?).

Local email synchronization

If your email service exists solely online, there’s a big email-shaped hole in your life. If you can’t log in to your email for any reason – an outage on their end, a billing error, or your Internet is down – you’ll have no way to access your messages for however long your exile lasts. If you think about all the things you do via email in a day, I think the appropriate reaction to not having local copies is 🤦.

Download an open source email client like Thunderbird. Follow instructions to install Thunderbird and set it up with your existing online email service. Your online service provider may have a help document that shows you how to set up Thunderbird.

You can maximize your privacy by turning off Thunderbird’s telemetry.

To ensure that Thunderbird downloads your email messages and stores them locally on your machine:

  1. Click the “hamburger” overflow menu and go to Account Settings
  2. Choose Synchronization & Storage in the sidebar
  3. Ensure that under Message Synchronizing, the checkbox for Keep messages in all folders for this account on this computer is checked.

You may need to visit each of your folders in order to trigger the initial download.

Some other settings you may want to update:

  1. Choose Composition & Addressing and uncheck the box next to Compose messages in HTML format to send plaintext emails instead.
  2. Under Return Receipts choose Global Preferences. Select the radio button for Never send a return receipt.

You don’t need to start using Thunderbird for all your email tasks. Just make sure you open it up regularly so that your messages sync and download to your machine.


I strongly believe you should have your own independent website for reasons that go beyond redundancy. To truly make your site resilient, it’s important to have your own domain.

If you know that my website is at the address, for example, it doesn’t matter whether I’m hosting it on GitHub Pages, AWS, Wordpress, or from a server in my basement. If my hosting provider becomes unavailable, my website won’t go down with it. Getting back up and running would be as simple as updating my DNS configuration to point to a new host.

Price is hardly an excuse, either. You can buy a domain for less than a cup of coffee with my Namecheap affiliate link (thanks!). Namecheap also handles your DNS settings, so it’s a one-stop shop.

With your own domain, you can build resiliency for your email address as well. Learn how to set up your custom domain with your email provider. If you need to switch providers in the future, your email address ports to the new service with you. Here are a few quick links for providers I’d recommend:

Build your digital resiliency

I hope you’ve found this article useful on your path to building digital resiliency. If you’re interested in more privacy topics, you might like to learn about the VPN I use or great apps for outsourcing security.

If your threat model includes anonymity or censorship, building digital resiliency is just a first step. The rest is outside the scope of my blog, but here are a few great resources I’ve come across:

Create a self-hosted chat service with your own Matrix server, 15 Feb 2021 01:38:07 (Victoria Drake) speed-run introduction to Matrix via Dendrite. <img src="" width="640" height="466"/>Matrix is an open standard for decentralized real-time communication. The specification is production-ready and bridges to tons of silo products like Slack, Gitter, Telegram, Discord, and even Facebook Messenger. This lets you use Matrix to link together disjoint communities in one place, or create an alternative communication method that works with, but is independent of, communication silos.

You can create your own self-hosted Matrix chat for as little as $3.50 USD per month on an AWS Lightsail instance. Your homeserver can federate with other Matrix servers, giving you a reliable and fault-tolerant means of communication.

Matrix is most widely installed via its Synapse homeserver implementation written in Python 3. Dendrite, its second-generation homeserver implementation written in Go, is currently released in beta. Dendrite will provide more memory efficiency and reliability out-of-the-box, making it an excellent choice for running on a virtual instance.

Here’s how to set up your own homeserver on AWS Lightsail with Dendrite. You can also contribute to Dendrite today.

Create a Lightsail instance

Spin up a new Lightsail instance on AWS with Debian as your operating system. It’s a good idea to create a new per-instance key for use with SSH. You can do this by with the SSH key pair manager on the instance creation page. Don’t forget to download your private key and .gitignore your secrets.

Click Create Instance. Wait for the status of your instance to change from Pending to Running, then click its name to see further information. You’ll need the Public IP address.

To enable people including yourself to connect to the instance, go to the Networking tab and add a firewall rule for HTTPS. This will open 443 so you can connect over IPv4. You can also do this for IPv6.

Connect DNS

Give your instance a catchier address by buying a domain at Namecheap and setting up DNS records.

  1. On your domain management page in the Nameservers section, choose Namecheap BasicDNS.
  2. On the Advanced DNS tab, click Add New Record.

Add an A Record to your Lightsail Public IP. You can use a subdomain if you want one, for example,

  • Type: A Record
  • Host: matrix
  • Value:

This points to your Lightsail instance.

Set up your Matrix homeserver

Change permissions on the private key you downloaded:

chmod 600 <path/to/key>

Then SSH to your Public IP:

ssh -i <path/to/key> admin@<public ip>

Welcome to your instance! You can make it more interesting by downloading some packages you’ll need for Dendrite. It’s a good idea ot use apt for this, but first you’ll want to make sure you’re getting the latest stuff.

Change your sources list in order to get the newest version of Go:

sudo vim /etc/apt/sources.list

Delete everything except these two lines:

deb buster main
deb-src buster main

Then replace the distributions:

:%s/buster main/testing main contrib non-free/g

Run sudo apt dist-upgrade. If you’re asked about modified configuration files, choose the option to “keep the local version currently installed.”

Once the upgrade is finished, restart your instance with sudo shutdown -r now.

Go make some coffee, then SSH back in. Get the packages you’ll need with:

sudo apt update
sudo apt upgrade
sudo apt install -y git golang nginx python3-certbot-nginx

You’re ready to get Dendrite.

Get Dendrite

Clone Dendrite and follow the README instructions to get started. You’ll need to choose whether you want your Matrix instance to be federating. For simplicity, here’s how to set up a non-federating deployment to start:

git clone
cd dendrite

# Generate a Matrix signing key for federation (required)
./bin/generate-keys --private-key matrix_key.pem

# Generate a self-signed certificate (optional, but a valid TLS certificate is normally
# needed for Matrix federation/clients to work properly!)
./bin/generate-keys --tls-cert server.crt --tls-key server.key

# Copy and modify the config file - you'll need to set a server name and paths to the keys
# at the very least, along with setting up the database connection strings.
cp dendrite-config.yaml dendrite.yaml

Configure Dendrite

Modify the configuration file you just copied:

sudo vim dendrite.yaml

At minimum, set:

  • server name to your shiny new domain name, e.g.
  • disable_federation to true or false
  • registration_disabled to true or false

You might like to read the Dendrite FAQ.

Configure nginx

Get the required packages if you didn’t already install them above:

sudo apt install nginx python3-certbot-nginx

Create your site’s configuration file under sites-available with:

cd /etc/nginx/sites-available
ln -s /etc/nginx/sites-available/<sitename> /etc/nginx/sites-enabled/<sitename>
sudo cp default <sitename>

Edit your site configuration. Delete the root and index lines if you don’t need them, and input your server name.

Your location block should look like:

location / {
    proxy_pass https://localhost:8448;

Remove the default with: sudo rm /etc/nginx/sites-enabled/default.

Create self-signed certificates

You can use Certbot to generate self-signed certificates with Let’s Encrypt.

sudo certbot --nginx -d <>

If you don’t want to give an email, add the --register-unsafely-without-email flag.

Test your configuration and restart nginx with:

sudo nginx -t
sudo systemctl restart nginx

Then start up your Matrix server.

# Build and run the server:
./bin/dendrite-monolith-server --tls-cert server.crt --tls-key server.key --config dendrite.yaml

Your Matrix server is up and running at your web address! If you disabled registration in your configuration, you may need to create a user. You can do this by running the included dendrite/bin/createuser.

You can log on to your new homeserver with any Matrix client, or Matrix-capable applications like Pidgin with the Matrix plugin.

Other troubleshooting

Log files

If you get an error such as:

... [] setupFileHook
  Couldn't create directory /var/log/dendrite: "mkdir /var/log/dendrite: permission denied"

You’ll need to create a spot for your log files. Avoid the bad practice of running stuff with sudo whenever you can. Instead, create the necessary file with the right permissions:

sudo mkdir /var/log/dendrite
sudo chown admin:admin /var/log/dendrite

# Build and run the server:

Unable to decrypt

If you see: Unable to decrypt: The sender's device has not sent us the keys for this message. you may need to verify a user (sometimes yourself).

  1. In your client, open the user’s profile. Click the lock icon if there is one, or otherwise look for a way to verify them.
  2. You may be asked to see if some emojis presented to both users match if you’re using certain clients like Element.
  3. You can then re-request encryption keys for any sent messages.

Set up your own Matrix server today

I hope you found this introduction to setting up your own Matrix homeserver to be helpful! If you have anything to add, feel free to reply via Webmention.

Do I raise or return errors in Python?, 09 Feb 2021 05:34:48 (Victoria Drake), return, and how to never fail silently in Python.I hear this question a lot: “Do I raise or return this error in Python?”

This is probably because the right answer will depend on your situation and the goals of your application logic. Either choice can help you ensure your Python code doesn’t fail silently, saving you and your teammates from having to hunt down deeply entrenched errors.

Here’s the difference between raise and return when handling failures in Python, and how to ensure your code doesn’t fail silently.

When to raise

The raise statement allows the programmer to force a specific exception to occur. (8.4 Raising Exceptions)

Use raise when you know you want a specific behavior, such as:

raise TypeError("Wanted strawberry, got grape.")

Raising an exception terminates the flow of your program, allowing the exception to bubble up the call stack. In the above example, this would let you explicitly handle TypeError later. If TypeError goes unhandled, code execution stops and you’ll get an unhandled exception message.

Raise is useful in cases where you want to define a certain behavior to occur. For example, you may choose to disallow certain words in a text field:

if "raisins" in text_field:
    raise ValueError("That word is not allowed here")

Raise takes an instance of an exception, or a derivative of the Exception class. Here are all of Python’s built-in exceptions.

Raise can help you avoid writing functions that fail silently. For example, this code will not raise an exception if JAM doesn’t exist:

import os

def sandwich_or_bust(bread: str) -> str:
    jam = os.getenv("JAM")
    return bread + str(jam) + bread

s = sandwich_or_bust("\U0001F35E")
# Prints "🍞None🍞" which is not very tasty.

To cause the sandwich_or_bust() function to actually bust, add a raise:

import os

def sandwich_or_bust(bread: str) -> str:
    jam = os.getenv("JAM")
    if not jam:
        raise ValueError("There is no jam. Sad bread.")
    return bread + str(jam) + bread

s = sandwich_or_bust("\U0001F35E")
# ValueError: There is no jam. Sad bread.

Any time your code interacts with an external variable, module, or service, there is a possibility of failure. You can use raise in an if statement to help ensure those failures aren’t silent.

Raise in try and except

To handle a possible failure by taking an action if there is one, use a tryexcept statement.

    s = sandwich_or_bust("\U0001F35E")
except ValueError:

This lets you buy_more_jam() before re-raising the exception. If you want to propagate a caught exception, use raise without arguments to avoid possible loss of the stack trace.

If you don’t know that the exception will be a ValueError, you can also use a bare except: or catch any derivative of the Exception class with except Exception:. Whenever possible, it’s better to raise and handle exceptions explicitly.

Use else for code to execute if the try does not raise an exception. For example:

    s = sandwich_or_bust("\U0001F35E")
except ValueError:
    print("Congratulations on your sandwich.")

You could also place the print line within the try block, however, this is less explicit.

When to return

When you use return in Python, you’re giving back a value. A function returns to the location it was called from.

While it’s more idiomatic to raise errors in Python, there may be occasions where you find return to be more applicable.

For example, if your Python code is interacting with other components that do not handle exception classes, you may want to return a message instead. Here’s an example using a tryexcept statement:

from typing import Union

def share_sandwich(sandwich: int) -> Union[float, Exception]:
        bad_math = sandwich / 0
        return bad_math
    except Exception as e:
        return e

s = share_sandwich(1)
# Prints "division by zero"

Note that when you return an Exception class object, you’ll get a representation of its associated value, usually the first item in its list of arguments. In the example above, this is the string explanation of the exception. In some cases, it may be a tuple with other information about the exception.

You may also use return to give a specific error object, such as with HttpResponseNotFound in Django. For example, you may want to return a 404 instead of a 403 for security reasons:

if object.owner != request.user:
    return HttpResponseNotFound

Using return can help you write appropriately noisy code when your function is expected to give back a certain value, and when interacting with outside elements.

Make your errors noisy

Silent failures create some of the most frustrating bugs to find and fix. You can help create a pleasant development experience for yourself and your team by using raise and return to handle errors in Python.

What tech leaders do before going on vacation, 01 Feb 2021 04:02:54 (Victoria Drake) to be a responsible leader even while you're away. <img src="" width="640" height="320"/>As a technical person who leads a technical team, I know firsthand that it can be easy to get lost in finishing up your own work before a vacation. It takes a bit of dedicated attention to ensure you don’t neglect the day-to-day tasks that don’t vanish while you’re away.

Here’s a pre-PTO checklist to make sure you’ve taken care of those responsibilities before you take off for a much-deserved vacation.

Does everyone know who’s supposed to do what?

A lot of information lives in your head alone, despite valiant efforts to document all the things. That’s the nature of the work. Instead of attempting to disseminate everything you’re thinking through, focus on the work your team will do while you’re away.

Hopefully, you already have a centralized list of priorities. Ensure it’s up to date, and that all the tasks that could conceivably be done during your time off have been assigned a caretaker.

Are there any decisions waiting on me?

Review notes, discussion boards, and ask your team directly if anyone is waiting on an answer from you. If you’re able to make a decision before your vacation, do so. If not, delegate the decision to someone else with a clear explanation of your overall goals and any applicable parameters.

Don’t neglect internal auto-responders

Set up auto-responders for communications from the public, as well as from your team! While a response from your public email may say that you’re out of the office and when you expect to be back, an internal auto-responder is an opportunity to provide even more value.

It could take the form of an email response for your internal inbox, but a post on your team’s message board or chat channel also works. Let people know where to look for information they might need, who to turn to if they need help while you’re away, and where to find your centralized priority list so they can decide what to work on next.

Tidy up loose ends

Finally, close out work that depends on you and that no one else can do. If it’s work you can delegate with some written guidance attached, you might choose this route instead of attempting to finish it yourself in a hurry.

If you hand off any work, ensure that you communicate clear instructions as well as any deadlines.

Set expectations and follow them

Your vacation is precious time to recoup, relax, and make room for those creative moments that only visit a quiet mind. Ensure you set expectations with your team for how often you might check in, if at all.

If you do decide to log in while you’re away, be sure to protect your privacy and your company data with a privacy-focused VPN on all your devices.

Responsible vacation planning for technology leaders

I hope you benefit from the ideas in this post. Here’s an easy way to put them into practice right now: grab this post as a pre-PTO Notion checklist template and instantly gain conscientious leadership powers!

Get checklist as a Notion template

Don’t leave your team members hanging while you’re away! If you have other ideas for good things to do before your vacation, I’d love to hear about it.

Add search to Hugo static sites with Lunr, 26 Jan 2021 09:25:17 (Victoria Drake) your static site searchable with a client-side search index.Yes, you can have an interactive search feature on your static site! No need for servers or paid subscriptions here. Thanks to the open source Lunr and the power of Hugo static site generator, you can create a client-side search index with just a template and some JavaScript.

A number of my readers have been kind enough to tell me that you find my blog useful, but there’s something that you don’t know. Up until I recently implemented a search feature on, I had been my own unhappiest user.

My blog exists for all to read, but it’s also my own personal Internet brain. I frequently pull up a post I’ve written when trying to re-discover some bit of knowledge that I may have had the foresight to record. Without a search, finding it again took a few clicks and more than a few guesses. Now, all my previous discoveries are conveniently at my fingertips, ready to be rolled into even more future work.

If you’d like to make your own personal Internet brain more useful, here’s how you can implement your own search feature on your static Hugo site.

Get Lunr

While you can install lunr.js via npm or include it from a CDN, I chose to vendorize it to minimize network impact. This means I host it from my own site files by placing the library in Hugo’s static directory.

You can save your visitors some bandwidth by minifying lunr.js, which I did just by downloading lunr.js from source and using the JS & CSS Minifier Visual Studio Code extension on the file. That brought the size down roughly 60% from 97.5 KB to 39.35 KB.

Save this as static/js/lunr.min.js.

Create a search form partial

To easily place your search form wherever you like on your site, create the form as a partial template at layouts/partials/search-form.html

<form id="search"
    action='{{ with .GetPage "/search" }}{{.Permalink}}{{end}}' method="get">
    <label hidden for="search-input">Search site</label>
    <input type="text" id="search-input" name="query"
    placeholder="Type here to search">
    <input type="submit" value="search">

Include your search form in other templates with:

{{ partial "search-form.html" . }}

Create a search page

For your search to be useful, you’ll need a way to trigger one. You can create a (static!) /search page that responds to a GET request, runs your search, and displays results.

Here’s how to create a Hugo template file for a search page and get it to render.

Create layouts/search/list.html with the following minimum markup, assuming you’re inheriting from a base template:

{{ define "main" }}
{{ partial "search-form.html" . }}

<ul id="results">
        Enter a keyword above to search this site.
{{ end }}

In order to get Hugo to render the template, a matching content file must be available. Create content/search/ to satisfy this requirement. The file just needs minimal front matter to render:

title: Search me!

You can run hugo serve and navigate to /search to see if everything builds as expected.

A few libraries exist to help you build a search index and implement Lunr. You can find them here on the Hugo site. If you want to fully understand the process, however, you’ll find it’s not complicated do this without additional dependencies, thanks to the power of Hugo’s static site processing.

Build your search index

Here’s how to build an index for Lunr to search using Hugo’s template rendering power. Use range to loop over the pages you want to make searchable, and capture your desired parameters in an array of documents. One way to do this is to create layouts/partials/search-index.html with:

<script> = {
    // You can specify your blog section only:
    // For all pages in your site, use "range .Site.Pages"
    // You can use any unique identifier here
    "{{ .Permalink }}": {
        // You can customize your searchable fields using any .Page parameters
        "title": "{{ .Title  }}",
        "tags": [{{ range .Params.Tags }}"{{ . }}",{{ end }}],
        "content": {{ .Content | plainify }}, // Strip out HTML tags
        "url": "{{ .Permalink }}"
    {{ end }}
<!-- Include Lunr and code for your search function,
which you'll write in the next section -->
<script src="/js/lunr.min.js"></script>
<script src="/js/search.js"></script>

When Hugo renders your site, it will build your search index in much the same way as a List page is built, creating a document for each page with its parameters.

The last piece of the puzzle is the code to handle the search process: taking the search query, getting Lunr to perform the search, and displaying the results.

Perform the search and show results

Create static/js/search.js to hold the JavaScript that ties it all together. This file has three main tasks: get the search query, perform the search with Lunr, and display the results.

Get query parameters with JavaScript

This part’s straightforward thanks to URLSearchParams:

const params = new URLSearchParams(
const query = params.get('q')

Search for the query with Lunr

Define and configure an index for Lunr. This tells Lunr what you’d like to search with, and you can optionally boost elements that are more important.

const idx = lunr(function () {
    // Search these fields
    this.field('title', {
        boost: 15
    this.field('content', {
        boost: 10

    // Add the documents from your search index to
    // provide the data to idx
    for (const key in {
        id: key,

You can then execute the search and store results with:

const results =

Display results

You’ll need a function that builds a list of results and displays them on your search page. Recall the id you gave your ul element in layouts/search/list.html and store it as a variable:

const searchResults = document.getElementById('results')

If a search results in some results (🥁), you can iterate over them and build a <li> element for each one.

if (results.length) { // Length greater than 0 is truthy
    let resultList = ''
    for (const n in results) {
      // Use the unique ref from the results list to get the full item
      // so you can build its <li>
      const item = store[results[n].ref]
      resultList += '<li><p><a href="' + item.url + '">' + item.title + '</a></p>'
      // Add a short clip of the content
      resultList += '<p>' + item.content.substring(0, 150) + '...</p></li>'
    searchResults.innerHTML = resultList

For each of your results, this produces a list item similar to:

        <a href=".../blog/add-search-to-hugo-with-lunr/">
        Add search to Hugo static sites with Lunr
    <p>Yes, you can have an interactive search feature on your static site!...</p>

If there are no results, ham-handedly insert a message instead.

else {
    searchResults.innerHTML = 'No results found.'

Full code for search.js

Here’s what static/js/search.js could look like in full.

search.js full code

function displayResults (results, store) {
  const searchResults = document.getElementById('results')
  if (results.length) {
    let resultList = ''
    // Iterate and build result list elements
    for (const n in results) {
      const item = store[results[n].ref]
    searchResults.innerHTML = resultList
  } else {
    searchResults.innerHTML = 'No results found.'

// Get the query parameter(s)
const params = new URLSearchParams(
const query = params.get('query')

// Perform a search if there is a query
if (query) {
  // Retain the search input in the form when displaying results
  document.getElementById('search-input').setAttribute('value', query)

  const idx = lunr(function () {
    this.field('title', {
      boost: 15
    this.field('content', {
      boost: 10

    for (const key in {
        id: key,

  // Perform the search
  const results =
  // Update the list with results

Make it go

You now have Lunr, the search index, and the code that displays results. Since these are all included in layouts/partials/search-index.html, add this partial on all pages with a search form. In your page footer, place:

{{ partial "search-index.html" . }}

You can see what this looks like when it’s all put together by trying it out on my blog.

Make it go faster

Since your site is static, it’s possible to pre-build your search index as a JSON data file for Lunr to load. This is where those aforementioned libraries may be helpful, since a JSON-formatted search index would need to be built outside of running hugo to generate your site.

You can maximize your search speed by minifying assets, and minimizing computationally expensive or blocking JavaScript in your code.

Static sites get search, too!

I hope this helps you make your Internet brain more useful for yourself and others, too! Don’t worry if you haven’t got the time to implement a search feature today – you can find this tutorial again when you visit and search for this post! 🥁

Make your own independent website, 16 Jan 2021 08:41:27 (Victoria Drake) to make 2021 the year of the independent web.The web that raised me was a digital playground in the truest sense. It was made up of HTML experiments frankensteined together by people still figuring it all out.

The beauty of not completely knowing what you’re doing is a lack of premature judgement. Without a standard to rise to, you’re free to go sideways. Explore. Try things that don’t work, without any expectation they will work. An open world with a beginner’s mindset.

The web that raised me was a little broken. Things didn’t always display the way they were supposed to. That too is part of the beauty. It was just broken enough to make you think for yourself.

1991 was the year of the individual on the web, the first year any layperson could open a web browser and access the new hypermedia dimension. There were no go-to, search-suggested, centralized websites. There were newsgroups. You had what you made and what your meatspace contacts sent you. In 2021, I think we need a return to that level of individualism. We need to make 2021 the year of the independent web.

That’s not to say I think the massive monopolistic platforms are going anywhere. Twitter, Facebook, mainstream “news” media sites – they’re all a kind of utility now, like plumbing and electricity. They’ll find their place in regulation and history. But they are not your website.

Your website is the one you create. Where the content, top-to-bottom, is yours alone to shape and present as you please. Your website is your place of self-expression, without follower counts or statistics to game. Your website is for creation, not reaction.

It’s all yours, but it doesn’t have to seem lonely. Your site can interact with the entire online world through syndication and protocols made possible by this thing we call the Internet. See:

Your website is your beginning point. The one source of truth for your identity online, from which you can generate and distribute disposable copies to any platform you please. This is what it means to truly own your content. And on the Internet, your content is you.

This is my website. When I first created it, I did so for myself. I had no expectation of visitors. I just knew I’d rather have these thoughts and things I’ve learned here, out here, made indelible in the folds of the public Internet, instead of on some dark corner of my machine, to be lost forever once I am.

Make your own website. You’ll grow your own sense of well-deserved accomplishment and contribute to your independence on the web. You’ll learn by doing, by scratching your own itch.

Learn about web technologies. Use them as you would if you were a child holding a pencil or paintbrush for the first time. Experiment, with no expectations other than discovering what you can do to make it delight you.

These sites and articles inspired this post and helped me implement webmentions!

How to get hired as a software developer, 12 Jan 2021 05:50:53 (Victoria Drake) to know before applying for a software developer job.I’m asked this question a lot, so let me be the first to give you the good news: there’s no one right answer. As general tech-literacy increases, the culture of the coding industry is steadily, thankfully, moving away from a checklist approach. Instead of degrees and pre-requisites when it comes to deciding whether you’re qualified to be hired as a software developer, companies (including my own) are far more concerned with just one question. What can you do?

There are some general best practices that will make you a far more attractive hire than the majority of applicants, and I’ll discuss those in this post. For the most part, however, demonstrating what you’re capable of is the best way to increase your chances of getting to the interview and beyond. Here’s how to get hired as a software developer.

First, build projects

Companies who are primarily focused on getting products built want to see that you’ve built products. They don’t need to be flashy or for-profit, but they do need to work. I’m far more likely to consider a candidate with a colorful bouquet of working code in their GitHub or GitLab or CodePen portfolio. Here are some basic ideas to get you started:

The best projects you could showcase are ones directly related to the specialty you want to apply for. Show that you have competency with the fundamentals. For instance, if you see yourself focusing on front end, demonstrate that you can build interactive web pages with no fancier tools than HTML, CSS, and vanilla JS. For back end focused developers, show that you know how to create a fundamental tool like an API for a service on a local server. Want to be well-rounded? Create an API with a web page UI.

Spend some time creating a good README. Use screenshots, highlight code snippets, include well-written instructions for local set up. Show that you care about and fully understand your own work.

Explore specific frameworks and libraries if they interest you, but keep in mind that those won’t be interesting to a company unless the company already wants to use that framework.

You maximize your chances of getting hired by demonstrating that you already have the ability to learn on your own, build, and then present projects to the world. Out of everything else in this article, this is the one fundamental trait that a company won’t want to have to teach you.

Next, stand out

Familiarity with the following topics, along with demonstrating that understanding in your own code, will put you miles ahead of most applicants.

Reusable code

Companies that build products are concerned with getting the most bang for their buck. This is the same idea as wanting to save yourself time when you’re creating something individually. If you put a week of effort into building something, it would be nice if you could keep easily using it for a long time afterwards. How can you maximize the return on your efforts?

Be familiar with DRY code. Avoid creating highly customized pieces that only fit a particular use case, peppered with hard-coded variables and dependent on a particular input structure. Avoid writing code that is hard to update in the future. Recognize when you’re writing a script or library that could apply in many different situations, and understand how to turn it into a reusable module.

Types and mutability

Besides building projects, debugging them can be a company’s most expensive task. It takes a lot of time to hunt down and fix bugs, but you can help reduce that cost by understanding the subtler ways that a lot of bugs occur. Understanding types and mutability – whether and how an object can be changed – can help open the door to even greater technical proficiency.

Get familiar with at least one type system. If there’s a linter available for your language, use it. Understand how immutable and mutable objects work in the language you use. Be able to describe specific use cases for either one. Understand at a general level how mutability impacts system resources, the difference between referencing and copying, and what it means to be thread-safe.

Follow-on effects

Keep in mind that organizations are made up of people. When you work together with colleagues, your work has an effect on someone else’s. Being aware of these effects and demonstrating conscientiousness in this area will help show potential employers that you’d benefit the team as a whole.

Consider the second and third-order effects of code you write. Avoid writing code that will unnecessarily slow down a larger system. Understand what blocking means and how to use concurrency or parallelism in your project. Include your thoughts on follow-on effects in your READMEs. Show that you always have the larger project, effort, costs, or organization in mind.

Other nice-to-haves

If you’ve fully taken advantage of the points above, you’re most of the way to getting hired already. Seal the deal with these easy wins.

Be a friendly open source participant

There’s no better way to show a potential employer you can work well on a team than by providing plenty of examples. Get involved with open source projects, contribute features and fixes, and interact with contributors and maintainers. Create a real-life simulation for your future colleagues that leaves no doubt about what you’d be like to work with. The further back this history goes, the better, so start right away.

Communicate with care

If you’re participating in open source or working remotely, most of your communication with your colleagues is going to take place online in text. Without facial expressions, tone or inflection, this form of communication leaves a lot to be desired. Some extra care on your part can help make sure your message always comes across as intended.

Get into a habit of drafting most everything you write, especially for long-form communication. Putting yourself in the mindset of creating a draft first lets you take all the time you need to craft your message. You can make sure you’re choosing appropriate words and coming across with the emotions you intend to convey. Feeling hurried? Remember the golden rule of online communication: you never need to reply right away. Take a breath, then take your time.

My list of things I use has some great apps I use for drafting. While some people may disagree with me, I say: use emojis liberally. 🤩

Finally, use your imagination

Software developers are creative people by necessity. Before you can write code, build a project, or design a page, you first have to be able to imagine it! So put that skill to good use.

In every application, every email, every chat message with your potential employer, imagine yourself in their position. What do they care about right now? What current goals does the company have? What information about yourself can you share that would make them feel comfortable hiring you?

Take your best guess, and then ask if you got it right. “I think the company is looking for someone to [insert guess here], is that accurate?” Show that you have both the capability to anticipate future needs and the desire to identify and solve them.

Get yourself hired

Admittedly, this post is my own wishlist. Good candidates for software development positions are hard to come by, and people who can rightly say they do everything above are rare. I don’t think the discrepancy is due to a lack of ability; perhaps just a lack of information.

I’ve seen both sides of the virtual interview table, and this post is a result of me figuring things out the long and circuitous way. I hope this helps you to take a more direct route to getting yourself hired as a software developer.

How to become a software developer, 05 Jan 2021 04:50:07 (Victoria Drake) article I wish I had read when I started coding. <img src="" width="640" height="320"/>As a Director of Engineering, I’m a software developer who hires and leads other software developers. It’s not surprising then that I get asked this question a lot, in various forms:

  • How do I become a software developer?
  • What language or framework should I learn first?
  • Where do I start?

While I’m certain there’s no one right answer for everyone, I’m also certain that the world needs more software developers and systems thinkers.

The best thing I can do to help you lead yourself, learn to code, and become a software developer is to share the most efficient parts of how I did it myself. This is the article I wish I had read when I started coding.

Depth matters

Comic of an iceberg with &ldquo;Enough to build a web app&rdquo; on top and &ldquo;The really interesting stuff&rdquo; on the bottom.

Software is exceedingly complex. Like a good novel that you wish you’d never finish reading, there’s always more to discover and learn. If you don’t want to miss the best parts, don’t be satisfied with surface-level explanations. Always go deeper! Ask why, why, and why again until you get to the fundamentals. Soon enough, you’ll start to see patterns.

By digging deeper, you’ll begin to understand the fundamentals of how things connect, what makes things “fast,” and facets of software operation that you probably can’t even imagine exist. It’s like peeking behind the curtain and seeing a whole world of systems and processes that most people are never aware of.

Going in-depth can expand your mind and your capacity for learning. Keep asking why. Follow every link. Let your curiosity guide you.

Hard stuff matters

Giving yourself the chance to be delighted through discovery doesn’t come for free. It takes a lot of hard work to read and compress complicated ideas into your meat brain.

Comic of hammering hard things into your head

It’s important not to gloss over the hard stuff. In fact, if something seems too hard to understand, you might benefit from doing it first. You might have to get creative to find ways to explain things to yourself, but when you succeed, it makes everything else easier later on.

Analogies are helpful for understanding hard concepts, but they’ll only help you start to understand concepts at a surface level. Remember to go in-depth. Don’t stop at the analogy.

Writing matters

Write right away. Create a habit of explaining everything you learn to yourself in long-form writing. Better than bullet points, writing with a conversational tone engages parts of your brain that help you to process and remember new information. It’s why humans like and remember stories, and it’s a superpower you get for free.

Start by writing for yourself. Write about what interests you. Try something new, even if it seems rudimentary, and write in-depth about what you learn. (One of my most popular posts is about iteration in Python. When I first wrote it, I considered myself a complete beginner.)

If you want to go a step further, share your writing with the world. Learn in public, like I do. I often get questions like, “how do I choose a theme for my blog?” or “what platform should I use?” or “what popular language/framework/topic should I focus on?” My answer is: don’t worry about it.

Don’t fret too much about your blog theme or platform. Pick the easiest option for you to get started with for now. All of that will change and improve as you learn, practice, and find your focus. Just start writing, ideally, yesterday.

Write for yourself by explaining what you’re doing, as if it were past-you teaching future-you — because it is. You will be your first reader, and the first judge of how useful your blog can be. Seek to impress yourself!

The language, framework, or version doesn’t matter

Why pigeonhole your abilities before you even start? Pick any software language, framework, or technology that seems to make sense to you when you first read it. Start there.

Remember that it’s important to dig deep and understand the fundamentals. Basic concepts of software transcend languages. Whichever first language you choose, understand functions, variables, return values, iteration, and how immutability works. You’ll find that learning these concepts will make it easier to recognize them in your second language, and learn that too.

Your portfolio doesn’t matter

If your first objective is to build a portfolio, you may be trying to run before you walk. Building a portfolio to showcase to potential employers is a great goal, but a terrible first step.

If you think of creating a polished portfolio as a first step, you’re liable to spend too much time making it pretty and presentable before focusing on the content. As someone who hires software developers, I can tell you wholeheartedly that I’d rather see clean and well-written code than a flashy front page.

A comic of an in-progress portfolio, text reads, &ldquo;Portfolio in progress. Watch me build at;

Don’t confuse building a portfolio with building projects. Absolutely build projects, right from the beginning. There’s no better way to see the practical application of what you’re learning. Just treat them as first drafts, as training ground, and don’t worry about packaging them up for professional consumption.

By allowing yourself to build some draft projects first, you allow yourself the breathing room to learn from them. Focus on iteration, on making one small thing better each time, and you’ll build a portfolio without even realizing it.

Focus on what matters

Don’t follow this advice blindly; rather, incorporate it into your own systems. Experiment, make it work better than when you found it, then pay it forward by writing down what you’ve learned for someone else to read!

Here are my favorite books for reading or listening to if you want to cultivate a learning mindset. See non-coding books for coders.

If this article benefits you in some way, I encourage you to write about it! The process of learning how to learn is never finished. You can be the next iteration.

Be brave and build in public, 24 Dec 2020 04:57:31 (Victoria Drake) every day feel like Christmas when you participate in open source communities.I used to think that when I wanted to make updates to a project, I ought to hold back and do a big re-launch with all the changes at once. I figured that seeing big changes would feel like Christmas!

Unfortunately, Christmas only comes once a year. After years in the tech and cybersecurity world, my perspective has changed.

I’ve found that people, including myself, value receiving small, constant, incremental improvements far more than big changes once or a few times a year. It makes sense if you think about it. The former constantly delights in small, unexpected ways that make the user experience better. The latter is invisible, except for a few times a year.

There are occasions when big changes make sense. Say you’re re-launching functionality, or coinciding with an event that deserves all the fanfare of an unveiling.

Other than that, and for most of us, holding back doesn’t serve us at all. It may even come from something far more insidious: fear of judgement.

Being brave

Thinking such as “I’ll show it to the world when it’s ready,” always leaves out the most important detail. What does “ready” mean?

Your MVP (a chandelier) vs. What Users Want (a lightbulb) comic

If you haven’t written down your definition of “ready,” consider that you may be holding back for no good reason. What’s the worst that could happen, anyway, if you make your work public when it’s less than perfect?

I decided to find out when I started to build in public. Instead of holding back work, I released a first version as soon as it functioned as intended. I leaned on v0.0.* tags as a way to say, “This is available, but still in progress.” Or, I’d say so outright, in the README.

In the world of open source, building in public can be scary stuff. It feels like making yourself vulnerable. It’s opening up part of yourself, a creative part, for scrutiny and nitpicking – by strangers. Of course it’s not comfortable.

Once I overcame the discomfort, once I decided to be brave and appreciate even possibly negative feedback, something amazing happened.

I suddenly had help.

Yes, there was scrutiny and nitpicking – but I don’t think any of it was ill-intentioned. I found that there existed whole communities of people who wanted to help me build a project that they thought was interesting. In some cases, I was utterly amazed when people submitted pull requests for issues I’d opened on my own projects describing enhancements I’d like to have.

I’ve been fortunate to have wonderful experiences with open source so far. Based on these experiences, I’d like to share with you what I’ve discovered to be the most effective ways to be brave and generous when it comes to open source.

‘Tis the season to share generously

When unprompted strangers submit helpful comments, issues, and pull requests on your projects, it feels like Christmas. You can give the gift of helpfulness in your contributions as well.

Illustration of a Santa contributor helping to fix a typo

Treat comments like a face-to-face conversation. Greet the person you’re addressing. Use full sentences. Think about whether what you’re writing will make someone’s day better or worse, and be nice.

When writing issues, include as much technical detail as possible. Screenshots, console logs, screenshots of console logs, your operating system, browser, screen resolution – all these can help maintainers quickly diagnose a root cause.

Pull requests are the best presents ever. They make maintainers happy when well done, and it’s a gift that gives back when your contribution gets merged! 🎉 Give your PR the best chance of getting accepted by looking for and following any project contribution guidelines.

Recognize the human

Our brains are slightly lacking in an evolutionary sense when it comes to interacting with other humans through tiny screens. It can be easy to forget that the actions you put out there will eventually reach one or more other people.

You can help to maintain a great open source community by remembering the humans that make it exist. When commenting, take the time to do it well (see below) and recognize the time that someone else has put in. When closing a thread or merging a contribution, remember to say thank you to the people who pitched in to help. I try to use first names instead of screen names, whenever possible.

You can build personal relationships, too. If you’re a project maintainer, you may choose to give people a way to contact you directly to ask questions or hash out complicated plans. Establishing one-to-one communications with regular contributors is also a great way to build a community around your project.

Recognizing the humans behind the open source community is a simple and meaningful way to give back.

Don’t rush

The vast majority of open source participants are volunteers, which means they don’t get paid for the time they spend building up projects. That sometimes means that other work takes priority. It’s okay if this describes you, too.

It’s important to remember that in most cases, a well-done contribution later is preferred over a half-done contribution sooner. If you’re too short on time now to write a thoughtful comment – don’t! Either draft a quick note and set it aside for later, or comment something along the lines of:

Hi there! Just wanted to let you know that I’ve seen this and I plan to help! I’ll respond in full as soon as I have the time to write a thoughtful comment.

Showing that you think a comment is worth the time to do well is something that open source contributors and repository maintainers both appreciate.

Build generously

When open source participants act with conscientiousness, every day feels like Christmas. Regardless of your type of contribution, you can help build this generous global community year-round.

The humans of open source, by self selection, mostly consist of good people who want to help. If you build openly, share feedback generously, and try to do good in general, I think you fit in here.

I hope you have a very happy holiday season and give many gifts that keep on giving!

An illustration of santa sitting on a pile of presents

So you're the family tech support, 21 Dec 2020 08:42:24 (Victoria Drake) and online security to-dos for the home-for-the-holidays tech support hero.🎄🌟 Happy holidays! 🌟🎄

For those of you seeing relatives this season, chances are that you’re the designated family tech support. If part of your time home for the holidays is spent on software updates and troubleshooting WiFi, here are a few other quick wins to help boost your family’s online privacy and security.

1. Set up a VPN

Using a VPN is Online Safety 101, but the biggest barrier to effectiveness is not remembering to turn it on. ExpressVPN’s Network Lock kill switch feature lets you set-it-and-forget-it, making it seamless to add VPN protection to your online activities. I go in-depth on this privacy-focused VPN here.

2. Introduce a password manager

If your family member uses the same password everywhere (<petname>+<house number>, same as last year) because passwords are hard to remember, introduce them to their new best friend, 1Password. Help your family get set up with secure passwords they don’t have to write down on Post-It notes – just one master pass(phrase) is all you need.

When choosing a passphrase, avoid using information easily found on social media accounts, like pet names, favorite sports teams, favorite brands, or birthdays.

3. Switch to DuckDuckGo

Help fight the Internet search monopoly by getting your family to use a search engine that respects their privacy. Go to your browser Settings and set your Default Search Engine (that uses the URL bar) to DuckDuckGo. Break the ice with an instant answer feature, like searching “calendar” so you can count down to Christmas.

(You might want to search for “classic cocktails cheat sheet” after all this.)

4. Install a better browser and blocker

While I prefer a Pi-hole, setting one up can be complex. Instead, help set up a privacy-preserving browser like Firefox or a wide-spectrum blocking extension like uBlock Origin (GitHub source).

Your family will get faster page load times, less advertisements interrupting articles and videos, and fewer sneaky trackers leaking browsing habits to big tech, all with near-zero maintenance.

Be a home-for-the-holidays hero!

Help improve your family’s security posture this holiday season. A little beefed-up cybersecurity may be one of the best gifts you can give!

I’m keeping it short-and-sweet this week. My annual Christmas post drops on December 24, full of warm fuzzy goodness and a tech tip or two. Thank you for being a subscriber – stay tuned!

How to write good documentation, 14 Dec 2020 04:53:10 (Victoria Drake) secret to never forgetting how your project works, in three steps. <img src="" width="640" height="320"/>If you’ve ever half-written a software project before taking a few days off, this is the article you’ll discover you needed when you reopen that IDE.

Your project on Friday (a finished puzzle) vs Monday (a pile of puzzle pieces) comic

In the technology teams I lead, we make a constant effort to document all the things. Documentation lives alongside the code as an equal player. This helps ensure that no one needs to make assumptions about how something works, or is calling lengthy meetings to gain working knowledge of a feature. Good documentation saves us a lot of time and hassle.

That said, and contrary to popular belief, the most valuable software documentation is not primarily written for other people. As I said in this well-received tweet:

With holidays around the corner, it’d be prudent to prepare for the possibility of some eggnog-induced programming paralysis. (Pecan pie and Python make a great combination.)

Here are three concrete steps you can take to write good documentation before it’s too late.

1. Start with accurate notes

As you work out ideas in code, ensure you don’t soon forget important details by starting with accurate notes. While you will want to explain things to yourself in long-form later, short-form notes will suffice to capture details without interrupting your coding session flow.

Keep a document open alongside your code and write down things like commands, decisions, and sources you use. This can include:

  • Terminal commands you typed in
  • Why you chose a particular method over another
  • Links you visited for help or coughcopy-pastecough inspiration
  • The order in which you did things

Don’t worry about full sentences at this point. Just ensure you accurately capture context, relevant code snippets, and helpful URLs. It can also be helpful to turn on any auto-save option available.

2. Explain decisions in long form

The ideal time to tackle this step is when you take a break from coding, but before you completely go out to lunch on whatever it is you’re working on at the moment. You want to ensure that context, ideas, and decisions are all still fresh in your mind when you explain them to yourself.

Go over the short-form notes you took and start expanding them into conversational writing. Be your own rubber duck. Describe what you’re doing as if you were teaching it to someone else. You might cover topics such as:

  • Quirky-looking decisions: “I would normally do it this way, but I chose to do something different because…”
  • Challenges you ran into and how you overcame them
  • Architectural decisions that support your project goals

Stick to the main points. Long-form writing doesn’t mean you’ll be paid by the word! Just use full sentences, and write as if explaining your project to a colleague. You’re explaining to future you, after all.

3. Don’t neglect prerequisite knowledge

This step is best done after a long lunch break, or even the next day (but probably not two). Re-read your document and fill in any blanks that become apparent after putting some distance between yourself and the project.

Take extra care to fill in or at least link to prerequisite knowledge, especially if you frequently use different languages or tools. Even an action as small as pasting in a link to the API documentation you used can save hours of future searching.

Write down or link to READMEs, installation steps, and relevant support issues. For frequently performed command-line actions, you can use a self-documenting Makefile to avoid having to man common tasks each time you come back to a project.

It’s easy to forget supporting details after even just a short break from your project. Capture anything you found helpful this time around.

Document all the things

The next time you catch yourself thinking, “I’m sure I’ll remember this part, no need to write it down,” just recall this emoji: 🤦‍♀️

Software projects are made up of a lot more than just their code. To best set up your future self for success, document all the things! Whether it’s a process you’ve established, Infrastructure as Code, or a fleeting future roadmap idea — write it down! Future you will thank you for it.

If you enjoyed this post, there’s a lot more where that came from! I write about computing, cybersecurity, and leading great technical teams. You can subscribe to see new articles first.

Make your team more productive by literally doing one thing, 07 Dec 2020 15:01:25 (Victoria Drake) giving your development team too many things to do first. <img src="" width="640" height="351"/>In the tech teams I lead, “priority” has no plural form.

Whether you’re leading a team of people or leading yourself, it’s important to take account of all the important things that need doing in your organization. This does not mean that everything can be equally important.

Logically, everything can’t be. Tasks are typically interdependent, and there’s always one task on which another depends. Tasks can be time-sensitive. Certain tasks might block a logical path towards a goal.

It’s the duty of a leader to make hard calls and decide which tasks are most important out of everything that needs doing. This necessitates comparing one to another, which is much easier to do with a centralized to-do list.

Here’s how this one simple change to your perspective on to-do lists can help to build happier and more productive teams.

Keep a central prioritized to-do list

A cartoon of a stick figure swinging on a rope ro plant a post-it note

Avoid working in silos. A single centralized list can make it easier for you and your team members to see what’s being worked on. With all tasks out in the open, it’s easier for people to spot opportunities for helping each other out and where they can contribute.

Encouraging a culture of openness can help people feel more comfortable asking questions, asking for help, and proposing ideas and improvements. Tracking work in the open also means that no one is left wondering what status a task is currently in.

For team leaders, a single list makes it easier to compare and prioritize tasks. This benefits team members by providing a completely unambiguous and transparent accounting of what needs doing next. Whichever task is most important, for the whole organization, is on top.

Priorities with autonomy

A single priority doesn’t necessarily pigeonhole someone into doing a task they don’t feel cut out for. Each member of your team has different strengths, skill sets, and diverse ways of thinking. You can take full advantage of this by encouraging autonomy in task selection.

A cartoon of a stick figure climbing a ladder to reach a post-it note

Have people choose whichever task is nearest to the top that they’d like to tackle. They might pick the highest priority task that’s in their wheelhouse, or experiment with a higher one that’s in a domain they’d like to improve their skills at.

Embrace opportunities for cross-training. If tasks high up on the list fall in a category that only one or a few people on your team are experts in, have your experts partner up with another team member who’s taking on the task. By pooling your resources to cross-train across domains, you multiply the capabilities of each team member and your team as a result.

When a task is especially time-sensitive, have several team members swarm on it and distribute the work according to their interests or strengths.

Make yourself redundant

Working off a single prioritized to-do list works best when your team members can take on tasks as independently as possible. This is especially important in remote teams where people work asynchronously.

If you’re a leader and find that your team members frequently ask you what they should do next, you could be making your team dependent on you. Ask yourself if you’re unnecessarily gatekeeping information that would let your team be more autonomous.

A team that overly depends on their leader is not an efficient one. Individual people, such as yourself, don’t scale. Don’t become a bottleneck to your team’s productivity. A successful leader should be able to take several days off on short notice without productivity grinding to a halt.

A cartoon of a stick figure carrying books to a wall of post-it notes

To support your team’s ability to work without you, make your team, product, and company goals painfully available. Put them where people hang out – your team’s message board, chat channel, or document repository, for example. No one should be at a loss when asked what the team wants to achieve next, and why.

Make any applicable resources, style guides, product documents, or links to external documentation painfully available as well. If your team makes a decision about how something should be done, write it down. Don’t rely on yours or anyone else’s meat brain to remember an important decision, nor make yourself the only resource for recalling it.

Make yourself redundant when it comes to day-to-day work. Doing so empowers your team members to do work without you, think through solutions on their own, and propose paths of action that you probably wouldn’t have thought of yourself.

Build happier and more productive teams

From first-hand experience as both a team member and leader, I’ve seen how encouraging a culture of openness, cross-training, and autonomy makes for happier team members and more productive teams. A single prioritized to-do list, coupled with available documentation and resources, opens the gates to let your technical team be maximally productive.

By removing bottlenecks, you allow people to make more decisions on their own and take ownership of their work. That’s a technical team I’d be proud to lead.

OWASP Web Security Testing Guide v4.2 released, 03 Dec 2020 16:02:33 (Victoria Drake) the release of a new version of the OWASP WSTG.I’m very happy and proud to share that the Open Web Application Security Project (OWASP) Web Security Testing Guide v4.2 is now available! This update is the result of a lot of hard work by the repository team and many dedicated contributors. With a team like this, I’m honored to be a core maintainer and co-author.

Here’s a reprint of the announcement I wrote for If you’re interested in security testing for web applications and APIs, this is an update you’ll definitely want to check out!

You can become a contributor yourself by joining us on GitHub!

Web Security Testing Guide v4.2 Released

Thursday, December 3, 2020

The OWASP Web Security Testing Guide team is proud to announce version 4.2 of the Web Security Testing Guide (WSTG)! In keeping with a continuous delivery mindset, this new minor version adds content as well as improves the existing tests.

In recent years, the Web Security Testing Guide has sought to remain your foremost open source resource for web application testing. Our previous release marked a move from a cumbersome wiki platform to the highly collaborative world of GitHub. Since then, over 61 new contributors pushing over 600 commits have helped to make the WSTG better than ever.

Version 4.2 of the Web Security Testing Guide introduces new testing scenarios, updates existing chapters, and offers an improved reading experience with a clearer writing style and chapter layout. Readers will enjoy easier navigation and consistent testing instructions.

With new improvements to our development workflow, new contributors will find it easier than ever to help build future versions of the WSTG. A clear and concise contributor’s guide and style guide can help you write new tests or ensure existing scenarios stay current. Core maintainers Rick Mitchell, Elie Saad, Rejah Rehim, and Victoria Drake have implemented modern processes like continuous integration with GitHub Actions. New workflows help to build PDFs and make reviewing new additions and updates easier.

We couldn’t be happier to share this new version with you, and we don’t plan to slow down anytime soon. The dedicated volunteers who’ve made this release possible are already hard at work on the next major version of the WSTG. Come join us and become a contributor!

You can read the Web Security Testing Guide v4.2 online or download a PDF on our project page. We greatly appreciate all the authors, editors, reviewers, and readers who make this open source security endeavor worthwhile.

Thank you for being a part of the WSTG!

What is TCP/IP? Layers and protocols explained, 29 Nov 2020 04:01:22 (Victoria Drake) titled, "Why the Internet Protocol Suite is an imaginary rainbow layer cake" <img src="" width="640" height="320"/>A significant part of the process of creation is the ability to imagine things that do not yet exist. This skill was instrumental to the creation of the Internet. If no one had imagined the underlying technology that most now take for granted every day, there would be no cat memes.

To make the Internet possible, two things that needed imagining are layers and protocols. Layers are conceptual divides that group similar functions together. The word “protocol,” means “the way we’ve agreed to do things around here,” more or less. In short, both layers and protocols can be explained to a five-year-old as “ideas that people agreed sounded good, and then they wrote them down so that other people could do things with the same ideas.”



Layers are imaginary








Link cake layer cartoon


Internet layer protocols


Internet cake layer cartoon

The most widely known protocol in this layer gives TCP/IP its last two letters. IP is a connectionless protocol, meaning that it provides no guarantee that packets are sent or received in the right order, along the same path, or even in their entirety. Reliability is handled by other protocols in the suite, such as in the transport layer.

There are currently two versions of IP in use: IPv4, and IPv6. Both versions describe how devices on the Internet are assigned IP addresses, which are used when navigating to cat memes. IPv4 is more widely used, but has only 32 bits for addressing, allowing for about 4.3 billion (ca. 4.3×109) possible addresses. These are running out, and IPv4 and will eventually suffer from address exhaustion as more and more people use more devices on the Internet.


Transport layer protocols


Transport cake layer cartoon


Application layer protocols

Application cake layer cartoon


Also included in the application layer are the Hypertext Transfer Protocol (HTTP) and its successor, Hypertext Transfer Protocol Secure (HTTPS). HTTPS is secured by Transport Layer Security, or TLS, which can be said to be the top-most layer of the networking model described by the Internet protocol suite. If you’d like to further understand TLS and how this protocol secures your cat meme viewing, I invite you read my article about TLS and cryptography.

The Internet cake is still baking



If you enjoyed learning from this post, there’s a lot more where this came from! I write about computing, cybersecurity, and building great technical teams. Subscribe to see new articles first.

Responsive pages and color themes with minimal CSS, 17 Nov 2020 06:04:58 (Victoria Drake) a responsive website with color themes? Start at the root. <img src="" width="640" height="107"/>Hello, do come in! If you’re reading this on my website, you may notice I’ve spruced up a bit. can now better respond to your devices and preferences!

Most modern devices and web browsers allow users to choose either a light or dark theme for the user interface. With CSS media queries, you can have your own website’s styles change to match this user setting!

Media queries are also a common way to have elements on web pages change to suit different screen sizes. This is an especially powerful tool when combined with custom properties set on the root element.

Here’s how to use CSS media queries and custom properties to improve your visitor’s browsing experience with just a few lines of CSS.

Catering to color preferences

The prefers-color-scheme media feature can be queried to serve up your user’s color scheme of choice. The light option is the go-to version if no active preference is set, and it has decent support across modern browsers.

Additionally, users reading on certain devices can also set light and dark color themes based on a schedule. For example, my phone uses light colors throughout its UI during the daytime, and dark colors at night. You can make your website follow suit!

Avoid repeating a lot of CSS by setting custom properties for your color themes on your :root pseudo-class. You can specify the themes available with the color-scheme property (currently part of a draft specification, but I like to write my articles to age well). Create a version for each theme you wish to support. Here’s a quick example you can build on:

:root {
    color-scheme: light dark;

@media (prefers-color-scheme: light) {
    :root {
        --text-primary: #24292e;
        --background: white;
        --shadow: rgba(0, 0, 0, 0.15) 0px 2px 5px 0px;

@media (prefers-color-scheme: dark) {
    :root {
        --text-primary: white;
        --background: #24292e;
        --shadow: rgba(0, 0, 0, 0.35) 0px 2px 5px 0px;

As you can see, you can use custom properties to set all kinds of values. To use these as variables with other CSS elements, use the var() function:

header {
    color: var(--text-primary);
    background-color: var(--background);
    box-shadow: var(--shadow);

In this quick example, the header element will now display your user’s preferred colors according to their browser settings!

Preferred color schemes are set by the user in different ways, depending on the browser. Here are a couple examples.


You can test out light and dark modes in Firefox by typing about:config into the address bar. Accept the warning if it pops up, then type ui.systemUsesDarkTheme into the search.

Choose a Number value for the setting, then input a 1 for dark or 0 for light.

A screenshot of setting the color theme in Firefox


If you’re using Brave, find color theme settings in Settings > Appearance > Brave colors.

A screenshot of setting the color theme in Brave

Variable scaling

You can also use a custom property to effortlessly adjust the size of text or other elements depending on your user’s screen size. The width media feature tests the width of the viewport. While width: _px will match an exact size, you can also use min and max to create ranges.

Query with min-width: _px to match anything over _ pixels, and max-width: _px to match anything up to _ pixels.

Use these queries to set a custom property on the :root to create a ratio:

@media (min-width: 360px) {
    :root {
        --scale: 0.8;

@media (min-width: 768px) {
    :root {
        --scale: 1;

@media (min-width: 1024px) {
    :root {
        --scale: 1.2;

Then make an element responsive by using the calc() function. Here are a few examples:

h1 {
    font-size: calc(42px * var(--scale));

h2 {
    font-size: calc(26px * var(--scale));

img {
    width: calc(200px * var(--scale));

In this example, multiplying an initial value by your --scale custom property allows the size of headings and images to magically adjust to your user’s device width.

The relative unit rem will have a similar effect. You can use it to define sizes for elements relative to the font size declared at the root element.

h1 {
    font-size: calc(5rem * var(--scale));

h2 {
    font-size: calc(1.5rem * var(--scale));

p {
    font-size: calc(1rem * var(--scale));

Of course, you can also multiply two custom properties. For example, setting the --max-img as a custom property on the :root can help to save you time later on by not having to update a pixel value in multiple places:

img {
    max-width: calc(var(--max-img) * var(--scale));

Raise your responsiveness game

Try out these easy wins for a website that caters to your visitor’s devices and preferences. I’ve put them to good use now on I invite you to let me know how you like it!

Build your own serverless subscriber list with Go and AWS, 10 Nov 2020 04:52:50 (Victoria Drake) to build your own newsletter list with DynamoDB and SES email sign up confirmations. <img src="" width="640" height="640"/>You can now subscribe to my email list on! Here’s how I lovingly built a subscription sign up flow with email confirmation that doesn’t suck. You can too.

Introducing Simple Subscribe

If you’re interested in managing your own mailing list or newsletter, you can set up Simple Subscribe on your own AWS resources to collect email addresses. This open source API is written in Go, and runs on AWS Lambda. Visitors to your site can sign up to your list, which is stored in a DynamoDB table, ready to be queried or exported at your leisure.

When someone signs up, they’ll receive an email asking them to confirm their subscription. This is sometimes called “double opt-in,” although I prefer the term “verified.” Simple Subscribe works on serverless infrastructure and uses an AWS Lambda to handle subscription, confirmation, and unsubscribe requests.

You can find the Simple Subscribe project, with its fully open-source code, on GitHub. I encourage you to pull up the code and follow along! In this post I’ll share each build step, the thought process behind the API’s single-responsibility functions, and security considerations for an AWS project like this one.

Building a verified subscription flow

A non-verified email sign up process is straightforward. Someone puts their email into a box on your website, then that email goes into your database. However, if I’ve taught you anything about not trusting user input, the very idea of a non-verified sign up process should raise your hackles. Spam may be great when fried in a sandwich, but no fun when it’s running up your AWS bill.

While you can use a strategy like a CAPTCHA or puzzle for is-it-a-human verification, these can create enough friction to turn away your potential subscribers. Instead, a confirmation email can help to ensure both address correctness and user sentience.

To build a subscription flow with email confirmation, create single-responsibility functions that satisfy each logical step. Those are:

  1. Accept an email address and record it.
  2. Generate a token associated with that email address and record it.
  3. Send a confirmation email to that email address with the token.
  4. Accept a verification request that has both the email address and token.

To achieve each of these goals, Simple Subscribe uses the official AWS SDK for Go to interact with DynamoDB and SES.

At each stage, consider what the data looks like and how you store it. This can help to handle conundrums like, “What happens if someone tries to subscribe twice?” or even threat-modeling such as, “What if someone subscribes with an email they don’t own?”

Ready? Let’s break down each step and see how the magic happens.


The subscription process begins with a humble web form, like the one on my site’s main page. A form input with attributes type="email" required helps with validation, thanks to the browser. When submitted, the form sends a GET request to the Simple Subscribe subscription endpoint.

Simple Subscribe receives a GET request to this endpoint with a query string containing the intended subscriber’s email. It then generates an id value and adds both email and id to your DynamoDB table.

The table item now looks like:

email confirm id timestamp false uuid-xxxxx 2020-11-01 00:27:39

The confirm column, which holds a boolean, indicates that the item is a subscription request that has not yet been confirmed. To verify an email address in the database, you’ll need to find the correct item and change confirm to true.

As you work with your data, consider the goal of each manipulation and how you might compare an incoming request to existing data.

For example, if someone made a subsequent subscription request for the same email address, how would you handle it? You might say, “Create a new line item with a new id,” however, this might not be best strategy when your serverless application database is paid for by request volume.

Since DynamoDB Pricing depends on how much data you read and write to your tables, it’s advantageous to avoid piling on excess data.

With that in mind, it would be prudent to handle subscription requests for the same email by performing an update instead of adding a new line. Simple Subscribe actually uses the same function to either add or update a database item. This is typically referred to as, “update or insert.”

In a database like SQLite this is accomplished with the UPSERT syntax. In the case of DynamoDB, you use an update operation. For the Go SDK, its syntax is UpdateItem.

When a duplicate subscription request is received, the database item is matched on the email only. If an existing line item is found, its id and timestamp are overridden, which updates the existing database record and avoids flooding your table with duplicate requests.

Verifying email addresses

After submitting the form, the intended subscriber then receives an email from SES containing a link. This link is built using the email and id from the table, and takes the format:


In this set up, the id is a UUID that acts as a secret token. It provides an identifier that you can match that is sufficiently complex and hard to guess. This approach deters people from subscribing with email addresses they don’t control.

Visiting the link sends a request to your verification endpoint with the email and id in the query string. This time, it’s important to compare both the incoming email and id values to the database record. This verifies that the recipient of the confirmation email is initiating the request.

The verification endpoint ensures that these values match an item in your database, then performs another update operation to set confirm to true, and update the timestamp. The item now looks like:

email confirm id timestamp true uuid-xxxxx 2020-11-01 00:37:39

Querying for emails

You can now query your table to build your email list. Depending on your email sending solution, you might do this manually, with another Lambda, or even from the command line.

Since data for requested subscriptions (where confirm is false) is stored in the table alongside confirmed subscriptions, it’s important to differentiate this data when querying for email addresses to send to. You’ll want to ensure you only return emails where confirm is true.

Similar to verifying an email address, Simple Subscribe uses email and id as arguments to the function that deletes an item from your DynamoDB table in order to unsubscribe an email address. To allow people to remove themselves from your list, you’ll need to provide a URL in each email you send that includes their email and id as a query string to the unsubscribe endpoint. It would look something like:


When the link is clicked, the query string is passed to the unsubscribe endpoint. If the provided email and id match a database item, that item will be deleted.

Proving a method for your subscribers to automatically remove themselves from your list, without any human intervention necessary, is part of an ethical and respectful philosophy towards handling the data that’s been entrusted to you.

Caring for your data

Once you decide to accept other people’s data, it becomes your responsibility to care for it. This is applicable to everything you build. For Simple Subscribe, it means maintaining the security of your database, and periodically pruning your table.

In order to avoid retaining email addresses where confirm is false past a certain time frame, it would be a good idea to set up a cleaning function that runs on a regular schedule. This can be achieved manually, with an AWS Lambda function, or using the command line.

To clean up, find database items where confirm is false and timestamp is older than a particular point in time. Depending on your use case and request volumes, the frequency at which you choose to clean up will vary.

Also depending on your use case, you may wish to keep backups of your data. If you are particularly concerned about data integrity, you can explore On-Demand Backup or Point-in-Time Recovery for DynamoDB.

Build your independent subscriber base

Building your own subscriber list can be an empowering endeavor! Whether you intend to start a newsletter, send out notifications for new content, or want to create a community around your work, there’s nothing more personal or direct than an email from me to you.

I encourage you to start building your subscriber base with Simple Subscribe today! Like most of my work, it’s open source and free for your personal use. Dive into the code at the GitHub repository or learn more at

WPA Key, WPA2, WPA3, and WEP Key: Wi-Fi security explained, 19 Oct 2020 04:02:27 (Victoria Drake) one should you be using? Why Wi-Fi security matters. <img src="" width="640" height="320"/>Setting up new Wi-Fi? Picking the type of password you need can seem like an arbitrary choice. After all, WEP, WPA, WPA2, and WPA3 all have mostly the same letters in them. A password is a password, so what’s the difference?

About 60 seconds to billions of years, as it turns out.

All Wi-Fi encryption is not created equal. Let’s explore what makes these four acronyms so different, and how you can best protect your home and organization Wi-Fi.

Wired Equivalent Privacy (WEP)

In the beginning, there was WEP.

cartoon of WEP letters

Not to be confused with the name of a certain rap song.

Wired Equivalent Privacy is a deprecated security algorithm from 1997 that was intended to provide equivalent security to a wired connection. “Deprecated” means, “Let’s not do that anymore.”

Even when it was first introduced, it was known not to be as strong as it could have been, for two reasons: one, its underlying encryption mechanism; and two, World War II.

During World War II, the impact of code breaking (or cryptanalysis) was huge. Governments reacted by attempting to keep their best secret-sauce recipes at home. Around the time of WEP, U.S. Government restrictions on the export of cryptographic technology caused access point manufacturers to limit their devices to 64-bit encryption. Though this was later lifted to 128-bit, even this form of encryption offered a very limited possible key size.

This proved problematic for WEP. The small key size resulted in being easier to brute-force, especially when that key doesn’t often change.

WEP’s underlying encryption mechanism is the RC4 stream cipher. This cipher gained popularity due to its speed and simplicity, but that came at a cost. It’s not the most robust algorithm. WEP employs a single shared key among its users that must be manually entered on an access point device. (When’s the last time you changed your Wi-Fi password? Right.) WEP didn’t help matters either by simply concatenating the key with the initialization vector – which is to say, it sort of mashed its secret-sauce bits together and hoped for the best.

Initialization Vector (IV): fixed-size input to a low-level cryptographic algorithm, usually random.

Combined with the use of RC4, this left WEP particularly susceptible to related-key attack. In the case of 128-bit WEP, your Wi-Fi password can be cracked by publicly-available tools in a matter of around 60 seconds to three minutes.

While some devices came to offer 152-bit or 256-bit WEP variants, this failed to solve the fundamental problems of WEP’s underlying encryption mechanism.

So, yeah. Let’s not do that anymore.

Wi-Fi Protected Access (WPA)

WPA illustration

A new, interim standard sought to temporarily “patch” the problem of WEP’s (lack of) security. The name Wi-Fi Protected Access (WPA) certainly sounds more secure, so that’s a good start; however, WPA first started out with another, more descriptive name.

Ratified in a 2004 IEEE standard, Temporal Key Integrity Protocol (TKIP) uses a dynamically-generated, per-packet key. Each packet sent has a unique temporal 128-bit key, (See? Descriptive!) that solves the susceptibility to related-key attacks brought on by WEP’s shared key mashing.

TKIP also implements other measures, such as a message authentication code (MAC). Sometimes known as a checksum, a MAC provides a cryptographic way to verify that messages haven’t been changed. In TKIP, an invalid MAC can also trigger rekeying of the session key. If the access point receives an invalid MAC twice within a minute, the attempted intrusion can be countered by changing the key an attacker is trying to crack.

Unfortunately, in order to preserve compatibility with the existing hardware that WPA was meant to “patch,” TKIP retained the use of the same underlying encryption mechanism as WEP – the RC4 stream cipher. While it certainly improved on the weaknesses of WEP, TKIP eventually proved vulnerable to new attacks that extended previous attacks on WEP. These attacks take a little longer to execute by comparison: for example, twelve minutes in the case of one, and 52 hours in another. This is more than sufficient, however, to deem TKIP no longer secure.

WPA, or TKIP, has since been deprecated as well. So let’s also not do that anymore.

Which brings us to…

Wi-Fi Protected Access II (WPA2)

WPA2 illustration

Rather than spend the effort to come up with an entirely new name, the improved Wi-Fi Protected Access II (WPA2) standard instead focuses on using a new underlying cipher. Instead of the RC4 stream cipher, WPA2 employs a block cipher called Advanced Encryption Standard (AES) to form the basis of its encryption protocol. The protocol itself, abbreviated CCMP, draws most of its security from the length of its rather long name (I’m kidding): Counter Mode Cipher Block Chaining Message Authentication Code Protocol, which shortens to Counter Mode CBC-MAC Protocol, or CCM mode Protocol, or CCMP. 🤷

CCM mode is essentially a combination of a few good ideas. It provides data confidentiality through CTR mode, or counter mode. To vastly oversimplify, this adds complexity to plaintext data by encrypting the successive values of a count sequence that does not repeat. CCM also integrates CBC-MAC, a block cipher method for constructing a MAC.

AES itself is on good footing. The AES specification was established in 2001 by the U.S. National Institute of Standards and Technology (NIST) after a five-year competitive selection process during which fifteen proposals for algorithm designs were evaluated. As a result of this process, a family of ciphers called Rijndael (Dutch) was selected, and a subset of these became AES. For the better part of two decades, AES has been used to protect every-day Internet traffic as well as certain levels of classified information in the U.S. Government.

While possible attacks on AES have been described, none have yet been proven to be practical in real-world use. The fastest attack on AES in public knowledge is a key-recovery attack that improved on brute-forcing AES by a factor of about four. How long would it take? Some billions of years.

Wi-Fi Protected Access III (WPA3)

WPA3 illustration

The next installment of the WPA trilogy has been required for new devices since July 1, 2020. Expected to further enhance the security of WPA2, the WPA3 standard seeks to improve password security by being more resilient to word list or dictionary attacks.

Unlike its predecessors, WPA3 will also offer forward secrecy. This adds the considerable benefit of protecting previously exchanged information even if a long-term secret key is compromised. Forward secrecy is already provided by protocols like TLS by using asymmetric keys to establish shared keys. You can learn more about TLS in this post.

As WPA2 has not been deprecated, both WPA2 and WPA3 remain your top choices for Wi-Fi security.

If the other ones suck, why are they still around?

You may be wondering why your access point even allows you to choose an option other than WPA2 or WPA3. The likely reason is that you’re using legacy hardware, which is what tech people call your mom’s router.

Since the deprecation of WEP and WPA occurred (in old-people terms) rather recently, it’s possible in large organizations as well as your parent’s house to find older hardware that still uses these protocols. Even newer hardware may have a business need to support these older protocols.

While I may be able to convince you to invest in a shiny new top-of-the-line Wi-Fi appliance, most organizations are a different story. Unfortunately, many just aren’t yet cognizant of the important role cybersecurity plays in meeting customer needs and boosting that bottom line. Additionally, switching to newer protocols may require new internal hardware or firmware upgrades. Especially on complex systems in large organizations, upgrading devices can be financially or strategically difficult.

Boost your Wi-Fi security

If it’s an option, choose WPA2 or WPA3. Cybersecurity is a field that evolves by the day, and getting stuck in the past can have dire consequences.

If you can’t use WPA2 or WPA3, do the best you can to take additional security measures. The best bang for your buck is to use a Virtual Private Network (VPN). Using a VPN is a good idea no matter which type of Wi-Fi encryption you have. On open Wi-Fi (coffee shops) and using WEP, it’s plain irresponsible to go without a VPN. Kind of like shouting out your bank details as you order your second cappuccino.

A cartoon of shouting out your bank details at a coffeeshop.

Choose a VPN provider that offers a feature like ExpressVPN's Network Lock with automatic connection. This is a kill switch that blocks your network traffic if your VPN becomes disconnected, preventing you from accidentally transmitting information on an insecure connection like open Wi-Fi or WEP. I personally use ExpressVPN. Their auto-connect feature works well on Linux, and I never worry about remembering to start it up first. I wrote more about choosing a VPN in this post.

When possible, ensure you only connect to known networks that you or your organization control. Many cybersecurity attacks are executed when victims connect to an imitation public Wi-Fi access point, also called an evil twin attack, or Wi-Fi phishing. These fake hotspots are easily created using publicly accessible programs and tools. A VPN can help mitigate damage from these attacks as well, but it’s always better not to take the risk. If you travel often, consider purchasing a portable hotspot that uses a cellular data plan, or using data SIM cards for all your devices.

Much more than just acronyms

WEP, WPA, WPA2, and WPA3 mean a lot more than a bunch of similar letters – in some cases, it’s a difference of billions of years minus about 60 seconds.

On more of a now-ish timescale, I hope I’ve taught you something new about the security of your Wi-Fi and how you can improve it!

If you’d like to thank me, you can protect yourself with ExpressVPN with my referral. I only refer products I use personally, and this one is solid.

Know someone who’d benefit from some beefed up cybersecurity? Share the cybersecurity starter pack!

Three rules for choosing a VPN that takes your privacy seriously, 18 Oct 2020 04:41:37 (Victoria Drake) lesser-known risks of ISPs and why I chose ExpressVPN. <img src="" width="640" height="320"/>Most people know that a VPN is meant to protect your privacy on public or open Wi-Fi. A lesser-known purpose is to protect your privacy right in your own home, from your own internet service provider (ISP).

A set of Federal Communications Commission (FCC) rules entitled “Protecting the Privacy of Customers of Broadband and Other Telecommunications Services” were unfortunately struck down in 2017. These rules would have prevented ISPs from using and selling your sensitive personal data, such as precise geographic location, health and financial information, web browsing history, and even the content of the messages you send.

I’m not comfortable having that data stored anywhere. Handing it over to my ISP makes me even less comfortable, since these treasure troves of sensitive personal data are a frequent and profitable target for ill-intentioned hackers as well.

Your online activities shouldn’t be anyone’s business, and certainly not in a literal money-making sense. Using a VPN helps to keep your private information where it belongs: between you and the person you’re sending it to.

Of course, if you type any flavor of “VPN vs VPN” into a search you’ll get a smorgasbord of comparison blogs and providers vying for your attention. How do you know what makes a VPN “good”? (Answer: lots of research.)

If you’re a regular reader, you know I’m big on security and privacy. (If you’re not yet, welcome! Hi!) Since I’ve built my career in the cybersecurity industry, I take my VPN fairly seriously. Here are the top three things I look for when choosing my own VPN provider:

  1. No DNS leaks
  2. A real commitment to privacy, with no logs
  3. Ease of use across all operating systems

I’ve written about why a VPN is important and even how to deal with the challenges of DNS leaks when using OpenVPN to set up my own. The response I often get to articles like these includes the question, “Which VPN do you use?”

The answer is ExpressVPN. Here’s how my privacy philosophy got me there, and why these three points matter so much.

TL;DR: ExpressVPN is built on privacy. You can set-it-and-forget-it with three extra months free.

Why you don’t want a DNS leak

In a previous Linux-flavored adventure, I created my own VPN using OpenVPN and AWS EC2. While I’ve been told my post was helpful, this was definitely not a plug-and-play solution. After reinstalling a new OS, I once failed to follow my own guide to the letter. It took a few months before I discovered I had a DNS leak.

A diagram of a DNS leak

Using a VPN prevents your ISP from collecting your sensitive personal data, including your web browsing history, but only as long as you don’t have a DNS leak. A DNS leak means that your ISP still sees all the URLs that you visit: their servers resolve them for you. This is plenty of information to build a picture of who you are, what your interests might be, any health issues you might have, what you like to spend money on, and much more.

Protocols like DNS over HTTPS will help, but they rely on co-operation between many entities that is still in its early stages. In the meantime, I want my VPN to do everything it can to avoid using DNS servers that could collect or sell my browsing history.

At time of writing, there’s really only one fool-proof solution to ensuring that your browsing records aren’t accidentally shared: run your own private DNS server. So ExpressVPN did just that.

Of course, this only works in my favor when the VPN itself doesn’t keep a record of my activities. Which is why…

A no-logs philosophy matters

VPN providers do not all value your privacy, and some are no better than your ISP. Many VPN providers, especially free ones, elect to log your personal data and sell it to data brokers and marketers. Using a VPN that does any kind of logging simply transfers the risk from your ISP to the VPN provider.

At a minimum, you want a VPN provider to clearly state a strict no-logs policy. Of course, this still means you’ll have to trust that they aren’t being cagey with their definition of “logs,” and still writing your personal data to disk under a pretense.

A more trustworthy solution would be to remove the possibility of writing any personal data to disk in the first place. So ExpressVPN got rid of the disks.

A cartoon of a ram eating logs. Text reads: Hey look, data! Nom. What am I eating again? Eh nvm.

Dad joke. I know.

I was pretty thrilled to learn about what ExpressVPN calls TrustedServer, which runs only on random-access memory, or RAM, and not on hard drives. Unlike a disk meant for long-term, fault-tolerant storage, RAM is volatile memory. It requires constant power to operate, which guarantees that all data is lost when the server is rebooted.

While you wouldn’t want a laptop that runs entirely on RAM, volatile memory is perfectly suited to an ephemeral, no-logs VPN server. The entire software stack including the OS must be re-installed from a central, signed image each time the server boots. This also means it’s always installing the most up-to-date security patches and configuration. That’s clever.

This post goes into more technical detail on TrustedServer, which was independently audited by PricewaterhouseCoopers.

As a Director of Engineering myself, I have a deep appreciation for a company that builds its technology on its philosophy.

That said, the technology only works if you actually use it.

The best VPN is one you actually use

None of what I’ve said so far would matter one iota if my chosen VPN was even just a little bit inconvenient to use.

My preferred platforms are Linux and iOS. I’ve had my fair share of struggles finding all kinds of software that works equally well on just these two. ExpressVPN seems to offer one of the few applications I’ve come across that isn’t trapped in an ecosystem.

There’s a dedicated app for every major platform, including even smart TVs and game consoles. Unlike my experiences with other VPNs, ExpressVPN's Linux app just works, out-of-the-box, the way they said it would.

ExpressVPN on all your devices image

All the devices!

I especially appreciate the Network Lock kill switch feature, which prevents me from accidentally sending unprotected network traffic when I first open up my laptop and it reconnects to Wi-Fi. It prevents my ISP from seeing anything I do, and only takes a few seconds to reconnect.

ExpressVPN connects fast and then gets out of my way. I haven’t noticed any reduced speeds or blocked sites. I gave a lot of thought to choosing my VPN so I wouldn’t have to think about it on a day-to-day basis. I use ExpressVPN constantly, and it just works.

Privacy is more than personal

When you protect yourself and your family with a VPN, you improve more than your own personal cybersecurity. The less data your ISP can collect, the less they have to lose, sell, or profit from. One day, the risk and cost for ISPs will outweigh the payoff. When you take action to prevent ISPs from scooping up your family’s sensitive personal data, everyone’s privacy can benefit.

If you found this article helpful, I invite you to help support my work by signing up for ExpressVPN. It only takes a few minutes (assuming you remember where you left your credit card) and will give you the best possible set-it-and-forget-it privacy protection that I can recommend.

Your cybersecurity starter pack, 04 Oct 2020 04:30:12 (Victoria Drake) security best practices to share with your non-technical friend. <img src="" width="640" height="320"/>Readers of my blog typically know more about technology and cybersecurity than most people. This article is for most people. If someone you know could benefit from a simple and straightforward introduction to cybersecurity tools, please share this article with them – it benefits everyone!

If you’ve ever said to yourself:

“There’s no one targeting lil ol' me."
“I have nothing to hide, anyway."
“I’m too busy to learn all this stuff. Why can’t someone just give me a simple summary of best practices that I can skim in approximately seven minutes?"

First of all, you might want to stop talking to yourself in public. Secondly, here is a simple summary of best practices that you can skim in approximately seven minutes.

Introducing your three-step starter pack

While there are many different degrees of security, privacy, and anonymity, these three basics are accessible to all:

  1. Use a VPN
  2. Use multifactor authentication
  3. Develop a healthy sense of skepticism

I’ll discuss each of these and help you get started with your security upgrade. But first…

Why is cybersecurity important?

Would you let just anyone walk into your house, or even look through your open doorway from across the street? If not, you might appreciate that the cybersecurity practices we’ll discuss today are not that different from locking your front door.

Cybersecurity isn’t about finding some magic spell that completely secures your online activities – that would be nice, but it’s unrealistic. Good security practices are about employing some thoughtful habits that make your online activities more secure than the next guy, in much the same way as you learned to lock your front door.

Security breaches and incidents happen every day. Most of them occur because an automated scanner cast a wide net and found a person or company with lax security that a hacker could then exploit. Don’t be that guy.

1. Use a VPN

Let’s say you send a lot of mail, but never bother to put your letters in envelopes or even fold them in half. Anyone who bothers to look can read all your dirty secrets (not that you have any).

When you use a Virtual Private Network, or VPN, especially if you often connect to public WiFi, it’s like putting your letters into cryptographically-sealed envelopes and sending them via a special invisible courier service. No one but the intended recipient can read your letters, and no one but you and the courier know to whom the letters are sent.

An illustration of a locked envelope

Encrypted mail still won’t stop you from the accidental reply all, unfortunately.

VPNs prevent others from reading your communications. This may include opportunistic attackers who scan open WiFi, and even your own internet service provider (ISP) who may sell your usage data for advertising dollars.

Choosing a VPN

A few important differentiating factors can help you choose a VPN provider.

  1. Is it free? VPNs cost money to operate; if one is offered for free, consider what they might be doing in order to cover their costs. Generally, I recommend avoiding free VPN apps and services; they’ll typically cost you much more than you’ll know. Expect to pay between $5-$10 USD monthly for the service.

  2. Where is it based? Understand where your VPN provider is based, and what that country’s laws allow them to do with your data.

  3. Do they keep logs? Part of the philosophy of using a VPN is that no one has any business getting into your business when it comes to online activities. When a VPN provider keeps logs of your usage, that defeats the purpose. Instead of your ISP knowing just what you’re up to online, that knowledge is simply transferred to the logging VPN. Look for VPN providers with a strict no-logging policy.

I use ExpressVPN. I go into greater detail about choosing a VPN in this post.

2. Use multifactor authentication

Passwords are dead. Computationally, they are a solved problem. Cracking your password is just a matter of time.

Unfortunately, many people still help to speed up the process by using the same compromised passwords for multiple accounts, putting themselves at further risk.

The answer, at least for now, is multifactor authentication (MFA). MFA is made up of three kinds of authentication factors:

  1. Something you know, like a pass phrase;
  2. Something you have, like a chip pin card or phone; and
  3. Something that you are, like your face or fingerprint.
An illustration of the letters MFA

Also the name of my next beatboxing team.

Two or more of these factors are infinitely better than a password alone, especially if your password is on this list.

Multiple authentication factors are now widely supported by account providers and social media sites. If you have the choice, avoid using text messages, or SMS, as a way of receiving authentication codes. SMS authentication leaves you vulnerable to the SIM swap attack - please direct further questions to Jack Dorsey.

Instead, use a One Time Password (OTP) app such as Authy to generate codes on your device. This ensures that you alone, using that particular device, will have the correct authentication code.

You can also use hardware authentication keys such as the YubiKey, but these aren’t yet as widely supported as OTP apps.

3. Develop a healthy sense of skepticism

Social engineering, sometimes SE, is the use of psychological persuasion to get an unwitting target to give up access or information. This can take the form of phishing emails, letters, or phone calls (vishing) as well as far more sophisticated spear-phishing attacks of high-value targets, like company executives.

While some attacks are easier to spot, others use cognitive biases very effectively and are difficult even for security professionals to avoid. No human is immune.

Ultimately, the weakest link in your cybersecurity defense is you. All the VPNs and MFA on the Internet won’t protect you if a scam can trick you into opening the front gates. Always look a Trojan gift horse in the mouth.

An illustration of a Trojan horse

Yes, I know it’s a very nice looking wooden horse. Also free. Did you order it? No? Then it can stay outside.

Develop the habit of second-guessing things delivered to your virtual doorstep. Email, phone, and messaging scams range in sophistication. Even security professionals can fall for a good scam.

One way to protect yourself is to practice a healthy sense of skepticism. Question communications that ask you to click on links or visit a website, even if they come from someone you know or a company you use.

If you’re not certain that your bank or mother sent this email, pick up the phone and call them. Even if you think you are certain, pick up the phone and double check. You don’t call your mother enough, anyway.

Oh, and if the person on the phone is from your local tax office or the IRS or the CRA and they’re about to freeze your accounts because a case of mistaken identity has resulted in you being criminally charged for not repaying a loan on a 600-foot yacht in Malibu, just hang up. You know better than that. Tax agencies don’t have phones.

A safer Internet

Congratulations! You now have three tools to make your personal cybersecurity better than the next guy’s. If enough people do that, the whole neighborhood (or in this case, the Internet) will benefit as a result.

If this article piqued your interest, you can go further and outsource your security with a password manager and temporary virtual credit cards.

Cheat sheets and other resources

I’ll leave you with a few resources that I’ve enjoyed:

  • The Electronic Frontier Foundation website Surveillance Self Defense offers many great guides and how-to’s, such as setting up the encrypted messaging app Signal on your mobile device, and protecting yourself on social media.
  • The Cybersecurity and Infrastructure Security Agency (CISA) offers many shareable starter resources.
  • Working from home? The National Security Agency Central Security Service has Telework and Mobile Security Guides that discuss best practices for an unprecedented era of remote work.
Increase developer confidence with a great Django test suite, 01 Oct 2020 05:50:37 (Victoria Drake) to write tests for your Django applications that are painless and productive. <img src="" width="640" height="320"/>Done correctly, tests are one of your application’s most valuable assets.

The Django framework in particular offers your team the opportunity to create an efficient testing practice. Based on the Python standard library unittest, proper tests in Django are fast to write, faster to run, and can offer you a seamless continuous integration solution for taking the pulse of your developing application.

With comprehensive tests, developers have higher confidence when pushing changes. I’ve seen firsthand in my own teams that good tests can boost development velocity as a direct result of a better developer experience.

In this article, I’ll share my own experiences in building useful tests for Django applications, from the basics to the best possible execution. If you’re using Django or building with it in your organization, you might like to read the rest of my Django series.

What to test

Tests are extremely important. Far beyond simply letting you know if a function works, tests can form the basis of your team’s understanding of how your application is intended to work.

Here’s the main goal: if you hit your head and forgot everything about how your application works tomorrow, you should be able to regain most of your understanding by reading and running the tests you write today.

Here are some questions that may be helpful to ask as you decide what to test:

  • What is our customer supposed to be able to do?
  • What is our customer not supposed to be able to do?
  • What should this method, view, or logical flow achieve?
  • When, how, or where is this feature supposed to execute?

Tests that make sense for your application can help build developer confidence. With these sensible safeguards in place, developers make improvements more readily, and feel confident introducing innovative solutions to product needs. The result is an application that comes together faster, and features that are shipped often and with confidence.

A cartoon for peanut butter and jelly sandwich tests

Where to put tests

If you only have a few tests, you may organize your test files similarly to Django’s default app template by putting them all in a file called This straightforward approach is best for smaller applications.

As your application grows, you may like to split your tests into different files, or test modules. One method is to use a directory to organize your files, such as projectroot/app/tests/. The name of each test file within that directory should begin with test, for example,

Besides being aptly named, Django will find these files using built-in test discovery based on the unittest module. All files in your application with names that begin with test will be collected into a test suite.

This convenient test discovery allows you to place test files anywhere that makes sense for your application. As long as they’re correctly named, Django’s test utility can find and run them.

How to document a test

Use docstrings to explain what a test is intended to verify at a high level. For example:

def test_create_user(self):
    """Creating a new user object should also create an associated profile object"""
    # ...

These docstrings help you quickly understand what a test is supposed to be doing. Besides navigating the codebase, this helps to make it obvious when a test doesn’t verify what the docstring says it should.

Docstrings are also shown when the tests are being run, which can be helpful for logging and debugging.

What a test needs to work

Django tests can be quickly set up using data created in the setUpTestData() method. You can use various approaches to create your test data, such as utilizing external files, or even hard-coding silly phrases or the names of your staff. Personally, I much prefer to use a fake-data-generation library, such as faker.

The proper set up of arbitrary testing data can help you ensure that you’re testing your application functionality instead of accidentally testing test data. Because generators like faker add some degree of unexpectedness to your inputs, it can be more representative of real-world use.

Here is an example set up for a test:

from django.test import TestCase
from faker import Faker

from app.models import MyModel, AnotherModel

fake = Faker()

class MyModelTest(TestCase):
    def setUpTestData(cls):
        """Quickly set up data for the whole TestCase"""
        cls.user_first = fake.first_name()
        cls.user_last = fake.last_name()

    def test_create_models(self):
        """Creating a MyModel object should also create AnotherModel object"""
        # In test methods, use the variables created above
        test_object = MyModel.objects.create(
            # ...
        another_model = AnotherModel.objects.get(my_model=test_object)
        self.assertEqual(another_model.first_name, self.user_first)
        # ...

Tests pass or fail based on the outcome of the assertion methods. You can use Python’s unittest methods, and Django’s assertion methods.

For further guidance on writing tests, see Testing in Django.

Best possible execution for running your tests

Django’s test suite is manually run with:

./ test

I rarely run my Django tests this way.

The best, or most efficient, testing practice is one that occurs without you or your developers ever thinking, “I need to run the tests first.” The beauty of Django’s near-effortless test suite set up is that it can be seamlessly run as a part of regular developer activities. This could be in a pre-commit hook, or in a continuous integration or deployment workflow.

I’ve previously written about how to use pre-commit hooks to improve your developer ergonomics and save your team some brainpower. Django’s speedy tests can be run this way, and they become especially efficient if you can run tests in parallel.

Tests that run as part of a CI/CD workflow, for example, on pull requests with GitHub Actions, require no regular effort from your developers to remember to run tests at all. I’m not sure how plainly I can put it – this one’s literally a no-brainer.

Testing your way to a great Django application

Tests are extremely important, and underappreciated. They can catch logical errors in your application. They can help explain and validate how concepts and features of your product actually function. Best of all, tests can boost developer confidence and development velocity as a result.

The best tests are ones that are relevant, help to explain and define your application, and are run continuously without a second thought. I hope I’ve now shown you how testing in Django can help you to achieve these goals for your team!

Django project best practices to keep your developers happy, 22 Sep 2020 04:55:19 (Victoria Drake) Makefiles, pre-commit, and GitHub Actions to help create a happy development team. <img src="" width="640" height="320"/>Do you want your team to enjoy your development workflow? Do you think building software should be fun and existentially fulfilling? If so, this is the post for you!

I’ve been developing with Django for years, and I’ve never been happier with my Django project set up than I am right now. Here’s how I’m making a day of developing with Django the most relaxing and enjoyable development experience possible for myself and my engineering team.

A custom CLI tool for your Django project

Instead of typing:

python3 -m venv env
source env/bin/activate
pip install -r requirements.txt
python3 makemigrations
python3 migrate
python3 collectstatic
python3 runserver

Wouldn’t it be much nicer to type:

make start

…and have all that happen for you? I think so!

We can do that with a self-documenting Makefile! Here’s one I frequently use when developing my Django applications, like

VENV := env
BIN := $(VENV)/bin
PYTHON := $(BIN)/python
SHELL := /bin/bash

include .env

.PHONY: help
help: ## Show this help
    @egrep -h '\s##\s' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'

.PHONY: venv
venv: ## Make a new virtual environment
    python3 -m venv $(VENV) && source $(BIN)/activate

.PHONY: install
install: venv ## Make venv and install requirements
    $(BIN)/pip install --upgrade -r requirements.txt

freeze: ## Pin current dependencies
    $(BIN)/pip freeze > requirements.txt

migrate: ## Make and run migrations
    $(PYTHON) makemigrations
    $(PYTHON) migrate

db-up: ## Pull and start the Docker Postgres container in the background
    docker pull postgres
    docker-compose up -d

db-shell: ## Access the Postgres Docker database interactively with psql. Pass in DBNAME=<name>.
    docker exec -it container_name psql -d $(DBNAME)

.PHONY: test
test: ## Run tests
    $(PYTHON) test application --verbosity=0 --parallel --failfast

.PHONY: run
run: ## Run the Django server
    $(PYTHON) runserver

start: install migrate run ## Install requirements, apply migrations, then start development server

You’ll notice the presence of the line include .env above. This ensures make has access to environment variables stored in a file called .env. This allows Make to utilize these variables in its commands, for example, the name of my virtual environment, or to pass in $(DBNAME) to psql.

What’s with that weird “##” comment syntax? A Makefile like this gives you a handy suite of command-line aliases you can check in to your Django project. It’s very useful so long as you’re able to remember what all those aliases are.

The help command above, which runs by default, prints a helpful list of available commands when you run make or make help:

help                 Show this help
venv                 Make a new virtual environment
install              Make venv and install requirements
migrate              Make and run migrations
db-up                Pull and start the Docker Postgres container in the background
db-shell             Access the Postgres Docker database interactively with psql
test                 Run tests
run                  Run the Django server
start                Install requirements, apply migrations, then start development server

All the usual Django commands are covered, and we’ve got a test command that runs our tests with the options we prefer. Brilliant.

You can read my full post about self-documenting Makefiles here, which also includes an example Makefile using pipenv.

Save your brainpower with pre-commit hooks

I previously wrote about some technical ergonomics that can make it a lot easier for teams to develop great software.

One area that’s a no-brainer is using pre-commit hooks to lint code prior to checking it in. This helps to ensure the quality of the code your developers check in, but most importantly, ensures that no one on your team is spending time trying to remember if it should be single or double quotes or where to put a line break.

The confusingly-named pre-commit framework is an otherwise fantastic way to keep hooks (which are not included in cloned repositories) consistent across local environments.

Here is my configuration file, .pre-commit-config.yaml, for my Django projects:

fail_fast: true
  - repo:
    rev: v3.1.0
      - id: detect-aws-credentials
  - repo:
    rev: 19.3b0
      - id: black
  - repo:
    rev: v1.7.0
      - id: blacken-docs
        additional_dependencies: [black==19.3b0]
  - repo: local
      - id: markdownlint
        name: markdownlint
        description: "Lint Markdown files"
        entry: markdownlint '**/*.md' --fix --ignore node_modules --config "./.markdownlint.json"
        language: node
        types: [markdown]

These hooks check for accidental secret commits, format Python files using Black, format Python snippets in Markdown files using blacken-docs, and lint Markdown files as well. To install them, just type pre-commit install.

There are likely even more useful hooks available for your particular use case: see supported hooks to explore.

Useful gitignores

An underappreciated way to improve your team’s daily development experience is to make sure your project uses a well-rounded .gitignore file. It can help prevent files containing secrets from being committed, and can additionally save developers hours of tedium by ensuring you’re never sifting through a git diff of generated files.

To efficiently create a gitignore for Python and Django projects, Toptal’s can be a nice resource for generating a robust .gitignore file.

I still recommend examining the generated results yourself to ensure that ignored files suit your use case, and that nothing you want ignored is commented out.

Continuous testing with GitHub Actions

If your team works on GitHub, setting up a testing process with Actions is low-hanging fruit.

Tests that run in a consistent environment on every pull request can help eliminate “works on my machine” conundrums, as well as ensure no one’s sitting around waiting for a test to run locally.

A hosted CI environment like GitHub Actions can also help when running integration tests that require using managed services resources. You can use encrypted secrets in a repository to grant the Actions runner access to resources in a testing environment, without worrying about creating testing resources and access keys for each of your developers to use.

I’ve written on many occasions about setting up Actions workflows, including using one to run your Makefile, and how to integrate GitHub event data. GitHub even interviewed me about Actions once.

For Django projects, here’s a GitHub Actions workflow that runs tests with a consistent Python version whenever someone opens a pull request in the repository.

name: Run Django tests

on: pull_request


    runs-on: ubuntu-latest

      - uses: actions/checkout@v2
      - name: Set up Python
        uses: actions/setup-python@v2
          python-version: '3.8'
      - name: Install dependencies
        run: make install
      - name: Run tests
        run: make test

For the installation and test commands, I’ve simply utilized the Makefile that’s been checked in to the repository. A benefit of using your Makefile commands in your CI test workflows is that you only need to keep them updated in one place – your Makefile! No more “why is this working locally but not in CI??!?” headaches.

If you want to step up your security game, you can add Django Security Check as an Action too.

Set up your Django project for success

Want to help keep your development team happy? Set them up for success with these best practices for Django development. Remember, an ounce of brainpower is worth a pound of software!

Manipulating data with Django migrations, 14 Sep 2020 02:12:57 (Victoria Drake) to update Django models and manipulate existing data using migrations. <img src="" width="640" height="320"/>Growing, successful applications are a lovely problem to have. As a product develops, it tends to accumulate complication the way your weekend cake project accumulates layers of frosting. Thankfully, Django, my favorite batteries-included framework, handles complexity pretty well.

Django models help humans work with data in a way that makes sense to our brains, and the framework offers plenty of classes you can inherit to help you rapidly develop a robust application from scratch. As for developing on existing Django applications, there’s a feature for that, too. In this article, we’ll cover how to use Django migrations to update your existing models and database.

What’s under the hood

Django migrations are Python files that help you add and change things in your database tables to reflect changes in your Django models. To understand how Django migrations help you work with data, it may be helpful to understand the underlying structures we’re working with.

What’s a database table

If you’ve laid eyes on a spreadsheet before, you’re already most of the way to understanding a database table. In a relational database, for example, a PostgreSQL database, you can expect to see data organized into columns and rows. A relational database table may have a set number of columns and any number of rows.

In Django, each model is its own table. For example, here’s a Django model:

from django.db import models

class Lunch(models.Model):
    left_side = models.CharField(max_length=100, null=True)
    center = models.CharField(max_length=100, null=True)
    right_side = models.CharField(max_length=100, null=True)

Each field is a column, and each row is a Django object instance of that model. Here’s a representation of a database table for the Django model “Lunch” above. In the database, its name would be lunch_table.

id left_side center right_side
1 Fork Plate Spoon

The model Lunch has three fields: left_side, center, and right-side. One instance of a Lunch object would have “Fork” for the left_side, a “Plate” for the center, and “Spoon” for the right_side. Django automatically adds an id field if you don’t specify a primary key.

If you wanted to change the name of your Lunch model, you would do so in your code. For example, change “Lunch” to “Dinner,” then run python makemigrations. You’ll see:

python makemigrations
Did you rename the backend.Lunch model to Dinner? [y/N] y
Migrations for 'backend':
    - Rename model Lunch to Dinner

Django automatically generates the appropriate migration files. The relevant line of the generated migrations file in this case would look like:

migrations.RenameModel(old_name="Lunch", new_name="Dinner"),

This operation would rename our “Lunch” model to “Dinner” while keeping everything else the same. But what if you also wanted to change the structure of the database table itself, its schema, as well as make sure that existing data ends up in the right place on your Dinner table?

Let’s explore how to turn our Lunch model into a Dinner model that looks like this:

from django.db import models

class Dinner(models.Model):
    top_left = models.CharField(max_length=100, null=True)
    top_center = models.CharField(max_length=100, null=True)
    top_right = models.CharField(max_length=100, null=True)
    bottom_left = models.CharField(max_length=100, null=True)
    bottom_center = models.CharField(max_length=100, null=True)
    bottom_right = models.CharField(max_length=100, null=True)

…with a database table that would look like this:

id top_left top_center top_right bottom_left bottom_center bottom_right
1 Bread plate Spoon Glass Fork Plate Knife

Manipulating data with Django migrations

Before you begin to manipulate your data, it’s always a good idea to create a backup of your database that you can restore in case something goes wrong. There are various ways to do this depending on the database you’re using. You can typically find instructions by searching for <your database name> and keywords like backup, recovery, or snapshot.

In order to design your migration, it’s helpful to become familiar with the available migration operations. Migrations are run step-by-step, and each operation is some flavor of adding, removing, or altering data. Like a strategic puzzle, it’s important to make model changes one step at a time so that the generated migrations have the correct result.

We’ve already renamed our model successfully. Now, we’ll rename the fields that hold the data we want to retain:

class Dinner(models.Model):
    bottom_left = models.CharField(max_length=100, null=True)
    bottom_center = models.CharField(max_length=100, null=True)
    top_center = models.CharField(max_length=100, null=True)

Django is sometimes smart enough to determine the old and new field names correctly. You’ll be asked for confirmation:

python makemigrations
Did you rename to dinner.bottom_center (a CharField)? [y/N] y
Did you rename dinner.left_side to dinner.bottom_left (a CharField)? [y/N] y
Did you rename dinner.right_side to dinner.top_center (a CharField)? [y/N] y
Migrations for 'backend':
    - Rename field center on dinner to bottom_center
    - Rename field left_side on dinner to bottom_left
    - Rename field right_side on dinner to top_center

In some cases, you’ll want to try renaming the field and running makemigrations one at a time.

Now that the existing fields have been migrated to their new names, add the remaining fields to the model:

class Dinner(models.Model):
    top_left = models.CharField(max_length=100, null=True)
    top_center = models.CharField(max_length=100, null=True)
    top_right = models.CharField(max_length=100, null=True)
    bottom_left = models.CharField(max_length=100, null=True)
    bottom_center = models.CharField(max_length=100, null=True)
    bottom_right = models.CharField(max_length=100, null=True)

Running makemigrations again now gives us:

python makemigrations
Migrations for 'backend':
    - Add field bottom_right to dinner
    - Add field top_left to dinner
    - Add field top_right to dinner

You’re done! By generating Django migrations, you’ve successfully set up your dinner_table and moved existing data to its new spot.

Additional complexity

You’ll notice that our Lunch and Dinner models are not very complex. Out of Django’s many model field options, we’re just using CharField. We also set null=True to let Django store empty values as NULL in the database.

Django migrations can handle additional complexity, such as changing field types, and whether a blank or null value is permitted. I keep Django’s model field reference handy as I work with varying types of data and different use cases.

De-mystified migrations

I hope this article has helped you better understand Django migrations and how they work!

Now that you can change models and manipulate existing data in your Django application, be sure to use your powers wisely! Backup your database, research and plan your migrations, and always run tests before working with customer data. By doing so, you have the potential to enable your application to grow – with manageable levels of complexity.

What is TLS? Transport Layer Security encryption explained in plain english, 05 Sep 2020 04:48:39 (Victoria Drake) TLS, digital certificates, and sessions help keep communications secure. <img src="" width="640" height="320"/>If you want to have a confidential conversation with someone you know, you might meet up in person and find a private place to talk. If you want to send data confidentially over the Internet, you might have a few more considerations to cover.

TLS, or Transport Layer Security, refers to a protocol. “Protocol” is a word that means, “the way we’ve agreed to do things around here,” more or less. The “transport layer” part of TLS simply refers to host-to-host communication, such as how a client and a server interact, in the Internet protocol suite model.

The TLS protocol attempts to solve these fundamental problems:

  • How do I know you are who you say you are?
  • How do I know this message from you hasn’t been tampered with?
  • How can we communicate securely?

Here’s how TLS works, explained in plain English. As with many successful interactions, it begins with a handshake.

Getting to know you

The basic process of a TLS handshake involves a client, such as your web browser, and a server, such as one hosting a website, establishing some ground rules for communication. It begins with the client saying hello. Literally. It’s called a ClientHello message.

The ClientHello message tells the server which TLS protocol version and cipher suites it supports. While “cipher suite” sounds like a fancy hotel upgrade, it just refers to a set of algorithms that can be used to secure communications. The server, in a similarly named ServerHello message, chooses the protocol version and cipher suite to use from the choices offered. Other data may also be sent, for example, a session ID if the server supports resuming a previous handshake.

A cartoon of a client and server saying hello

Depending on the cipher suite chosen, the client and server exchange further information in order to establish a shared secret. Often, this process moves the exchange from asymmetric cryptography to symmetric cryptography with varying levels of complexity. Let’s explore these concepts at a general level and see why they matter to TLS.

Asymmetric beginnings

This is asymmetry:

A small egg and a big egg

Small egg, big egg.

Asymmetric cryptography is one method by which you can perform authentication. When you authenticate yourself, you answer the fundamental question, “How do I know you are who you say you are?”

In an asymmetric cryptographic system, you use a pair of keys in order to achieve authentication. These keys are asymmetric. One key is your public key, which, as you would guess, is public. The other is your private key, which – well, you know.

Typically, during the TLS handshake, the server will provide its public key via its digital certificate, sometimes still called its SSL certificate, though TLS replaces the deprecated Secure Sockets Layer (SSL) protocol. Digital certificates are provided and verified by trusted third parties known as Certificate Authorities (CA), which are a whole other article in themselves.

While anyone may encrypt a message using your public key, only your private key can then decrypt that message. The security of asymmetric cryptography relies only on your private key staying private, hence the asymmetry. It’s also asymmetric in the sense that it’s a one-way trip. Alice can send messages encrypted with your public key to you, but neither of your keys will help you send an encrypted message to Alice.

Symmetric secrets

Asymmetric cryptography also requires more computational resources than symmetric cryptography. Thus when a TLS handshake begins with an asymmetric exchange, the client and server will use this initial communication to establish a shared secret, sometimes called a session key. This key is symmetric, meaning that both parties use the same shared secret and must maintain that secrecy for the encryption to be secure.

A cartoon of public-key cryptography vs. shared-key cryptography

Wise man say: share your public key, but keep your shared keys private.

By using the initial asymmetric communication to establish a session key, the client and server can rely on the session key being known only to them. For the rest of the session, they’ll both use this same shared key to encrypt and decrypt messages, which speeds up communication.

Secure sessions

A TLS handshake may use asymmetric cryptography or other cipher suites to establish the shared session key. Once the session key is established, the handshaking portion is complete and the session begins.

The session is the duration of encrypted communication between the client and server. During this time, messages are encrypted and decrypted using the session key that only the client and server have. This ensures that communication is secure.

The integrity of exchanged information is maintained by using a checksum. Messages exchanged using session keys have a message authentication code (MAC) attached. This is not the same thing as your device’s MAC address. The MAC is generated and verified using the session key. Because of this, either party can detect if a message has been changed before being received. This solves the fundamental question, “How do I know this message from you hasn’t been tampered with?”

Sessions can end deliberately, due to network disconnection, or from the client staying idle for too long. Once a session ends, it must be re-established via a new handshake or through previously established secrets called session IDs that allow resuming a session.

TLS and you

Let’s recap:

  • TLS is a cryptographic protocol for providing secure communication.
  • The process of creating a secure connection begins with a handshake.
  • The handshake establishes a shared session key that is then used to secure messages and provide message integrity.
  • Sessions are temporary, and once ended, must be re-established or resumed.

This is just a surface-level skim of the very complex cryptographic systems that help to keep your communications secure. For more depth on the topic, I recommend exploring cipher suites and the various supported algorithms.

The TLS protocol serves a very important purpose in your everyday life. It helps to secure your emails to family, your online banking activities, and the connection by which you’re reading this article. The HTTPS communication protocol is encrypted using TLS. Every time you see that little lock icon in your URL bar, you’re experiencing firsthand all the concepts you’ve just read about in this article. Now you know the answer to the last question: “How can we communicate securely?”

Deceptively simple search-and-replace across multiple files, 25 Aug 2020 04:48:39 (Victoria Drake) to interactively search-and-replace across many files with just two commands, thanks to Vim. <img src="" width="640" height="256"/>While a multitude of methods exist to search for and replace words in a single file, what do you do when you’ve got a string to update across multiple unrelated files, all with different names? You harness the power of command line tools, of course!

First, you’ll need to find all the files you want to change. Stringing together what are effectively search queries for find is really only limited by your imagination. Here’s a simple example that finds Python files:

find . -name '*.py'

The -name test searches for a pattern, such as all files ending in .py, but find can do a lot more with other test conditions, including -regex tests. Run find --help to see the multitude of options.

Further tune your search by using grep to get only the files that contain the string you want to change, such as by adding:

grep -le '\<a whale\>'

The -l option gives you just the file names for all files containing a pattern (denoted with -e) that match “a whale”.

Using Vim’s impressive :bufdo lets you run the same command across multiple buffers, interactively working with all of these files without the tedium of opening, saving, and closing each file, one at a time.

Let’s plug your powerful find+grep results into Vim with:

vim `find . -name '*.py' \
-exec grep -le '\<a whale\>' {} \;`

Using backtick-expansion to pass our search to Vim opens up multiple buffers ready to go. (Do :h backtick-expansion in Vim for more.) Now you can apply the Vim command :bufdo to all of these files and perform actions such as interactive search-and-replace:

:bufdo %s/a whale/a bowl of petunias/gce

The g for “global” will change occurrences of the pattern on all lines. The e will omit errors if the pattern is not found. The c option makes this interactive; if you’re feeling confident, you can omit it to make the changes without reviewing each one.

If one of the patterns contains a / character, you can substitute the separator in the above command to make it more readable. Vim will assume the character following the %s is the separator, so for example:

:bufdo %s_a whale_a bowl of peonies/petunias_gce

When you’ve finished going through all the buffers, save all the work you’ve completed with:

:bufdo wq!

Then bask in the glory of your saved time and effort.

How GitHub Codespaces increases productivity and lowers barriers, 15 Aug 2020 16:08:08 (Victoria Drake) look at how GitHub Codespaces can increase productivity and help new teammates and contributors start developing fast. <img src="" width="640" height="246"/>The most recent integration between Visual Studio Code and GitHub can help make development accessible and welcoming: Codespaces in GitHub!

Now in beta, GitHub Codespaces provide an online, in-the-browser IDE powered by Visual Studio Code. This lets you use this full-featured IDE, complete with extensions, terminal, Git commands, and all the settings you’re accustomed to, on any machine. You can now bring your development workflow anywhere using a tablet or other browser-based device.

Codespaces is great news for open source contributors, too. Adding a codespace configuration to your project is a great way to invite new folks to easily start contributing.

A new open source contributor or new hire at your organization can quickly fire up a codespace and get hacking on a good first issue with no local environment set up or installations necessary!

Starting a new codespace

We’ve added codespace configuration settings over at the OWASP Web Security Testing Guide (WSTG). Want to take it for a spin? See our open issues.

Configuring Codespaces

You can use Visual Studio Code’s .devcontainer folder to configure a development container for your repository as well.

Many pre-built containers are available – just copy the .devcontainer you need to your repository root. If your repository doesn’t have one, a default base Linux image will be used.

Here’s a reason to remove .vscode from your .gitignore file. Any new codespaces created in your repository will now respect settings found at .vscode/settings.json. This means that your online IDE can have the same Workspace configuration as you have on your local machine. Isn’t that useful!

Making Codespaces personal

For next-level dotfiles personalization, consider committing relevant files from your local dotfiles folder as a public GitHub repository at yourusername/dotfiles.

When you create a new codespace, this brings in your configurations, such as shell aliases and preferences, by creating symlinks to dotfiles in your codespace $HOME. This personalizes all the codespaces you create in your account.

Need some inspiration? Browse my dotfiles repository on GitHub.

Developing in a codespace is a familiar experience for Visual Studio Code users, right down to running an application locally.

Thanks to port forwarding, when I run an application in a codespace terminal, clicking on the resulting localhost URL takes me to the appropriate port as output from my codespace.

When I’m working on this website in my codespace, for example, I run hugo serve then click the provided localhost:1313 link to see a preview of my changes in another browser tab.

Want to stay in sync between devices? There’s an extension for that. You can connect to your codespace from Visual Studio Code on your local machine so you can always pick up right where you left off.

Develop anywhere

Codespaces is a super exciting addition to my GitHub workflow. It allows me to access my full development process pretty much anywhere, using devices like my iPad.

It’ll also make it easier for new open source contributors or new hires at your organization to hit the ground running with a set-up IDE. If you have access to the limited beta, I invite you to spin up a codespace and try contributing to the WSTG, or to an issue on one of my open source projects.

I’m looking forward to general availability and seeing what the open source community will dream up for GitHub Codespaces next!

And yes – codespaces support your favorite Visual Studio Code theme. 😈

A screenshot of my codespace

Screenshot of a codespace with the Kabukichō theme for Visual Studio Code

How to create a self-documenting Makefile, 05 Aug 2020 08:55:19 (Victoria Drake) your workflow and DevOps with command-line aliases you can check in <img src="" width="640" height="256"/>My new favorite way to completely underuse a Makefile? Creating personalized, per-project repository workflow command aliases that you can check in.

Can a Makefile improve your DevOps and keep developers happy? How awesome would it be if a new developer working on your project didn’t start out by copying and pasting commands from your README? What if instead of:

pip3 install pipenv
pipenv shell --python 3.8
pipenv install --dev
npm install
pre-commit install --install-hooks
# look up how to install Framework X...
# copy and paste from README...
npm run serve

… you could just type:

make start

…and then start working?

Making a difference

I use make every day to take the tedium out of common development activities like updating programs, installing dependencies, and testing. To do all this with a Makefile (GNU make), we use Makefile rules and recipes. Similar parallels exist for POSIX flavor make, like Target Rules; here’s a great article on POSIX-compatible Makefiles.

Here’s some examples of things we can make easier (sorry):

update: ## Do apt upgrade and autoremove
    sudo apt update && sudo apt upgrade -y
    sudo apt autoremove -y

    pip3 install pipenv
    pipenv shell --python 3.8

install: ## Install or update dependencies
    pipenv install --dev
    npm install
    pre-commit install --install-hooks

serve: ## Run the local development server
    hugo serve --enableGitInfo --disableFastRender --environment development

initial: update env install serve ## Install tools and start development server

Now we have some command-line aliases that you can check in! Great idea! If you’re wondering what’s up with that weird ## comment syntax, it gets better.

A self-documenting Makefile

Aliases are great, if you remember what they all are and what they do without constantly typing cat Makefile. Naturally, you need a help command:

.PHONY: help
help: ## Show this help

With a little command-line magic, this egrep command takes the output of MAKEFILE_LIST, sorts it, and uses awk to find strings that follow the ## pattern. It then prints a helpful formatted version of the comments.

We’ll put it at the top of the file so it’s the default target. Now to see all our handy shortcuts and what they do, we just run make, or make help:

help                 Show this help
initial              Install tools and start development server
install              Install or update dependencies
serve                Run the local development server
update               Do apt upgrade and autoremove

Now we have our very own personalized and project-specific CLI tool!

The possibilities for improving your DevOps flow with a self-documenting Makefile are almost endless. You can use one to simplify any workflow and produce some very happy developers.

Please enjoy the (live!) Makefile I use to manage and develop this Hugo site. I hope it inspires you!

My Hugo site Makefile

SHELL := /bin/bash
.PHONY: help env install upgrade-hugo serve build start initial

help: ## Show this help

	pip3 install pipenv

shell: ## Enter the virtual environment
	pipenv shell

install: ## Install or update dependencies
	pipenv install --dev
	npm install
	npm install -g markdownlint-cli
	pre-commit install --install-hooks

HUGO_VERSION:=$(shell curl -s | grep 'tag_name' | cut -d '"' -f 4 | cut -c 2-)

upgrade-hugo: ## Get the latest Hugo
	mkdir tmp/ && \
	cd tmp/ && \
	curl -sSL$(HUGO_VERSION)/hugo_extended_$(HUGO_VERSION)_Linux-64bit.tar.gz | tar -xvzf- && \
	sudo mv hugo /usr/local/bin/ && \
	cd .. && \
	rm -rf tmp/
	hugo version

dev: ## Run the local development server
	hugo serve --enableGitInfo --disableFastRender --environment development

future: ## Run the local development server in the future
	hugo serve --enableGitInfo --buildFuture --disableFastRender --environment development

build: ## Lock dependencies and build site
	pipenv lock
	hugo --minify --cleanDestination

start: upgrade-hugo serve ## Update Hugo and start development server

initial: env install upgrade-hugo serve ## Install tools and start development server

Go automate your GitHub profile README, 25 Jul 2020 10:51:15 (Victoria Drake) a dynamic Markdown file with Go and GitHub Actions <img src="" width="640" height="256"/>GitHub’s new profile page README feature is having the wonderful effect of bringing some personality to the Myspace pages of the developer Internet. Though Markdown lends itself best to standard static text content, that’s not stopping creative folks from working to create a next-level README. You can include GIFs and images to add some motion and pizazz (they’re covered in GitHub Flavor Markdown), but I’m thinking of something a little more dynamic.

At front-and-center on your GitHub profile, your README is a great opportunity to let folks know what you’re about, what you find important, and to showcase some highlights of your work. You might like to show off your latest repositories, tweet, or blog post. Keeping it up to date doesn’t have to be a pain either, thanks to continuous delivery tools like GitHub Actions.

My current README refreshes itself daily with a link to my latest blog post. Here’s how I’m creating a self-updating with Go and GitHub actions.

Reading and writing files with Go

I’ve been writing a lot of Python lately, but for some things I really like using Go. You could say it’s my go-to language for just-for-func projects. Sorry. Couldn’t stop myself.

To create my, I’m going to get some static content from an existing file, mash it together with some new dynamic content that we’ll generate with Go, then bake the whole thing at 400 degrees until something awesome comes out.

Here’s how we read in a file called and put it in string form:

// Unwrap Markdown content
content, err := ioutil.ReadFile("")
if err != nil {
    log.Fatalf("cannot read file: %v", err)
    return err

// Make it a string
stringyContent := string(content)

The possibilities for your dynamic content are only limited by your imagination! Here, I’ll use the package to read the RSS feed from my blog and get the newest post.

fp := gofeed.NewParser()
feed, err := fp.ParseURL("")
if err != nil {
// Get the freshest item
rssItem := feed.Items[0]

To join these bits together and produce stringy goodness, we use fmt.Sprintf() to create a formatted string.

// Whisk together static and dynamic content until stiff peaks form
blog := "Read my latest blog post: **[" + rssItem.Title + "](" + rssItem.Link + ")**"
data := fmt.Sprintf("%s\n%s\n", stringyContent, blog)

Then to create a new file from this mix, we use os.Create(). There are more things to know about deferring file.Close(), but we don’t need to get into those details here. We’ll add file.Sync() to ensure our README gets written.

// Prepare file with a light coating of os
file, err := os.Create("")
if err != nil {
    return err
defer file.Close()

// Bake at n bytes per second until golden brown
_, err = io.WriteString(file, data)
if err != nil {
    return err
return file.Sync()

View the full code here in my README repository.

Mmmm, doesn’t that smell good? 🍪 Let’s make this happen on the daily with a GitHub Action.

Running your Go program on a schedule with Actions

You can create a GitHub Action workflow that triggers both on a push to your master branch as well as on a daily schedule. Here’s a slice of the .github/workflows/update.yaml that defines this:

      - master
    - cron: '0 11 * * *'

To run the Go program that rebuilds our README, we first need a copy of our files. We use actions/checkout for that:

    - name: 🍽️ Get working copy
      uses: actions/checkout@master
        fetch-depth: 1

This step runs our Go program:

- name: 🍳 Shake & bake README
  run: |
    cd ${GITHUB_WORKSPACE}/update/
    go run main.go    

Finally, we push the updated files back to our repository. Learn more about the variables shown at Using variables and secrets in a workflow.

- name: 🚀 Deploy
  run: |
    git config "${GITHUB_ACTOR}"
    git config "${GITHUB_ACTOR}"
    git add .
    git commit -am "Update dynamic content"
    git push --all -f https://${{ secrets.GITHUB_TOKEN }}${GITHUB_REPOSITORY}.git    

View the full code for this Action workflow here in my README repository.

Go forth and auto-update your README

Congratulations and welcome to the cool kids' club! You now know how to build an auto-updating GitHub profile README. You may now go forth and add all sorts of neat dynamic elements to your page – just go easy on the GIFs, okay?

Writing efficient Django, 09 Jul 2020 04:02:47 (Victoria Drake) to use Django's views, models, and queries to build a better application. <img src="" width="640" height="271"/>I like Django. It’s a well-considered and intuitive framework with a name I can pronounce out loud. You can use it to quickly spin up a weekend-sized project, and you can still use it to run full-blown production applications at scale. I’ve done both these things, and over the years I’ve discovered how to use some of Django’s features for maximum efficiency. These are:

Understanding these main features are the building blocks for maximizing development efficiency with Django. They’ll build the foundation for you to test efficiently and create an awesome development experience for your engineers. Let’s look at how these tools let you create a performant Django application that’s pleasant to build and maintain.

Class-based versus function-based views

Remember that Django is all Python under the hood. When it comes to views, you’ve got two choices: view functions (sometimes called “function-based views”), or class-based views.

Years ago when I first built ApplyByAPI, it was initially composed entirely of function-based views. These offer granular control, and are good for implementing complex logic; just as in a Python function, you have complete control (for better or worse) over what the view does. With great control comes great responsibility, and function-based views can be a little tedious to use. You’re responsible for writing all the necessary methods for the view to work - this is what allows you to completely tailor your application.

In the case of ApplyByAPI, there were only a sparse few places where that level of tailored functionality was really necessary. Everywhere else, function-based views began making my life harder. Writing what is essentially a custom view for run-of-the-mill operations like displaying data on a list page became tedious, repetitive, and error-prone.

With function-based views, you’ll need figure out which Django methods to implement in order to handle requests and pass data to views. Unit testing can take some work to write. In short, the granular control that function-based views offer also requires some granular tedium to properly implement.

I ended up holding back ApplyByAPI while I refactored the majority of the views into class-based views. This was not a small amount of work and refactoring, but when it was done, I had a bunch of tiny views that made a huge difference. I mean, just look at this one:

class ApplicationsList(ListView):
    model = Application
    template_name = "applications.html"

It’s three lines. My developer ergonomics, and my life, got a lot easier.

You may think of class-based views as templates that cover most of the functionality any app needs. There are views for displaying lists of things, for viewing a thing in detail, and editing views for performing CRUD (Create, Read, Update, Delete) operations. Because implementing one of these generic views takes only a few lines of code, my application logic became dramatically succinct. This gave me less repeated code, fewer places for something to go wrong, and a more manageable application in general.

Class-based views are fast to implement and use. The built-in class-based generic views may require less work to test, since you don’t need to write tests for the base view Django provides. (Django does its own tests for that; no need for your app to double-check.) To tweak a generic view to your needs, you can subclass a generic view and override attributes or methods. In my case, since I only needed to write tests for any customizations I added, my test files became dramatically shorter, as did the time and resources it took to run them.

When you’re weighing the choice between function-based or class-based views, consider the amount of customization the view needs, and the future work that will be necessary to test and maintain it. If the logic is common, you may be able to hit the ground running with a generic class-based view. If you need sufficient granularity that re-writing a base view’s methods would make it overly complicated, consider a function-based view instead.

Django models

Models organize your Django application’s central concepts to help make them flexible, robust, and easy to work with. If used wisely, models are a powerful way to collate your data into a definitive source of truth.

Like views, Django provides some built-in model types for the convenience of implementing basic authentication, including the User and Permission models. For everything else, you can create a model that reflects your concept by inheriting from a parent Model class.

class StaffMember(models.Model):
    user = models.OneToOneField(User, on_delete=models.CASCADE)
    company = models.OneToOneField(Company, on_delete=models.CASCADE)

    def __str__(self):
        return + " - " +

When you create a custom model in Django, you subclass Django’s Model class and take advantage of all its power. Each model you create generally maps to a database table. Each attribute is a database field. This gives you the ability to create objects to work with that humans can better understand.

You can make a model useful to you by defining its fields. Many built-in field types are conveniently provided. These help Django figure out the data type, the HTML widget to use when rendering a form, and even form validation requirements. If you need to, you can write custom model fields.

Database relationships can be defined using a ForeignKey field (many-to-one), or a ManyToManyField (give you three guesses). If those don’t suffice, there’s also a OneToOneField. Together, these allow you to define relations between your models with levels of complexity limited only by your imagination. (Depending on the imagination you have, this may or may not be an advantage.)

Retrieving objects with queries

Use your model’s Manager (objects by default) to construct a QuerySet. This is a representation of objects in your database that you can refine, using methods, to retrieve specific subsets. All available methods are in the QuerySet API and can be chained together for even more fun.


Some methods return new QuerySets, such as filter(), or exclude(). Chaining these can give you powerful queries without affecting performance, as QuerySets aren’t fetched from the database until they are evaluated. Methods that evaluate a QuerySet include get(), count(), len(), list(), or bool().

Iterating over a QuerySet also evaluates it, so avoid doing so where possible to improve query performance. For instance, if you just want to know if an object is present, you can use exists() to avoid iterating over database objects.

Use get() in cases where you want to retrieve a specific object. This method raises MultipleObjectsReturned if something unexpected happens, as well as the DoesNotExist exception, if, take a guess.

If you’d like to get an object that may not exist in the context of a user’s request, use the convenient get_object_or_404() or get_list_or_404() which raises Http404 instead of DoesNotExist. These helpful shortcuts are suited to just this purpose. To create an object that doesn’t exist, there’s also the convenient get_or_create().

Efficient essentials

You’ve now got a handle on these three essential tools for building your efficient Django application – congratulations! You can make Django work even better for you by learning about manipulating data with migrations, testing effectively, and setting up your team’s Django development for maximum happiness.

If you’re going to build on GitHub, you may like to set up my django-security-check GitHub Action. In the meantime, you’re well on your way to building a beautiful software project.

Look mom, I'm a GitHub Action Hero, 27 Jun 2020 09:06:33 (Victoria Drake) GitHub blog interviews me for their GitHub Action Hero series. <img src="" width="640" height="336"/>GitHub recently interviewed me for their blog editorial entitled GitHub Action Hero: Victoria Drake. Here’s a behind-the-scenes peek at the original interview questions and my answers.

Among the several Actions I’ve built, I have two current favorites. One is hugo-remote, which lets you continuously deploy a Hugo static site from a private source repository to a public GitHub Pages repository. This keeps the contents of the source repository private, such as your unreleased drafts, while still allowing you to have a public open source site using GitHub Pages.

The second is django-security-check. It’s an effortless way to continuously check that your production Django application is free from a variety of security misconfigurations. You can think of it as your little CI/CD helper for busy projects – a security linter!

Tell us a little bit more about yourself—how did you get started in software tools?

When I was a kid, I spent several summer vacations coding a huge medieval fantasy world MUD (Multi-User Dungeon, like a multiplayer role-playing game) written in LPC, with friends. It was entirely text-based, and built and played via Telnet. I fell in love with the terminal and learned a lot about object-oriented programming and prototype-based programming early on.

I became a freelance developer and had the privilege of working on a wide variety of client projects. Realizing the difficulty that companies have with hiring experienced developers, I built to help. As you might imagine, it allows candidates to apply for jobs via API, instead of emailing a resume. It’s based on the Django framework, so in the process, I learned even more about building reusable units of software.

When I became a co-author and a core maintainer for the Open Web Application Security Project (OWASP) Web Security Testing Guide (WSTG), I gained an even broader appreciation for how a prototype-based, repeatable approach can help build secure web applications. Organizations worldwide consider the WSTG the foremost open source resource for testing the security of web applications. We’ve applied this thinking via the use of GitHub Actions in our repository – I’ll tell you more about that later.

Whether I’m creating an open source tool or leading a development team, my childhood experience still informs how I think about programming today. I strive to create repeatable units of software like GitHub Actions – only now, I make them for large enterprises in the real world!

What is the story behind your built GitHub Action? (Why did you build this?)

Developers take on a lot of responsibility when it comes to building secure applications these days. I’m a full-time senior software developer at a cybersecurity company. I’ve found that I’m maximally productive when I create systems and processes that help myself and my team make desired outcomes inevitable. So I spend my free time building tools that make it easy for other developers to build secure software as well. My Actions help to automate contained, repeatable units of work that can make a big difference in a developer’s day.

Do you have future plans for this or other Actions?

Yes! I’m always finding ways for tools like GitHub Actions to boost the velocity of technical teams, whether at work or in my open source projects. Remember the Open Web Application Security Project? In the work I’ve lead with OWASP, I’ve championed the effort to increase automation using GitHub Actions to maintain quality, securely deploy new versions to the web, and even build PDFs of the WSTG. We’re constantly looking into new ways that GitHub Actions can make our lives easier and our readers' projects more secure.

What has been your favorite feature of GitHub Actions?

I like that I can build an Action using familiar and portable technologies, like Docker. Actions are easy for collaborators to work on too, since in the case of a Dockerized Action, you can use any language your team is comfortable with. This is especially useful in large organizations with polyglot teams and environments. There aren’t any complicated dependencies for running these portable tasks, and you don’t need to learn any special frameworks to get started.

One of my first blog posts about GitHub Actions even describes how I used an Action to run a Makefile! This is especially useful for large legacy applications that want to modernize their pipeline by using GitHub Actions.

What are the biggest challenges you’ve faced while building your GitHub Action?

The largest challenge of GitHub Actions isn’t really in GitHub Actions, but in the transition of legacy software and company culture.

Migrating legacy software is always challenging, particularly with large legacy applications. Moving to modern CI/CD processes requires changes at the software level, team level, and even a shift in thinking when it comes to individual developers. It can help to have a tool like GitHub Actions, which is at once seamlessly modern and familiar, when transitioning legacy code to a modern pipeline.

Anything else you would like to share about your experience? Any stories or lessons learned through building your Action?

I’m happiest when I’m solving a challenge that makes developing secure software less challenging in the future, both for myself and for the technology organization I’m leading. With tools like GitHub Actions, a lot of mental overhead can be offloaded to automatic processes – like getting a whole other brain, for free! This can massively help organizations that are ready to scale up their development output.

In the realm of cybersecurity, not only does creating portable and reusable software make developers' lives easier, it helps to make whole workflows repeatable, which in turn makes software development processes more secure. With smart processes in place, technical teams are happier. As an inevitable result, they’ll build better software for customers, too.

Technical ergonomics for the efficient developer, 22 Jun 2020 06:33:28 (Victoria Drake) you knew these tools existed, you'd probably be using them by now. <img src="" width="640" height="256"/>This article isn’t going to tell you about saving your neck with a Roost stand, or your wrists with a split keyboard - I’ve already done that. This article is about saving your brain.

When I first began to program full time, I found myself constantly tired from the mental exertion. Programming is hard! Thankfully, you can take some solace in knowing it gets easier with practice, and with a great supporting cast. Some very nice folks who preceded us both came up with tools to make the difficult bits of communicating with computers much easier on our poor human meatbrains.

I invite you to explore these super helpful technical tools. They’ll improve your development set up and alleviate much of the mental stress of programming. You soon won’t believe you could have done without them.

Not your average syntax highlighting

If you’re still working with syntax highlighting that just picks out variable and class names for you, that’s cute. Time to turn it up a notch.

My current VSC theme and syntax highlighting

In all seriousness, syntax highlighting can make it much easier to find what you’re looking for on your screen: the current line, where your current code block starts and ends, or the absolute game-changing which-bracket-set-am-I-in highlight. I primarily use Visual Studio Code, but similar extensions can be found for the major text editors.

Here are my favorites:

  • Bracket Pair Colorizer highlights sequential bracket pairs in different matching colors, making the pain of picking through nested brackets and parentheses a thing of the past.
  • TODO Highlight effectively removes any excuse you may have had for unintentionally committing TODO and FIXME comments by making them really easy to see. You can even add your own custom keywords to be highlighted (I suggest, wtf, but you didn’t hear it from me.)
  • Indented Block Highlighting puts an easy-to-distinguish but unobtrusive highlight behind your current indented code block, so you can see just where that if ends and why that last else isn’t doing anything at all.
  • Highlight Line puts a (slightly too) bright line where you last left your cursor. You can customize the line’s appearance - I set the borderWidth of mine to 1px.

The theme pictured in Visual Studio Code above is Kabukichō. I made it.

Use Git hooks

I previously brought you an interactive pre-commit checklist in the style of infomercials that’s both fun and useful for reinforcing the quality of your commits. But that’s not all!

Git hooks are scripts that run automatically at pre-determined points in your workflow. Use them well, and you can save a ton of brainpower. A pre-commit hook remembers to do things like lint and format code, and even runs local tests for you before you indelibly push something embarrassing. Hooks can be a little annoying to share (the .git/hooks directory isn’t tracked and thus omitted when you clone or fork a repository) but there’s a framework for that: the confusingly-named pre-commit framework, which allows you to create a sharable configuration file of Git hook plugins, not just for pre-commit.

I spend a majority of my time these days coding in Python, so here is my current .pre-commit-config.yaml:

fail_fast: true
  - repo:
    rev: v0.1.3
    - id: markdownlint-cli2
      name: markdownlint-cli2
      description: "Checks the style of Markdown/CommonMark files."
      entry: markdownlint-cli2
      language: node
      types: [markdown]
      minimum_pre_commit_version: 0.15.0

There are tons of supported hooks to explore.

Use a type system

If you write in languages like Python and JavaScript, get yourself an early birthday present and start using a static type system. Not only will this help improve the way you think about code, it can help make type errors clear before running a single line.

For Python, I like using mypy for static type checking. You can set it up as a pre-commit hook (see above) and it’s supported in Visual Studio Code too.

TypeScript is my preferred way to write JavaScript. You can run the compiler on the command line using Node.js (see instructions in the repo), it works pretty well with Visual Studio Code out of the box, and of course there are multiple options for extension integrations.

Quit unnecessarily beating up your meatbrain

I mean, you wouldn’t stand on your head all day to do your work. It would be rather inconvenient to read things upside down all the time (at least until your brain adjusted), and in any case you’d likely get uncomfortably congested in short order. Working without taking advantage of the technical ergonomic tools I’ve given you today is a little like unnecessary inversion - why would you, if you don’t have to?

How to choose and care for a secure open source project, 25 May 2020 05:53:09 (Victoria Drake) few tricks for assessing the security of an open source project. <img src="" width="640" height="256"/>There is a rather progressive sect of the software development world that believes that most people would be a lot happier and get a lot more work done if they just stopped building things that someone else has already built and is offering up for free use. They’re called the open source community. They want you to take their stuff.

A comic I drew about using other people&rsquo;s stuff, with the wheel as an example.

Besides existing without you having to lift a finger, open source tools and software have some distinct advantages. Especially in the case of well-established projects, it’s highly likely that someone else has already worked out all the most annoying bugs for you. Thanks to the ease with which users can view and modify source code, it’s also more likely that a program has been tinkered with, improved, and secured over time. When many developers contribute, they bring their own unique expertise and experiences. This can result in a product far more robust and capable than one a single developer can produce.

Of course, being as varied as the people who build them, not all open source projects are created equal, nor maintained to be equally secure. There are many factors that affect a project’s suitability for your use case. Here are a few general considerations that make a good starting point when choosing an open source project.

How to choose an open source project

As its most basic requirements, a good software project is reliable, easy to understand, and has up-to-date components and security. There are several indicators that can help you make an educated guess about whether an open source project satisfies these criteria.

Who’s using it

Taken in context, the number of people already using an open source project may be indicative of how good it is. If a project has a hundred users, for instance, it stands to reason that someone has tried to use it at least a hundred times before you found it. Thus by the ancient customs of “I don’t know what’s in that cave, you go first,” it’s more likely to be fine.

You can draw conclusions about a project’s user base by looking at available statistics. Depending on your platform, these may include the number of downloads, reviews, issues or tickets, comments, contributions, forks, or “stars,” whatever those are.

Evaluate social statistics on platforms like GitHub with a grain of salt. They can help you determine how popular a project may be, but only in the same way that restaurant review apps can help you figure out if you should eat at Foo’s Grill & Bar. Depending on where Foo’s Grill & Bar is, when it opened, and how likely people are to be near it when the invariable steak craving should call, having twenty-six reviews may be a good sign or a terrible one. While you would not expect a project that addresses a very obscure use case or technology to have hundreds of users, having a few active users is, in such a case, just as confidence-inspiring.

External validation can also be useful. For example, packages that are included in a Linux operating system distribution (distro) must conform to stringent standards and undergo vetting. Choosing software that is included in a distro’s default repositories can mean it’s more likely to be secure.

Perhaps one of the best indications to look for is whether a project’s development team is using their own project. Look for issues, discussions, or blog posts that show that the project’s creators and maintainers are using what they’ve built themselves. Commonly referred to as “eating your own dog food," or “dogfooding,” it’s an indicator that the project is most likely to be well-maintained by its developers.

Who’s building it

The main enemy of good open source software is usually a lack of interest. The parties involved in an open source project can make the difference between a flash-in-the-pan library and a respected long-term utility. Multiple committed maintainers, even making contributions in their spare time, have a much higher success rate of sustaining a project and generating interest.

Projects with healthy interest are usually supported by, and in turn cultivate, a community of contributors and users. New contributors may be actively welcomed, clear guides are available explaining how to help, and project maintainers are available and approachable when people have inevitable questions. Some communities even have chat rooms or forums where people can interact outside of contributions. Active communities help sustain project interest, relevance, and its ensuing quality.

In a less organic fashion, a project can also be sustained through organizations that sponsor it. Governments and companies with financial interest are open source patrons too, and a project that enjoys public sector use or financial backing has added incentive to remain relevant and useful.

How alive is it

The recency and frequency of an open source project’s activity is perhaps the best indicator of how much attention is likely paid to its security. Look at releases, commit history, changelogs, or documentation revisions to determine if a project is active. As projects vary in size and scope, here are some general things to look for.

Maintaining security is an ongoing endeavor that requires regular monitoring and updates, especially for projects with third-party components. These may be libraries or any part of the project that relies on something outside itself, such as a payment gateway integration. An inactive project is more likely to have outdated code or use outdated versions of components. For a more concrete determination, you can research a project’s third-party components and compare their most recent patches or updates with the project’s last updates.

Projects without third-party components may have no outside updates to apply. In these cases, you can use recent activity and release notes to determine how committed a project’s maintainers may be. Generally, active projects should show updates within the last months, with a notable release within the last year. This can be a good indication of whether the project is using an up-to-date version of its language or framework.

You can also judge how active a project may be by looking at the project maintainers themselves. Active maintainers quickly respond to feedback or new issues, even if it’s just to say, “We’re on it.” If the project has a community, its maintainers are a part of it. They may have a dedicated website or write regular blogs. They may offer ways to contact them directly and privately, especially to raise security concerns.

Can you understand it

Having documentation is a baseline requirement for a project that’s intended for anyone but its creator to use. Good open source projects have documentation that is easy to follow, honest, and thorough.

Having well-written documentation is one way a project can stand out and demonstrate the thoughtfulness and dedication of its maintainers. A “Getting Started” section may detail all the requirements and initial set up for running the project. An accurate list of topics in the documentation enables users to quickly find the information they need. A clear license statement leaves no doubt as to how the project can be used, and for what purposes. These are characteristic aspects of documentation that serves its users.

A project that is following sound coding practices likely has code that is as readable as its documentation. Code that is easy to read lends itself to being understood. Generally, it has clearly defined and appropriately-named functions and variables, a logical flow, and apparent purpose. Readable code is easier to fix, secure, and build upon.

How compatible is it

A few factors will determine how compatible a project is with your goals. These are objective qualities, and can be determined by looking at a project’s repository files. They include:

  • Code language
  • Specific technologies or frameworks
  • License compatibility

Compatibility doesn’t necessarily mean a direct match. Different code languages can interact with each other, as can various technologies and frameworks. You should carefully read a project’s license to understand if it permits usage for your goal, or if it is compatible with a license you would like to use.

Ultimately, a project that satisfies all these criteria may still not quite suit your use case. Part of the beauty of open source software, however, is that you may still benefit from it by making alterations that better suit your usage. If those alterations make the project better for everyone, you can pay it back and pay it forward by contributing your work to the project.

Proper care and feeding of an open source project

Once you adopt an open source project, a little attention is required to make sure it continues to be a boon to your goals. While its maintainers will look after the upstream project files, you alone are responsible for your own copy. Like all software, your open source project must be well-maintained in order to remain as secure and useful as possible.

Have a system that provides you with notifications when updates for your software are made available. Update software promptly, treating each patch as if it were vital to security; it may well be. Keep in mind that open source project creators and maintainers are, in most cases, acting only out of the goodness of their own hearts. If you’ve got a particularly awesome one, its developers may make updates and security patches available on a regular basis. It’s up to you to keep tabs on updates and promptly apply them.

As with most things in software, keeping your open source additions modular can come in handy. You might use git submodules, branches, or environments to isolate your additions. This can make it easier to apply updates or pinpoint the source of any bugs that arise.

So although an open source project may cost no money, caveat emptor, which means, “Jimmy, if we get you a puppy, it’s your responsibility to take care of it.”

If you want to build a treehouse, start at the bottom, 11 May 2020 05:46:47 (Victoria Drake) threat modeling and pushing left help create a stable foundation for secure software. <img src="" width="640" height="256"/>If you’ve ever watched a kid draw a treehouse, you have some idea of how applications are built when security isn’t made a priority. It’s far more fun to draw the tire swing, front porch, and swimming pool than to worry about how a ten-thousand-gallon bucket of water stays suspended in midair. With too much attention spent on fun and flashy features, foundations suffer.

A comic I drew about building castles with poor foundations. It&rsquo;s not that funny.

Of course, spending undue hours building a back end like Fort Knox may not be necessary for your application, either. Being an advocate for security doesn’t mean always wearing your tinfoil hat (although you do look dashing in it) but does mean building in an appropriate amount of security.

How much security is appropriate? The answer, frustratingly, is, “it depends.” The right amount of security for your application depends on who’s using it, what it does, and most importantly, what undesirable things it could be made to do. It takes some analysis to make decisions about the kinds of risks your application faces and how you’ll prepare to handle them. Okay, now’s a good time to don your tinfoil hat. Let’s imagine the worst.

Threat modeling: what’s the worst that could happen

A threat model is a stuffy term for the result of trying to imagine the worst things that could happen to an application. Using your imagination to assess risks (fittingly called risk assessment) is a conveniently non-destructive method for finding ways an application can be attacked. You won’t need any tools; just an understanding of how the application might work, and a little imagination. You’ll want to record your results with pen and paper. For the younger folks, that means the notes app on your phone.

A few different methodologies for application risk assessment can be found in the software world, including the in-depth NIST Special Publication 800-30. Each method’s framework has specific steps and output, and will go into various levels of detail when it comes to defining threats. If following a framework, first choose the one you’re most likely to complete. You can always add more depth and detail from there.

Even informal risk assessments are beneficial. Typically taking the form of a set of questions, they may be oriented around possible threats, the impact to assets, or ways a vulnerability could be exploited. Here are some examples of questions addressing each orientation:

  • What kind of adversary would want to break my app? What would they be after?
  • If the control of x fell into the wrong hands, what could an attacker do with it?
  • Where could a x vulnerability occur in my app?

A basic threat model explains the technical, business, and human considerations for each risk. It will typically detail:

  • The vulnerabilities or components that can cause the risk
  • The impact that a successful execution of the risk would have on the application
  • The consequences for the application’s users or organization

The result of a risk assessment exercise is your threat model; in other words, a list of things you would very much like not to occur. It is usually sorted in a hierarchy of risks, from the worst to the mildest. The worst risks have the most negative impact, and are most important to protect against. The mildest risks are the most acceptable - while still an undesirable outcome, they have the least negative impact on the application and users.

You can use this resulting hierarchy as a guide to determine how much of your cybersecurity efforts to apply to each risk area. An appropriate amount of security for your application will eliminate (where possible) or mitigate the worst risks.

Pushing left

Although it sounds like a dance move meme, pushing left refers instead to building in as much of your planned security as possible in the early stages of software development.

Building software is a lot like building a treehouse, just without the pleasant fresh air. You start with the basic supporting components, such as attaching a platform to a tree. Then comes the framing, walls, and roof, and finally, your rustic-modern Instagram-worthy wall hangings and deer bust.

The further along in the build process you are, the harder and more costly it becomes to make changes to a component that you’ve already installed. If you discover a problem with the walls only after the roof is put in place, you may need to change or remove the roof in order to fix it. Similar parallels can be drawn for software components, only without similar ease in detangling the attached parts.

In the case of a treehouse, it’s rather impossible to start with decorations or even a roof, since you can’t really suspend them in midair. In the case of software development, it is, unfortunately, possible to build many top-layer components and abstractions without a sufficient supporting architecture. A push-left approach views each additional layer as adding cost and complication. Pushing left means attempting to mitigate security risks as much as possible at each development stage before proceeding to the next.

Building bottom-to-top

By considering your threat model in the early stages of developing your application, you reduce the chances of necessitating a costly remodel later on. You can make choices about architecture, components, and code that support the main security goals of your particular application.

While it’s not possible to foresee all the functionality your application may one day need to support, it is possible to prepare a solid foundation that allows additional functionality to be added more securely. Building in appropriate security from the bottom to the top will help make mitigating security risks much easier in the future.

Hugo vs Jekyll: an epic battle of static site generator themes, 27 Apr 2020 06:34:41 (Victoria Drake) comparison of nuances of creating themes for the top two static site generators.I recently took on the task of creating a documentation site theme for two projects. Both projects needed the same basic features, but one uses Jekyll while the other uses Hugo.

In typical developer rationality, there was clearly only one option. I decided to create the same theme in both frameworks, and to give you, dear reader, a side-by-side comparison.

This post isn’t a comprehensive theme-building guide, but intended to familiarize you with the process of building a theme in either generator. Here’s what we’ll cover:

Here’s a crappy wireframe of the theme I’m going to create.

A sketch of the finished page

If you’re planning to build-along, it may be helpful to serve the theme locally as you build it; both generators offer this functionality. For Jekyll, run jekyll serve, and for Hugo, hugo serve.

There are two main elements: the main content area, and the all-important sidebar menu. To create them, you’ll need template files that tell the site generator how to generate the HTML page. To organize theme template files in a sensible way, you first need to know what directory structure the site generator expects.

How theme files are organized

Jekyll supports gem-based themes, which users can install like any other Ruby gems. This method hides theme files in the gem, so for the purposes of this comparison, we aren’t using gem-based themes.

When you run jekyll new-theme <name>, Jekyll will scaffold a new theme for you. Here’s what those files look like:

├── assets
├── Gemfile
├── _includes
├── _layouts
│   ├── default.html
│   ├── page.html
│   └── post.html
├── LICENSE.txt
├── _sass
└── <name>.gemspec

The directory names are appropriately descriptive. The _includes directory is for small bits of code that you reuse in different places, in much the same way you’d put butter on everything. (Just me?) The _layouts directory contains templates for different types of pages on your site. The _sass folder is for Sass files used to build your site’s stylesheet.

You can scaffold a new Hugo theme by running hugo new theme <name>. It has these files:

├── archetypes
│   └──
├── layouts
│   ├── 404.html
│   ├── _default
│   │   ├── baseof.html
│   │   ├── list.html
│   │   └── single.html
│   ├── index.html
│   └── partials
│       ├── footer.html
│       ├── header.html
│       └── head.html
├── static
│   ├── css
│   └── js
└── theme.toml

You can see some similarities. Hugo’s page template files are tucked into layouts/. Note that the _default page type has files for a list.html and a single.html. Unlike Jekyll, Hugo uses these specific file names to distinguish between list pages (like a page with links to all your blog posts on it) and single pages (like one of your blog posts). The layouts/partials/ directory contains the buttery reusable bits, and stylesheet files have a spot picked out in static/css/.

These directory structures aren’t set in stone, as both site generators allow some measure of customization. For example, Jekyll lets you define collections, and Hugo makes use of page bundles. These features let you organize your content multiple ways, but for now, lets look at where to put some simple pages.

Where to put content

To create a site menu that looks like this:

    Getting Started
Advanced Usage
    All Configuration Settings
    Help and Support

You’ll need two sections (“Introduction” and “Advanced Usage”) containing their respective subsections.

Jekyll isn’t strict with its content location. It expects pages in the root of your site, and will build whatever’s there. Here’s how you might organize these pages in your Jekyll site root:

├── 404.html
├── assets
├── Gemfile
├── _includes
├── index.markdown
├── intro
│   ├──
│   ├──
│   ├──
│   └──
├── _layouts
│   ├── default.html
│   ├── page.html
│   └── post.html
├── LICENSE.txt
├── _sass
├── <name>.gemspec
└── usage

You can change the location of the site source in your Jekyll configuration.

In Hugo, all rendered content is expected in the content/ folder. This prevents Hugo from trying to render pages you don’t want, such as 404.html, as site content. Here’s how you might organize your content/ directory in Hugo:

├── intro
│   ├──
│   ├──
│   ├──
│   └──
└── usage

To Hugo, and mean different things. It can be helpful to know what kind of Page Bundle you want for each section: Leaf, which has no children, or Branch.

Now that you have some idea of where to put things, let’s look at how to build a page template.

How templating works

Jekyll page templates are built with the Liquid templating language. It uses braces to output variable content to a page, such as the page’s title: {{ page.title }}.

Hugo’s templates also use braces, but they’re built with Go Templates. The syntax is similar, but different: {{ .Title }}.

Both Liquid and Go Templates can handle logic. Liquid uses tags syntax to denote logic operations:

{% if user %}
  Hello {{ }}!
{% endif %}

And Go Templates places its functions and arguments in its braces syntax:

{{ if .User }}
    Hello {{ .User }}!
{{ end }}

Templating languages allow you to build one skeleton HTML page, then tell the site generator to put variable content in areas you define. Let’s compare two possible default page templates for Jekyll and Hugo.

Jekyll’s scaffold default theme is bare, so we’ll look at their starter theme Minima. Here’s _layouts/default.html in Jekyll (Liquid):

<!DOCTYPE html>

  {%- include head.html -%}


    {%- include header.html -%}

    <main class="page-content" aria-label="Content">
      <div class="wrapper">
        {{ content }}

    {%- include footer.html -%}



Here’s Hugo’s scaffold theme layouts/_default/baseof.html (Go Templates):

<!DOCTYPE html>
    {{- partial "head.html" . -}}
        {{- partial "header.html" . -}}
        <div id="content">
        {{- partial "footer.html" . -}}

Different syntax, same idea. Both templates pull in reusable bits for head.html, header.html, and footer.html. These show up on a lot of pages, so it makes sense not to have to repeat yourself. Both templates also have a spot for the main content, though the Jekyll template uses a variable ({{ content }}) while Hugo uses a block ({{- block "main" . }}{{- end }}). Blocks are just another way Hugo lets you define reusable bits.

Now that you know how templating works, you can build the sidebar menu for the theme.

Creating a top-level menu with the pages object

You can programmatically create a top-level menu from your pages. It will look like this:

Advanced Usage

Let’s start with Jekyll. You can display links to site pages in your Liquid template by iterating through the site.pages object that Jekyll provides and building a list:

    {% for page in site.pages %}
    <li><a href="{{ page.url | absolute_url }}">{{ page.title }}</a></li>
    {% endfor %}

This returns all of the site’s pages, including all the ones that you might not want, like 404.html. You can filter for the pages you actually want with a couple more tags, such as conditionally including pages if they have a section: true parameter set:

    {% for page in site.pages %}
    {%- if page.section -%}
    {%- endif -%}
    {% endfor %}

You can achieve the same effect with slightly less code in Hugo. Loop through Hugo’s .Pages object using Go Template’s range action:

{{ range .Pages }}
{{ end }}

This template uses the .Pages object to return all the top-level pages in content/ of your Hugo site. Since Hugo uses a specific folder for the site content you want rendered, there’s no additional filtering necessary to build a simple menu of site pages.

Both site generators can use a separately defined data list of links to render a menu in your template. This is more suitable for creating nested links, like this:

    Getting Started
Advanced Usage
    All Configuration Settings
    Help and Support

Jekyll supports data files in a few formats, including YAML. Here’s the definition for the menu above in _data/menu.yml:

  - page: Introduction
    url: /intro
      - page: Getting Started
        url: /intro/quickstart
      - page: Configuration
        url: /intro/config
      - page: Deploying
        url: /intro/deploy
  - page: Advanced Usage
    url: /usage
      - page: Customizing
        url: /usage/customizing
      - page: All Configuration Settings
        url: /usage/settings
      - page: Help and Support
        url: /usage/support

Here’s how to render the data in the sidebar template:

{% for a in %}
<a href="{{ a.url }}">{{ }}</a>
    {% for b in a.subsection %}
    <li><a href="{{ b.url }}">{{ }}</a></li>
    {% endfor %}
{% endfor %}

This method allows you to build a custom menu, two nesting levels deep. The nesting levels are limited by the for loops in the template. For a recursive version that handles further levels of nesting, see Nested tree navigation with recursion.

Hugo does something similar with its menu templates. You can define menu links in your Hugo site config, and even add useful properties that Hugo understands, like weighting. Here’s a definition of the menu above in config.yaml:

sectionPagesMenu: main

    - identifier: intro
      name: Introduction
      url: /intro/
      weight: 1
    - name: Getting Started
      parent: intro
      url: /intro/quickstart/
      weight: 1
    - name: Configuration
      parent: intro
      url: /intro/config/
      weight: 2
    - name: Deploying
      parent: intro
      url: /intro/deploy/
      weight: 3
    - identifier: usage
      name: Advanced Usage
      url: /usage/
    - name: Customizing
      parent: usage
      url: /usage/customizing/
      weight: 2
    - name: All Configuration Settings
      parent: usage
      url: /usage/settings/
      weight: 1
    - name: Help and Support
      parent: usage
      url: /usage/support/
      weight: 3

Hugo uses the identifier, which must match the section name, along with the parent variable to handle nesting. Here’s how to render the menu in the sidebar template:

    {{ range .Site.Menus.main }}
    {{ if .HasChildren }}
        <a href="{{ .URL }}">{{ .Name }}</a>
    <ul class="sub-menu">
        {{ range .Children }}
        {{ end }}
    {{ else }}
    {{ end }}
    {{ end }}

The range function iterates over the menu data, and Hugo’s .Children variable handles nested pages for you.

Putting the template together

With your menu in your reusable sidebar bit (_includes/sidebar.html for Jekyll and partials/sidebar.html for Hugo), you can add it to the default.html template.

In Jekyll:

<!DOCTYPE html>

{%- include head.html -%}

    {%- include sidebar.html -%}

    {%- include header.html -%}

    <div id="content" class="page-content" aria-label="Content">
        {{ content }}

    {%- include footer.html -%}



In Hugo:

<!DOCTYPE html>
{{- partial "head.html" . -}}

    {{- partial "sidebar.html" . -}}

    {{- partial "header.html" . -}}
    {{- partial "footer.html" . -}}


When the site is generated, each page will contain all the code from your sidebar.html.

Create a stylesheet

Both site generators accept Sass for creating CSS stylesheets. Jekyll has Sass processing built in, and Hugo uses Hugo Pipes. Both options have some quirks.

Sass and CSS in Jekyll

To process a Sass file in Jekyll, create your style definitions in the _sass directory. For example, in a file at _sass/style-definitions.scss:

$background-color: #eef !default;
$text-color: #111 !default;

body {
  background-color: $background-color;
  color: $text-color;

Jekyll won’t generate this file directly, as it only processes files with front matter. To create the end-result filepath for your site’s stylesheet, use a placeholder with empty front matter where you want the .css file to appear. For example, assets/css/style.scss. In this file, simply import your styles:


@import "style-definitions";

This rather hackish configuration has an upside: you can use Liquid template tags and variables in your placeholder file. This is a nice way to allow users to set variables from the site _config.yml, for example.

The resulting CSS stylesheet in your generated site has the path /assets/css/style.css. You can link to it in your site’s head.html using:

<link rel="stylesheet" href="{{ "/assets/css/style.css" | relative_url }}" media="screen">

Sass and Hugo Pipes in Hugo

Hugo uses Hugo Pipes to process Sass to CSS. You can achieve this by using Hugo’s asset processing function, resources.ToCSS, which expects a source in the assets/ directory. It takes the SCSS file as an argument. With your style definitions in a Sass file at assets/sass/style.scss, here’s how to get, process, and link your Sass in your theme’s head.html:

{{ $style := resources.Get "/sass/style.scss" | resources.ToCSS }}
<link rel="stylesheet" href="{{ $style.RelPermalink }}" media="screen">

Hugo asset processing requires extended Hugo, which you may not have by default. You can get extended Hugo from the releases page.

Configure and deploy to GitHub Pages

Before your site generator can build your site, it needs a configuration file to set some necessary parameters. Configuration files live in the site root directory. Among other settings, you can declare the name of the theme to use when building the site.

Configure Jekyll

Here’s a minimal _config.yml for Jekyll:

title: Your awesome title
  Google search results) and in your feed.xml site description.
theme: # for gem-based themes
remote_theme: # for themes hosted on GitHub, when used with GitHub Pages

With remote_theme, any Jekyll theme hosted on GitHub can be used with sites hosted on GitHub Pages.

Jekyll has a default configuration, so any parameters added to your configuration file will override the defaults. Here are additional configuration settings.

Configure Hugo

Here’s a minimal example of Hugo’s config.yml:

baseURL: # The full domain your site will live at
languageCode: en-us
title: Hugo Docs Site
theme: # theme name

Hugo makes no assumptions, so if a necessary parameter is missing, you’ll see a warning when building or serving your site. Here are all configuration settings for Hugo.

Deploy to GitHub Pages

Both generators build your site with a command.

For Jekyll, use jekyll build. See further build options here.

For Hugo, use hugo. You can run hugo help or see further build options here.

You’ll have to choose the source for your GitHub Pages site; once done, your site will update each time you push a new build. Of course, you can also automate your GitHub Pages build using GitHub Actions. Here’s one for building and deploying with Hugo, and one for building and deploying Jekyll.


All the substantial differences between these two generators are under the hood; all the same, let’s take a look at the finished themes, in two color variations.

Here’s Hugo:

OpenGitDocs theme for Hugo

Here’s Jekyll:

OpenGitDocs theme for Jekyll


Wait who won


Both Hugo and Jekyll have their quirks and conveniences.

From this developer’s perspective, Jekyll is a workable choice for simple sites without complicated organizational needs. If you’re looking to render some one-page posts in an available theme and host with GitHub Pages, Jekyll will get you up and running fairly quickly.

Personally, I use Hugo. I like the organizational capabilities of its Page Bundles, and it’s backed by a dedicated and conscientious team that really seems to strive to facilitate convenience for their users. This is evident in Hugo’s many functions, and handy tricks like Image Processing and Shortcodes. They seem to release new fixes and versions about as often as I make a new cup of coffee.

If you still can’t decide, don’t worry. Many themes are available for both Hugo and Jekyll! Start with one, switch later if you want. That’s the benefit of having options.

Outsourcing security with 1Password, Authy, and Privacy.com, 16 Mar 2020 08:12:32 (Victoria Drake) some work off your plate while beefing up security with three changes you can make today. <img src="" width="640" height="256"/>We’ve already got enough to deal with without worrying about our cybersecurity. When humans are busy and under stress, we tend to get lax in less-obviously-pressing areas, like the integrity of our online accounts. These areas only become an obvious problem when it’s too late for prevention.

Cybersecurity can be fiddly and time-consuming. You might need to reset forgotten passwords, transfer multifactor authentication (MFA) codes to different devices, or deal with the fallout of compromised payment details in the event one of your accounts is still breached.

Thankfully, most of the work necessary to keep up our cybersecurity measures can be outsourced.

Here are three changes you can make to significantly reduce the chances of needing to fiddle with any of these things again.


1Password on an iPhone

I’ve historically avoided password managers because of an irrational knee-jerk reaction to putting all my eggs in one basket. You know what’s great for irrational reactions? Education.

To figure out if putting all my passwords into a password manager is more secure than not using one, I set out to see what some smart people wrote about it.

First, we need to know a thing or two about passwords. Troy Hunt figured out almost a decade ago that trying to remember strong passwords doesn’t work. In more recent times, Alex Weinert expanded on this in Your Pa$$word doesn’t matter. TL;DR: our brains aren’t better at passwords than computers, and please use MFA.

So passwords don’t matter, but complicated passwords are still better than memorable and guessable ones. Since I’ve next to no hope of remembering a dozen variations of p/q2-q4! (I’m not a chess player), this is a task I can outsource to 1Password. I’ll still need to remember one, long, complicated master password - 1Password uses this to encrypt my data, so I really can’t lose it - but I can handle just one.

Using 1Password specifically has another, decidedly obvious, advantage. I chose 1Password because of their Watchtower feature. Thanks to Troy Hunt’s Have I Been Pwned, Watchtower will alert you if any of your passwords show up in a breach so you can change them. Passwords still don’t completely work, but this is probably the best band-aid there is.

One last bonus is that using a password manager is a heck of a lot more convenient. I don’t need to take a few tries to type in a complicated password. I don’t end up spending time resetting passwords I’ve forgotten on sites I only rarely use.

When tasked with remembering all their own passwords, people typically create simpler passwords that are easier to remember – and easier to hack. This occurs most frequently on sites that are considered unimportant. Using 1Password and generated passwords, those sites are now also first-class citizens in the land of strong passwords, instead of being half-abandoned and half-open attack vectors.

So, yes, all my eggs are in one basket. A well-protected, complex, and monitored basket.


Okay - so it’s more like one-and-a-half baskets. 🤷🏻

Authy&rsquo;s logo

Authy, from the folks over at Twilio, provides a 2FA solution that’s more secure than SMS. Unlike Google Authenticator, you can choose to back up your 2FA codes in case you lose or change your phone. (1Password offers 2FA functionality as well - but, you know, redundancies.)

With Authy, your back up is encrypted with your password, similarly to how 1Password works. This makes it the second password you can’t forget, if you don’t want to lose access to your codes. If you reset your account, they all go away. I can deal with remembering two passwords; I’ll take that trade.

I’ve tried other methods of MFA, including hardware keys, which can make accessing accounts on your phone more complicated than I care to put up with. I find the combination of 1Password and Authy to be the most practical combination of convenience and security that yet exists to my knowledge.

Screenshot of Privacy card

Finally, there’s one last line of defense you can put in place in the unfortunate event that one of your accounts is still compromised. All the strong passwords and MFA in the world won’t help if you open the doors yourself, and scams and phishing are a thing.

Since it’s rather impractical to use a different real credit card every place you shop, virtual cards are just a great idea. There’s no good reason to spend an afternoon (or more) resetting your payment information on every account just to thwart a misbehaving merchant or patch up a data breach from that online shop for cute salt shakers you made a purchase at last year (just me?).

As a bonus, a partnership between 1Password and lets you easily create virtual credit cards using the 1Password extension.

By setting up a separate virtual card for each merchant, in the event that one of those merchants is compromised, you can simply pause or delete that card. None of your other accounts or actual bank details are caught up in the process. Cards can have time-based limits or be one-off burner numbers, making them ideal for setting up subscriptions.

This is the sort of basic functionality that I hope, one day, becomes more prevalent from banks and credit cards. In the meantime, I’ll keep using That’s my referral link; if you’d like to thank me by using it, we’ll both get five bucks as a bonus.

Outsource better security

All together, implementing these changes will probably take up an afternoon, depending on how many accounts you have. It’s worth it for the time you’d otherwise spend resetting passwords, setting up new devices, or (knock on wood) recovering from compromised banking details. Best of all, you’ll have continual protection just running in the background.

We have the technology. Free up some brain cycles to focus on other things - or simply remove some unnecessary stress from your life by outsourcing the fiddly bits.

Want to give the gift of cybersecurity to someone you know? Get them started with a cybersecurity starter pack.

SQLite in production with WAL, 05 Mar 2020 10:14:43 (Victoria Drake) underappreciated candidate for light and fast database transactions. <img src="" width="640" height="256"/>Update: read the HackerNews discussion.

SQLite (“see-quell-lite”) is a lightweight Sequel, or Structured Query Language (SQL), database engine. Instead of using the client-server database management system model, SQLite is self-contained in a single file. It is library, database, and data, all in one package.

For certain applications, SQLite is a solid choice for a production database. It’s lightweight, ultra-portable, and has no external dependencies. Remember when MacBook Air first came out? It’s nothing like that.

SQLite is best suited for production use in applications that:

  • Desire fast and simple set up.
  • Require high reliability in a small package.
  • Have, and want to retain, a small footprint.
  • Are read-heavy but not write-heavy.
  • Don’t need multiple user accounts or features like multiversion concurrency snapshots.

If your application can benefit from SQLite’s serverless convenience, you may like to know about the different modes available for managing database changes.

With and without WAL

POSIX system call fsync() commits buffered data (data saved in the operating system cache) referred to by a specified file descriptor to permanent storage or disk. This is relevant to understanding the difference between SQLite’s two modes, as fsync() will block until the device reports the transfer is complete.

For efficiency, SQLite uses atomic commits to batch database changes into a single transaction. This enables the apparent writing of many transactions to a database file simultaneously. Atomic commits are performed using one of two modes: a rollback journal, or a write-ahead log (WAL).

Rollback journal

A rollback journal is essentially a back-up file created by SQLite before write changes occur on a database file. It has the advantage of providing high reliability by helping SQLite restore the database to its original state in case a write operation is compromised during the disk-writing process.

Assuming a cold cache, SQLite first needs to read the relevant pages from a database file before it can write to it. Information is read out into the operating system cache, then transferred into user space. SQLite obtains a reserved lock on the database file, preventing other processes from writing to the database. At this point, other processes may still read from the database.

SQLite creates a separate file, the rollback journal, with the original content of the pages that will be changed. Initially existing in the cache, the rollback journal is written to persistent disk storage with fsync() to enable SQLite to restore the database should its next operations be compromised.

SQLite then obtains an exclusive lock preventing other processes from reading or writing, and writes the page changes to the database file in cache. Since writing to disk is slower than interaction with the cache, writing to disk doesn’t occur immediately. The rollback journal continues to exist until changes are safely written to disk, with a second fsync(). From a user-space process point of view, the change to the disk (the COMMIT, or end of the transaction) happens instantaneously once the rollback journal is deleted - hence, atomic commits. However, the two fsync() operations required to complete the COMMIT make this option, from a transactional standpoint, slower than SQLite’s lesser known WAL mode.

Write-ahead logging (WAL)

While the rollback journal method uses a separate file to preserve the original database state, the WAL method uses a separate WAL file to instead record the changes. Instead of a COMMIT depending on writing changes to disk, a COMMIT in WAL mode occurs when a record of one or more commits is appended to the WAL. This has the advantage of not requiring blocking read or write operations to the database file in order to make a COMMIT, so more transactions can happen concurrently.

WAL mode introduces the concept of the checkpoint, which is when the WAL file is synced to persistent storage before all its transactions are transferred to the database file. You can optionally specify when this occurs, but SQLite provides reasonable defaults. The checkpoint is the WAL version of the atomic commit.

In WAL mode, write transactions are performed faster than in the traditional rollback journal mode. Each transaction involves writing the changes only once to the WAL file instead of twice - to the rollback journal, and then to disk - before the COMMIT signals that the transaction is over.

The simplicity of SQLite

For medium-sized read-heavy applications, SQLite may be a great choice. Using SQLite in WAL mode may make it an even better one. Benchmarks on the smallest EC2 instance, with no provisioned IOPS, put this little trooper at 400 write transactions per second, and thousands of reads. That’s some perfectly adequate capability, in a perfectly compact package.

Multithreaded Python: slithering through an I/O bottleneck, 28 Feb 2020 09:31:02 (Victoria Drake) taking advantage of parallelism in Python can make your software orders of magnitude faster. <img src="" width="640" height="256"/>I recently developed a project that I called Hydra: a multithreaded link checker written in Python. Unlike many Python site crawlers I found while researching, Hydra uses only standard libraries, with no external dependencies like BeautifulSoup. It’s intended to be run as part of a CI/CD process, so part of its success depended on being fast.

Multiple threads in Python is a bit of a bitey subject (not sorry) in that the Python interpreter doesn’t actually let multiple threads execute at the same time. Python’s Global Interpreter Lock, or GIL, prevents multiple threads from executing Python bytecodes at once. Each thread that wants to execute must first wait for the GIL to be released by the currently executing thread. The GIL is pretty much the microphone in a low-budget conference panel, except where no one gets to shout.

This has the advantage of preventing race conditions. It does, however, lack the performance advantages afforded by running multiple tasks in parallel. (If you’d like a refresher on concurrency, parallelism, and multithreading, see Concurrency, parallelism, and the many threads of Santa Claus.) While I prefer Go for its convenient first-class primitives that support concurrency (see Goroutines), this project’s recipients were more comfortable with Python. I took it as an opportunity to test and explore!

Simultaneously performing multiple tasks in Python isn’t impossible; it just takes a little extra work. For Hydra, the main advantage is in overcoming the input/output (I/O) bottleneck.

In order to get web pages to check, Hydra needs to go out to the Internet and fetch them. When compared to tasks that are performed by the CPU alone, going out over the network is comparatively slower. How slow?

Here are approximate timings for tasks performed on a typical PC:

Task Time
CPU execute typical instruction 1/1,000,000,000 sec = 1 nanosec
CPU fetch from L1 cache memory 0.5 nanosec
CPU branch misprediction 5 nanosec
CPU fetch from L2 cache memory 7 nanosec
RAM Mutex lock/unlock 25 nanosec
RAM fetch from main memory 100 nanosec
Network send 2K bytes over 1Gbps network 20,000 nanosec
RAM read 1MB sequentially from memory 250,000 nanosec
Disk fetch from new disk location (seek) 8,000,000 nanosec (8ms)
Disk read 1MB sequentially from disk 20,000,000 nanosec (20ms)
Network send packet US to Europe and back 150,000,000 nanosec (150ms)

Peter Norvig first published these numbers some years ago in Teach Yourself Programming in Ten Years. Since computers and their components change year over year, the exact numbers shown above aren’t the point. What these numbers help to illustrate is the difference, in orders of magnitude, between operations.

Compare the difference between fetching from main memory and sending a simple packet over the Internet. While both these operations occur in less than the blink of an eye (literally) from a human perspective, you can see that sending a simple packet over the Internet is over a million times slower than fetching from RAM. It’s a difference that, in a single-thread program, can quickly accumulate to form troublesome bottlenecks.

In Hydra, the task of parsing response data and assembling results into a report is relatively fast, since it all happens on the CPU. The slowest portion of the program’s execution, by over six orders of magnitude, is network latency. Not only does Hydra need to fetch packets, but whole web pages! One way of improving Hydra’s performance is to find a way for the page fetching tasks to execute without blocking the main thread.

Python has a couple options for doing tasks in parallel: multiple processes, or multiple threads. These methods allow you to circumvent the GIL and speed up execution in a couple different ways.

Multiple processes

To execute parallel tasks using multiple processes, you can use Python’s ProcessPoolExecutor. A concrete subclass of Executor from the concurrent.futures module, ProcessPoolExecutor uses a pool of processes spawned with the multiprocessing module to avoid the GIL.

This option uses worker subprocesses that maximally default to the number of processors on the machine. The multiprocessing module allows you to maximally parallelize function execution across processes, which can really speed up compute-bound (or CPU-bound) tasks.

Since the main bottleneck for Hydra is I/O and not the processing to be done by the CPU, I’m better served by using multiple threads.

Multiple threads

Fittingly named, Python’s ThreadPoolExecutor uses a pool of threads to execute asynchronous tasks. Also a subclass of Executor, it uses a defined number of maximum worker threads (at least five by default, according to the formula min(32, os.cpu_count() + 4)) and reuses idle threads before starting new ones, making it pretty efficient.

Here is a snippet of Hydra with comments showing how Hydra uses ThreadPoolExecutor to achieve parallel multithreaded bliss:

# Create the Checker class
class Checker:
    # Queue of links to be checked
    TO_PROCESS = Queue()
    # Maximum workers to run
    THREADS = 100
    # Maximum seconds to wait for HTTP response
    TIMEOUT = 60

    def __init__(self, url):
        # Create the thread pool
        self.pool = futures.ThreadPoolExecutor(max_workers=self.THREADS)

def run(self):
    # Run until the TO_PROCESS queue is empty
    while True:
            target_url = self.TO_PROCESS.get(block=True, timeout=2)
            # If we haven't already checked this link
            if target_url["url"] not in self.visited:
                # Mark it as visited
                # Submit the link to the pool
                job = self.pool.submit(self.load_url, target_url, self.TIMEOUT)
        except Empty:
        except Exception as e:

You can view the full code in Hydra’s GitHub repository.

Single thread to multithread

If you’d like to see the full effect, I compared the run times for checking my website between a prototype single-thread program, and the multiheadedmultithreaded Hydra.

time python3

real    17m34.084s
user    11m40.761s
sys     0m5.436s

time python3

real    0m15.729s
user    0m11.071s
sys     0m2.526s

The single-thread program, which blocks on I/O, ran in about seventeen minutes. When I first ran the multithreaded version, it finished in 1m13.358s - after some profiling and tuning, it took a little under sixteen seconds. Again, the exact times don’t mean all that much; they’ll vary depending on factors such as the size of the site being crawled, your network speed, and your program’s balance between the overhead of thread management and the benefits of parallelism.

The more important thing, and the result I’ll take any day, is a program that runs some orders of magnitude faster.

Breaking bottlenecks 🍾, 25 Feb 2020 12:50:29 (Victoria Drake) talk on the benefits of non-blocking functions for programs, developers, and organizations. <img src="" width="640" height="256"/>I recently gave a lecture on the benefits of building non-blocking processes. This is a write-up of the full talk, minus any “ums” that may have occurred. You can view the slides here.

I’ve been helping out a group called the Open Web Application Security Project (OWASP). They’re a non-profit foundation that produces some of the foremost application testing guides and cybersecurity resources. OWASP’s publications, checklists, and reference materials are a help to security professionals, penetration testers, and developers all over the world. Most of the individual teams that create these materials are run almost entirely by volunteers.

OWASP is a great group doing important work. I’ve seen this firsthand as part of the core team that produces the Web Security Testing Guide. However, while OWASP inspires in its large volunteer base, it lacks in the area of central organization.

This lack of organization was most recently apparent in the group’s website, A big organization with an even bigger website to match, enjoys hundreds of thousands of visitors. Unfortunately, many of its pages - individually managed by disparate projects - are infrequently updated. Some are abandoned. The website as a whole lacks a centralized quality assurance process, and as a result, is peppered with broken links.

Customers don’t like broken links; attackers really do. That’s because broken links are a security vulnerability. Broken links can signal opportunities for attacks like broken link hijacking and subdomain takeovers. At their least effective, these attacks can be embarrassing; at their worst, severely damaging to businesses and organizations. One OWASP group, the Application Security Verification Standard (ASVS) project, writes about integrity controls that can help to mitigate the likelihood of these attacks. This knowledge, unfortunately, has not yet propagated throughout the rest of OWASP yet.

This is the story of how I created a fast and efficient tool to help OWASP solve this problem.

The job

I took on the task of creating a program that could run as part of a CI/CD process to detect and report broken links. The program needed to:

  • Find and enumerate all the broken links on in a report.
  • Keep track of the parent pages the broken links were on so they could be fixed.
  • Run efficiently as part of a CI/CD pipeline.

Essentially; I need to build a web crawler.

My original journey through this process was also in Python, as that was a comfortable language choice for everyone in the OWASP group. Personally, I prefer to use Go for higher performance as it offers more convenient concurrency primitives. Between the task and this talk, I wrote three programs: a prototype single-thread Python program, a multithreaded Python program, and a Go program using goroutines. We’ll see a comparison of how each worked out near the end of the talk - first, let’s explore how to build a web crawler.

Prototyping a web crawler

Here’s what our web crawler will need to do:

  1. Get the HTML data of the first page of the website (for example,
  2. Check all of the links on the page
    1. Keep track of the links we’ve already visited so we don’t end up checking them twice
    2. Record any broken links we find
  3. Fetch more HTML data from any valid links on the page, as long as they’re in the same domain ( and not, for instance)
  4. Repeat step #2 until all of the links on the site have been checked

Here’s what the execution flow will look like:

A flow chart showing program execution

As you can see, the nodes “GET page” -> “HTML” -> “Parse links” -> “Valid link” -> “Check visited” all form a loop. These are what enable our web crawler to continue crawling until all the links on the site have been accounted for in the “Check visited” node. When the crawler encounters links it’s already checked, it will “Stop.” This loop will become more important in a moment.

For now, the question on everyone’s mind (I hope): how do we make it fast?

How fast can you do the thing

Here are some approximate timings for tasks performed on a typical PC:

Type Task Time
CPU execute typical instruction 1/1,000,000,000 sec = 1 nanosec
CPU fetch from L1 cache memory 0.5 nanosec
CPU branch misprediction 5 nanosec
CPU fetch from L2 cache memory 7 nanosec
RAM Mutex lock/unlock 25 nanosec
RAM fetch from main memory 100 nanosec
RAM read 1MB sequentially from memory 250,000 nanosec
Disk read 1MB sequentially from disk 20,000,000 nanosec (20ms)

Peter Norvig first published these numbers some years ago in Teach Yourself Programming in Ten Years. They typically crop up now and then in articles titled along the lines of, “Latency numbers every developer should know.”



Bottleneck: network latency

The numbers above mean that the difference in time it takes to send something over the Internet compared to fetching data from main memory is over six orders of magnitude. Remember the loop in our execution chart? The “GET page” node, in which our crawler fetches page data over the network, is going to be a million times slower than the next slowest thing in the loop!

We don’t need to run our prototype to see what that means in practical terms; we can estimate it. Let’s take, which has upwards of 12,000 links, as an example:

      150 milliseconds
 x 12,000 links
1,800,000 milliseconds (30 minutes)

A whole half hour, just for the network tasks. It may even be much slower than that, since web pages are frequently much larger than a packet. This means that in our single-thread prototype web crawler, our biggest bottleneck is network latency. Why is this problematic?

Feedback loops

I previously wrote about feedback loops. In essence, in order to improve at doing anything, you first need to be able to get feedback from your last attempt. That way, you have the necessary information to make adjustments and get closer to your goal on your next iteration.

As a software developer, bottlenecks can contribute to long and inefficient feedback loops. If I’m waiting on a process that’s part of a CI/CD pipeline, in our bottlenecked web crawler example, I’d be sitting around for a minimum of a half hour before learning whether or not changes in my last push were successful, or whether they broke master (hopefully staging).

Multiply a slow and inefficient feedback loop by many runs per day, over many days, and you’ve got a slow and inefficient developer. Multiply that by many developers in an organization bottlenecked on the same process, and you’ve got a slow and inefficient company.

The cost of bottlenecks

To add insult to injury, not only are you waiting on a bottlenecked process to run; you’re also paying to wait. Take the serverless example - AWS Lambda, for instance. Here’s a chart showing the cost of functions by compute time and CPU usage.

Chart showing Total Lambda compute cost by function execution

Source: Understanding and Controlling AWS Lambda Costs

Again, the numbers change over the years, but the main concepts remain the same: the bigger the function and the longer its compute time, the bigger the cost. For applications taking advantage of serverless, these costs can add up dramatically.

Bottlenecks are a recipe for failure, for both productivity and the bottom line.

The good news is that bottlenecks are mostly unnecessary. If we know how to identify them, we can strategize our way out of them. To understand how, let’s get some tacos.

Tacos and threading

Everyone, meet Bob. He’s a gopher who works at the taco stand down the street as the cashier. Say “Hi,” Bob.

🌮                                          🌳
🌮   ╔══════════════╗
🌮      Hi I'm Bob                          🌳
🌮   ╚══════════════╝ \
🌮                     🐹 🌮
🌮                                          🌳

Bob works very hard at being a cashier, but he’s still just one gopher. The customers who frequent Bob’s taco stand can eat tacos really quickly; but in order to get the tacos to eat them, they’ve got to order them through Bob. Here’s what our bottlenecked, single-thread taco stand currently looks like:

🌮                                          🌳
🌮                                          🌳
🌮      🐹 🧑💵🧑💵🧑💵🧑💵🧑💵🧑💵🧑💵🧑💵🧑💵
🌮                                          🌳

As you can see, all the customers are queued up, right out the door. Poor Bob handles one customer’s transaction at a time, starting and finishing with that customer completely before moving on to the next. Bob can only do so much, so our taco stand is rather inefficient at the moment. How can we make Bob faster?

We can try splitting the queue:

🌮                                          🌳
🌮         🧑💵🧑💵🧑💵🧑💵
🌮                                          🌳
🌮      🐹
🌮         🧑💵🧑💵🧑💵🧑💵🧑💵
🌮                                          🌳

Now Bob can do some multitasking. For example, he can start a transaction with a customer in one queue; then, while that customer counts their bills, Bob can pop over to the second queue and get started there. This arrangement, known as a concurrency model, helps Bob go a little bit faster by jumping back and forth between lines. However, it’s still just one Bob, which limits our improvement possibilities. If we were to make four queues, they’d all be shorter; but Bob would be very thinly stretched between them. Can we do better?

We could get two Bobs:

🌮                                          🌳
🌮                                          🌳
🌮      🐹 🧑💵🧑💵🧑💵🧑💵
🌮                                          🌳
🌮      🐹 🧑💵🧑💵🧑💵🧑💵🧑💵
🌮                                          🌳
🌮                                          🌳

With twice the Bobs, each can handle a queue of his own. This is our most efficient solution for our taco stand so far, since two Bobs can handle much more than one Bob can, even if each customer is still attended to one at a time.

We can do even better than that:

🌮                                          🌳
🌮      🐹 🧑💵🧑💵
🌮                                          🌳
🌮      🐹 🧑💵🧑💵
🌮                                          🌳
🌮      🐹 🧑💵🧑💵
🌮                                          🌳
🌮      🐹 🧑💵🧑💵🧑💵
🌮                                          🌳

With quadruple the Bobs, we have some very short queues, and a much more efficient taco stand. In computing, the concept of having multiple workers do tasks in parallel is called multithreading.

In Go, we can apply this concept using goroutines. Here are some illustrative snippets from my Go solution.

Setting up a Go web crawler

In order to share data between our goroutines, we’ll need to create some data structures. Our Checker structure will be shared, so it will have a Mutex (mutual exclusion) to allow our goroutines to lock and unlock it. The Checker structure will also hold a list of brokenLinks results, and visitedLinks. The latter will be a map of strings to booleans, which we’ll use to directly and efficiently check for visited links. By using a map instead of iterating over a list, our visitedLinks lookup will have a constant complexity of O(1) as opposed to a linear complexity of O(n), thus avoiding the creation of another bottleneck. For more on time complexity, see my coffee-break introduction to time complexity of algorithms article.

type Checker struct {
    startDomain             string
    brokenLinks             []Result
    visitedLinks            map[string]bool
    workerCount, maxWorkers int
// Page allows us to retain parent and sublinks
type Page struct {
    parent, loc, data string

// Result adds error information for the report
type Result struct {
    reason string
    code   int

To extract links from HTML data, here’s a parser I wrote on top of package html:

// Extract links from HTML
func parse(parent, data string) ([]string, []string) {
    doc, err := html.Parse(strings.NewReader(data))
    if err != nil {
        fmt.Println("Could not parse: ", err)
    goodLinks := make([]string, 0)
    badLinks := make([]string, 0)

    var f func(*html.Node)
    f = func(n *html.Node) {
        if n.Type == html.ElementNode && checkKey(string(n.Data)) {
            for _, a := range n.Attr {
                if checkAttr(string(a.Key)) {
                    j, err := formatURL(parent, a.Val)
                    if err != nil {
                        badLinks = append(badLinks, j)

                    } else {
                        goodLinks = append(goodLinks, j)
    return goodLinks, badLinks

If you’re wondering why I didn’t use a more full-featured package for this project, I highly recommend the story of left-pad. The short of it: more dependencies, more problems.

Here are snippets of the main function, where we pass in our starting URL and create a queue (or channels, in Go) to be filled with links for our goroutines to process.

func main() {
    startURL := flag.String("url", "", "full URL of site")

    firstPage := Page{
        parent: *startURL,
        loc:    *startURL,

    toProcess := make(chan Page, 1)
    toProcess <- firstPage

    var wg sync.WaitGroup

The last significant piece of the puzzle is to create our workers, which we’ll do here:

for i := range toProcess {
    🐹 go worker(i, &checker, &wg, toProcess)
    if checker.workerCount > checker.maxWorkers {
        time.Sleep(1 * time.Second) // throttle down

A WaitGroup does just what it says on the tin: it waits for our group of goroutines to finish. When they have, we’ll know our Go web crawler has finished checking all the links on the site.

Did we do the thing fast

Here’s a comparison of the three programs I wrote on this journey. First, the prototype single-thread Python version:

time python3

real 17m34.084s
user 11m40.761s
sys     0m5.436s

This finished crawling my website in about seventeen-and-a-half minutes, which is rather long for a site at least an order of magnitude smaller than

The multithreaded Python version did a bit better:

time python3

real 1m13.358s
user 0m13.161s
sys     0m2.826s

My multithreaded Python program (which I dubbed Hydra) finished in one minute and thirteen seconds.

How did Go do?

time ./go-link-check --url=

real 0m7.926s
user 0m9.044s
sys     0m0.932s

At just under eight seconds, I found the Go version to be extremely palatable.

Breaking bottlenecks

As fun as it is to simply enjoy the speedups, we can directly relate these results to everything we’ve learned so far. Consider taking a process that used to soak up seventeen minutes and turning it into an eight-second-affair instead. Not only will that give developers a much shorter and more efficient feedback loop, it will give companies the ability to develop faster, and thus grow more quickly - while costing less. To drive the point home: a process that runs in seventeen-and-a-half minutes when it could take eight seconds will also cost over a hundred and thirty times as much to run!

A better work day for developers, and a better bottom line for companies. There’s a lot of benefit to be had in making functions, code, and processes as efficient as possible - by breaking bottlenecks.

Command line tricks for managing your messy open source repository, 17 Feb 2020 08:05:06 (Victoria Drake) handy toolbox for the terminal to help open source maintainers make their projects sparkle. <img src="" width="640" height="320"/>Effective collaboration, especially in open source software development, starts with effective organization. To make sure that nothing gets missed, the general rule, “one issue, one pull request” is a nice rule of thumb.

Instead of opening an issue with a large scope like, “Fix all the broken links in the documentation,” open source projects will have more luck attracting contributors with several smaller and more manageable issues. In the preceding example, you might scope broken links by section or by page. This allows more contributors to jump in and dedicate small windows of their time, rather than waiting for one person to take on a larger and more tedious contribution effort.

Smaller scoped issues also help project maintainers see where work has been completed and where it hasn’t. This reduces the chances that some part of the issue is missed, assumed to be completed, and later leads to bugs or security vulnerabilities.

That’s all well and good; but what if you’ve already opened several massively-scoped issues, some PRs have already been submitted or merged, and you currently have no idea where the work started or stopped?

It’s going to take a little sorting out to get the state of your project back under control. Thankfully, there are a number of command line tools to help you scan, sort, and make sense of a messy repository. Here’s a small selection of ones I use.

Jump to:

Interactive search-and-replace with vim

You can open a file in Vim, then interactively search and replace with:


The % indicates to look in all lines of the current file; s is for substitute; \<word\> matches the whole word; and the g for “global” is for every occurrence. The c at the end will let you view and confirm each change before it’s made. You can run it automatically, and much faster, without c; however, you put yourself at risk of complicating things if you’ve made a pattern-matching error.

The markdown-link-check node module has a great CLI buddy.

I use this so often I turned it into a Bash alias function. To do the same, add this to your .bashrc:

# Markdown link check in a folder, recursive
function mlc () {
    find $1 -name \*.md -exec markdown-link-check -p {} \;

Then run with mlc <filename>.

List subdirectories with or without a git repository with find

Print all subdirectories that are git repositories, or in other words, have a .git in them:

find . -maxdepth 1 -type d -exec test -e '{}/.git' ';' -printf "is git repo: %p\n"

To print all subdirectories that are not git repositories, negate the test with !:

find . -maxdepth 1 -type d -exec test '!' -e '{}/.git' ';' -printf "not git repo: %p\n"

Pull multiple git repositories from a list with xargs

I initially used this as part of automatically re-creating my laptop with Bash scripts, but it’s pretty handy when you’re working with cloud instances or Dockerfiles.

Given a file, repos.txt with a repository’s SSH link on each line (and your SSH keys set up), run:

xargs -n1 git clone < repos.txt

If you want to pull and push many repositories, I previously wrote about how to use a Bash one-liner to manage your repositories.

List issues by number with jot

I’m a co-author and maintainer for the OWASP Web Security Testing Guide repository where I recently took one large issue (yup, it was “Fix all the broken links in the documentation” - how’d you guess?) and broke it up into several smaller, more manageable issues. A whole thirty-seven smaller, more manageable issues.

I wanted to enumerate all the issues that the original one became, but the idea of typing out thirty-seven issue numbers (#275 through #312) seemed awfully tedious and time-consuming. So, in natural programmer fashion, I spent the same amount of time I would have used to type out all those numbers and crafted a way to automate it instead.

The jot utility (apt install athena-jot) is a tiny tool that’s a big help when you want to print out some numbers. Just tell it how many you want, and where to start and stop.

# jot [ reps [ begin [ end ] ] ]
jot 37 275 312

This prints each number, inclusively, from 275 to 312 on a new line. To make these into issue number notations that GitHub and many other platforms automatically recognize and turn into links, you can pipe the output to awk.

jot 37 275 312 | awk '{printf "#"$0", "}'

#275, #276, #277, #278, #279, #280, #281, #282, #283, #284, #285, #286, #287, #288, #289, #290, #291, #292, #293, #295, #296, #297, #298, #299, #300, #301, #302, #303, #304, #305, #306, #307, #308, #309, #310, #311, #312

You can also use jot to generate random or redundant data, mainly for development or testing purposes.

CLI-powered open source organization

A well-organized open source repository is a well-maintained open source project. Save this post for handy reference, and use your newfound CLI superpowers for good! 🚀

Why PixelFed won't save us from Instagram, 16 Feb 2020 19:23:20 (Victoria Drake) decentralized photo sharing won't save us from Instagram - but it might help. <img src="" width="640" height="427"/>PixelFed is a decentralized photo sharing network based on the ActivityPub protocol, the same one that Mastodon uses. For a lot of people divorced (or wanting to be) from Instagram over mental health concerns and issues like forced consent to post-GDPR terms, a decentralized social network like PixelFed sounds like an exciting and promising alternative.

Personally, I stopped using Instagram once I accepted the fact that its core premise and integral structure of social interaction was encouraging me to form habits that were harmful to my life goals. I’m not alone - studies have shown that people are happier after deleting apps like Facebook. The reasons for this don’t differ greatly from why any social network can be bad for you - they’re just found in much greater intensity on photo sharing sites, and specifically Instagram.

It is still early days for PixelFed. As I write this I have no way to know what kind of network it will become, or even if it will survive at all. I do know, however, that there are many glaring and fundamental problems that a decentralized photo sharing network like PixelFed won’t solve. To elaborate, I’m going to discuss what makes Instagram so poisonous to health, why centralized social networks aren’t likely to ever be healthy, and why decentralized social networks have a very slim chance of being better.

Why Instagram is bad for your health

Let’s start with the basics. Your brain responds very differently to reading text than it does to looking at images.

It doesn’t take more than a quick search to find hundreds of articles and studies about how reading can make you smarter, more empathetic, and stave off cognitive decline by improving brain connectivity. In essence, reading involves a multitude of brain regions including the temporal and frontal lobes. There’s still a lot to be discovered about the human brain, but here’s what we think we know. The frontal lobes control important cognitive skills like emotional expression, problem solving, memory, language, judgement, and sexual behaviors. The temporal lobes handle important functions such as the encoding of memory, and processing emotions and visual perception.

In other words, reading text - even on social media - stimulates your brain and makes you think about the information you’re taking in. To react to words on a page, you first have to read them and form thoughts about them.

Unlike reading, looking at an image has a very different effect on your brain. Here’s an infographic about infographics that covers some of these effects. Basically, millennia of evolution have produced human brains hardwired to respond quickly to visual stimuli - in less than 1/10 of a second. As the infographic will literally show you, almost 50% of the brain is involved in visual processing, and 70% of all our sensory receptors are in our eyes. That’s a lot of resources devoted to quickly processing visuals. Why could this be bad?

Unlike times past, we’re no longer (day-to-day) concerned with spotting a tiger in the bushes about to pounce on us. The near-instant processing time needed to discern if that shivering tree branch is the just wind or impending mortal danger is outdated in our current living arrangements. Our brain, however, doesn’t know that. It hasn’t evolved faster than our technologies or society. The downside to this is that anyone with a little knowledge of this fundamental flaw in the human mind is able to exploit it.

Advertisers call this exploitation, “visual marketing.”

These linked articles are stuffed with the same factoids over and over again. “The brain processes images 60,000 times faster than text.” “90% of the information sent to the brain is visual.” Whether or not these numbers are accurate, it’s clearly provable that visual marketing is on the whole more effective than advertisements without images. There’s a reason for it, and it should scare you.

Unlike reading, which involves regions of your brain responsible for comprehension, decision making, and emotional control, images are processed by different areas of the brain. Visual input travels from our eyes through our optic nerves to the thalamus (or LGN, Lateral Geniculate Nucleus) and the superior colliculus. From the thalamus, it proceeds to the visual cortex at the rear of our brains, where the image is processed. Effectively, viewing images does not make us think in the same way that reading does. In other words, it’s easy to do.

Let me be clear. This difference in the way words and images are processed is not, in itself, bad. A photo-centric social network is not, in itself, bad. Images and words alike have the power to evoke strong emotions, send powerful messages, spark revolutions, and spur progress. This is good… if it’s used for good.

Instagram, a photo-centric network chock full of product placements, paid sponsorships, and outright advertisements, is a social network primarily designed to bypass your cognitive thinking and sell you stuff.

I don’t think Instagram started out with the same motivations it has now. Along with all the photo sharing networks that blossomed when Instagram first got popular, I still believe its initial vision was to make sharing photos with your friends fun and easy.

It just got too popular.

Why centralized networks are bad for your health

In the wake of privacy concerns over the last few years, new uproar over algorithm-driven timelines, and the #DeleteFacebook, #DeleteTwitter, and #DeleteInstagram movements, more people today are aware of how networks that make their money on your data are bad for your health. This is in part due to their centralized nature - one hierarchy of authority makes decisions for the whole system, and at the same time, has to support it. It’s expensive to support millions of users, so it’s no wonder that the network’s main concern (and let’s just consider the most innocent case) is to remain profitable.

What’s a good way to remain profitable?

Take a human desire, preferably one that has been around for a really long time… Identify that desire and use modern technology to take out steps. – Evan Williams, co-founder of Twitter and Blogger

Quoted in Wired article, 2013, “Twitter Founder Reveals Secret Formula for Getting Rich Online"

There’s a book called Hooked: How to Build Habit-Forming Products which, if you’re ever in the mood for a good horror flick, you should curl up in bed with some popcorn and read.

The book details a simple model for a habit-forming product. The model is cyclical, and has the following key points: a trigger, an action, variable reward, and investment. In summary, if a product can get you to think of it, leading to some action that is easier to do than to think about, give you a reward for that action some of the time, and then compel you to commit or invest in it - you’re hooked.

If you’re paying attention, you might notice I’ve described Instagram. And Twitter. And Facebook. And every other social network.

There’s a reason it’s easy to use Instagram, easy to post a tweet, easy to browse Facebook. These products have been designed to make it easy for you to use them. They’ve been designed to alter your behavior to better suit the product’s goals.

This industry employs some of the smartest people, thousands of Ph.D. designers, statisticians, engineers. They go to work every day to get us to do this one thing, to undermine our willpower. – James Williams, co-founder of Time Well Spent

Quoted in Nautilus article, 2017, “Modern Media Is a DoS Attack on Your Free Will"

At the heart of the idea of getting you hooked is the concept of a dopamine feedback loop. Dopamine, an organic chemical neurotransmitter in your brain, is thought to be responsible for allowing us to anticipate the reward to an action. It inspires us to get a glass of water when we’re thirsty, for example, and may help us to feel good when we take actions towards doing so. Where dopamine is so effectively misused is in the practice of providing variable rewards to drive social media addiction.

Unlike getting a glass of water when you’re thirsty, variable rewards are random. It’s as if drinking water sometimes, but not always, cured your thirst. This effectively programs your mind to pursue the action that results in the unpredictable reward. Since getting the reward isn’t guaranteed, you need to make more attempts to achieve success. Social media is designed to make these variable dopamine hits easy to obtain. It’s designed to hijack your intellectual independence in order to keep you on the network.

Especially when the main goal of a centralized social network is to make a profit, that network is exploiting evolutionary flaws in your brain to make that profit from you. You are literally being hacked.

Now combine this information with the knowledge of how a product comprised primarily of images bypasses your cognitive thinking. Not only are you being hacked, but your main defense system is being easily, laughably, circumvented.

Exploiting users is a particularly compelling temptation for any social networks under pressure to make a profit, and this pressure is amplified in organizations with a centralized structure. Not all centralized networks do this, but undoubtedly, the very successful ones do.

Decentralization is by no means a fix for exploitation and greed, but a decentralized social network might have a few things going for it.

Why decentralized networks might be slightly better for your health

The main issues present in Twitter, Facebook, and Instagram as pertains to social media addiction do not go away on decentralized networks. I’m personally, currently, using both Twitter and Mastodon. The former is centralized and the latter is decentralized, but the the same motivations that could get me in trouble on one platform apply to both. Decentralization does not fix the problem.

It might help.

Unlike a centralized, single-hierarchy, definitely-for-profit social network, decentralization has one thing going for it: more people. Specifically, more instance owners who are in control of their instances.

Running a Mastodon instance is a responsibility, should you choose to accept it. Besides the server itself, instances require their own sets of rules and code of conduct, and like the often adopted code of conduct, it can be collaboratively drafted by the community. Mastodon provides instance owners with moderation tools and provides users with reporting tools, and there’s an expectation that they’ll both be used. As with other decentralized social networks, it is the responsibility of the instance owner to moderate and foster a social environment that serves the best interests of the instance users.

Instances typically run on donations, and in the grand scheme of things, are inexpensive to support. Decentralization means that instance owners individually have to bear smaller costs. There’s no central body being pressured to make a profit in order to run servers that support millions of users. The effect of this many-owners structure is that decisions that concern any particular instance and rules that it might want to adopt are made by that instance’s community, or the instance owner. If a user disagrees with the direction taken, they can communicate directly with the instance owner, or simply move to another instance. There’s no “take it or leave it,” and no forced acceptance of terms. Users always have somewhere else to go.

This, in general, means that over many instances, and via many moderators, more people from diverse backgrounds with a collection of both overlapping and contrasting interests are able to have a voice in how the social network evolves.

If instance owners have their users' best interests, not addiction, in mind; if moderators act responsibly, and according to their instance rules, moderate for good; and if a wide and varying selection of instances with differing interests, political viewpoints, and topics continue to be available; then decentralized social networks might be better for your health.

Where this leaves PixelFed (and all social networks)

All social networks have the potential to do more good than harm, but it is up to those who control them to put in the constant, proactive effort required to make that happen. Twitter has recently been making some steps towards becoming a healthier network, like banning political ads and highlighting manipulated media. I think they’re ahead of the curve. With decentralized social networks, there’s at least more chances for the possibility that instance owners truly want to do more good than harm with their own little piece of the whole.

While photo sharing networks will, by their essential nature, bypass cognitive thinking and have an advantage over their users that way, there are many design considerations that PixelFed can implement in order to make the network healthier. Features such as comments, likes, timelines, and push notifications can be designed to provide utility more than drive addiction, and there are designers more qualified than I who can tell you how.

These networks will have to constantly resist the temptation to take the easy route. They will have to work to avoid success based on the exploitation of their users' desires to chase the easy dopamine hit. They will have to prioritize the ability of the social network to add real value to the lives of its users - at the expense of its own potential to garner mindless, meaningless popularity.

This is in no way a condemnation of PixelFed or any other decentralized photo sharing network. Personally, I sincerely hope they succeed in giving users a healthy, safe, and free-as-in-freedom network for sharing photos with friends, and with the rest of the federated community. It will require considered design with mental health at the forefront; the active, caring effort of moderators and instance owners; and ongoing collaboration from the federated community at large to work together to build for the greater good.

A photo-sharing social media network that does more good than harm? It’s possible. But it won’t be easy.

The past ten years, or, how to get better at anything, 31 Dec 2019 08:27:31 (Victoria Drake) on learning software development, technical blogging, and what the past ten years have taught me. <img src="" width="640" height="256"/>If you want to get better at anything:

  1. Solve your own problems,
  2. Write about it,
  3. Teach others.

1. Searching, a decade ago

I was a young graduate with newly-minted freedoms, and I was about to fall in love. I had plenty of imagination, a couple handfuls of tenacity, and no sense of direction at all.

For much of my youth, when I encountered a problem, I just sort of bumped up against it. I tried using whatever was in my head from past experiences or my own imagination to find a solution. For some problems, like managing staff duties at work, my experience was sufficient guidance. For other, more complicated problems, it wasn’t.

When you don’t have a wealth of experience to draw upon, relying on it is a poor strategy. Like many people at my age then, I thought I knew enough. Like many people at my age now, I recognize how insufficient “enough” can be. A lack of self-directed momentum meant being dragged in any direction life’s currents took me. When falling in love turned out to mean falling from a far greater height than I had anticipated, I tumbled on, complacent. When higher-ups at work handed me further responsibilities, I accepted them without considering if I wanted them at all. When, inevitably, life became more and more complicated, I encountered even more problems I didn’t know how to solve. I felt stuck.

Though I was morbidly embarrassed about it at the time, I’m not shy to say it now. At one point, it had to be pointed out to me that I could search the Internet for the solution to any of my problems. Anything I wanted to solve - interactions with people at work, a floundering relationship, or the practicalities of filing taxes - I was lucky enough to have the greatest collection of human knowledge ever assembled at my disposal.

Instead of bumbling along in the floatsam of my own trial and error, I started to take advantage of the collective experiences of all those who have been here before me. They weren’t always right, and I often found information only somewhat similar to my own experience. Still, it always got me moving in the right direction. Eventually, I started to steer.

There’s a learning curve, even when just searching for a problem. Distilling the jumble of confusion in your head to the right search terms is a learned skill. It helped me to understand how search engines like Google work:


When crawlers find a webpage, our systems render the content of the page, just as a browser does. We take note of key signals — from keywords to website freshness — and we keep track of it all in the Search index.

Sometimes, I find what I need by using the right keyword. Other times, I discover the keyword by searching for text that might surround it on the content of the page. For software development, I search for the weirdest word or combination of words attached to what I’m trying to learn. I rarely find whole solutions in my search results, but I always find direction for solving the problem myself.

Solving my own problems, even just a few little ones at a time, gave me confidence and built momentum. I began to pursue the experiences I wanted, instead of waiting for experiences to happen to me.

2. Updating the Internet, some years ago

I’d solved myself out of a doomed relationship and stagnant job. I found myself, rather gleefully, country-hopping with just one backpack of possessions. I met, though I didn’t know it at the time, my future husband. I found a new sense of freedom, of having options, that I knew I never wanted to give up. I had to find a means to sustain myself by working remotely.

When I first tried to make a living on the Internet, I felt like a right amateur. Sitting on the bed, hunched over my laptop, I started a crappy Wordpress blog with a modified theme that didn’t entirely work. I posted about how I tried and failed to start a dropshipping business. My site was terrible, and I knew it. My first forays into being a “real” developer were to solve my own problems: how to get my blog working, how to set up a custom domain, how to get and use a security certificate. I found some guidance in blogs and answers that others had written, but much of it was outdated, or not entirely correct. Still, it helped me.

I can’t imagine a world in which people did nothing to pass on their knowledge to future generations. Our stories are all we have beyond instinct and determination.

I stopped posting about dropshipping and started writing about the technical problems I was solving. I wrote about what I tried, and ultimately what worked. I started hearing from people who thanked me for explaining the solution they were looking for. Even in posts where all I’d done was link to the correct set of instructions on some other website, people thanked me for leading them to it. I still thought my website was terrible, but I realized I was doing something useful. The more problems I solved, the better I got at solving them, and the more I wrote about it in turn.

One day, someone offered me money for one of my solutions. To my great delight, they weren’t the last to do so.

As I built up my skills, I started taking on more challenging offers to solve problems. I discovered, as others have before me, that especially in software development, not every solution is out there waiting for you. The most frustrating part of working on an unsolved problem is that, at least to your knowledge, there’s no one about to tell you how to solve it. If you’re lucky, you’ve at least got a heading from someone’s cold trail in an old blog post. If you’re lucky and tenacious, you’ll find a working solution.

Don’t leave it scribbled in the corner of a soon-forgotten notepad, never to ease the path of someone who comes along later. Update that old blog post by commenting on it, or sending a note to the author. Put your solution on the Internet, somewhere. Ideally, blog about it yourself in as much detail as you can recall. Some of the people who find your post might have the same problem, and might even be willing to pay you to solve it. And, if my own experience and some scattered stories hold true, one of the people to who’ll come along later, looking for that same solution, will be you.

3. Paying it forwards, backwards, and investing; two years ago

Already being familiar with how easy it is to stop steering and start drifting, I sought new ways to challenge myself and my skills. I wanted to do more than just sustain my lifestyle. I wanted to offer something to others; something that mattered.

A strange thing started happening when I decided, deliberately, to write an in-depth technical blog about topics I was only beginning to become familiar with. I started to deeply understand some fundamental computer science topics - and trust me, that was strange enough - but odder than that was that others started to see me as a resource. People asked me questions because they thought I had the answers. I didn’t, at least, not always - but I knew enough now to not let that stop me. I went to find the answers, to test and understand them, and then I wrote about them to teach those who had asked. I hardly noticed, along the way, that I was learning too.

When someone’s outdated blog post leads you to an eventual solution, you can pay them back by posting an update, or blogging about it yourself. When you solve an unsolved problem, you pay it forward by recording that solution for the next person who comes along (sometimes you). In either case, by writing about it - honestly, and with your best effort to be thorough and correct - you end up investing in yourself.

Explaining topics you’re interested in to other people helps you find the missing pieces in your own knowledge. It helps you fill those gaps with learning, and integrate the things you learn into a new, greater understanding. Teaching something to others helps you become better at it yourself. Getting better at something - anything - means you have more to offer.

The past decade, and the next decade

It’s the end of a decade. I went from an aimless drift through life to being captain of my ship. I bettered my environment, learned new skills, made myself a resource, and became a wife to my best friend. I’m pretty happy with all of it.

It’s the end of 2019. Despite a whole lot of life happening just this year, I’ve written one article on this blog for each week since I started in July. That’s 23 articles for 23 weeks, plus one Christmas bonus. I hear from people almost every day who tell me that an article I wrote was helpful to them, and it makes me happy and proud to think that I’ve been doing something that matters. The first week of January will make this blog two years old.

The past several months have seen me change tack, slightly. I’ve become very interested in cybersecurity, and have been lending my skills to the Open Web Application Security Project. I’m now an author and maintainer of the Web Security Testing Guide, version 5. I’m pretty happy with that, too.

Next year, I’ll be posting a little less, though writing even more, as I pursue an old dream of publishing a book, as well as develop my new cybersecurity interests. I aim to get better at quite a few things. Thankfully, I know just how to do it - and now, so do you:

  1. Solve your own problems,
  2. Write about it,
  3. Teach others.

Have a very happy new decade, dear reader.

Three healthy cybersecurity habits, 26 Dec 2019 08:27:31 (Victoria Drake) helpful cybersecurity new year resolutions. Happy holidays! <img src="" width="640" height="256"/>In a similar fashion to everyone getting the flu now and again, the risk of catching a cyberattack is a common one. Both a sophisticated social engineering attack or grammatically-lacking email phishing scam can cause real damage. No one who communicates over the Internet is immune.

Like proper hand washing and getting a flu shot, good habits can lower your risk of inadvertently allowing cybergerms to spread. Since the new year is an inspiring time for beginning new habits, I offer a few suggestions for ways to help protect yourself and those around you.

1. Get a follow-up

Recognizing a delivery method for cyberattack is getting more difficult. Messages with malicious links do not always come from strangers. They may appear to be routine communications, or seem to originate from someone you know or work with. Attacks use subtle but deeply-engrained cognitive biases to override your common sense. Your natural response ensures you click.

Thankfully, there’s a simple low-tech habit you can use to deter these attacks: before you act, follow-up.

You may get an email from a friend that needs help, or from your boss who’s about to get on a plane. It could be as enticing and mysterious as a direct message from an acquaintance who sends a link asking, “Lol. Is this you?” It takes presence of mind to override the panic these attacks prey on, but the deterrent itself is quick and straightforward. Send a text message, pick up the phone and call, or walk down the hall, and ask, “Did you send me this?”

If the message is genuine, there’s no harm in a few extra minutes to double check. If it’s not, you’ll immediately alert the originating party that they may be compromised, and you may have deterred a cyberattack!

2. Use, and encourage others to use, end-to-end encrypted messaging

When individuals in a neighborhood get the flu shot, others in that neighborhood are safer for it. Encryption is similarly beneficial. Encourage your friends, coworkers, and Aunt Matilda to switch to an app like Signal. By doing so, you’ll reduce everyone’s exposure to more exploitable messaging systems.

This doesn’t mean that you must stop using other methods of communication entirely. Instead, think of it as a hierarchy. Use Signal for important messages that should be trusted, like requests for money or making travel arrangements. Use all other methods of messaging, like SMS or social sites, only for “unimportant” communications. Now, if requests or links that seem important come to you through your unimportant methods, you’ll be all the more likely to second-guess them.

3. Don’t put that dirty USB plug into your ***

You wouldn’t brush your teeth with a toothbrush you found on the sidewalk. Why would you plug in a USB device if you don’t know where it’s been?! While we might ascribe putting a random found USB drive in your computer to a clever exploitation of natural human curiosity, we’re no sooner likely to suspect using a public phone-charging station or a USB cable we bought ourselves. Even seemingly-innocuous USB peripherals or rechargeable devices can be a risk.

Unlike email and some file-sharing services that scan and filter files before they reach your computer, plugging in via USB is as direct and unprotected as connection gets. Once this connection is made, the user doesn’t need to do anything else for a whole host of bad things to happen. Through USB connections, problems like malware and ransomware can easily infect your computer or phone.

There’s no need to swear off the convenience of USB connectivity, or to avoid these devices altogether. Instead of engaging in questionable USB behavior, don’t cheap out on USB devices and cables. If it’s going to get plugged into your computer, ensure you’re being extra cautious. Buy it from the manufacturer (like the Apple Store) or from a reputable company or reseller with supply chain control. When juicing up USB-rechargeables, don’t plug them into your computer. Use a wall charger with a USB port instead.

Practice healthy cybersecurity habits

Keeping your devices healthy and happy is a matter of practicing good habits. Like battling the flu, good habits can help protect yourself and those around you. Incorporate some conscientious cybersecurity practices in your new year resolutions - or start them right away.

Have a safe and happy holiday!

Concurrency, parallelism, and the many threads of Santa Claus 🎅, 23 Dec 2019 19:29:01 (Victoria Drake) seasonal exploration of the difference between concurrent, parallel, and multithreaded processes. <img src="" width="640" height="256"/>Consider the following: Santa brings toys to all the good girls and boys.

There are 7,713,468,100 people in the world in 2019, around 26.3% of which are under 15 years old. This works out to 2,028,642,110 children (persons under 15 years of age) in the world this year.

Santa doesn’t seem to visit children of every religion, so we’ll generalize and only include Christians and non-religious folks. Collectively that makes up approximately 44.72% of the population. If we assume that all kids take after their parents, then 907,208,751.6 children would appear to be Santa-eligible.

What percentage of those children are good? It’s impossible to know; however, we can work on a few assumptions. One is that Santa Claus functions more on optimism than economics and would likely have prepared for the possibility that every child is a good child in any given year. Thus, he would be prepared to give a toy to every child. Let’s assume it’s been a great year and that all 907,208,751.6 children are getting toys.

That’s a lot of presents, and, as we know, they’re all made by Santa’s elves at his North China Pole workshop. Given that there are 365 days in a year and one of them is Christmas, let’s assume that Santa’s elves collectively have 364 days to create and gift wrap 907,208,752 (rounded up) presents. That works out to 2,492,331.74 presents per day.

Almost two-and-a-half million presents per day is a heavy workload for any workshop. Let’s look at two paradigms that Santa might employ to hit this goal: concurrency, and parallelism.

A sequential process

Suppose that Santa’s workshop is staffed by exactly one, very hard working, very tired elf. The production of one present involves four steps:

  1. Cutting wood
  2. Assembly and glueing
  3. Painting
  4. Gift-wrapping

With a single elf, only one step for one present can be happening at any instance in time. If the elf were to produce one present at a time from beginning to end, that process would be executed sequentially. It’s not the most efficient method for producing two-and-a-half million presents per day; for instance, the elf would have to wait around doing nothing while the glue on the present was drying before moving on to the next step.

Illustration of sequence


In order to be more efficient, the elf works on all presents concurrently.

Instead of completing one present at a time, the elf first cuts all the wood for all the toys, one by one. When everything is cut, the elf assembles and glues the toys together, one after the other. This concurrent processing means that the glue from the first toy has time to dry (without needing more attention from the elf) while the remaining toys are glued together. The same goes for painting, one toy at a time, and finally wrapping.

Illustration of concurrency

Since one elf can only do one task at a time, a single elf is using the day as efficiently as possible by concurrently producing presents.


Hopefully, Santa’s workshop has more than just one elf. With more elves, more toys can be built simultaneously over the course of a day. This simultaneous work means that the presents are being produced in parallel. Parallel processing carried out by multiple elves means more work happens at the same time.

Illustration of parallel processes

Elves working in parallel can also employ concurrency. One elf can still tackle only one task at a time, so it’s most efficient to have multiple elves concurrently producing presents.

Of course, if Santa’s workshop has, say, two-and-a-half million elves, each elf would only need to finish a maximum of one present per day. In this case, working sequentially doesn’t detract from the workshop’s efficiency. There would still be 7,668.26 elves left over to fetch coffee and lunch.

Santa Claus, and threading

After all the elves' hard work is done, it’s up to Santa Claus to deliver the presents – all 907,208,752 of them.

Santa doesn’t need to make a visit to every kid; just to the one household tree. So how many trees does Santa need to visit? Again with broad generalization, we’ll say that the average number of children per household worldwide is 2.45, based on the year’s predicted fertility rates. That makes 370,289,286.4 houses to visit. Let’s round that up to 370,289,287.

How long does Santa have? The lore says one night, which means one earthly rotation, and thus 24 hours. NORAD confirms.

This means Santa must visit 370,289,287 households in 24 hours (86,400 seconds), at a rate of 4,285.75 households per second, nevermind the time it takes to put presents under the tree and grab a cookie.

Clearly, Santa doesn’t exist in our dimension. This is especially likely given that despite being chubby and plump, he fits down a chimney (with a lit fire, while remaining unhurt) carrying a sack of toys containing presents for all the household’s children. We haven’t even considered the fact that his sleigh carries enough toys for every believing boy and girl around the world, and flies.

Does Santa exist outside our rules of physics? How could one entity manage to travel around the world, delivering packages, in under 24 hours at a rate of 4,285.75 households per second, and still have time for milk and cookies and kissing mommy?

One thing is certain: Santa uses the Internet. No other technology has yet enabled packages to travel quite so far and quite so quickly. Even so, attempting to reach upwards of four thousand households per second is no small task, even with even the best gigabit Internet hookup the North Pole has to offer. How might Santa increase his efficiency?

There’s clearly only one logical conclusion to this mystery: Santa Claus is a multithreaded process.

A single thread

Let’s work outward. Think of a thread as one particular task, or the most granular sequence of instructions that Santa might execute. One thread might execute the task, put present under tree. A thread is a component of a process, in this case, Santa’s process of delivering presents.

If Santa Claus is single-threaded, he, as a process, would only be able to accomplish one task at a time. Since he’s old and a bit forgetful, he probably has a set of instructions for delivering presents, as well as a schedule to abide by. These two things guide Santa’s thread until his process is complete.

A single Santa Claus emoji

Single-threaded Santa Claus might work something like this:

  1. Land sleigh at Timmy’s house
  2. Get Timmy’s present from sleigh
  3. Enter house via chimney
  4. Locate Christmas tree
  5. Place Timmy’s present under Christmas tree
  6. Exit house via chimney
  7. Take off in sleigh

Rinse and repeat… another 370,289,286 times.


Multithreaded Santa Claus, by contrast, is the Doctor Manhattan of the North Pole. There’s still only one Santa Claus in the world; however, he has the amazing ability to multiply his consciousness and accomplish multiple instruction sets of tasks simultaneously. These additional task workers, or worker threads, are created and controlled by the main process of Santa delivering presents.

Multiple Santa threads

Each worker thread acts independently to complete its instructions. Since they all belong to Santa’s consciousness, they share Santa’s memory and know everything that Santa knows, including what planet they’re running around on, and where to get the presents from.

With this shared knowledge, each thread is able to execute its set of instructions in parallel with the other threads. This multithreaded parallelism makes the one and only Santa Claus as efficient as possible.

If an average present delivery run takes an hour, Santa need only spawn 4,286 worker threads. With each making one delivery trip per hour, Santa will have completed all 370,289,287 trips by the end of the night.

Of course, in theory, Santa could even spawn 370,289,287 worker threads, each flying to one household to deliver presents for all the children in it! That would make Santa’s process extremely efficient, and also explain how he manages to consume all those milk-dunked cookies without getting full. 🥛🍪🍪🍪

An efficient and merry multithreaded Christmas

Thanks to modern computing, we now finally understand how Santa Claus manages the seemingly-impossible task of delivering toys to good girls and boys the world-over. From my family to yours, I hope you have a wonderful Christmas. Don’t forget to hang up your stockings on the router shelf.

Of course, none of this explains how reindeer manage to fly.

Word bugs in software documentation and how to fix them, 18 Dec 2019 09:01:23 (Victoria Drake) collection of mistakes that make documentation less awesome than it could be. <img src="" width="640" height="256"/>I’ve been an editor longer than I’ve been a developer, so this topic for me is a real root issue. 🥁 When I see a great project with poorly-written docs, it hits close to /home. Okay, okay, I’m done.

I help the Open Web Application Security Project (OWASP) with their Web Security Testing Guide (WSTG). I was recently tasked with writing a style guide and article template that show how to write technical instruction for testing software applications.

I thought parts of the guide would benefit more people than just OWASP’s contributors, so I’m sharing some here.

Many of the projects I participate in are open source. This is a wonderful way for people to share solutions and to build on each others' ideas. Unfortunately, it’s also a great way for misused and non-existent words to catch on. Here’s an excerpt of the guide with some mistakes I’ve noticed and how you can fix them in your technical documents.

Use Correct Words

The following are frequently misused words and how to correct them.


While sometimes used in legal documents, and/or leads to ambiguity and confusion in technical writing. Instead, use or, which in the English language includes and. For example:

Bad: “The code will output an error number and/or description.” Good: “The code will output an error number or description.”

The latter sentence does not exclude the possibility of having both an error number and description.

If you need to specify all possible outcomes, use a list:

“The code will output an error number, or a description, or both.”

frontend, backend

While it’s true that the English language evolves over time, these are not yet words.

When referring to nouns, use front end and back end. For example:

Security is equally important on the front end as it is on the back end.

As a descriptive adverb, use the hyphenated front-end and back-end.

Both front-end developers and back-end developers are responsible for application security.

whitebox, blackbox, greybox

These are not words.

As nouns, use white box, black box, and grey box. These nouns rarely appear in connection with cybersecurity.

My cat enjoys jumping into that grey box.

As adverbs, use the hyphenated white-box, black-box, and grey-box. Do not use capitalization unless the words are in a title.

While white-box testing involves knowledge of source code, black-box testing does not. A grey-box test is somewhere in-between.

ie, eg

These are letters.

The abbreviation i.e. refers to the Latin id est, which means “in other words.” The abbreviation e.g. is for exempli gratia, translating to “for example.” To use these in a sentence:

Write using proper English, i.e. correct spelling and grammar. Use common words over uncommon ones, e.g. “learn” instead of “glean.”


These are also letters.

The Latin phrase et cetera translates to “and the rest.” It is abbreviated etc. and typically placed at the end of a list that seems redundant to complete:

WSTG authors like rainbow colors, such as red, yellow, green, etc.

In technical writing, the use of etc. is problematic. It assumes the reader knows what you’re talking about, and they may not. Violet is one of the colors of the rainbow, but the example above does not explicitly tell you if violet is a color that WSTG authors like.

It is better to be explicit and thorough than to make assumptions of the reader. Only use etc. to avoid completing a list that was given in full earlier in the document.


The ellipsis punctuation mark can indicate that words have been left out of a quote:

Linus Torvalds once said, “Once you realize that documentation should be laughed at… THEN, and only then, have you reached the level where you can safely read it and try to use it to actually implement a driver.”

As long as the omission does not change the meaning of the quote, this is acceptable usage of ellipsis in the WSTG.

All other uses of ellipsis, such as to indicate an unfinished thought, are not.


While this is a word, it is likely not the word you are looking for. The word ex has particular meaning in the fields of finance and commerce, and may refer to a person if you are discussing your past relationships. None of these topics should appear in the WSTG.

The abbreviation ex. may be used to mean “example” by lazy writers. Please don’t be lazy, and write example instead.

Go forth and write docs

If these reminders are helpful, please share them freely and use them when writing your own READMEs and documentation! If there’s some I’ve missed, I’d love to know.

And if you’re here for the comments…

Change my mind meme

There are none on my blog. You can still @ me.

If you’d like to help contribute to the OWASP WSTG, please read the contribution guide. See the full style guide here.

Secure web forms for the front-end developer, 11 Dec 2019 08:27:31 (Victoria Drake) to design secure web forms: validate, sanitize, and control. <img src="" width="640" height="256"/>While cybersecurity is often thought of in terms of databases and architecture, much of a strong security posture relies on elements in the domain of the front-end developer. For certain potentially devastating vulnerabilities like SQL injection and Cross-Site Scripting (XSS), a well-considered user interface is the first line of defense.

Here are a few areas of focus for front-end developers who want to help fight the good fight.

Control user input

A whole whack of crazy things can happen when developers build a form that fails to control user input. To combat vulnerabilities like injection, it’s important to validate or sanitize user input.

Input can be validated by constraining it to known values, such as by using semantic input types or validation-related attributes in forms. Frameworks like Django also help by providing field types for this purpose. Sanitizing data can be done by removing or replacing contextually-dangerous characters, such as by using a whitelist or escaping the input data.

While it may not be intuitive, even data that a user submits to their own area on a site should be validated. One of the fastest viruses to proliferate was the Samy worm on MySpace (yes, I’m old), thanks to code that Samy Kamkar was able to inject into his own profile page. Don’t directly return any input to your site without thorough validation or santization.

For some further guidance on battling injection attacks, see the OWASP Injection Prevention Cheat Sheet.

Beware hidden fields

Adding type="hidden" is an enticingly convenient way to hide sensitive data in pages and forms, but unfortunately not an effective one. With tools like ZapProxy and even inspection tools in plain ol' web browsers, users can easily click to reveal tasty bits of invisible information. Hiding checkboxes can be a neat hack for creating CSS-only switches, but hidden fields do little to contribute to security.

Carefully consider autofill fields

When a user chooses to give you their Personally Identifiable Information (PII), it should be a conscious choice. Autofill form fields can be convenient - for both users and attackers. Exploits using hidden fields can harvest PII previously captured by an autocomplete field.

Many users aren’t even aware what information their browser’s autofill has stored up. Use these fields sparingly, and disable autofilled forms for particularly sensitive data.

It’s important to also weigh your risk profile against its trade-offs. If your project must be WCAG compliant, disabling autocomplete can break your input for different modalities. For more, see 1.3.5: Identify Input Purpose in WCAG 2.1.

Keep errors generic

While it may seem helpful to let users know whether a piece of data exists, it’s also very helpful to attackers. When dealing with accounts, emails, and PII, it’s most secure to err (🥁) on the side of less. Instead of returning “Your password for this account is incorrect,” try the more ambiguous feedback “Incorrect login information,” and avoid revealing whether the username or email is in the system.

In order to be more helpful, provide a prominent way to contact a human in case an error should arise. Avoid revealing information that isn’t necessary. If nothing else, for heaven’s sake, don’t suggest data that’s a close match to the user input.

Be a bad guy

When considering security, it’s helpful to take a step back, observe the information on display, and ask yourself how a malicious attacker would be able to utilize it. Play devil’s advocate. If a bad guy saw this page, what new information would they gain? Does the view show any PII?

Ask yourself if everything on the page is actually necessary for a genuine user. If not, redact or remove it. Less is safer.

Security starts at the front door

These days, there’s a lot more overlap between coding on the front end and the back end. To create a well-rounded and secure application, it helps to have a general understanding of ways attackers can get their foot in the front door.

The surprisingly difficult task of printing newlines in a terminal, 04 Dec 2019 09:17:35 (Victoria Drake) guide to string interpolation quirks that confound the best of us. <img src="" width="640" height="256"/>Surprisingly, getting computers to give humans readable output is no easy feat. With the introduction of standard streams and specifically standard output, programs gained a way to talk to each other using plain text streams; humanizing and displaying stdout is another matter. Technology throughout the computing age has tried to solve this problem, from the use of ASCII characters in video computer displays to modern shell commands like echo and printf.

These advancements have not been seamless. The job of printing output to a terminal is fraught with quirks for programmers to navigate, as exemplified by the deceptively nontrivial task of expanding an escape sequence to print newlines. The expansion of the placeholder \n can be accomplished in a multitude of ways, each with its own unique history and complications.

Using echo

From its appearance in Multics to its modern-day Unix-like system ubiquity, echo remains a familiar tool for getting your terminal to say “Hello world!” Unfortunately, inconsistent implementations across operating systems make its usage tricky. Where echo on some systems will automatically expand escape sequences, others require the -e option to do the same:

echo "the study of European nerves is \neurology"
# the study of European nerves is \neurology

echo -e "the study of European nerves is \neurology"
# the study of European nerves is
# eurology

Because of these inconsistencies in implementations, echo is considered non-portable. Additionally, its usage in conjunction with user input is relatively easy to corrupt through shell injection attack using command substitutions.

In modern systems, it is retained only to provide compatibility with the many programs that still use it. The POSIX specification recommends the use of printf in new programs.

Using printf

Since 4th Edition Unix, the portable printf command has essentially been the new and better echo. It allows you to use format specifiers to humanize input. To interpret backslash escape sequences, use %b. The character sequence \n ensures the output ends with a newline:

printf "%b\n" "Many females in Oble are \noblewomen"
# Many females in Oble are
# oblewomen

Though printf has further options that make it a far more powerful replacement of echo, this utility is not foolproof and can be vulnerable to an uncontrolled format string attack. It’s important for programmers to ensure they carefully handle user input.

Putting newlines in variables

In an effort to improve portability amongst compilers, the ANSI C Standard was established in 1983. With ANSI-C quoting using $'...', escape sequences are replaced in output according to the standard.

This allows us to store strings with newlines in variables that are printed with the newlines interpreted. You can do this by setting the variable, then calling it with printf using $:


printf "%b\n" "These words started with n but don't make $puns"

# These words started with n but don't make
# umber
# arrow
# ether
# ice

The expanded variable is single-quoted, which is passed literally to printf. As always, it is important to properly handle the input.

Bonus round: shell parameter expansion

In my article explaining Bash and braces, I covered the magic of shell parameter expansion. We can use one expansion, ${parameter@operator}, to interpret escape sequences, too. We use printf’s %s specifier to print as a string, and the E operator will properly expand the escape sequences in our variable:

printf "%s\n" ${puns@E}

# umber
# arrow
# ether
# ice

The ongoing challenge of humanizing output

String interpolation continues to be a chewy problem for programmers. Besides getting languages and shells to agree on what certain placeholders mean, properly using the correct escape sequences requires an eye for detail.

Poor string interpolation can lead to silly-looking output, as well as introduce security vulnerabilities, such as from injection attacks. Until the next evolution of the terminal has us talking in emojis, we’d best pay attention when printing output for humans.

The care and feeding of an IoT device, 27 Nov 2019 08:59:35 (Victoria Drake) IoT devices are, basically, puppies, and whether or not you should give somebody one for Christmas. <img src="" width="640" height="389"/>Giving someone a puppy for Christmas might work really well in a movie, but in real life often comes hitched to a multitude of responsibilities that the giftee may not be fully prepared to take on. The same is true for Internet of Things (IoT) devices, including Amazon’s Alexa-enabled devices, Google Home, and other Internet-connected appliances like cameras, lightbulbs, and toasters. Yes, they have those now.

Like puppies, IoT devices are still young. Many contain known vulnerabilities that remote attackers can use to gain access to device owners' networks. These attacks are sometimes as laughably simple as using a default username and password that the device owner cannot change.

Does all this mean you shouldn’t give Grandma Mabel a new app-enabled coffee maker or Ring doorbell for Christmas? Probably, although not necessarily. Like puppies, properly-maintained IoT devices are capable of warming your heart without causing too much havoc; but they take a lot of work to care for. Here are a few responsibilities to keep in mind for the care and feeding of an IoT device.

Immature security

Many manufacturers of IoT devices have not made security a priority. There aren’t yet any enforced security requirements for this industry, which leaves the protection of your device and the network it’s connected to in the hands of the manufacturer.

It’s not just obscure no-name toasters, either; malicious third-party apps have snuck onto Amazon’s and Google’s more reputable devices and enabled attackers to eavesdrop on unsuspecting owners.

Until security regulations are put in place and enforced, it’s buyer beware for both devices and third-party applications. To the extent possible, potential owners must do ample research to weed out vulnerable devices and untrustworthy apps.

Protecting your network

If you think hackers aren’t likely to find your device in the vast expanse of the Internet, you might be wrong. These days, obscurity doesn’t provide security. It’s no longer left up to a potential attacker’s fallible human eyes to find your insecure front door camera in a cacophony of wireless traffic; IoT search engines like Shodan will do that for them. Thankfully, these search engines are also used for good, enabling white hat hackers and penetration testers to find and fix insecure devices.

Just like locking your own front door, IoT owners are responsible for locking down access to their devices. This may mean searching through device settings to make sure default credentials are changed, or checking to make sure that a device used on your private home network doesn’t by default have public Internet access.

Where the options are available, HTTPS and multifactor authentication should be enabled. The use of a VPN can also keep your devices from being found.

Keeping them patched

Unlike puppies, many IoT devices are “headless” and have no inherent way of interfacing with a human. An app-controlled lightbulb, for example, may be all but useless without the software that makes it shine. As convenient as it may be to have your 1500K mood lighting come on automatically at dusk, it also means automatically ceding control of the device to its software developers.

When vulnerabilities in your phone’s operating system are discovered and patched, it’s likely that automatic updates are pushed and installed overnight, possibly without you even knowing. Your IoT device, on the other hand, may have no such support. In those cases, it’s completely up to the user to discover that an update is needed, find and download the patch, then correctly update their device. Even for owners with some technical expertise, this process takes significant effort. Many device owners aren’t even aware that their software is dangerously outdated.

In practical terms, this means that users without the time, knowledge, or willingness to keep their devices updated should reconsider owning them. Alternatively, some research can help prospective owners choose devices that receive automatic push updates from their (hopefully responsible) manufacturers over WiFi.

Being responsible

Raising a healthy and happy IoT device is no small task, especially for potential owners with little time or willingness to put in the required effort. With the proper attention and maintenance, your Internet-connected appliance can bring joy and convenience to your life; but without, it introduces a potential security risk and a whole lot of trouble.

Before getting or giving IoT, be sure the potential owner is up to the task of caring for it.

You can learn more about basic cybersecurity for IoT (as a user or maker) by reading NIST’s draft guidelines publication.

Bash and shell expansions: lazy list-making, 18 Nov 2019 07:07:24 (Victoria Drake) tour of brace expansion, shell parameter expansions, and playing with substrings in Bash. <img src="" width="640" height="309"/>It’s that time of year again! When stores start putting up colourful sparkly lit-up plastic bits, we all begin to feel a little festive, and by festive I mean let’s go shopping. Specifically, holiday gift shopping! (Gifts for yourself are still gifts, technically.)

Just so this doesn’t all go completely madcap, you ought to make some gift lists. Bash can help.

Brace expansion

These are not braces: ()

Neither are these: []

These are braces: {}

Braces tell Bash to do something with the arbitrary string or strings it finds between them. Multiple strings are comma-separated: {a,b,c}. You can also add an optional preamble and postscript to be attached to each expanded result. Mostly, this can save some typing, such as with common file paths and extensions.

Let’s make some lists for each person we want to give stuff to. The following commands are equivalent:

touch /home/me/gift-lists/Amy.txt /home/me/gift-lists/Bryan.txt /home/me/gift-lists/Charlie.txt
touch /home/me/gift-lists/{Amy,Bryan,Charlie}.txt
tree gift-lists

├── Amy.txt
├── Bryan.txt
└── Charlie.txt

Oh darn, “Bryan” spells his name with an “i.” I can fix that.

mv /home/me/gift-lists/{Bryan,Brian}.txt

renamed '/home/me/gift-lists/Bryan.txt' -> '/home/me/gift-lists/Brian.txt'

Shell parameter expansions

Shell parameter expansion allows us to make all sorts of changes to parameters enclosed in braces, like manipulate and substitute text.

There are a few stocking stuffers that all our giftees deserve. Let’s make that a variable:

STUFF=$'socks\nlump of coal\nwhite chocolate'

echo "$STUFF"
lump of coal
white chocolate

Now to add these items to each of our lists with some help from the tee command to get echo and expansions to play nice.

echo "$STUFF" | tee {Amy,Brian,Charlie}.txt

cat {Amy,Brian,Charlie}.txt

lump of coal
white chocolate
lump of coal
white chocolate
lump of coal
white chocolate

Pattern match substitution

On second thought, maybe the lump of coal isn’t such a nice gift. You can replace it with something better using a pattern match substitution in the form of ${parameter/pattern/string}:

echo "${STUFF/lump of coal/candy cane}" | tee {Amy,Brian,Charlie}.txt

cat {Amy,Brian,Charlie}.txt

candy cane
white chocolate
candy cane
white chocolate
candy cane
white chocolate

This replaces the first instance of “lump of coal” with “candy cane.” To replace all instances (if there were multiple), use ${parameter//pattern/string}. This doesn’t change our $STUFF variable, so we can still reuse the original list for someone naughty later.


While we’re improving things, our giftees may not all like white chocolate. We’d better add some regular chocolate to our lists just in case. Since I’m super lazy, I’m just going to hit the up arrow and modify a previous Bash command. Luckily, the last word in the $STUFF variable is “chocolate,” which is nine characters long, so I’ll tell Bash to keep just that part using ${parameter:offset}. I’ll use tee’s -a flag to append to my existing lists:

echo "${STUFF: -9}" | tee -a {Amy,Brian,Charlie}.txt

cat {Amy,Brian,Charlie}.txt

candy cane
white chocolate
candy cane
white chocolate
candy cane
white chocolate

You can also:

Do this With this
Get substring from n characters onwards ${parameter:n}
Get substring for x characters starting at n ${parameter:n:x}

There! Now our base lists are finished. Let’s have some eggnog.

Testing variables

You know, it may be the eggnog, but I think I started a list for Amy yesterday and stored it in a variable that I might have called amy. Let’s see if I did. I’ll use the ${parameter:?word} expansion. It’ll write word to standard error and exit if there’s no amy parameter.

echo "${amy:?no such}"

bash: amy: no such

I guess not. Maybe it was Brian instead?

echo "${brian:?no such}"


You can also:

Do this With this
Substitute word if parameter is unset or null ${parameter:-word}
Substitute word if parameter is not unset or null ${parameter:+word}
Assign word to parameter if parameter is unset or null ${parameter:=word}

Changing case

That’s right! Brian said he wanted some lederhosen and so I made myself a note. This is pretty important, so I’ll add it to Brian’s list in capital letters with the ${parameter^^pattern} expansion. The pattern part is optional. We’re only writing to Brian’s list, so I’ll just use >> instead of tee -a.

echo "${brian^^}" >> Brian.txt

cat Brian.txt

candy cane
white chocolate

You can also:

Do this With this
Capitalize the first letter ${parameter^pattern}
Lowercase the first letter ${parameter,pattern}
Lowercase all letters ${parameter,,pattern}

Expanding arrays

You know what, all this gift-listing business is a lot of work. I’m just going to make an array of things I saw at the store:

gifts=(sweater gameboy wagon pillows chestnuts hairbrush)

I can use substring expansion in the form of ${parameter:offset:length} to make this simple. I’ll add the first two to Amy’s list, the middle two to Brian’s, and the last two to Charlie’s. I’ll use printf to help with newlines.

printf '%s\n' "${gifts[@]:0:2}" >> Amy.txt
printf '%s\n' "${gifts[@]:2:2}" >> Brian.txt
printf '%s\n' "${gifts[@]: -2}" >> Charlie.txt
cat Amy.txt

candy cane
white chocolate

cat Brian.txt

candy cane
white chocolate

cat Charlie.txt

candy cane
white chocolate

There! Now we’ve got a comprehensive set of super personalized gift lists. Thanks Bash! Too bad it can’t do the shopping for us, too.

A cron job that could save you from a ransomware attack, 13 Nov 2019 08:27:31 (Victoria Drake) a simple scheduled job can help you quickly recover from ransomware. <img src="" width="640" height="178"/>It’s 2019, and ransomware has become a thing.

Systems that interact with the public, like companies, educational institutions, and public services, are most susceptible. While delivery methods for ransomware vary from the physical realm to communication via social sites and email, all methods only require one person to make one mistake in order for ransomware to proliferate.

Ransomware, as you may have heard, is a malicious program that encrypts your files, rendering them unreadable and useless to you. It can include instructions for paying a ransom, usually by sending cryptocurrency, in order to obtain the decryption key. Successful ransomware attacks typically exploit vital, time-sensitive systems. Victims like public services and medical facilities are more likely to have poor or zero recovery processes, leaving governments or insurance providers to reward attackers with ransom payments.

Individuals, especially less-than-tech-savvy ones, are no less at risk. Ransomware can occlude personal documents and family photos that may only exist on one machine.

Thankfully, a fairly low-tech solution exists for rendering ransomware inept: back up your data!

You could achieve this with a straightforward system like plugging in an external hard drive and dragging files over once a day, but this method has a few hurdles. Manually transferring files may be slow or incomplete, and besides, you’ll first have to remember to do it.

In my constant pursuit of automating all the things, there’s one tool I often return to for its simplicity and reliability: cron. Cron does one thing, and does it well: it runs commands on a schedule.

I first used it a few months shy of three years ago (Have I really been blogging that long?!) to create custom desktop notifications on Linux. Using the crontab configuration file, which you can edit by running crontab -e, you can specify a schedule for running any commands you like. Here’s what the scheduling syntax looks like, from the Wikipedia cron page:

# │ │ │ │ │
# │ │ │ │ │
# │ │ │ │ │
# * * * * * command to execute

For example, a cron job that runs every day at 00:00 would look like:

0 0 * * *

To run a job every twelve hours, the syntax is:

0 */12 * * *

This great tool can help you wrap your head around the cron scheduling syntax.

What’s a scheduler have to do with backing up? By itself, not much. The simple beauty of cron is that it runs commands - any shell commands, and any scripts that you’d normally run on the command line. As you may have gleaned from my other posts, I’m of the strong opinion that you can do just about anything on the command line, including backing up your files. Options for storage in this area are plentiful, from near-to-free local and cloud options, as well as paid managed services too numerous to list. For CLI tooling, we have utilitarian classics like rsync, and CLI tools for specific cloud providers like AWS.

Backing up with rsync

The rsync utility is a classic choice, and can back up your files to an external hard drive or remote server while making intelligent determinations about which files to update. It uses file size and modification times to recognize file changes, and then only transfers changed files, saving time and bandwidth.

The rsync syntax can be a little nuanced; for example, a trailing forward slash will copy just the contents of the directory, instead of the directory itself. I found examples to be helpful in understanding the usage and syntax.

Here’s one for backing up a local directory to a local destination, such as an external hard drive:

rsync -a /home/user/directory /media/user/destination

The first argument is the source, and the second is the destination. Reversing these in the above example would copy files from the mounted drive to the local home directory.

The a flag for archive mode is one of rsync’s superpowers. Equivalent to flags -rlptgoD, it:

  • Syncs files recursively through directories (r);
  • Preserves symlinks (l), permissions (p), modification times (t), groups (g), and owner (o); and
  • Copies device and special files (D).

Here’s another example, this time for backing up the contents of a local directory to a directory on a remote server using SSH:

rsync -avze ssh /home/user/directory/

The v flag turns on verbose output, which is helpful if you like realtime feedback on which files are being transferred. During large transfers, however, it can tend to slow things down. The z flag can help with that, as it indicates that files should be compressed during transfer.

The e flag, followed by ssh, tells rsync to use SSH according to the destination instructions provided in the final argument.

Backing up with AWS CLI

Amazon Web Services offers a command line interface tool for doing just about everything with your AWS set up, including a straightforward s3 sync command for recursively copying new and updated files to your S3 storage buckets. As a storage method for back up data, S3 is a stable and inexpensive choice. You can even turn on versioning in your bucket.

The syntax for interacting with directories is fairly straightforward, and you can directly indicate your S3 bucket as an S3Uri argument in the form of s3://mybucket/mykey. To back up a local directory to your S3 bucket, the command is:

aws s3 sync /home/user/directory s3://mybucket

Similar to rsync, reversing the source and destination would download files from the S3 bucket.

The sync command is intuitive by default. It will guess the mime type of uploaded files, as well as include files discovered by following symlinks. A variety of options exist to control these and other defaults, even including flags to specify the server-side encryption to be used.

Setting up your cronjob back up

You can edit your machine’s cron file by running:

crontab -e

Intuitive as it may be, it’s worth mentioning that your back up commands will only run when your computer is turned on and the cron daemon is running. With this in mind, choose a schedule for your cronjob that aligns with times when your machine is powered on, and maybe not overloaded with other work.

To back up to an S3 bucket every day at 8AM, for example, you’d put a line in your crontab that looks like:

0 8 * * * aws s3 sync /home/user/directory s3://mybucket

If you’re curious whether your cron job is currently running, find the PID of cron with:

pstree -ap | grep cron

Then run pstree -ap <PID>.

This rabbit hole goes deeper; a quick search can reveal different ways of organizing and scheduling cronjobs, or help you find different utilities to run cronjobs when your computer is asleep. To protect against the possibility of ransomware-affected files being transferred to your back up, incrementally separated archives are a good idea. In essence, however, this basic set up is all you really need to create a reliable, automatic back up system.

Don’t feed the trolls

Humans are fallible; that’s why cyberattacks work. The success of a ransomware attack depends on the victim having no choice but to pay up in order to return to business as usual. A highly accessible recent back up undermines attackers who depend on us being unprepared. By blowing away a system and restoring from yesterday’s back up, we may lose a day of progress; ransomers, however, gain nothing at all.

For further resources on ransomware defense for users and organizations, check out CISA’s advice on ransomware.

Publishing GitHub event data with GitHub Actions and Pages, 04 Nov 2019 09:13:23 (Victoria Drake) guide to overcoming the GitHub event data horizon with a little command-line magic. <img src="" width="640" height="270"/>Teams who work on GitHub rely on event data to collaborate. The data recorded as issues, pull requests, and comments, become vital to understanding the project.

With the general availability of GitHub Actions, we have a chance to programmatically access and preserve GitHub event data in our repository. Making the data part of the repository itself is a way of preserving it outside of GitHub, and also gives us the ability to feature the data on a front-facing website, such as with GitHub Pages, through an automated process that’s part of our CI/CD pipeline.




Accessing GitHub event data






ID=${{ }}


AUTHOR=${{ github.event.comment.user.login }}
DATE=${{ github.event.comment.created_at }}


Preserving event data: using artifacts







Preserving event data: pushing workflow files to the repository


Creating files in the workflow


- uses: actions/checkout@master
    fetch-depth: 1


- name: Turn comment into file
  run: |
    ID=${{ }}
    AUTHOR=${{ github.event.comment.user.login }}
    DATE=${{ github.event.comment.created_at }}

    printf '%b\n' "<div class=\"comment\"><p>${AUTHOR} says:</p><p>${NO_TAGS//$'\n'/\<\/p\>\<p\>}</p><p>${DATE}</p></div>\r\n" > ${FOLDER}/${ID}.html    


<div class="comment">
    <p>victoriadrake says:</p>
    <p>This is a comment!</p>



- name: Assemble page
  run: |
    cat header.html > index.html
    cat footer.html >> index.html    

Committing changes to the repository



- name: Push changes to repo
  run: |
    REMOTE=https://${{ secrets.GITHUB_TOKEN }}${{ github.repository }}
    git config "${{ }}"
    git config "${{ }}"

    git pull ${REMOTE}
    git checkout master
    git add .
    git status
    git commit -am "Add new comment"
    git push ${REMOTE} master    

The remote, in fact, our repository, is specified using the github.repository context variable. For our workflow to be allowed to push to master, we give the remote URL using the default secrets.GITHUB_TOKEN variable.

Since the workflow environment is shiny and newborn, we need to configure Git. In the above example, I’ve used the context variable to input the username of the account initiating the workflow. The email is similarly configured using the default noreply GitHub email address.

Displaying event data

If you’re using GitHub Pages with the default secrets.GITHUB_TOKEN variable and without a site generator, pushing changes to the repository in the workflow will only update the repository files. The GitHub Pages build will fail with an error, “Your site is having problems building: Page build failed.”

To enable Actions to trigger a Pages site build, you’ll need to create a Personal Access Token. This token can be stored as a secret in the repository settings and passed into the workflow in place of the default secrets.GITHUB_TOKEN variable. I wrote more about Actions environment and variables in this post.

With the use of a Personal Access Token, a push initiated by the Actions workflow will also update the Pages site. You can see it for yourself by leaving a comment in my guestbook! The comment creation event triggers the workflow, which then takes around 30 seconds to run and update the guestbook page.


- name: Run Makefile
    TOKEN: ${{ secrets.GITHUB_TOKEN }}
  run: make all


No more event data horizon


  4. An awesome 90s guestbook page.


A lightweight, tool-agnostic CI/CD flow with GitHub Actions, 28 Oct 2019 08:28:52 (Victoria Drake) to take advantage of a simple GitHub Actions workflow without sacrificing agnostic tooling. <img src="" width="640" height="202"/>Agnostic tooling is the clever notion that you should be able to run your code in various environments. With many continuous integration and continuous development (CI/CD) apps available, agnostic tooling gives developers a big advantage: portability.





Appreciating the shell





Using a Makefile with GitHub Actions




name: make-master

      - master
    - cron: '20 13 * * *'

    runs-on: ubuntu-latest
      - uses: actions/checkout@master
          fetch-depth: 1
      - name: Run Makefile
          TOKEN: ${{ secrets.TOKEN }}
        run: make all

I’ll explain the components that make this work.

Triggering the workflow



One hacky thing


nothing to commit, working tree clean
On branch master

nothing to commit, working tree clean
Makefile:62: recipe for target 'deploy' failed
make: *** [deploy] Error 1
##[error]Process completed with exit code 2.


Environment and variables



Running Makefile
##[error]Process completed with exit code 2.


Portable tooling



A portable Makefile for continuous delivery with Hugo and GitHub Pages, 21 Oct 2019 09:09:06 (Victoria Drake) Makefile for building this site, optimizing images, and running my CI/CD GitHub Actions flow. <img src="" width="640" height="390"/>Fun fact: I first launched this GitHub Pages site 1,018 days ago.

Since then, we’ve grown together. From early cringe-worthy commit messages, through eighty-six versions of Hugo, and up until last week, a less-than-streamlined multi-app continuous integration and deployment (CI/CD) workflow.

If you know me at all, you know I love to automate things. I’ve been using a combination of AWS Lambda, Netlify, and Travis CI to automatically build and publish this site. My workflow for the task includes:

Thanks to the introduction of GitHub Actions, I’m able to do all the above with just one portable Makefile.

Next week I’ll cover my Actions set up; today, I’ll take you through the nitty-gritty of my Makefile so you can write your own.

Makefile portability

POSIX-standard-flavour Make runs on every Unix-like system out there. Make derivatives, such as GNU Make and several flavours of BSD Make also run on Unix-like systems, though their particular use requires installing the respective program. To write a truly portable Makefile, mine follows the POSIX standard. (For a more thorough summation of POSIX-compatible Makefiles, I found this article helpful: A Tutorial on Portable Makefiles.) I run Ubuntu, so I’ve tested the portability aspect using the BSD Make programs bmake, pmake, and fmake. Compatibility with non-Unix-like systems is a little more complicated, since shell commands differ. With derivatives such as Nmake, it’s better to write a separate Makefile with appropriate Windows commands.

While much of my particular use case could be achieved with shell scripting, I find Make offers some worthwhile advantages. I enjoy the ease of using variables and macros, and the modularity of rules when it comes to organizing my steps.

The writing of rules mostly comes down to shell commands, which is the main reason Makefiles are as portable as they are. The best part is that you can do pretty much anything in a terminal, and certainly handle all the workflow steps listed above.

My continuous deployment Makefile

Here’s the portable Makefile that handles my workflow. Yes, I put emojis in there. I’m a monster.


OPTIMIZE = find $(DESTDIR) -not -path "*/static/*" \( -name '*.png' -o -name '*.jpg' -o -name '*.jpeg' \) -print0 | \
xargs -0 -P8 -n2 mogrify -strip -thumbnail '1000>'

.PHONY: all
all: get_repository clean get build test deploy

.PHONY: get_repository
 @echo "🛎 Getting Pages repository"
 git clone $(DESTDIR)

.PHONY: clean
 @echo "🧹 Cleaning old build"
 cd $(DESTDIR) && rm -rf *

.PHONY: get
 @echo "❓ Checking for hugo"
 @if ! [ -x "$$(command -v hugo)" ]; then\
  echo "🤵 Getting Hugo";\
     wget -q -P tmp/$(HUGO_VERSION)/hugo_extended_$(HUGO_VERSION)_Linux-64bit.tar.gz;\
  tar xf tmp/hugo_extended_$(HUGO_VERSION)_Linux-64bit.tar.gz -C tmp/;\
  sudo mv -f tmp/hugo /usr/bin/;\
  rm -rf tmp/;\
  hugo version;\

.PHONY: build
 @echo "🍳 Generating site"
 hugo --gc --minify -d $(DESTDIR)

 @echo "🧂 Optimizing images"

.PHONY: test
 @echo "🍜 Testing HTML"
 docker run -v $(GITHUB_WORKSPACE)/$(DESTDIR)/:/mnt 18fgsa/html-proofer mnt --disable-external

.PHONY: deploy
 @echo "🎁 Preparing commit"
 @cd $(DESTDIR) \
 && git config "" \
 && git config "Victoria via GitHub Actions" \
 && git add . \
 && git status \
 && git commit -m "🤖 CD bot is helping" \
 && git push -f -q https://$(TOKEN) master
 @echo "🚀 Site is deployed!"

Sequentially, this workflow:

  1. Clones the public Pages repository;
  2. Cleans (deletes) the previous build files;
  3. Downloads and installs the specified version of Hugo, if Hugo is not already present;
  4. Builds the site;
  5. Optimizes images;
  6. Tests the built site with HTMLProofer, and
  7. Prepares a new commit and pushes to the public Pages repository.

If you’re familiar with command line, most of this may look familiar. Here are a couple bits that might warrant a little explanation.

Checking if a program is already installed

I think this bit is pretty tidy:


I use a negated if conditional in conjunction with command -v to check if an executable (-x) called hugo exists. If one is not present, the script gets the specified version of Hugo and installs it. This Stack Overflow answer has a nice summation of why command -v is a more portable choice than which.

Image optimization

My Makefile uses mogrify to batch resize and compress images in particular folders. It finds them automatically using the file extension, and only modifies images that are larger than the target size of 1000px in any dimension. I wrote more about the batch-processing one-liner in this post.

There are a few different ways to achieve this same task, one of which, theoretically, is to take advantage of Make’s suffix rules to run commands only on image files. I find the shell script to be more readable.

Using Dockerized HTMLProofer

HTMLProofer is installed with gem, and uses Ruby and Nokogiri, which adds up to a lot of installation time for a CI workflow. Thankfully, 18F has a Dockerized version that is much faster to implement. Its usage requires starting the container with the built site directory mounted as a data volume, which is easily achieved by appending to the docker run command.

docker run -v /absolute/path/to/site/:/mounted-site 18fgsa/html-proofer /mounted-site

In my Makefile, I specify the absolute site path using the default environment variable GITHUB_WORKSPACE. I’ll dive into this and other GitHub Actions features in the next post.

In the meantime, happy Making!

How to quickly batch resize, compress, and convert images with a Bash one-liner, 14 Oct 2019 08:27:49 (Victoria Drake) fast command line interface solution for batch image processing. <img src="" width="640" height="345"/>Part of my Hugo site continuous deployment workflow is the processing of 210 images, at time of writing.

Here’s my one-liner:

find public/ -not -path "*/static/*" \( -name '*.png' -o -name '*.jpg' -o -name '*.jpeg' \) -print0 | xargs -0 -P8 -n2 mogrify -strip -thumbnail '1000>' -format jpg

I use find to target only certain image file formats in certain directories. With mogrify, part of ImageMagick, I resize only the images that are larger than a certain dimension, compress them, and strip the metadata. I tack on the format flag to create jpg copies of the images.

Here’s the one-liner again (broken up for better reading):

# Look in the public/ directory
find public/ \
# Ignore directories called "static" regardless of location
-not -path "*/static/*" \
# Print the file paths of all files ending with any of these extensions
# Pipe the file paths to xargs and use 8 parallel workers to process 2 arguments
| xargs -0 -P8 -n2 \
# Tell mogrify to strip metadata, and...
mogrify -strip \
# ...compress and resize any images larger than the target size (1000px in either dimension)
-thumbnail '1000>' \
# Convert the files to jpg format
-format jpg

That’s it. That’s the post.

Secure application architecture basics: separation, configuration, and access, 30 Sep 2019 08:03:12 (Victoria Drake) starting point for building secure application architecture, for busy developers. <img src="" width="640" height="443"/>Software developers today are encouraged to focus on building, and that’s a great thing. There’s the benefit of maker culture, an attitude of “always be shipping,” open source collaboration, and a bevy of apps that help you prioritize and execute with maximum efficiency. It’s in an environment of constant creation, where both teams and solo entrepreneurs can be maximally productive.

Sometimes, this breakneck-speed productivity shows its downsides.

A lack of awareness of security seems to lead to a lack of prioritization of tasks that don’t directly support bringing the product to launch. The market seems to have made it more important to launch a usable product than a secure one, with the prevailing attitude being, “we can do the security stuff later.”

Cobbling together a foundation based more on expediency than longevity is a bad way to build applications and a great way to build security debt. Security debt, like technical debt, amasses when developers make decisions that can make it more difficult to secure the application later on. If you’re familiar with the concept of “pushing left” (or if you read my article about sensitive data exposure), you’ll know that when it comes to security, sometimes there isn’t a version of “later” that isn’t too late. It’s a shame, especially since following some basic security practices with high benefit yield early on in the development process doesn’t take significantly more time than not following them. Often, it comes down to having some basic but important knowledge that enables making the more secure decision.

While application architecture specifics vary, there are a few basic principles you can commonly apply. This article will provide a high-level overview of areas that I hope will help point developers in the right direction.

There must be a reason you call it application “architecture.” I like to think it’s because the architecture of software is similar in some basic ways to the architecture of a building. (Or at least, in my absolute zero building-building expertise, how I imagine a building to be built.) Here’s how I like to summarize three basic points of secure application architecture:

  1. Separated storage
  2. Customized configuration
  3. Controlled access and user scope

This is only a jumping-off point meant to get you started on the right foot; a complete picture of a fully-realized application’s security posture includes areas outside the scope of this article, including authentication, logging and monitoring, testing, and perhaps meeting compliance standards.

1. Separated storage

From a security standpoint, the concept of separation refers to storing files that serve different purposes in different places. When you’re constructing your building and deciding where all the rooms go, you similarly create the lobby on the ground floor and place administrative offices on higher floors, perhaps off the main path. While both are rooms, you understand that they serve different purposes, have different functional needs, and possibly very different security requirements.

Separation of building floors

When it comes to your files, the benefit is perhaps easiest to understand if you consider a simple file structure:

 │   └───index.html
 │   ├───images/
 │   │   ├───rainbows.jpg
 │   │   └───unicorns.jpg
 │   └───style.css

In this simplified example, let’s say that all your application’s images are stored in the application/assets/images/ directory. When one of your users creates a profile and uploads their picture to it, this picture is also stored in this folder. Makes sense, right? It’s an image, and that’s where the images go. What’s the issue?

If you’re familiar with navigating a file structure in a terminal, you may have seen this syntax before: ../../. The two dots are a handy way of saying, “go up one directory.” If you execute the command cd ../../ in the images/ directory of the simple file structure above, you’d go up into assets/, then up again to the root directory, application/. This is a problem because of a wee little vulnerability dubbed path traversal.

While the dot syntax saves you some typing, it also introduces the interesting advantage of not actually needing to know what the parent directory is called in order to go to it. Consider an attack payload script, delivered into the images/ folder of your insecure application via an uploaded file, that went up one directory using cd ../ and then sent everything it found to the attacker, on repeat. Eventually, it would reach the root application directory and access the super-secret-configurations/ folder. Not good.

While other measures should be in place to prevent path traversal and related user upload vulnerabilities, the simplest prevention by far is a separation of storage. Core application files and assets should not be combined with other data, and especially not with user input. It’s best to keep user-uploaded files and activity logs (which may contain juicy data and can be vulnerable to injection attacks) separate from the main application.

Separation can be achieved in a few ways, such as by using a different server, different instance, separate IP range, or separate domain.

2. Customized configuration

Security misconfiguration is listed in the OWASP Top 10. A surprising number of very preventable security incidents occur because a server, firewall, or administrative account is running in production with default settings. Upon the opening of your new building, you’d hopefully be more careful to ensure you haven’t left any keys in the locks.

Three keys

Usually, the victims of attacks related to default settings aren’t specifically targeted. Rather, they are found by automated scanning tools that attackers run over many possible targets, effectively prodding at many different systems to see if any roll over and expose some useful exploit. The automated nature of this attack means that it’s important for you to review settings for every piece of your architecture. Even if an individual piece doesn’t seem significant, it may provide a vulnerability that allows an attacker to use it as a gateway to your larger application.

In particular, examine architecture components for unattended areas such as:

  • Default accounts, especially with default passwords, left in service;
  • Example web pages, tutorial applications, or sample data left in the application;
  • Unnecessary ports left in service, or ports left open to the Internet;
  • Unrestricted permitted HTTP methods;
  • Sensitive information stored in automated logs;
  • Default configured permissions in managed services; and,
  • Directory listings, or sensitive file types, left accessible by default.

This list isn’t exhaustive. Specific architecture components, such as cloud storage or web servers, will have other configurable features that should be reviewed. In general, reduce the application’s attack surface by using minimal architecture components. If you use fewer components or don’t install modules you don’t need, you’ll have fewer possible attack entry points to configure and safeguard.

3. Controlled access and user scope

One of the more difficult security problems to test in an application is misconfigured access control. Automated testing tools have limited capability to find areas of an application that one user shouldn’t be able to access. Thus, this is often left to manual testing or source code review to discover. By considering this vulnerability early on in the software development lifecycle when architectural decisions are being made, you reduce the risk that it becomes a problem that’s harder to fix later. After all, you wouldn’t simply leave your master keys out of reach on a high ledge and hope no one comes along with a ladder.

A cartoon of a user attempting to elevate privilege

Broken access control is listed in the OWASP Top 10, which goes into more detail on its various forms. As a simple example, consider an application with two levels of access: administrators and users. You want to build a new feature - the ability to moderate or ban users - with the intention that only administrators would be allowed to use it.

If you’re aware of the possibility of access control misconfigurations or exploits, you may decide to build the moderation feature in a completely separate area from the user-accessible space, such as on a different domain, or as part of a model that users don’t share. This greatly reduces the risk that an access control misconfiguration or elevation of privilege vulnerability might allow a user to improperly access the moderation feature later on.

Of course, robust access control in your application needs more support to be effective. Consider factors such as sensitive tokens, or keys passed as URL parameters, or whether a control fails securely or insecurely. Nevertheless, by considering authorization at the architectural stage, you can set yourself up to make further reinforcements easier to implement.

Security basics for maximum benefit

Similar to avoiding racking up technical debt by choosing a well-vetted framework, developers can avoid security debt by becoming more aware of common vulnerabilities and the simple architectural decisions you can make to help mitigate them. For a much more detailed resource on how to bake security into your applications from the start, the OWASP Application Security Verification Standard is a robust guide.

Migrating to the cloud but without screwing it up, or how to move house, 23 Sep 2019 08:03:12 (Victoria Drake) practical guide to moving to cloud services with minimal downtime, using AWS examples. <img src="" width="640" height="468"/>For an application that’s ready to scale, not using managed cloud architecture these days is like insisting on digging your own well for water. It’s far more labour-intensive, requires buying all your own equipment, takes a lot more time, and there’s a higher chance you’re going to get it wrong because you don’t personally have a whole lot of experience digging wells, anyway.

That said - let’s just get this out of the way first - there is no cloud. It’s just someone else’s computer.

Of course, these days, cloud services go far beyond the utility we’d expect from a single computer. Besides being able to quickly set up and utilize the kind of computing power that previously required a new office lease agreement to house, there are now a multitude of monitoring, management, and analysis tools at our giddy fingertips. While it’s important to understand that the cloud isn’t a better option in every case, for applications that can take advantage of it, we can do more, do it faster, and do it for less money than if we were to insist on building our own on-premises infrastructure.

That’s all great, and easily said; moving to the cloud, however, can look from the outset like a pretty daunting task. How, exactly, do we go about shifting what may be years of on-premises data and built-up systems to someone else’s computer? You know, without being able to see it, touch it, and without completely screwing up our stuff.

While it probably takes less work and money than setting up or maintaining the same architecture on-premise, it does take some work to move to the cloud initially. It’s important that our application is prepared to migrate, and capable of using the benefits of cloud services once it gets there. To accomplish this, and a smooth transition, preparation is key. In fact, it’s a whole lot like moving to a new house.

In this article, we’ll take a high-level look at the general stages of taking an on-premise or self-hosted application and moving it to the cloud. This guide is meant to serve as a starting point for designing the appropriate process for your particular situation, and to enable you to better understand the cloud migration process. While cloud migration may not be the best choice for some applications - such as ones without scalable architecture or where very high computing resources are needed - a majority of modular and modern applications stand to benefit from a move to the cloud.

It’s certainly possible, as I discovered at a recent event put on by Amazon Web Services (AWS) Solutions Architects, to migrate smoothly and efficiently, with near-zero loss of availability to customers. I’ll specifically reference some services provided by AWS, however, similar functionality can be found with other cloud providers. I’ve found the offerings from AWS to be pleasantly modular in scope, which is why I use them myself and why they make good examples for discussing general concepts.

To have our move go as smoothly as possible, here are the things we’ll want to consider:

  1. The type of move we’re making;
  2. The things we’ll take, and the things we’ll clean up;
  3. How to choose the right type and size for the infrastructure we’re moving into; and
  4. How to do test runs to practice for the big day.

The type of move we’re making

While it’s important to understand why we’re moving our application to cloud services, we should also have an idea of what we’d like it to look like when it gets there. There are three main ways to move to the cloud: re-host, re-platform, or re-factor.


A re-host scenario is the the most straightforward type of move. It involves no change to the way our application is built or how it runs. For example, if we currently have Python code, use PostgreSQL, and serve our application with Apache, a re-host move would mean we use all the same components, combined in just the same way, only now they’re in the cloud. It’s a lot like moving into a new house that has the exact same floor plan as the current one. All the furniture goes into the same room it’s in now, and it’s going to feel pretty familiar when we get there.

The main draw of a re-host move is that it may offer the least amount of complication necessary in order to take advantage of going to the cloud. Scalable applications, for example, can gain the ability to automatically manage necessary application resources.

While re-hosting makes scaling more automatic, it’s important to note that it won’t in itself make an application scalable. If the application infrastructure is not organized in such a way that gives it the ability to scale, a re-factor may be necessary instead.


If a component of our current application set up isn’t working out well for us, we’re probably going to want to re-platform. In this case, we’re making a change to at least one component of our architecture; for example, switching our database from Oracle to MySQL on Amazon Relational Database Service (RDS).

Like moving from a small apartment in Tokyo to an equally small apartment in New York, a re-platform doesn’t change the basic nature of our application, but does change its appearance and environment. In the database change example, we’ll have all the same data, just organized or formatted a little differently. In most cases, we won’t have to make these changes manually. A tool such as Amazon Database Migration Service (DMS) can help to seamlessly shift our data over to the new database.

We might re-platform in order to enable us to better meet a business demand in the future, such as scaling up, integrating with other technological components, or choosing a more modern technology stack.


A move in which we re-factor our application is necessarily more complicated than our other options, however, it may provide the most overall benefit for companies or applications that have reason to make this type of move. As with code, refactoring is done when fundamental changes need to be made in order for our application to meet a business need. The specifics necessarily differ case-by-case, but typically involve changes to architectural components or how those components relate to one another. This type of move may also involve changing application code in order to optimize the application’s performance in a cloud environment. We can think of it like moving out from our parent’s basement in the suburbs and getting a nice townhouse in the city. There’s no way we’re taking that ancient hand-me-down sofa, so we’ll need some new furniture, and for our neighbour’s sake, probably window dressings.

Refactoring may enable us to modernize a dated application, or make it more efficient in general. With greater efficiency, we can better take advantage of services that cloud providers typically offer, like bursting resources or attaining deep analytical insight.

If a re-factor is necessary but time is scarce, it may be better to re-host or re-platform first, then re-factor later. That way, we’ll have a job well done later instead of a hasty, botched migration (and more problems) sooner.

What to take, and what to clean up

Over the years of living in one place, stuff tends to pile up unnoticed in nooks and crannies. When moving house, it’s usually a great opportunity to sort everything out and decide what is useful enough to keep, and what should be discarded or given away. Moving to the cloud is a similarly great opportunity to do the same when it comes to our application.

While cloud storage is inexpensive nowadays, there may be some things that don’t make sense to store any longer, or at least not keep stored with our primary application. If data cannot be discarded due to policy or regulations, we may choose a different storage class to house data that we don’t expect to need anytime soon outside of our main application.

In the case of Amazon’s Simple Storage Service (S3), we can choose to use different storage classes that accomplish this goal. While the data that our business relies on every day can take advantage of the Standard class 99.99% availability, data meant for long-term cold storage such as archival backups can be put into the Glacier class, which has longer retrieval time and lower cost.

The right type and size

Choosing the type and size of cloud infrastructure appropriate for our business is usually the part that can be the most confusing. How should we predict, in a new environment or for a growing company, the computing power we’ll need?

Part of the beauty of not procuring hardware on our own is that won’t have to make predictions like these. Using cloud storage and instances, expanding or scaling back resources can be done in a matter of minutes, sometimes seconds. With managed services, it can even be done automatically for us. With the proper support for scalability in our application, it’s like having a magical house that instantly generates any type of room and amenity we need at that moment. The ability to continually ensure that we’re using appropriate, cost-effective resources is at our fingertips, and often clearly visualized in charts and dashboards.

For applications new to the cloud, some leeway for experimentation may be necessary. While cloud services enables us to quickly spin up and try out different architectures, there’s no guarantee that all of those set ups will work well for our application. For example, running a single instance may be less expensive than going serverless, but we’d be hard pressed to know this until we tried it out.

As a starting point, we simply need enough storage and computing power to support the application as it is currently running, today. For example, in the case of storage, consider the size of the current database - the actual database data, not the total storage capacity of hardware on-premises. For a detailed cost exploration, AWS even offers a Simple Monthly Calculator with use case samples to help guide expectations.

Do test runs before the big day

Running a trial cloud migration may be an odd concept, but it is an essential component to ensuring that the move goes as planned with minimal service interruption. Imagine the time and energy that would be saved in the moving house example if we could automate test runs! Invariably, some box or still-hung picture is forgotten and left out of the main truck, necessitating additional trips in other vehicles. With multiple chances to ensure we’ve got it down pat, we minimize the possibility that our move causes any break in normal day-to-day business.

Generally, to do a test run, we create a duplicate version of our application. The more we can duplicate, the more thorough the test run will be, especially if our data is especially large. Though duplication may seem tedious, working with the actual components we intend to migrate is essential to ensuring the migration goes as planned. After all, if we only did a moving-house test run with one box, it wouldn’t be very representative.

Test runs can help to validate our migration plan against any challenges we may encounter. These challenges might include:

  • Downtime restrictions;
  • Encrypting data in transit and immediately when at rest on the target;
  • Schema conversion to a new target schema (the AWS Schema Conversion Tool can also help);
  • Access to databases, such as through firewalls or VPNs;
  • Developing a process to ensure that all the data successfully migrated, such as by using a hash function.

Test runs also help to give us a more accurate picture of the overall time that a migration will take, as well as affording us the opportunity to fine-tune it. Factors that may affect the overall speed of a migration include:

  • The sizes of the source and target instances;
  • Available bandwidth for moving data;
  • Schema configurations; and
  • Transaction pressure on the source, such as changes to the data and the volume of incoming transactions.

Once the duplicate application has been migrated via one or more options, we test the heck out of the application that’s now running in the cloud to ensure it performs as expected. Ideally, on the big day, we’d follow this same general process to move up-to-date duplicate data, and then seamlessly point the “real” application or web address to the new location in the cloud. This means that our customers experience near-zero downtime; essentially, only the amount of time that the change in location-pointing would need to propagate to their device.

In the case of very large or complex applications with many components or many teams working together at the same time, a more gradual approach may be more appropriate than the “Big Bang” approach, and may help to mitigate risk of any interruptions. This means migrating in stages, component by component, and running tests between stages to ensure that all parts of the application are communicating with each other as expected.

Preparation is essential to a smooth migration

I hope this article has enabled a more practical understanding of how cloud migration can be achieved. With thorough preparation, it’s possible to take advantage of all the cloud has to offer, with minimal hassle to get there.

My thanks to the AWS Solutions Architects who presented at Pop-Up Loft and shared their knowledge on these topics, in particular: Chandra Kapireddy, Stephen Moon, John Franklin, Michael Alpaugh, and Priyanka Mahankali.

One last nugget of wisdom, courtesy of John: “Friends don’t let friends use DMS to create schema objects.”

How users and applications stay safe on the Internet: it's proxy servers all the way down, 16 Sep 2019 09:35:28 (Victoria Drake) overview of how proxy servers form the basis of online anonymity, and how their use in various forms helps both users and web applications. <img src="" width="640" height="480"/>Both Internet users and Internet-connected applications can benefit from investing in cybersecurity. One core aspect of online privacy is the use of a proxy server, though this basic building block may not be initially visible underneath its more recognizable forms. Proxy servers are a useful thing to know about nowadays, for developers, software product owners, as well as the average dog on the Internet. Let’s explore what makes proxy servers an important piece of cybersecurity support.

“On the Internet, nobody knows you’re a dog.”

When Peter Steiner’s caption was first published in The New Yorker in 1993, it reportedly went largely unnoticed. Only later did the ominous and slightly frightening allusion to online anonymity touch the public consciousness with the icy fingers of the unknown. As Internet usage became more popular, users became concerned that other people could represent themselves online in any manner they chose, without anyone else knowing who they truly were.

This, to make a gross understatement, is no longer the case. Thanks to tracking cookies, browser fingerprinting, Internet Service Providers (ISPs) selling our browsing logs to advertisers, and our own inexplicable inclination to put our names and faces on social networks, online anonymity is out like last year’s LaCroix flavours. While your next-door neighbor may not know how to find you online (well, except for through that location-based secondhand marketplace app you’re using), you can be certain that at least one large advertising company has a series of zeroes and ones somewhere that represent you, the specific details of your market demographic, and all your online habits, including your preferred flavour of LaCroix.

There are ways to add some layers of obscurity, like using a corporate firewall that hides your IP, or using Tor. The underlying mechanism of both these methods is the same. Like being enshrouded in the layers of an onion, we’re using one or more proxy servers to shield our slightly sulfuric selves from third-party tracking.

What’s a proxy server, anyway

A proxy, in the traditional English definition, is the “authority or power to act for another.” (Merriam-Webster) A proxy server, in the computing context, is a server that acts on behalf of another server, or a user’s machine.

By using a proxy to browse the Internet, for example, a user can defer being personally identifiable. All of the user’s Internet traffic appears to come from the proxy server instead of their machine.

Proxy servers are for users

There are a few ways that we, as the client, can use a proxy server to conceal our identity when we go online. It’s important to know that these methods offer differing levels of anonymity, and that no single method will really provide true anonymity; if others are actively seeking to find you on the Internet, for whatever reason, further steps should be taken to make your activity truly difficult to identify. (Those steps are beyond the scope of this article, but you can get started with the Electronic Frontier Foundation’s (EFF) Surveillance Self-Defense resource.) For the average user, however, here is a small menu of options ranging from least to most anonymous.

Use a proxy in your web browser

Certain web browsers, including Firefox and Safari on Mac, allow us to configure them to send our Internet traffic through a proxy server. The proxy server attempts to anonymize our requests by replacing our originating IP address with the proxy server’s own IP. This provides us with some anonymity, as the website we’re trying to reach will not see our originating IP address; however, the proxy server that we choose to use will know exactly who originated the request. This method also doesn’t necessarily encrypt traffic, block cookies, or stop social media and cross-site trackers from following us around; on the upside, it’s the method least likely to prevent websites that use cookies from functioning properly.

A cartoon of a proxy server guarding a browser

Public proxy servers are out there, and deciding whether or not we should use any one of them is on par with deciding whether we should eat a piece of candy handed to us by a smiling stranger. If your academic institution or company provides a proxy server address, it is (hopefully) a private server with some security in place. My preferred method, if we have a little time and a few monthly dollars to invest in our security, is to set up our own virtual instance with a company such as Amazon Web Services or Digital Ocean and use this as our proxy server.

To use a proxy through our browser, we can edit our Connection Settings in Firefox, or set up a proxy server using Safari on Mac.

In regards to choosing a browser, I would happily recommend Firefox to any Internet user who wants to beef up the security of their browsing experience right out of the box. Mozilla has been a champion of privacy-first since I’ve heard of them, and recently made some well-received changes to Enhanced Tracking Protection in Firefox Browser that blocks social media trackers, cross-site tracking cookies, fingerprinters, and cryptominers by default.

Use a VPN on your device

In order to take advantage of a proxy server for all our Internet usage instead of just through one browser, we can use a Virtual Private Network (VPN). A VPN is a service, usually paid, that sends our Internet traffic through their servers, thus acting as a proxy. A VPN can be used on our laptop as well as phone and tablet devices, and since it encompasses all our Internet traffic, it doesn’t require much extra effort to use other than ensuring our device is connected. Using a VPN is an effective way to keep nosy ISPs from snooping on our requests.

A cartoon depicting a private VPN

To use a paid, third-party VPN service, we’d usually sign up on their website and download their app. It’s important to keep in mind that whichever provider we choose, we’re entrusting them with our data. VPN providers anonymize our activity from the Internet, but can themselves see all our requests. Providers vary in terms of their privacy policies and the data they choose to log, so a little research may be necessary to determine which, if any, we are comfortable trusting.

We can also roll our own VPN service by using a virtual instance and OpenVPN. OpenVPN is an open source VPN protocol, and can be used with a few virtual instance providers, such as Amazon VPC, Microsoft Azure, Google Cloud, and Digital Ocean Droplets. I previously wrote a tutorial on setting up your own personal VPN service with AWS using an EC2 instance. I’ve been running this solution personally for about a month, and it’s cost me almost $4 USD in total, which is a price I’m quite comfortable paying for some peace of mind.

Use Tor

Tor takes the anonymity offered by a proxy server and compounds it by forwarding our requests through a relay network of other servers, each called a “node.” Our traffic passes through three nodes on its way to a destination: the guard, middle, and exit nodes. At each step, the request is encrypted and anonymized such that the current node only knows where to send it, and nothing more about what the request contains. This separation of knowledge means that, of the options discussed, Tor provides the most complete version of anonymity. (For a more complete explanation, see Robert Heaton’s article on how Tor works, which is so excellently done that I wish I’d written it myself.)

Tor onion holding a Free Hugs sign

That said, this level of anonymity comes with its own cost. Not monetary, as Tor Browser is free to download and use. It is, however, slower than using a VPN or simple proxy server through a browser, due to the circuitous route our requests take.

Proxy servers are for servers too

We’re now familiar with proxy servers in the context of protecting users as they surf the web, but proxies aren’t just for clients. Websites and Internet-connected applications can use reverse proxy servers for obfuscation too. The “reverse” part just means that the proxy is acting on behalf of the server, instead of the client.

Why would a web server care about anonymity? Generally, they don’t, at least not in the same way some users do. Web servers can benefit from using a proxy for a few different reasons; for example, they typically offer faster service to users by caching or compressing content to optimize delivery. From a cybersecurity perspective, however, a reverse proxy can improve an application’s security posture by obfuscating the underlying infrastructure.

A cartoon making fun of the big words I used

Basically, by placing another web server (the “proxy”) in front of the web server that directly accesses all the files and assets, we make it more difficult for an attacker to pinpoint our “real” web server and mess with our stuff. Like when you want to see the store manager and the clerk you’re talking to says, “I speak for the manager,” and you’re not really sure there even is a manager, anyway, but you successfully exchange the hot pink My Little Pony they sold you for a fuchsia one, thankyouverymuch, so now you’re no longer concerned with who the manager is and whether or not they really exist, and if you passed them on the street you would not be able to stop them and call them out for passing off hot pink as fuchsia, and the manager is just fine with that.

Some common web servers can also act as reverse proxies, often with just a minimal and straightforward configuration change. While the best choice for your particular architecture is unknown to me, I will offer a couple common examples here.

Using NGINX as a reverse proxy

NGINX uses the proxy_pass directive in its configuration file (nginx.conf by default) to turn itself into a reverse proxy server. The set up requires the following lines to be placed in the configuration file:

location /requested/path/ {

This specifies that all requests for the path /requested/path/ are forwarded to The target can be a domain name or an IP address, the latter with or without a port.

The full guide to using NGINX as a reverse proxy is part of the NGINX documentation.

Using Apache httpd as a reverse proxy

Apache httpd similarly requires some straightforward configuration to act as a reverse proxy server. In the configuration file, usually httpd.conf, set the following directives:

ProxyPass "/requested/path/"  ""
ProxyPassReverse "/requested/path/"  ""

The ProxyPass directive ensures that all requests for the path /requested/path/ are forwarded to The ProxyPassReverse directive ensures that the headers sent by the web server are modified to point to the reverse proxy server instead.

The full reverse proxy guide for Apache HTTP server is available in their documentation.

Proxy servers most of the way down

I concede that my title is a little facetious, as cybersecurity best practices aren’t really some eternal infinite-regression mystery (though they may sometimes seem to be). Regardless, I hope this post has helped in your understanding of what proxy servers are, how they contribute to online anonymity for both clients and servers, and that they are an integral building block of cybersecurity practices.

If you’d like to learn more about personal best practices for online security, I highly recommend exploring the articles and resources provided by EFF. For a guide to securing web sites and applications, the OWASP Cheat Sheet Series is a fantastic resource.

Hackers are Googling your plain text passwords: preventing sensitive data exposure, 09 Sep 2019 09:10:11 (Victoria Drake) sensitive data controls need to be established long before you think you need them, as demonstrated by Google dorking. <img src="" width="640" height="454"/>Last week, I wrote about the importance of properly handling user input in our websites and applications. I alluded to an overarching security lesson that I hope to make explicit today: the security of our software, application, and customer data is built from the ground up, long before the product goes live.







Search Operators


  • site: will limit the search to the provided URL.








A screenshot of exposed api key in Google search

API keys in plain view.

A screenshot of exposed username and password in Google search

Log-in details out in the open.


But why





  1. what constitutes sensitive data,
  2. the difference between plain text and encrypted data, and
  3. how to properly transmit and store sensitive data.


Public repositories


A screenshot of a Google search for tokens on GitHub


But why





Pushing left to prevent sensitive data exposure


No results found in Google Search



SQL injection and XSS: what white hat hackers know about trusting user input, 02 Sep 2019 09:01:23 (Victoria Drake) primer on SQL injection and cross site scripting, and how to handle user input in software development. <img src="" width="640" height="374"/>Software developers have a lot on their minds. There are are myriad of questions to ask when it comes to creating a website or application: What technologies will we use? How will the architecture be set up? What functions do we need? What will the UI look like? Especially in a software market where shipping new apps seems more like a race for reputation than a well-considered process, one of the most important questions often falls to the bottom of the “Urgent” column: how will our product be secured?

If you’re using a robust, open-source framework for building your product (and if one is applicable and available, why wouldn’t you?) then some basic security concerns, like CSRF tokens and password encryption, may already be handled for you. Still, fast-moving developers would be well served to brush up on their knowledge of common threats and pitfalls, if only to avoid some embarrass ing rookie mistakes. Usually, the weakest point in the security of your software is you.

I’ve recently become more interested in information security in general, and practicing ethical hacking in particular. An ethical hacker, sometimes called “white hat” hacker, and sometimes just “hacker,” is someone who searches for possible security vulnerabilities and responsibly (privately) reports them to project owners. By contrast, a malicious or “black hat” hacker, also called a “cracker,” is someone who exploits these vulnerabilities for amusement or personal gain. Both white hat and black hat hackers might use the same tools and resources, and generally try to get into places they aren’t supposed to be; however, white hats do this with permission, and with the intention of fortifying defences instead of destroying them. Black hats are the bad guys.

When it comes to learning how to find security vulnerabilities, it should come as no surprise that I’ve been devouring whatever information I can get my hands on; this post is a distillation of some key areas that are specifically helpful to developers when handling user input. These lessons have been collectively gleaned from these excellent resources:

You may be familiar with the catchphrase, “sanitize your inputs!” However, as I hope this post demonstrates, developing an application with robust security isn’t quite so straightforward. I suggest an alternate phrase: pay attention to your inputs. Let’s elaborate by examining the most common attacks that take advantage of vulnerabilities in this area: SQL injection and cross site scripting.

SQL injection attacks

If you’re not yet familiar with SQL (Structured Query Language) injection attacks, or SQLi, here is a great explain-like-I’m-five video on SQLi. You may already know of this attack from xkcd’s Little Bobby Tables. Essentially, malicious actors may be able to send SQL commands that affect your application through some input on your site, like a search box that pulls results from your database. Sites coded in PHP can be especially susceptible to these, and a successful SQL attack can be devastating for software that relies on a database (as in, your Users table is now a pot of petunias).

A monitor with an SQL Select command that gets all your base

You have no chance to survive make your time.

You can test your own site to see if you’re susceptible to this kind of attack. (Please only test sites that you own, since running SQL injections where you don’t have permission to be doing so is, possibly, illegal in your locality; and definitely, universally, not very funny.) The following payloads can be used to test inputs:

  • ' OR 1='1 evaluates to a constant true, and when successful, returns all rows in the table.
  • ' AND 0='1 evaluates to a constant false, and when successful, returns no rows.

This video demonstrates the above tests, and does a great job of showing how impactful an SQL injection attack can be.

Thankfully, there are ways to mitigate SQL injection attacks, and they all boil down to one basic concept: don’t trust user input.

SQL injection mitigation

In order to effectively mitigate SQL injections, developers must prevent users from being able to successfully submit raw SQL commands to any part of the site.

Some frameworks will do most of the heavy lifting for you. For example, Django implements the concept of Object-Relational Mapping, or ORM, with its use of QuerySets. We can think of these as wrapper functions that help your application query the database using pre-defined methods that avoid the use of raw SQL.

Being able to use a framework, however, is never a guarantee. When dealing directly with a database, there are other methods we can use to safely abstract our SQL queries from user input, though they vary in efficacy. These are, by order of most to least preferred, and with links to relevant examples:

  1. Prepared statements with variable binding (or parameterized queries),
  2. Stored procedures; and
  3. Whitelisting or escaping user input.

If you want to implement the above techniques, the linked cheatsheets are a great starting point for digging deeper. Suffice to say, the use of these techniques to obtain data instead of using raw SQL queries helps to minimize the chances that SQL will be processed by any part of your application that takes input from users, thus mitigating SQL injection attacks.

The battle, however, is only half won…

Cross Site Scripting (XSS) attacks

If you’re a malicious coder, JavaScript is pretty much your best friend. The right commands will do anything a legitimate user could do (and even some things they aren’t supposed to be able to) on a web page, sometimes without any interaction on the part of an actual user. Cross Site Scripting attacks, or XSS, occur when JavaScript code is injected into a web page and changes that page’s behavior. Its effects can range from prank nuisance occurrences to more severe authentication bypasses or credential stealing. This incident report from Apache in 2010 is a good example of how XSS can be chained in a larger attack to take over accounts and machines.

An HTML dance party with a little JS cutting in

The annual DOM dance-off receives an unexpected guest);

XSS can occur on the server or on the client side, and generally comes in three flavors: DOM (Document Object Model) based, stored, and reflected XSS. The differences amount to where the attack payload is injected into the application.

DOM-based XSS

DOM-based XSS occurs when a JavaScript payload affects the structure, behavior, or content of the web page the user has loaded in their browser. These are most commonly executed through modified URLs, such as in phishing.

To see how easy it would be for injected JavaScript to manipulate a page, we can create a working example with an HTML web page. Try creating a file on your local system called xss-test.html (or whatever you like) with the following HTML and JavaScript code:

        <title>My XSS Example</title>
        <h1 id="greeting">Hello there!</h1>
                var name = new URLSearchParams('name');
                if (name !== 'null') {
                    document.getElementById('greeting').innerHTML = 'Hello ' + name + '!';

This web page will display the title “Hello there!” unless it receives a URL parameter from a query string with a value for name. To see the script work, open the page in a browser with an appended URL parameter, like so:


Fun, right? Our insecure (in the safety sense, not the emotional one) page takes the URL parameter value for name and displays it in the DOM. The page is expecting the value to be a nice friendly string, but what if we change it to something else? Since the page is owned by us and only exists on our local system, we can test it all we like. What happens if we change the name parameter to, say, <img+src+onerror=alert("pwned")>?

A screenshot of the XSS page example

This is just one example, largely based on one from Brute’s post, that demonstrates how an XSS attack could be executed. Funny pop-up alerts may be amusing, but JavaScript can do a lot of harm, including helping malicious attackers steal passwords and personal information.

Stored and reflected XSS

Stored XSS occurs when the attack payload is stored on the server, such as in a database. The attack affects a victim whenever that stored data is retrieved and rendered in the browser. For example, instead of using a URL query string, an attacker might update their profile page on a social site to include a hidden script in, say, their “About Me” section. The script, improperly stored on the site’s server, would successfully execute at a later time when another user views the attacker’s profile.

One of the most famous examples of this is the Samy worm that all but took over MySpace in 2005. It propagated by sending HTTP requests that replicated it onto a victim’s profile page whenever an infected profile was viewed. Within just 20 hours, it had spread to over a million users.

Reflected XSS similarly occurs when the injected payload travels to the server, however, the malicious code does not end up stored in a database. It is instead immediately returned to the browser by the web application. An attack like this might be executed by luring the victim to click a malicious link that sends a request to the vulnerable website’s server. The server would then send a response to the attacker as well as the victim, which may result in the attacker being able to obtain passwords, or perpetrate actions that appear to originate from the victim.

XSS attack mitigation

In all of these cases, XSS attacks can be mitigated with two key strategies: validating form fields, and avoiding the direct injection of user input on the web page.

Validating form fields

Frameworks can again help us out when it comes to making sure that user-submitted forms are on the up-and-up. One example is Django’s built-in Field classes, which provide fields that validate to some commonly used types and also specify sane defaults. Django’s EmailField, for instance, uses a set of rules to determine if the input provided is a valid email. If the submitted string has characters in it that are not typically present in email addresses, or if it doesn’t imitate the common format of an email address, then Django won’t consider the field valid and the form will not be submitted.

If relying on a framework isn’t an option, we can implement our own input validation. This can be accomplished with a few different techniques, including type conversion, for example, ensuring that a number is of type int(); checking minimum and maximum range values for numbers and lengths for strings; using a pre-defined array of choices that avoids arbitrary input, for example, months of the year; and checking data against strict regular expressions.

Thankfully, we needn’t start from scratch. Open source resources are available to help, such as the OWASP Validation Regex Repository, which provides patterns to match against for some common forms of data. Many programming languages offer validation libraries specific to their syntax, and we can find plenty of these on GitHub. Additionally, the XSS Filter Evasion Cheat Sheet has a couple suggestions for test payloads we can use to test our existing applications.

While it may seem tedious, properly implemented input validation can protect our application from being susceptible to XSS.

Avoiding direct injection

Elements of an application that directly return user input to the browser may not, on a casual inspection, be obvious. We can determine areas of our application that may be at risk by exploring a few questions:

  • How does data flow through our application?
  • What does a user expect to happen when they interact with this input?
  • Where on our page does data appear? Does it become embedded in a string or an attribute?

Here are some sample payloads that we can play with in order to test inputs on our site (again, only our own site!) courtesy of Hacker101. The successful execution of any of these samples can indicate a possible XSS vulnerability due to direct injection.

  • "><h1>test</h1>
  • '+alert(1)+'
  • "onmouserover="alert(1)
  • http://"onmouseover="alert(1)

As a general rule, if you are able to design around directly injecting input, do so. Alternatively, be sure to completely understand the effect of the methods you choose; for example, using innerText instead of innerHTML in JavaScript will ensure that content will be set as plain text instead of (potentially vulnerable) HTML.

Pay attention to your inputs

Software developers are at a marked disadvantage when it comes to competing with black hat, or malicious, hackers. For all the work we do to secure each and every input that could potentially compromise our application, an attacker need only find the one we missed. It’s like installing deadbolts on all the doors, but leaving a window open!

By learning to think along the same lines as an attacker, however, we can better prepare our software to stand up against bad actors. Exciting as it may be to ship features as quickly as possible, we’ll avoid racking up a lot of security debt if we take the time beforehand to think through our application’s flow, follow the data, and pay attention to our inputs.

How to set up OpenVPN on AWS EC2 and fix DNS leaks on Ubuntu 18.04 LTS, 26 Aug 2019 09:01:23 (Victoria Drake) guide for setting up your own private VPN service, and understanding and fixing a DNS leak. <img src="" width="640" height="388"/>Rolling your own Virtual Private Network (VPN) is complicated. I put a lot of time and effort into setting up OpenVPN on my own AWS EC2 instance, yet was still delighted to outsource the ongoing maintenance when I found my preferred privacy-focused VPN.

That said, there’s no better way to strive for maximum privacy than a VPN service you control, configure, and maintain yourself. Here’s a step-by-step tutorial for setting up your own OpenVPN on AWS EC2, and how to check for and fix DNS leaks.

Set up OpenVPN on AWS EC2

This post will cover how to set up the OpenVPN Access Server product on AWS Marketplace, running on an Amazon EC2 instance. Then, you’ll look at how to fix a known NetworkManager bug in Ubuntu 18.04 that might cause DNS leaks. The whole process should take about fifteen minutes, so grab a ☕ and let’s be configuration superheroes.

Note: IDs and IP addresses shown for demonstration in this tutorial are invalid.

1. Launch the OpenVPN Access Server on AWS Marketplace

The OpenVPN Access Server is available on AWS Marketplace. The Bring Your Own License (BYOL) model doesn’t actually require a license for up to two connected devices; to connect more clients, you can get bundled billing for five, ten, or twenty-five clients, or purchase a minimum of ten OpenVPN licenses a la carte for $15/device/year. For most of us, the two free connected devices will suffice; and if using an EC2 Micro instance, your set up will be AWS Free Tier eligible as well.

Start by clicking Continue to Subscribe for the OpenVPN Access Server, which will bring you to a page that looks like this:

Subscription details page for OpenVPN Access Server

Click Continue to Configuration.

Configure this software page for OpenVPN Access Server

You may notice that the EC2 instance type in the right side bar (and consequently, the Monthly Estimate) isn’t the one you want - that’s okay, you can change it soon. Just ensure that the Region chosen is where you want the instance to be located. Generally, the closer it is to the physical location of your client (your laptop, in this case), the faster your VPN will be. Click Continue to Launch.

Launch this software page

On this page, you’ll change three things:

1. The EC2 Instance type

Different types of EC2 (Elastic Compute Cloud) instances will offer you different levels of computing power. If you plan to use your instance for something more than just this VPN, you may want to choose something with higher memory or storage capacity, depending on how you plan to use it. You can view each instance offering on the Amazon EC2 Instance Types page.

For simple VPN use, the t2.nano or t2.micro instances are likely sufficient. Only the Micro instance is Free Tier eligible.

2. The Security Group settings

A Security Group is a profile, or collection of settings, that Amazon uses to control access to your instance. If you’ve set up other AWS products before, you may already have some groups with their own rules defined. You should be careful to understand the reasons for your Security Group settings, as these define how public or private your instance is, and consequently, who has access to it.

If you click Create New Based on Seller Settings, the OpenVPN server defines some recommended settings for a default Security Group.

Security group settings

The default recommended settings are all for TCP ports 22, 943, 443, and 945, and UDP port 1194. OpenVPN offers an explanation of how the ports are used on their website. With the default settings, all these ports are left open to support various features of the OpenVPN server. You may wish to restrict access to these ports to a specific IP address or block of addresses (like that of your own ISP) to increase the security of your instance. However, if your IP address frequently changes (like when you travel and connect to a different WiFi network), restricting the ports may not be as helpful as you hope.

In any case, your instance will require SSH keys to connect to, and the OpenVPN server will be password protected. Unless you have other specific security goals, it’s fine to accept the default settings for now.

Let’s give the Security Group a name and brief description, so you know what it’s for. Then click Save.

3. The Key Pair settings

The aforementioned SSH keys are access credentials that you’ll use to connect to your instance. You can create a key pair in this section, or you can choose a key pair you may already be using with AWS.

Key Pair Settings link

To create a new set of access credentials, click Create a key pair in EC2 to open a new window. Then, click the Create Key Pair blue button. Once you give your key pair a name, it will be created and the private key will automatically download to your machine. It’s a file ending with the extension .pem. Store this key in a secure place on your computer. You’ll need to refer to it when you connect to your new EC2 instance.

You can return to the previous window to select the key pair you just created. If it doesn’t show up, hit the little “refresh” icon next to the drop-down. Once it’s selected, hit the shiny yellow Launch button.

You should see a message like this:

Launch success message

Great stuff! Now that your instance exists, let’s make sure you can access it and start up your VPN. For a shortcut to the next step, click on the “EC2 Console” link in the success message.

2. Associate an Elastic IP

Amazon’s Elastic IP Addresses provides you with a public IPv4 address controlled by your account, unlike the public IP address tied to your EC2 instance. It’s considered a best practice to create one and associate it with your VPN instance. If anything should go wrong with your instance, or if you want to use a new instance for your VPN in the future, the Elastic IP can be disassociated from the current instance and reassociated with your new one. This makes the transition seamless for your connected clients. Think of the Elastic IP like a web domain address that you register - you can point it at whatever you choose.

We can create a new Elastic IP address on the Amazon EC2 Console. If you clicked the link from the success message above, we’re already there.

EC2 console

If you have more than one instance, take note of the Instance ID of the one you’ve just launched.

In the left sidebar under Network & Security, choose Elastic IPs. Then click the blue Allocate new address button.

Allocate new address page

Choose Amazon Pool, then click Allocate.

Allocate elastic IP success message

Success! Click Close to return to the Elastic IP console.

Associate elastic IP

Now that you have an Elastic IP, let’s associate it with your instance. Select the IP address, then click Actions, and choose Associate address.

Associate elastic IP with instance

Ensure the Instance option is selected, then click the drop-down menu. You should see your EC2 instance ID there. Select it, then click Associate.

Associate elastic IP success message

Success! Now that you’ll be able to access your VPN instance, let’s get your VPN service up and running.

3. Initialize OpenVPN on the EC2 server

First, you’ll need to connect to the EC2 instance via your terminal. You’ll use the private key you created earlier.

Open a new terminal window and navigate to the directory containing the private key .pem file. You’ll need to set its permissions with:

sudo chmod 400 <name>.pem

Be sure to substitute <name> with the name of your key.

This sets the file permissions to -r-------- so that it can only be read by the user (you). It may help to protect the private key from read and write operations by other users, but more importantly, will prevent AWS from throwing an error when you try to connect to your instance.

We can now do just that by running:

ssh -i <name>.pem openvpnas@<elastic ip>

The user openvpnas is set up by the OpenVPN Access Server to allow you to connect to your instance. Replace <elastic ip> with the Elastic IP address you just associated.

We may get a message saying that the authenticity of your host can’t be established. As long as you’ve typed the Elastic IP correctly, go ahead and answer yes to the prompt.

Upon the initial connection to the OpenVPN instance, a set up wizard called Initial Configuration Tool should automatically run. (If, for some reason, it doesn’t, or you panic-mashed a button, you can restart it with sudo ovpn-init –ec2.) You’ll be asked to accept the agreement, then the wizard will help to walk you through some configuration settings for your VPN server.

You may generally accept the default settings, however, there are a couple questions you may like to answer knowledgeably. They are:

Should client traffic be routed by default through the VPN?

Should client DNS traffic be routed by default through the VPN?

These answers depend on your privacy goals for your VPN.

When asked for your OpenVPN-AS license key, you can leave it blank to use the VPN with up to two clients. If you’ve purchased a key, enter it here.

Once the configuration wizard finishes running, you should see the message “Initial Configuration Complete!” Before you move on, you should set a password for your server’s administration account. To do this, run:

sudo passwd openvpn

Then enter your chosen password twice. Now we’re ready to get connected!

To close the SSH connection, type exit.

4. Connect the client to the VPN

To connect your client (in this case, your laptop) to the VPN and start reaping the benefits, you’ll need to do two things; first, obtain your connection profile; second, install the openvpn daemon.

1. Get your .ovpn connection profile

You’ll need to download a connection profile; this is like a personal configuration file with information, including keys, that the VPN server will need to allow your connection. You can do this by logging in with the password you just set at your Elastic IP address, port 943. This looks like:

https://<elastic ip>:943/

The https part is important; without it, the instance won’t send any data.

When you go to this URL, you may see a page warning you that this site’s certificate issuer is unknown or invalid. As long as you’ve typed your Elastic IP correctly, it’s safe to proceed. If you’re using Firefox, click Advanced, and then Accept the Risk and Continue. In Chrome, click Advanced, then Proceed to the elastic IP.

Security warning page

Log in with the username openvpn and the password you just set. You’ll now be presented with a link to download your user-locked connection profile:

Connection profile download page

When you click the link, a file named client.ovpn will download.

2. Install and start openvpn on your Ubuntu 18.04 client

The openvpn daemon will allow your client to connect to your VPN server. It can be installed through the default Ubuntu repositories. Run:

sudo apt install openvpn

In order for OpenVPN to automatically start when you boot up your computer, you’ll need to rename and move the connection profile file. I suggest using a symlink to accomplish this, as it leaves your original file more easily accessible for editing, and allows you to store it in any directory you choose. You can create a symlink by running this command in the directory where your file is located:

sudo ln -s client.ovpn /etc/openvpn/<name>.conf

This creates a symbolic link for the connection profile in the appropriate folder for systemd to find it. The <name> can be anything. When the Linux kernel has booted, systemd is used to initialize the services and daemons that the user has set up to run; one of these will now be OpenVPN. Renaming the file with the extension .conf will let the openvpn daemon know to use it as your connection file.

For now, you can manually start and connect to OpenVPN by running:

sudo openvpn --config client.ovpn

You’ll be asked for a username and password, which will be the same credentials you used before. Once the service finishes starting up, you’ll see “Initialization Sequence Complete.” If you now visit the DNS leak test website, you should see the Elastic IP and the location of your EC2 server. Yay!

If you’re on a later version of Ubuntu, you may check for DNS leaks by clicking on one of the test buttons. If all the ISPs shown are Amazon and none are your own service provider’s, congratulations! No leaks! You can move on to Step 3 in the second section below, after which, you’ll be finished.

If you’re using Ubuntu 18.04 LTS, however, we’re not yet done.

What a DNS leak looks like

Sites like the DNS leak test website can help you check your configuration and see if the Internet knows more about your location than you’d like. On the main page you’ll see a big hello, your IP address, and your location, so far as can be determined.

If you have a DNS leak, you can see what it looks like by clicking on one of the test buttons on the the DNS leak test page. When you do, you’ll see not only your IP addresses, but also your own ISP and location.

You can also see the leak by running systemd-resolve --status in your terminal. Your results will contain two lines under different interfaces that both have entries for DNS Servers. It’ll look something like this:

Link 7 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers:
          DNS Domain: ~.

Link 3 (wlp4s0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers:
          DNS Domain: ~.

The DNS leak problem in Ubuntu 18.04 stems from Ubuntu’s DNS resolver, systemd-resolved, failing to properly handle your OpenVPN configuration. In order to try and be a good, efficient DNS resolver, systemd-resolved will send DNS lookup requests in parallel to each interface that has a DNS server configuration, and then utilizes the fastest response. In your case, you only want to use your VPN’s DNS servers. Sorry, systemd-resolved. You tried.

How to fix OpenVPN DNS leak on Ubuntu 18.04

Luckily, there is a fix that you can implement. You’ll need to install a few helpers from the Ubuntu repositories, update your configuration file, then set up OpenVPN using NetworkManager. Let’s do it!

1. Install some helpers

To properly integrate OpenVPN with systemd-resolved, you’ll need a bit more help. In a terminal, run:

sudo apt install -y openvpn-systemd-resolved network-manager-openvpn network-manager-openvpn-gnome

This will install a helper script that integrates OpenVPN and systemd-resolved, a NetworkManager plugin for OpenVPN, and its GUI counterpart for GNOME desktop environment.

2. Add DNS implementation to your connection profile

You’ll need to edit the connection profile file you downloaded earlier. Since it’s symbolically linked, you can accomplish this by changing the .ovpn file, wherever it’s stored. Run vim <name>.ovpn to open it in Vim, then add the following lines at the bottom. Explanation in the comments:

# Allow OpenVPN to call user-defined scripts
script-security 2
# Tell systemd-resolved to send all DNS queries over the VPN
dhcp-option DOMAIN-ROUTE .

# Use the update-systemd-resolved script when TUN/TAP device is opened,
# and also run the script on restarts and before the TUN/TAP device is closed
up /etc/openvpn/update-systemd-resolved
down /etc/openvpn/update-systemd-resolved

For the full list of OpenVPN options, see OpenVPN Scripting and Environment Variables. You may also like more information about TUN/TAP.

3. Set up OpenVPN as NetworkManager system connection

Use the GUI to set up your VPN with NetworkManager. Open up Network Settings, which should look something like this:

Network Settings window on Ubuntu 18.04

Then click the plus sign (+) button. On the window that pops up, counterintuitively, choose Import from file… instead of the OpenVPN option.

Add VPN window

Navigate to, and then select, your .ovpn file. You should now see something like this:

The filled VPN connection settings

Add your username and password for the server (openvpn and the password you set in the first section’s Step 3), and your user key password (the same one again, if you’ve followed this tutorial), then click the “Add” button.

4. Edit your OpenVPN NetworkManager configuration

Nearly there! Now that you’ve added the VPN as a NetworkManager connection, you’ll need to make a quick change to it. You can see a list of NetworkManager connections by running:

ls -la /etc/NetworkManager/system-connections/*

The one for your VPN is probably called openvpn, so let’s edit it by running:

sudo vim /etc/NetworkManager/system-connections/openvpn

Under [ipv4], you’ll need to add the line dns-priority=-42. It should end up looking like this:

Connection settings for ipv4

Setting a negative number is a workaround that prioritizes this DNS server. The actual number is arbitrary (-1 should also work) but I like 42. ¯\_(ツ)_/¯

5. Restart, connect, profit

In a terminal, run:

sudo service network-manager restart

Then in the Network Settings, click the magic button that turns on the VPN:

Network Settings window

Finally, visit the DNS leak test website and click on Extended test to verify the fix. If everything’s working properly, you should now see a list containing only your VPN ISP.

Successful DNS leak test results

And we’re done! Congratulations on rolling your very own VPN server and stopping DNS leaks with OpenVPN. Enjoy surfing in (relative) privacy. Now your only worry at the local coffeeshop is who’s watching you surf from the seat behind you.

If you enjoyed this post, there’s a lot more where it came from! I write about computing, cybersecurity, and leading great technical teams. Subscribe on to see new articles first.

How to do twice as much with half the keystrokes using `.bashrc`, 21 Aug 2019 09:17:02 (Victoria Drake) overview of time-saving aliases, functions, and creating a useful Bash prompt. <img src="" width="640" height="245"/>In my recent post about setting up Ubuntu with Bash scripts, I briefly alluded to the magic of .bashrc. This didn’t really do it justice, so here’s a quick post that offers a bit more detail about what the Bash configuration file can do.

My current configuration hugely improves my workflow, and saves me well over 50% of the keystrokes I would have to employ without it! Let’s look at some examples of aliases, functions, and prompt configurations that can improve our workflow by helping us be more efficient with fewer key presses.

Bash aliases

A smartly written .bashrc can save a whole lot of keystrokes. You can take advantage of this in the literal sense by using bash aliases, or strings that expand to larger commands. For an indicative example, here is a Bash alias for copying files in the terminal:

# Always copy contents of directories (r)ecursively and explain (v) what was done
alias cp='cp -rv'

The alias command defines the string you’ll type, followed by what that string will expand to. You can override existing commands like cp above. On its own, the cp command will only copy files, not directories, and succeeds silently. With this alias, you need not remember to pass those two flags, nor cd or ls the location of our copied file to confirm that it’s there! Now, just those two key presses (for c and d) will do all of that for us.

Here are a few more .bashrc aliases for passing flags with common functions.

# List contents with colors for file types, (A)lmost all hidden files (without . and ..), in (C)olumns, with class indicators (F)
alias ls='ls --color=auto -ACF'
# List contents with colors for file types, (a)ll hidden entries (including . and ..), use (l)ong listing format, with class indicators (F)
alias ll='ls --color=auto -alF'

# Explain (v) what was done when moving a file
alias mv='mv -v'
# Create any non-existent (p)arent directories and explain (v) what was done
alias mkdir='mkdir -pv'
# Always try to (c)ontinue getting a partially-downloaded file
alias wget='wget -c'

Aliases come in handy when you want to avoid typing long commands, too. Here are a few I use when working with Python environments:

alias pym='python3'
alias mkenv='python3 -m venv env'
alias startenv='source env/bin/activate && which python3'
alias stopenv='deactivate'

For further inspiration on ways Bash aliases can save time, I highly recommend the examples in this article.

Bash functions

One downside of the aliases above is that they’re rather static - they’ll always expand to exactly the text declared. For a Bash alias that takes arguments, you’ll need to create a function. You can do this like so:

# Show contents of the directory after changing to it
function cd () {
    builtin cd "$1"
    ls -ACF

I can’t begin to tally how many times I’ve typed cd and then ls immediately after to see the contents of the directory I’m now in. With this function set up, it all happens with just those two letters! The function takes the first argument, $1, as the location to change directory to, then prints the contents of that directory in nicely formatted columns with file type indicators. The builtin part is necessary to get Bash to allow us to override this default command.

Bash functions are very useful when it comes to downloading or upgrading software, too.

Bash function for downloading extended Hugo

Thanks to the static site generator Hugo’s excellent ship frequency, I previously spent at least a few minutes every couple weeks downloading the new extended version. With a Bash function, I only need to pass in the version number, and the upgrade happens in a few seconds.

# Hugo install or upgrade
function gethugo () {
    wget -q -P tmp/"$@"/hugo_extended_"$@"_Linux-64bit.tar.gz
    tar xf tmp/hugo_extended_"$@"_Linux-64bit.tar.gz -C tmp/
    sudo mv -f tmp/hugo /usr/local/bin/
    rm -rf tmp/
    hugo version

The $@ notation simply takes all the arguments given, replacing its spot in the function. To run the above function and download Hugo version 0.57.2, you use the command gethugo 0.57.2.

Bash function for downloading a specific Go version

I’ve got one for Golang, too:

function getgolang () {
    sudo rm -rf /usr/local/go
    wget -q -P tmp/"$@".linux-amd64.tar.gz
    sudo tar -C /usr/local -xzf tmp/go"$@".linux-amd64.tar.gz
    rm -rf tmp/
    go version

Bash function for adding a GitLab remote

Or how about a function that adds a remote origin URL for GitLab to the current repository?

function glab () {
    git remote set-url origin --add"$@"/"${PWD##*/}".git
    git remote -v

With glab username, you can create a new origin URL for the current Git repository with our username on Pushing to a new remote URL automatically creates a new private GitLab repository, so this is a useful shortcut for creating backups!

Bash functions are really only limited by the possibilities of scripting, of which there are, practically, few limits. If there’s anything you do on a frequent basis that requires typing a few lines into a terminal, you can probably create a Bash function for it!

Bash prompt

Besides directory contents, it’s also useful to see the full path of the directory we’re in. The Bash prompt can show us this path, along with other useful information like our current Git branch. To make it more readable, you can define colours for each part of the prompt. Here’s how you can set up our prompt in .bashrc to accomplish this:

# Colour codes are cumbersome, so let's name them
txtcyn='\[\e[0;96m\]' # Cyan
txtpur='\[\e[0;35m\]' # Purple
txtwht='\[\e[0;37m\]' # White
txtrst='\[\e[0m\]'    # Text Reset

# Which (C)olour for what part of the prompt?

# Get the name of our branch and put parenthesis around it
gitBranch() {

# Build the prompt
export PS1="${pathC}\w ${gitC}\$(gitBranch) ${pointerC}\$${normalC} "


~/github/myrepo (master) $

Naming the colours helps to easily identify where one colour starts and stops, and where the next one begins. The prompt that you see in our terminal is defined by the string following export PS1, with each component of the prompt set with an escape sequence. Let’s break that down:

  • \w displays the current working directory,
  • \$(gitBranch) calls the gitBranch function defined above, which displays the current Git branch,
  • \$ will display a “$” if you are a normal user or in normal user mode, and a “#” if you are root.

The full list of Bash escape sequences can help us display many more bits of information, including even the time and date! Bash prompts are highly customizable and individual, so feel free to set it up any way you please.

Here are a few options that put information front and centre and can help us to work more efficiently.

For the procrastination-averse

Username and current time with seconds, in 24-hour HH:MM:SS format:

export PS1="${userC}\u ${normalC}at \t >"
user at 09:35:55 >

For those who always like to know where they stand

Full file path on a separate line, and username:

export PS1="${pathC}\w${normalC}\n\u:"

For the minimalist

export PS1=">"

We can build many practical prompts with just the basic escape sequences; once you start to integrate functions with prompts, as in the Git branch example, things can get really complicated. Whether this amount of complication is an addition or a detriment to your productivity, only you can know for sure!

Many fancy Bash prompts are possible with programs readily available with a quick search. I’ve intentionally not provided samples here because, well, if you can tend to get as excited about this stuff as I can, it might be a couple hours before you get back to what you were doing before you started reading this post, and I just can’t have that on my conscience. 🥺

We’ve hopefully struck a nice balance now between time invested and usefulness gained from our Bash configuration file! I hope you use your newly-recovered keystroke capacity for good.

How to set up a fresh Ubuntu desktop using only dotfiles and bash scripts, 19 Aug 2019 07:58:18 (Victoria Drake) settings, install programs, and customize your desktop environment with a single bash command. <img src="" width="640" height="354"/>One of my most favorite things about open source files on GitHub is the ability to see how others do (what some people might call) mundane things, like set up their .bashrc and other dotfiles. While I’m not as enthusiastic about ricing as I was when I first came to the Linux side, I still get pretty excited when I find a config setting that makes things prettier and faster, and thus, better.

I recently came across a few such things, particularly in Tom Hudson’s dotfiles. Tom seems to like to script things, and some of those things include automatically setting up symlinks, and installing Ubuntu repository applications and other programs. This got me thinking. Could I automate the set up of a new machine to replicate my current one?

Being someone generally inclined to take things apart in order to see how they work, I know I’ve messed up my laptop on occasion. (Usually when I’m away from home, and my backup hard drive isn’t.) On those rare but really inconvenient situations when my computer becomes a shell of its former self, (ba-dum-ching) it’d be quite nice to have a fast, simple way of putting Humpty Dumpty back together again, just the way I like.

In contrast to creating a disk image and restoring it later, a collection of bash scripts is easier to create, maintain, and move around. They require no special utilities, only an external transportation method. It’s like passing along the recipe, instead of the whole bundt cake. (Mmm, cake.)

Additionally, functionality like this would be super useful when setting up a virtual machine, or VM, or even just a virtual private server, or VPS. (Both of which, now that I write this, would probably make more forgiving targets for my more destructive experiments… live and learn!)

Well, after some grepping and Googling and digging around, I now have a suite of scripts that can do this:

This is the tail end of a test run of the set up scripts on a fresh Ubuntu desktop, loaded off a bootable USB. It had all my programs and settings restored in under three minutes!

This post will cover how to achieve the automatic set up of a computer running Ubuntu Desktop using bash scripts. This exact process was last used on Ubuntu 19.10; see my dotfiles master branch for the latest configuration. The majority of the information covered is applicable to all the Linux desktop flavours, though some syntax may differ. The bash scripts cover three main areas: linking dotfiles, installing software from Ubuntu and elsewhere, and setting up the desktop environment. We’ll cover each of these areas and go over the important bits so that you can begin to craft your own scripts.


Dotfiles are what most Linux enthusiasts call configuration files. They typically live in the user’s home directory (denoted in bash scripts with the builtin variable $HOME) and control the appearance and behavior of all kinds of programs. The file names begin with ., which denotes hidden files in Linux (hence “dot” files). Here are some common dotfiles and ways in which they’re useful.


The .bashrc file is a list of commands executed at startup by interactive, non-login shells. Interactive vs non-interactive shells can be a little confusing, but aren’t necessary for us to worry about here. For our purposes, any time you open a new terminal, see a prompt, and can type commands into it, your .bashrc was executed.

Lines in this file can help improve your workflow by creating aliases that reduce keystrokes, or by displaying a helpful prompt with useful information. It can even run user-created programs, like Eddie. For more ideas, you can have a look at my .bashrc file on GitHub.


The .vimrc dotfile configures the champion of all text editors, Vim. (If you haven’t yet wielded the powers of the keyboard shortcuts, I highly recommend a fun game to learn Vim with.)

In .vimrc, we can set editor preferences such as display settings, colours, and custom keyboard shortcuts. You can take a look at my .vimrc on GitHub.

Other dotfiles may be useful depending on the programs you use, such as .gitconfig or .tmux.conf. Exploring dotfiles on GitHub is a great way to get a sense of what’s available and useful to you!

Linking dotfiles

We can use a script to create symbolic links, or symlinks for all our dotfiles. This allows us to keep all the files in a central repository, where they can easily be managed, while also providing a sort of placeholder in the spot that our programs expect the configuration file to be found. This is typically, but not always, the user home directory. For example, since I store my dotfiles on GitHub, I keep them in a directory with a path like ~/github/dotfiles/ while the files themselves are symlinked, resulting in a path like ~/.vimrc.

To programmatically check for and handle any existing files and symlinks, then create new ones, we can use this elegant shell script. I compliment it only because I blatantly stole the core of it from Tom’s setup script, so I can’t take the credit for how lovely it is.

The script works by attempting to create symlinks for each dotfile in our $HOME. It first checks to see if a symlink already exists, or if a regular file or directory with the same name exists. In the former case, the symlink is removed and remade; in the latter, the file or directory is renamed, then the symlink is made.

Installing software

One of the beautiful things about exploring shell scripts is discovering how much can be achieved using only the command line. As someone whose first exposure to computers was through a graphical operating system, I find working in the terminal to be refreshingly fast.

With Ubuntu, most programs we likely require are available through the default Ubuntu software repositories. We typically search for these with the command apt search <program> and install them with sudo apt install <program>. Some software we’d like may not be in the default repositories, or may not be offered there in the most current version. In these cases, we can still install these programs in Ubuntu using a PPA, or Personal Package Archive. We’ll just have to be careful that the PPAs we choose are from the official sources.

If a program we’d like doesn’t appear in the default repositories or doesn’t seem to have a PPA, we may still be able to install it via command line. A quick search for “ installation command line” should get some answers.

Since bash scripts are just a collection of commands that we could run individually in the terminal, creating a script to install all our desired programs is as straightforward as putting all the commands into a script file. I chose to organize my installation scripts between the default repositories, which are installed by my script, and programs that involve external sources, handled with my script.

Setting up the desktop environment

On the recent occasions when I’ve gotten a fresh desktop (intentionally or otherwise) I always seem to forget how long it takes to remember, find, and then change all the desktop environment settings. Keyboard shortcuts, workspaces, sound settings, night mode… it adds up!

Thankfully, all these settings have to be stored somewhere in a non-graphical format, which means that if we can discover how that’s done, we can likely find a way to easily manipulate the settings with a bash script. Lo and behold the terminal command, gsettings list-recursively.

There are a heck of a lot of settings for GNOME desktop environment. We can make the list easier to scroll through (if, like me, you’re sometimes the type of person to say “Just let me look at everything and figure out what I want!") by piping to less: gsettings list-recursively | less. Alternatively, if we have an inkling as to what we might be looking for, we can use grep: gsettings list-recursively | grep 'keyboard'.

We can manipulate our settings with the gsettings set command. It can sometimes be difficult to find the syntax for the setting we want, so when we’re first building our script, I recommend using the GUI to make the changes, then finding the gsettings line we changed and recording its value.

For some inspiration, you can view my settings script on GitHub.

Putting it all together

Having modular scripts (one for symlinks, two for installing programs, another for desktop settings) is useful for both keeping things organized and for being able to run some but not all of the automated set up. For instance, if I were to set up a VPS in which I only use the command line, I wouldn’t need to bother with installing graphical programs or desktop settings.

In cases where I do want to run all the scripts, however, doing so one-by-one is a little tedious. Thankfully, since bash scripts can themselves be run by terminal commands, we can simply write another master script to run them all!

Here’s my master script to handle the set up of a new Ubuntu desktop machine:



## Get all upgrades
sudo apt upgrade -y

## See our bash changes
source ~/.bashrc

## Fun hello
figlet "... and we're back!" | lolcat

I threw in the upgrade line for good measure. It will make sure that the programs installed on our fresh desktop have the latest updates. Now a simple, single bash command will take care of everything!

You may have noticed that, while our desktop now looks and runs familiarly, these scripts don’t cover one very important area: our files. Hopefully, you have a back up method for those that involves some form of reliable external hardware. If not, and if you tend to put your work in external repository hosts like GitHub or GitLab, I do have a way to automatically clone and back up your GitHub repositories with bash one-liners.

Relying on external repository hosts doesn’t offer 100% coverage, however. Files that you wouldn’t put in an externally hosted repository (private or otherwise) consequently can’t be pulled. Git ignored objects that can’t be generated from included files, like private keys and secrets, will not be recreated. Those files, however, are likely small enough that you could fit a whole bunch on a couple encrypted USB flash drives (and if you don’t have private key backups, maybe you ought to do that first?).

That said, I hope this post has given you at least some inspiration as to how dotfiles and bash scripts can help to automate setting up a fresh desktop. If you come up with some settings you find useful, please help others discover them by sharing your dotfiles, too!

How to write Bash one-liners for cloning and managing GitHub and GitLab repositories, 06 Aug 2019 10:55:19 (Victoria Drake) xargs and awk in Bash to automate managing remote-hosted repositories. <img src="" width="640" height="343"/>Few things are more satisfying to me than one elegant line of Bash that automates hours of tedious work. As part of some recent explorations into automatically re-creating my laptop with Bash scripts, I wanted to find a way to easily clone my GitHub-hosted repositories to a new machine. After a bit of digging around, I wrote a one-liner that did just that. Then, in the spirit of not putting all our eggs in the same basket, I wrote another one-liner to automatically create and push to GitLab-hosted backups as well. Here they are.

A Bash one-liner to clone all your GitHub repositories

Caveat: you’ll need a list of the GitHub repositories you want to clone. The good thing about that is it gives you full agency to choose just the repositories you want on your machine, instead of going in whole-hog.

You can easily clone GitHub repositories without entering your password each time by using HTTPS with your 15-minute cached credentials or, my preferred method, by connecting to GitHub with SSH. For brevity I’ll assume we’re going with the latter, and our SSH keys are set up.

Given a list of GitHub URLs in the file gh-repos.txt, like this:

We run:

xargs -n1 git clone < gh-repos.txt

This clones all the repositories on the list into the current folder. This same one-liner works for GitLab repositories as well, if you substitute the appropriate URLs.

What’s going on here

There are two halves to this one-liner: the input, counterintuitively on the right side, and the part that makes stuff happen, on the left. We could make the order of these parts more intuitive (maybe?) by writing the same command like this:

<gh-repos.txt xargs -n1 git clone

To run a command for each line of our input, gh-repos.txt, we use xargs -n1. The tool xargs reads items from input and executes any commands it finds (it will echo if it doesn’t find any). By default, it assumes that items are separated by spaces; new lines also works and makes our list easier to read. The flag -n1 tells xargs to use 1 argument, or in our case, one line, per command. We build our command with git clone, which xargs then executes for each line. Ta-da.

A Bash one-liner to create and push many repositories on GitLab

GitLab, unlike GitHub, lets us do this nifty thing where we don’t have to use the website to make a new repository first. We can create a new GitLab repository from our terminal. The newly created repository defaults to being set as Private, so if we want to make it Public on GitLab, we’ll have to do that manually later.

The GitLab docs tell us to push to create a new project using git push --set-upstream, but I don’t find this to be very convenient for using GitLab as a backup. As I work with my repositories in the future, I’d like to run one command that pushes to both GitHub and GitLab without additional effort on my part.

To make this Bash one-liner work, we’ll also need a list of repository URLs for GitLab (ones that don’t exist yet). We can easily do this by copying our GitHub repository list, opening it up with Vim, and doing a search-and-replace:

cp gh-repos.txt gl-repos.txt
vim gl-repos.txt

This produces gl-repos.txt, which looks like:

We can create these repositories on GitLab, add the URLs as remotes, and push our code to the new repositories by running:

awk -F'\/|(\.git)' '{system("cd ~/FULL/PATH/" $2 " && git remote set-url origin --add " $0 " && git push")}' gl-repos.txt

Hang tight and I’ll explain it; for now, take note that ~/FULL/PATH/ should be the full path to the directory containing our GitHub repositories.

We do have to make note of a couple assumptions:

  1. The name of the directory on your local machine that contains the repository is the same as the name of the repository in the URL (this will be the case if it was cloned with the one-liner above);
  2. Each repository is currently checked out to the branch you want pushed, ie. master.

The one-liner could be expanded to handle these assumptions, but it is the humble opinion of the author that at that point, we really ought to be writing a Bash script.

What’s going on here

Our Bash one-liner uses each line (or URL) in the gl-repos.txt file as input. With awk, it splits off the name of the directory containing the repository on our local machine, and uses these pieces of information to build our larger command. If we were to print the output of awk, we’d see:

cd ~/FULL/PATH/first-repository && git remote set-url origin --add && git push
cd ~/FULL/PATH/second-repository && git remote set-url origin --add && git push
cd ~/FULL/PATH/third-repository && git remote set-url origin --add && git push

Let’s look at how we build this command.

Splitting strings with awk

The tool awk can split input based on field separators. The default separator is a whitespace character, but we can change this by passing the -F flag. Besides single characters, we can also use a regular expression field separator. Since our repository URLs have a set format, we can grab the repository names by asking for the substring between the slash character / and the end of the URL, .git.

One way to accomplish this is with our regex \/|(\.git):

  • \/ is an escaped / character;
  • | means “or”, telling awk to match either expression;
  • (\.git) is the capture group at the end of our URL that matches “.git”, with an escaped . character. This is a bit of a cheat, as “.git” isn’t strictly splitting anything (there’s nothing on the other side) but it’s an easy way for us to take this bit off.

Once we’ve told awk where to split, we can grab the right substring with the field operator. We refer to our fields with a $ character, then by the field’s column number. In our example, we want the second field, $2. Here’s what all the substrings look like:

2: first-repository

To use the whole string, or in our case, the whole URL, we use the field operator $0. To write the command, we just substitute the field operators for the repository name and URL. Running this with print as we’re building it can help to make sure we’ve got all the spaces right.

awk -F'\/|(\.git)' '{print "cd ~/FULL/PATH/" $2 " && git remote set-url origin --add " $0 " && git push"}' gl-repos.txt

Running the command

We build our command inside the parenthesis of system(). By using this as the output of awk, each command will run as soon as it is built and output. The system() function creates a child process that executes our command, then returns once the command is completed. In plain English, this lets us perform the Git commands on each repository, one-by-one, without breaking from our main process in which awk is doing things with our input file. Here’s our final command again, all put together.


Using our backups

By adding the GitLab URLs as remotes, we’ve simplified the process of pushing to both externally hosted repositories. If we run git remote -v in one of our repository directories, we’ll see:

origin (fetch)
origin (push)
origin (push)

Now, simply running git push without arguments will push the current branch to both remote repositories.

We should also note that git pull will generally only try to pull from the remote repository you originally cloned from (the URL marked (fetch) in our example above). Pulling from multiple Git repositories at the same time is possible, but complicated, and beyond the scope of this post. Here’s an explanation of pushing and pulling to multiple remotes to help get you started, if you’re curious. The Git documentation on remotes may also be helpful.

To elaborate on the succinctness of Bash one-liners

Bash one-liners, when understood, can be fun and handy shortcuts. At the very least, being aware of tools like xargs and awk can help to automate and alleviate a lot of tediousness in our work. However, there are some downsides.

In terms of an easy-to-understand, maintainable, and approachable tool, Bash one-liners suck. They’re usually more complicated to write than a Bash script using if or while loops, and certainly more complicated to read. It’s likely that when we write them, we’ll miss a single quote or closing parenthesis somewhere; and as I hope this post demonstrates, they can take quite a bit of explaining, too. So why use them?

Imagine reading a recipe for baking a cake, step by step. You understand the methods and ingredients, and gather your supplies. Then, as you think about it, you begin to realize that if you just throw all the ingredients at the oven in precisely the right order, a cake will instantly materialize. You try it, and it works!

That would be pretty satisfying, wouldn’t it?

A quick guide to changing your GitHub username, 28 Jul 2019 15:19:13 (Victoria Drake) additional steps to consider after making a change to your username on GitHub. <img src="" width="640" height="421"/>This being the 2,38947234th and probably last time I’ll change my username, (marriage is permanent, right?) I thought I’d better write a quick post on how this transition can be achieved as smoothly as possible. You can read official instructions on how to change your GitHub username here, and they will tell you how to do it and what happens. The following is a quick guide to some things to consider afterwards.

Where to make changes

  1. Change username in GitHub account settings.

Local file updates


  • (Links to your GitHub Page)
  • (Git config remote ssh urls)
  • (Travis badges in READMEs)


grep -rnw -e 'foobar'



Enjoy your new handle! (I hope it sticks.)

Two ways to deploy a public GitHub Pages site from a private Hugo repository, 22 Apr 2019 10:05:15 (Victoria Drake) your drafts out of the public eye by making use of continuous deployment tools to publish your public GitHub Pages site - from a separate private repository. <img src="" width="640" height="370"/>Tools like Travis CI and Netlify offer some pretty nifty features, like seamlessly deploying your GitHub Pages site when changes are pushed to its repository. Along with a static site generator like Hugo, keeping a blog up to date is pretty painless.

I’ve used Hugo to build my site for years, but until this past week I’d never hooked up my Pages repository to any deployment service. Why? Because using a tool that built my site before deploying it seemed to require having the whole recipe in one place - and if you’re using GitHub Pages with the free version of GitHub, that place is public. That means that all my three-in-the-morning bright ideas and messy unfinished (and unfunny) drafts would be publicly available - and no amount of continuous convenience was going to convince me to do that.

So I kept things separated, with Hugo’s messy behind-the-scenes stuff in a local Git repository, and the generated public/ folder pushing to my GitHub Pages remote repository. Each time I wanted to deploy my site, I’d have to get on my laptop and hugo to build my site, then cd public/ && git add . && git commit… etc etc. And all was well, except for the nagging feeling that there was a better way to do this.

I wrote another article a little while back about using GitHub and Working Copy to make changes to my repositories on my iPad whenever I’m out and about. It seemed off to me that I could do everything except deploy my site from my iPad, so I set out to change that.

A couple three-in-the-morning bright ideas and a revoked access token later (oops), I now have not one but two ways to deploy to my public GitHub Pages repository from an entirely separated, private GitHub repository. In this post, I’ll take you through achieving this with Travis CI or using Netlify and Make.

There’s nothing hackish about it - my public GitHub Pages repository still looks the same as it does when I pushed to it locally from my terminal. Only now, I’m able to take advantage of a couple great deployment tools to have the site update whenever I push to my private repo, whether I’m on my laptop or out and about with my iPad.

Hashtag: you did not push from there


This article assumes you have working knowledge of Git and GitHub Pages. If not, you may like to spin off some browser tabs from my articles on using GitHub and Working Copy and building a site with Hugo and GitHub Pages first.

Let’s do it!

Private-to-public GitHub Pages deployment with Travis CI

Travis CI has the built-in ability (♪) to deploy to GitHub Pages following a successful build. They do a decent job in the docs of explaining how to add this feature, especially if you’ve used Travis CI before… which I haven’t. Don’t worry, I did the bulk of the figuring-things-out for you.

  • Travis CI gets all its instructions from a configuration file in the root of your repository called .travis.yml
  • You need to provide a GitHub personal access token as a secure encrypted variable, which you can generate using travis on the command line
  • Once your script successfully finishes doing what you’ve told it to do (not necessarily what you want it to do but that’s a whole other blog post), Travis will deploy your build directory to a repository you can specify with the repo configuration variable.

Setting up the Travis configuration file

Create a new configuration file for Travis with the filename .travis.yml (note the leading “."). These scripts are very customizable and I struggled to find a relevant example to use as a starting point - luckily, you don’t have that problem!

Here’s my basic .travis.yml:

  depth: false

    - HUGO_VERSION="0.54.0"

  - wget -q${HUGO_VERSION}/hugo_${HUGO_VERSION}_Linux-64bit.tar.gz
  - tar xf hugo_${HUGO_VERSION}_Linux-64bit.tar.gz
  - mv hugo ~/bin/

  - hugo --gc --minify

  provider: pages
  skip-cleanup: true
  github-token: $GITHUB_TOKEN
  keep-history: true
  local-dir: public
  repo: gh-username/
  target-branch: master
  verbose: true
    branch: master

This script downloads and installs Hugo, builds the site with the garbage collection and minify flags, then deploys the public/ directory to the specified repo - in this example, your public GitHub Pages repository. You can read about each of the deploy configuration options here.

To add the GitHub personal access token as an encrypted variable, you don’t need to manually edit your .travis.yml. The travis gem commands below will encrypt and add the variable for you when you run them in your repository directory.

First, install travis with sudo gem install travis.

Then generate your GitHub personal access token, copy it (it only shows up once!) and run the commands below in your repository root, substituting your token for the kisses:

travis login --pro --github-token xxxxxxxxxxxxxxxxxxxxxxxxxxx
travis encrypt GITHUB_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxx --add env.matrix

Your encrypted token magically appears in the file. Once you’ve committed .travis.yml to your private Hugo repository, Travis CI will run the script and if the build succeeds, will deploy your site to your public GitHub Pages repo. Magic!

Travis will always run a build each time you push to your private repository. If you don’t want to trigger this behavior with a particular commit, add the skip command to your commit message.

Yo that’s cool but I like Netlify.

Okay fine.

Deploying to a separate repository with Netlify and Make

We can get Netlify to do our bidding by using a Makefile, which we’ll run with Netlify’s build command.

Here’s what our Makefile looks like:


.PHONY: all
all: clean get_repository build deploy

.PHONY: clean
 @echo "Removing public directory"

.PHONY: get_repository
 @echo "Getting public repository"
 git clone public

.PHONY: build
 @echo "Generating site"
 hugo --gc --minify

.PHONY: deploy
 @echo "Preparing commit"
 @cd $(OUTPUTDIR) \
 && git config "" \
 && git config "Your Name" \
 && git add . \
 && git status \
 && git commit -m "Deploy via Makefile" \
 && git push -f -q https://$(GITHUB_TOKEN) master

 @echo "Pushed to remote"

To preserve the Git history of our separate GitHub Pages repository, we’ll first clone it, build our new Hugo site to it, and then push it back to the Pages repository. This script first removes any existing public/ folder that might contain files or a Git history. It then clones our Pages repository to public/, builds our Hugo site (essentially updating the files in public/), then takes care of committing the new site to the Pages repository.

In the deploy section, you’ll notice lines starting with &&. These are chained commands. Since Make invokes a new sub-shell for each line, it starts over with every new line from our root directory. To get our cd to stick and avoid running our Git commands in the project root directory, we’re chaining the commands and using the backslash character to break long lines for readability.

By chaining our commands, we’re able to configure our Git identity, add all our updated files, and create a commit for our Pages repository.

Similarly to using Travis CI, we’ll need to pass in a GitHub personal access token to push to our public GitHub Pages repository - only Netlify doesn’t provide a straightforward way to encrypt the token in our Makefile.

Instead, we’ll use Netlify’s Build Environment Variables, which live safely in our site settings in the Netlify app. We can then call our token variable in the Makefile. We use it to push (quietly, to avoid printing the token in logs) to our Pages repository by passing it in the remote URL.

To avoid printing the token in Netlify’s logs, we suppress recipe echoing for that line with the leading @ character.

With your Makefile in the root of your private GitHub repository, you can set up Netlify to run it for you.

Setting up Netlify

Getting set up with Netlify via the web UI is straightforward. Once you sign in with GitHub, choose the private GitHub repository where your Hugo site lives. The next page Netlify takes you to lets you enter deploy settings:

Create a new site page

You can specify the build command that will run your Makefile (make all for this example). The branch to deploy and the publish directory don’t matter too much in our specific case, since we’re only concerned with pushing to a separate repository. You can enter the typical master deploy branch and public publish directory.

Under “Advanced build settings” click “New variable” to add your GitHub personal access token as a Build Environment Variable. In our example, the variable name is GITHUB_TOKEN. Click “Deploy site” to make the magic happen.

If you’ve already previously set up your repository with Netlify, find the settings for Continuous Deployment under Settings > Build & deploy.

Netlify will build your site each time you push to the private repository. If you don’t want a particular commit to trigger a build, add [skip ci] in your Git commit message.

Same same but different

One effect of using Netlify this way is that your site will be built in two places: one is the separate, public GitHub Pages repository that the Makefile pushes to, and the other is your Netlify site that deploys on their CDN from your linked private GitHub repository. The latter is useful if you’re going to play with Deploy Previews and other Netlify features, but those are outside the scope of this post.

The main point is that your GitHub Pages site is now updated in your public repo. Yay!

Go forth and deploy fearlessly

I hope the effect of this new information is that you feel more able to update your sites, wherever you happen to be. The possibilities are endless - at home on your couch with your laptop, out cafe-hopping with your iPad, or in the middle of a first date on your phone. Endless!

Don't update your site from your phone on a date

Don’t do stuff on your phone when you’re on a date. Not if you want a second one, anyway.

A remote sync solution for iOS and Linux: Git and Working Copy, 15 Mar 2019 11:55:28 (Victoria Drake) to set up a cross-platform solution for working with Git on iOS.I’m always looking for pockets of time in which I can be productive. If you add up the minutes you spend in limbo while waiting in line, commuting, or waiting for food delivery (just me?), you may just find an extra hour or two in your day.

To take full advantage of these bits of time, I needed a solution that let me pick up work on my Git repositories wherever I happen to be. That means a remote sync solution that bridges my iOS devices (iPad and iPhone) and my Linux machine.

After a lot of trial and error, I’ve found one that works really well. With synced Git repositories on iOS, I can seamlessly pick up work for any of my repositories on the go. This level of convenience has earned a permanent spot in the list of *ware I use.


  • Working Copy app ($15.99 one-time pro-unlock and well worth it)
  • iA Writer app ($8.99 one-time purchase for iOS, also available on Mac, Windows, and Android)
  • GitHub repositories

Get set up

Here are the steps to setting up that I’ll walk you through in this article.

  1. Create your remote repository
  2. Clone repository to iPad with Working Copy
  3. Open and edit files with iA Writer
  4. Push changes back to remote
  5. Pull changes from repository on your computer

This system is straightforward to set up whether you’re a command line whiz or just getting into Git. Let’s do it!

Create your remote repository

Create a public or private repository on GitHub.

Selection options for public and private repository

If you’re creating a new repository, you can follow GitHub’s instructions to push some files to it from your computer, or you can add files later from your iOS device.

Clone repository to iOS with Working Copy

Download Working Copy from the App Store. It’s a fantastic app. Developer Anders Borum has a steady track record of frequent updates and incorporating the latest features for iOS apps, like drag and drop on iPad. I think he’s fairly priced his product in light of the work he puts into maintaining and enhancing it.

In Working Copy, find the gear icon in the top left corner and touch to open Settings.

Settings menu in Working Copy

Tap on SSH Keys, and you’ll see this screen:

SSH Key for Working Copy on iPad

SSH keys, or Secure Shell keys, are access credentials used in the SSH protocol. Your key is a password that your device will use to securely connect with your remote repository host - GitHub, in this example. Since anyone with your SSH keys can potentially pretend to be you and gain access to your files, it’s important not to share them accidentally, like in a screenshot on a blog post.

Tap on the second line that looks like WorkingCopy@iPad-xxxxxxxx to get this screen:

Connect to GitHub or Bitbucket in Working Copy

Working Copy supports easy connection to GitHub. Tap Connect With GitHub to bring up some familiar sign-in screens that will authorize Working Copy to access your account(s).

Once connected, tap the + symbol in the top right of the side bar to add a new repository. Choose Clone repository to bring up this screen:

Loading repositories from remote

Here, you can either manually input the remote URL, or simply choose from the list of repositories that Working Copy fetches from your connected account. When you make your choice, the app clones the repository to your device and it will show up in the sidebar. You’re connected!

Open and edit files with iA Writer

One of the (many) reasons I adore iA Writer is its ability to select your freshly cloned remote repository as a Library Location. To enable this, first open your Files app. On the Browse screen, tap the overflow menu (three dots) in the top right and choose Edit.

Screenshot of the overflow menu

Turn on Working Copy as a location option:

Screenshot of toggle switch for Working Copy

Then in the iA Writer app:

  1. From the main Library list, in the top right of the sidebar, tap Edit.
  2. Tap Add Location….
  3. A helpful popup appears. Tap OK.
  4. From the Working Copy location, tap Select in the top right, then choose the repository folder.
  5. Tap Open, then Done.

Your remote repository now appears as a Location in the sidebar. Tap on it to work within this directory.

While inside this location, new files you create (by tapping the pencil-and-paper icon in the top right corner) will be saved to this folder locally. As you work, iA Writer automatically saves your progress. Next, we’ll look at pushing those files and changes back to your remote.

Push changes back to remote

Once you’ve made changes to your files, open Working Copy again. You should see a yellow dot on your changed repository.

Yellow dot indicating changes to repository

Tap on your repository name, then on Repository Status and Configuration at the top of the sidebar. Your changed files will be indicated by yellow dots or green + symbols. These mean that you’ve modified or added files, respectively.

Working Copy is a sweet iOS Git client, and you can tap on your files to see additional information including a comparison of changes (“diff”) as well as status and Git history. You can even edit files right within the app, with syntax highlighting for its many supported languages. For now, we’ll look at how to push your changed work to your remote repository.

Changes to commit

On the Repository Status and Configuration page, you’ll see right at the top that there are changes to be committed. If you’re new to Git, this is like “saving your changes” to your Git history, something typically done with the terminal command git commit. You can think of this as saving the files that we’ll want to send to the GitHub repository. Tap Commit changes.

Add a commit message and select files to commit

Enter your commit message, and select the files you want to add. Toggle the Push switch to send everything to your remote repository when you commit the files. Then tap Commit.

You’ll see a progress bar as your files are uploaded, and then a confirmation message on the status screen.

Commit success message

Congratulations! Your changes are now present in your remote repository on GitHub. You’ve successfully synced your files remotely!

Pull changes from repository on your computer

To bring your updated files full circle to your computer, you pull them from the GitHub repository. I prefer to use the terminal for this as it’s quick and easy, but GitHub also offers a graphical client if terminal commands seem a little alien for now.

If you started with the GitHub repository, you can clone it to a folder on your computer by following these instructions.

Staying in sync

When you update your work on your computer, you’ll use Git to push your changes to the remote repository. To do this, you can use GitHub’s graphical client, or follow these instructions.

On your iOS device, Working Copy makes pulling and pushing as simple as a single tap. On the Repository Status and Configuration page, tap on the remote name under Remotes.

List of Remotes in Working Copy

Then tap Synchronize. Working Copy will take care of the details of pushing your committed changes and/or pulling any new changes it finds from the remote repository.

Work anywhere

For a Git-based developer and work-anywhere-aholic like me, this set up couldn’t be more convenient. Working Copy really makes staying in sync with my remote repositories seamless, nevermind the ability to work with any of my GitHub repos on the go.

I most recently used this set up to get some writing done while hanging out in the atrium of Washington DC’s National Portrait Gallery, which is pleasantly photogenic.

The atrium of the National Portrait Gallery

Happy working! If you enjoyed this post, there’s a lot more where this came from! I write about computing, cybersecurity, and leading great technical teams. You can subscribe to see new articles first.

On doing great things, 08 Mar 2019 18:36:15 (Victoria Drake) thoughts inspired by International Women's Day, Grace Hopper, and making contributions to the world of tech. <img src="" width="640" height="480"/>It’s International Women’s Day, and I’m thinking about Grace Hopper.

Grace Hopper was an amazing lady who did great things. She envisioned and helped create programming languages that translate English terms into machine code. She persevered in her intention to join the US Navy from the time she was rejected at 34 years old, to being sworn in to the US Navy Reserve three years later, to retiring with the rank of commander at age 60… then was recalled (twice) and promoted to the rank of captain at the age of 67. She advocated for distributed networks and developed computer testing standards we use today, among other achievements too numerous to list here.

By my read, throughout her life, she kept her focus on her work. She did great things because she could do them, and felt some duty to do them. Her work speaks for itself.

I recently came across a sizeable rock denoting a rather small, quiet park. It looks like this:

Signage on a rock denoting Grace Murray Hopper Park

When I first saw this park, I thought it in no way did this great lady justice. But upon some reflection, its lack of assumption and grandeur grew on me. And today, it drew to the forefront something that’s been on my mind.

I try and contribute regularly to the wide world of technology, usually through building things, writing, and mentorship. I sometimes get asked to participate in female-focused tech events. I hear things like, “too few developers are women,” or “we need more women in blockchain,” or “we need more female coders.”

For some time I haven’t been sure how to respond, because while my answer isn’t “yes,” it’s not exactly “no,” either. It’s really, “no, because…” and it’s because I’m afraid. I’m afraid of misrepresenting myself, my values, and my goals.

Discrimination and racism are real things. They exist in the minds and attitudes of a very small percentage of very loud people, as they always will. These people aren’t, however, the majority. They are small.

I think that on the infrequent occasions when we encounter these people, we should do our best to lead by example. We should have open minds, tell our stories, listen to theirs. Try and learn something. That’s all.

When I present myself, I don’t point out that I’m a woman. I don’t align myself with “women in tech” or seek to represent them. I don’t go to women-only meetings or support organizations that discriminate against men, or anyone at all. It’s not because I’m insecure as a woman, or ashamed that I’m a woman, or some other inflammatory adjective that lately shows up in conjunction with being female. It’s because I’ve no reason to point out my gender, any more than needing to point out that my hair is black, or that I’m short. It’s obvious and simultaneously irrelevant.

When I identify with a group, I talk about the go-getters who wake up at 0500 every day and go work out - no matter the weather, or whether they feel like it. I tell stories about the people I met in different countries around the world, who left home, struck out on their own, and had an adventure, because they saw value in the experience. I identify with people who constantly build things, try things, design and make things, and then share those things with the world, because they love to do so. This is how I see myself. This is what matters to me.

Like the unassuming park named after an amazing woman, when truly great things are done, they are done relatively quietly. Not done for the fanfare of announcing them to the world, but for the love of the thing itself. So go do great things, please. The world still needs them.

Git commit practices your future self will thank you for, 06 Aug 2018 08:54:56 (Victoria Drake) how to easily create a clean and readable Git commit history with git squash, message templates, git stash, and tags. <img src="" width="640" height="320"/>A history of clean commits can be evidence of a lot of things: attention to detail, good work ethic, and genuine investment in the project. What do your Git commits say about you?

Here’s how you can create and maintain a clean and orderly Git commit history using message templates, learning how to squash commits, using git stash, and creating annotated commit tags.

What it means to commit responsibly

Whether our code will be seen by the entire open source community or just future versions of ourselves, either one will be grateful if we commit responsibly today. Being responsible can mean a lot of things to different people, so I enlisted some of and to help round out my list. From those (really great) threads, I distilled these main points:

Committing responsibly

  1. Provide and/or use tests to avoid committing bugs or broken builds
  2. Write clean code that meets style specifications
  3. Use descriptive commit messages that reference related discussion
  4. Make only one change per commit and avoid including unrelated changes

Some of the above is achieved through maintaining a short feedback loop that helps you improve your code quality while staying accountable to yourself. I wrote another article that discusses this in detail, especially the part about code review. Other items on this list have to do specifically with making commits in Git. There are some features of Git that can benefit us in these areas, as can harnessing tools like Vim. I’ll cover those topics here.

If the majority of your Git commits so far have been created with something like git commit -m "Bug fixes" then this is the article for you!

Write great Git commit messages with a template

I think Linus would be very happy if we didn’t use git commit -m "Fix bug" in a public repository ever again. As very well put in this classic post and the seven rules of a great Git commit message:

A properly formed Git commit subject line should always be able to complete the following sentence:

If applied, this commit will your subject line here

This other classic post also discusses three questions that the body of the commit message should answer:

Why is it necessary? How does it address the issue? What effects does the patch have?

This can be a lot to remember to cover, but there’s a slick way to have these prompts at hand right when you need it. You can set up a commit message template by using the commit.template configuration value.

To set it, configure Git to use a template file (for example, .gitmessage in your home directory), then create the template file with Vim:

git config --global commit.template ~/.gitmessage
vim ~/.gitmessage

When we run git commit without the -m message flag, the editor will open with our helpful template ready to go. Here’s my commit message template:

## If applied, this commit will...
## [Add/Fix/Remove/Update/Refactor/Document] [issue #id] [summary]

## Why is it necessary? (Bug fix, feature, improvements?)
## How does the change address the issue?
## What side effects does this change have?

I’m a fan of this format because commented lines are not included in the final message. I can simply fill in the blank lines with text and bullet points under the prompts, and it comes out looking something like this:

Fix #16 missing CSS variables
- Fix for unstyled elements
- Add background color, height for code blocks
- Only affects highlight class

Issue trackers in GitHub and Bitbucket both recognize the keywords close, fix, and resolve followed immediately by the issue or pull request number. These keywords conveniently help us close the referenced issue or pull request, and this helps maintain a clear trail of changes. GitLab, and issue trackers like Jira offer similar functionalities.

Use helpful Vim settings for git commit messages

By adding a few lines to our Vim configuration, we can make writing great git commit messages easy. We can add these lines to ~/.vimrc to turn on syntax highlighting in general, and spell check and text wrapping for commit messages in particular:

" Filetype detection, plugins, and indent rules
filetype plugin indent on

" Syntax highlighting
syntax on

" Spell check and line wrap just for git commit messages
autocmd Filetype gitcommit setlocal spell textwidth=72

If you’re curious, you can find my full ~/.vimrc in my dotfiles.

Other editors have settings that can help us out as well. I came across these for Sublime Text 3 and language specific settings for VS Code.

One change per commit: how to squash Git commits

A doodle of squash

Still life Git

Let’s get one thing out of the way first: rewriting Git history just for the sake of having a pretty tree, especially with public repositories, is generally not advisable. It’s kind of like going back in time, where changes you make to your version of the project cause it to look completely different from a version that someone else forked from a point in history that you’ve now erased - I mean, haven’t you seen Back to the Future Part II? (If you’d rather maintain that only one Back to the Future movie was ever made, thus sparing your future self from having to watch the sequels, I get it.)

Here’s the main point. If you’ve pushed messy commits to a public repository, I say go right ahead and leave them be, instead of complicating things further. (We all learn from our embarrassments, especially the public ones - I’m looking at you, past-Vicky.) If your messy commits currently only exist on your local version, great! We can tidy them up into one clean, well-described commit that we’ll be proud to push, and no one will be the wiser.

There are a couple different ways to squash commits, and choosing the appropriate one depends on what we need to achieve.

The following examples are illustrated using git log --graph, with some options for brevity. We can set a handy alias to see this log format in our terminal with:

git config --global alias.plog "log --graph --pretty=format:'%h -%d %s %n' --abbrev-commit --date=relative --branches"

Then we just do git plog to see the pretty log.

Method #1: one commit to rule the master branch

This is appropriate when:

  • We’re committing directly to master
  • We don’t intend to open a pull request to merge a feature
  • We don’t want to preserve history of branches or changes we haven’t yet pushed

This method takes a Git tree that looks like this:

*   3e8fd79 - (HEAD -> master) Fix a thing
*   4f0d387 - Tweak something
*   0a6b8b3 - Merge branch 'new-article'
| * 33b5509 - (new-article) Update article again again
| |
| * 1782e63 - Update article again
| |
| * 3c5b6a8 - Update article
| |
* | f790737 - (master) Tweak unrelated article
* 65af7e7 Add social media link
* 0e3fa32 (origin/master, origin/HEAD) Update theme

And makes it look like this:

* 7f9a127 - (HEAD -> master) Add new article
* 0e3fa32 - (origin/master, origin/HEAD) Update theme

Here’s how to do it - hold on to your hoverboards, it’s super complicated:

git reset --soft origin/master
git commit

Yup that’s all. We can delete the unwanted branch with git branch -D new-article.

Method #2: not that much

This is appropriate when:

  • We want to squash the last x commits but not all commits since origin/master
  • We want to open a pull request to merge a branch

This method takes a Git tree that looks like this:

* 13a070f - (HEAD -> new-article) Finish new article
* 78e728a - Edit article draft
* d62603c - Add example
* 1aeb20e - Update draft
* 5a8442a - Add new article draft
| * 65af7e7 - (master) Add social media link

And makes it look like this:

* 90da69a - (HEAD -> new-article) Add new article
| * 65af7e7 - (master) Add social media link

To squash the last five commits on branch new-article into one, we use:

git reset --soft HEAD~5
git commit -m "New message for the combined commit"

Where --soft leaves our files untouched and staged, and 5 can be thought of as “the number of previous commits I want to combine.”

We can then do git merge master and create our pull request.

Method #3: getting picky

Say we had a really confusing afternoon and our Git tree looks like this:

* dc89918 - (HEAD -> master) Add link
* 9b6780f - Update image asset
* 6379956 - Fix CSS bug
*   16ee1f3 - Merge master into branch
| |
| * ccec365 - Update list page
| |
* | 033dee7 - Fix typo
| |
* | 90da69a - Add new article

We want to retain some of this history, but clean up the commits. We also want to change the messages for some of the commits. To achieve this, we’ll use git rebase.

This is appropriate when:

  • We want to squash only some commits
  • We want to edit previous commit messages
  • We want to delete or reorder specific commits

Git rebase is a powerful tool, and handy once we’ve got the hang of it. To change all the commits since origin/master, we do:

git rebase -i origin/master

Or, we can do:

git rebase -i 0e3fa32

Where the commit hash is the last commit we want to retain as-is.

The -i option lets us run the interactive rebase tool, which launches our editor with, essentially, a script for us to modify. We’ll see a list of our commits in reverse order to the git log, with the oldest at the top:

pick 90da69a Add new article
pick 033dee7 Fix typo
pick ccec365 Update list page
pick 6379956 Fix CSS bug
pick 9b6780f Update image asset
pick dc89918 Add link

# Rebase 0e3fa32..dc89918 onto 0e3fa32 (6 commands)
# Commands:
# p, pick = use commit
# e, edit = use commit, but stop for amending
# d, drop = remove commit
# Note that empty commits are commented out

The comments give us a handy guide as to what we’re able to do. For now, let’s squash the commits with small changes into the more significant commits. In our editor, we change the script to look like this:

pick 90da69a Add new article
squash 033dee7 Fix typo
pick ccec365 Update list page
squash 6379956 Fix CSS bug
squash 9b6780f Update image asset
squash dc89918 Add link

Once we save the changes, the interactive tool continues to run. It will execute our instructions in sequence. In this case, we see the editor again with the following:

# This is a combination of 2 commits.
# This is the 1st commit message:

Add new article

# This is the commit message #2:

Fix typo

# Please enter the commit message for your changes. Lines starting
# interactive rebase in progress; onto 0e3fa32
# Last commands done (2 commands done):
#    pick 90da69a Add new article
#    squash 033dee7 Fix typo
# Next commands to do (4 remaining commands):
#    pick ccec365 Update list page
#    squash 6379956 Fix CSS bug
# You are currently rebasing branch 'master' on '0e3fa32'.
# Changes to be committed:
#       modified:   ...

Here’s our chance to create a new commit message for this first squash, if we want to. Once we save it, the interactive tool will go on to the next instructions. Unless…

[detached HEAD 3cbad01] Add new article
 1 file changed, 129 insertions(+), 19 deletions(-)
Auto-merging content/dir/
CONFLICT (content): Merge conflict in content/dir/
error: could not apply ccec365... Update list page

Resolve all conflicts manually, mark them as resolved with

Could not apply ccec365... Update list page

Again, the tool offers some very helpful instructions. Once we fix the merge conflict, we can resume the process with git rebase --continue. Our interactive rebase picks up where it left off.

Once all the squashing is done, our Git tree looks like this:

* 3564b8c - (HEAD -> master) Update list page
* 3cbad01 - Add new article

Phew, much better.

Git stash

If we’re in the middle of some work and it’s not a good time to commit, but we need to switch branches, stashing can be a good option. Stashing lets us save our unfinished work without needing to create a half-assed commit. It’s like that pile of paper on your desk representing all the stuff you’ve been in the middle of doing since two weeks ago. Yup, that one.

It’s as easy as typing git stash:

Saved working directory and index state WIP on master: 3564b8c Update list page

The dirty work we’re in the midst of is safely tucked away, and our working directory is clean - just as it was after our last commit. To see what’s in our stash stack, we do git stash list:

stash@{0}: WIP on master: 3564b8c Update list page
stash@{1}: WIP on master: 90da69a Add new article
stash@{2}: WIP on cleanup: 0e3fa32 Update theme

To restore our work in progress, we use git stash apply. Git will try and apply our most recent stashed work. To apply an older stash, we use git stash apply stash@{1} where 1 is the stash to apply. If changes since stashing our work prevent the stash from reapplying cleanly, Git will give us a merge conflict to resolve.

Applying a stash doesn’t remove it from our list. To remove a stash from our stack, we do git stash drop stash@{0} where 0 is the one we want to remove.

We can also use git stash pop to apply the most recent stash and then immediately remove it from the stack.

Tag release versions using annotated Git tags

In the spirit of having a beautiful, clean Git history, there’s one more thing we can do to help make our commit log inspire infinite joy in its viewers. If you’ve never heard of git tag, your master branch history might look like this…

* 0377782 - Update theme
* ecf8128 - Add about page (#25)
* 33e432f - Fix #23 navigation bug
* 08b853b - Create blog section
* 63d18b4 - Add theme (#12)
* 233e23f - Add main content (#6)

Wouldn’t it be nice if it looked like this instead?

* 0377782 - (tag: v2.1.0) Update theme
* ecf8128 - Add about page (#25)
* 33e432f - Fix #23 navigation bug
* 08b853b - (tag: v2.0.0) Create blog section
* 63d18b4 - Add theme (#12)
* 233e23f - (tag: v1.1.0) Add main content (#6)

We can tag Git commits with anything, but tags are especially helpful for semantic versioning of releases. Sites like GitHub and GitLab have pages for repositories that list tags, letting viewers of our project browse the release versions. This can be helpful for public projects to differentiate major releases, updates with bug fixes, or beta versions.

There are two types of Git tags: lightweight and annotated. For adding a version tag to commits, we use annotated Git tags.

The Git tag documentation explains it this way:

Tag objects (created with -a, -s, or -u) are called “annotated” tags; they contain a creation date, the tagger name and e-mail, a tagging message, and an optional GnuPG signature. Whereas a “lightweight” tag is simply a name for an object (usually a commit object).


We can think of lightweight tags as bookmarks, and annotated tags as signed releases.

For public repositories, annotated tags allow us to:

  • See who tagged the commit, which may differ from the commit author
  • See all the tags with git describe
  • Avoid conflicting tag names

To create an annotated Git tag and attach it to our current (last) commit, we do:

git tag -a v1.2.0 -m "Clever release title"

This tags the commit on our local repository. To push all annotated tags to the remote, we do:

git push --follow-tags

We can also set our Git configuration to push our annotated tags by default:

git config --global push.followTags true

If we then want to skip pushing tags this time, we pass --no-follow-tags.

Practice responsible commits

A little time invested in getting familiar with these tools and practices can make your commits even more useful and well-crafted. With a little practice, these processes will become second nature. You can make it even easier by creating a personal commit checklist on paper to keep handy while you work - or if that isn’t fun enough, make it an interactive pre-commit hook.

Creating clean, useful, and responsible Git commits says a lot about you. Especially in remote work, Git commits may be a primary way that people interact with you over projects. With a little practice and effort, you can make your commit habits an even better reflection of your best work - work that is evidently created with care and pride.

If you enjoyed this post, there’s a lot more where it came from! I write about computing, cybersecurity, and leading great technical teams. Subscribe on to see new articles first, and check out the ones below!

An automatic interactive pre-commit checklist, in the style of infomercials, 23 Jul 2018 09:38:09 (Victoria Drake) to set up an interactive checklist using a Git pre-commit hook script. <img src="" width="640" height="320"/>What’s that, you say? You’ve become tired of regular old boring paper checklists? Well, my friend, today is your lucky day! You, yes, you, can become the proud owner of a brand-spanking-new automatic interactive pre-commit hook checklist! You’re gonna love this! Your life will be so much easier! Just wait until your friends see you.

What’s a pre-commit hook

Did you know that nearly 1 out of 5 coders are too embarrassed to ask this question? Don’t worry, it’s perfectly normal. In the next 60 seconds we’ll tell you all you need to know to pre-commit with confidence.

A Git hook is a feature of Git that triggers custom scripts at useful moments. They can be used for all kinds of reasons to help you automate your work, and best of all, you already have them! In every repository that you initialize with git init, you’ll have a set of example scripts living in .git/hooks. They all end with .sample and activating them is as easy as renaming the file to remove the .sample part.

Git hooks are not copied when a repository is cloned, so you can make them as personal as you like.

The useful moment in particular that we’ll talk about today is the pre-commit. This hook is run after you do git commit, and before you write a commit message. Exiting this hook with a non-zero status will abort the commit, which makes it extremely useful for last-minute quality checks. Or, a bit of fun. Why not both!

How do I get a pre-commit checklist

I only want the best for my family and my commits, and that’s why I choose an interactive pre-commit checklist. Not only is it fun to use, it helps to keep my projects safe from unexpected off-spec mistakes!

It’s so easy! I just write a bash script that can read user input, and plop it into .git/hooks as a file named pre-commit. Then I do chmod +x .git/hooks/pre-commit to make it executable, and I’m done!

Oh look, here comes an example bash script now!


echo "Would you like to play a game?"

# Read user input, assign stdin to keyboard
exec < /dev/tty

while read -p "Have you double checked that only relevant files were added? (Y/n) " yn; do
    case $yn in
        [Yy] ) break;;
        [Nn] ) echo "Please ensure the right files were added!"; exit 1;;
        * ) echo "Please answer y (yes) or n (no):" && continue;
while read -p "Has the documentation been updated? (Y/n) " yn; do
    case $yn in
        [Yy] ) break;;
        [Nn] ) echo "Please add or update the docs!"; exit 1;;
while read -p "Do you know which issue or PR numbers to reference? (Y/n) " yn; do
    case $yn in
        [Yy] ) break;;
        [Nn] ) echo "Better go check those tracking numbers!"; exit 1;;

exec <&-

Take my money

Don’t delay! Take advantage right now of this generous one-time offer! An interactive pre-commit hook checklist can be yours, today, for the low, low price of… free? Wait, who wrote this script?

How to set up a short feedback loop as a solo coder, 02 Jul 2018 10:08:41 (Victoria Drake) for continuous improvement when you're a freelance developer. <img src="" width="640" height="288"/>I’ve spent the last couple years as a solo freelance developer. Comparing this experience to previously working in companies, I’ve noticed that those of us who work alone can have fewer iterative opportunities for improvement than developers who work on teams. Integral to having opportunities to improve is the concept of a short feedback loop: a process of incorporating new learning from observation and previous experience continuously over a short period of time. This process has to be manufactured by people working mostly alone, instead of, as is often the case, adopted when you join a team.

In this post I hope to share what I’ve learned about setting yourself up to improve quickly and continuously as a solo coder.

About feedback loops

United States Air Force Colonel John Boyd developed the concept of the OODA loop, OODA being an acronym for observe, orient, decide, act. In military operations, this illustrates a process of decision-making based on the constant ingestion of new information:

Observe: Obtain raw information about unfolding circumstances and the current environment. Orient: Put raw observations in context. Consider such things as relevancy to the current situation and previously gained knowledge and expertise. Decide: Make a plan for moving towards your goal. Act: Execute the plan.

Since it’s a loop, the act stage leads directly back into the observe stage. This is the critical “feed back” concept that enables increasingly successful iterations. It’s widely applicable beyond military operations - you may recognize it as the origin of the PDCA (plan-do-check-act) method.

I like the OODA loop for being a succinct illustration of a general feedback loop. Many concepts and working methods build on the idea of feedback loops, including DevOps and Agile software development methods.

Development team feedback loop

Let’s look at what some components of a feedback loop for a developer on a team might look like:

  1. Direction from product owners or reviews from users
  2. Daily scrum/standup with whole team
  3. Prioritization with developer team
  4. Individual coding and testing
  5. Peer code review
  6. Deployment and performance monitoring

Implicit in these steps is the support of co-workers and management - in other words, someone to answer to. How can a solo freelance developer create a similar environment of accountability?

Solo developer feedback loop

Here are some possible steps that an individual freelance developer can implement to create a short feedback loop:

  1. Build discipline
  2. Clarify concrete top-level goals
  3. Prioritize and plan mid-level and low-level goals
  4. Automate your work
  5. Block out time for code review
  6. Block out time for process review
  7. Update your goals and processes with the results of your reviews

I’ll cover each of these stages in detail below.

Build discipline

More of a prerequisite than a stage in itself, building discipline is what enables our short feedback loop to work. Nothing else in this article will be helpful unless we have the skill to do something we don’t want to do. Discipline is most certainly a skill. It can be learned, trained, and improved just like any other.

Why is discipline so important? Because when we’re crunching to get a project completed this Friday evening, we’re not going to want to write a good commit message. We’re not going to want to clean up the code comments. We just want to see the darn thing go, Hello, git push -f. It’s in those moments that discipline enables us to not miss an opportunity to practice, learn, and improve our work process. Discipline helps us avoid Friday night commits that turn into Monday morning git reset --hards.

Clarify concrete top-level goals

Envisioning a peanut butter and jelly sandwich

Whether working for a client or bootstrapping our own best-new-app-ever, we won’t be able to measure any progress or improvements without something to measure them against.

When I’m discussing a new project with a client, I always speak in terms of concrete achievements. This could take the form of accomplishing a specific feature by a certain date, or deciding what the MVP looks like to a user. This is as much to my benefit as my client’s. By agreeing, in writing, what will be achieved and when, my client and I have clearly defined top-level goals and can both assess how the project is progressing. When I’m working for myself, I treat myself as I would a client. I make a commitment, in writing, describing what will be achieved, and when. This can be something as simple as a goals list for the week, or as detailed as a kanban board.

The point of having a concrete goal, however, is not to stick to it at all costs. It’s important to set an expectation, with ourselves and with our clients, that the goals will be revisited at mutually-agreeable dates over the course of the project. This enables the all-important “feed back” part of the loop.

Prioritize and plan mid-level and low-level goals

The components of a peanut butter and jelly sandwich

Few goals are achieved all in one step. Even the simple process of making a peanut butter and jelly sandwich (a favourite computer programming teaching example) can be broken down into successively smaller, more precise instructions. While we humans may not require the granularity that a computer program does, goals that are chunked into time-boxed, achievable steps are much more easily digested. 🥪

Start with the mid-level goals, and make each step concrete. If the goal is to release a new open source web app, for example, the steps might look like this:

  1. Complete app JavaScript
  2. Create front end and stylesheet
  3. Do local tests
  4. Set up cloud server
  5. Deploy app to cloud
  6. Do tests
  7. Add repository to GitHub
  8. Post on Hacker News
  9. Profit!!!

Each of the above examples encapsulates many smaller, low-level goals - we can think of these as our to-do list items. For example, “Set up cloud server” might involve:

  1. Research cloud providers
  2. Decide on service and sign up
  3. Set up server/instance
  4. Add integrations
  5. Test deployment

Our parameters for chunk sizes and what constitutes a “step” may be different from one another, and will likely change from project to project. If your mid-level and low-level steps clearly define a concrete path for achieving the top-level goals you set, then you’re in good shape. Later on, evaluating the decision process that brought us to these mid-level and low-level goals enables us to bring our feedback loop full circle.

Automate your work

Peanut butter and jelly sandwich robot

I recently read a great article entitled Manual Work is a Bug. It discusses a process by which successful developers document and eventually automate their work. The beauty of this idea is in its simplicity. By writing down the things we do manually, we’re able to correct and refine our processes. By refining our processes, we can more easily translate them into code snippets and scripts. With a collection of scripts that we can string together, we can automate our work.

Automating work isn’t only about saving time. It reduces haven’t-had-my-coffee-yet errors, minimizes cognitive load allowing more room for creativity, and allows our processes to be repeatable across collaborators and projects. It help shorten our feedback loop by ensuring we aren’t doing the same thing three times in three different ways.

We can begin to automate by starting our own personal wiki. If we build a habit of writing down every manual thing we do, no matter how basic it may seem at the time, we give ourselves more opportunities to spot patterns, and thus possible integrations and improvements.

The first time we do something manually, we write out the steps. The second time, we follow the steps. This gives us the opportunity to correct and refine them based on what we’ve learned since the first time. Over successive iterations, we might replace parts of manual commands with variables; we might find handy snippets of bash scripts that automate just a part of our task. As long as we keep revising and improving our personal wiki, we’re moving towards automation.

Block out time for code review

Reviewing a peanut butter and jelly sandwich with a clipboard

It’s all too easy to commit messy code when we work alone. We think, who’s going to see it? I’ll fix it later. Each time that happens, though, we’re building a habit. It’s a bad one.

Working alone means there’s no one likely to give feedback on our commits when we’re doing something that doesn’t make sense, or that could be improved. Instead, we have to actively seek out opportunities to improve. Open source communities are amazing for this. There’s a wealth of information available to us in terms of coding styles, examples of refactored code, and a smorgasbord of snippets that achieve that-thing-we-were-trying-to-do but in fewer lines. We can learn all we please, if we just block out the time to do it.

Schedule your own code review at a time that makes sense for you and the project you’re working on. This might be each time you finish a fix or feature, or at regular intervals daily or weekly. If you have someone who can help, book them. There are also chatrooms full of people happy to lend a hand.

Do some research on basic best practices for what you’re working on. Set yourself a time limit though, and take what you read with a grain of salt. There’s a lot of rabbit holes in that field. As a starting point, I’d recommend learning about DRY code, and watching Uncle Bob demand professionalism in software development.

Code review checklist

Here’s my personal code review checklist, based off some general best practices. Feel free to use it as a starting point for your own!

Victoria’s Code Review Extravaganza!

  • This solves a high-priority item.
  • This is a complete implementation that follows the specification.
  • Off-topic changes were not included and have been added to backlog.
  • Variable names are meaningful and there are no magic numbers.
  • Correct and useful error messages are returned at every opportunity.
  • No debugging print statements were left in.
  • This code is DRY and modular.
  • This code is secure. Private and public code are well separated.
  • This code is its own documentation, or the documentation is up to date.
  • A five-year-old could follow this, seriously it’s that readable.
  • Unit tests successfully pass.
  • Master was merged into the branch and tested.
  • Formatting follows style guidelines.
  • I cannot find any further edge cases or known defects.
  • I would be happy if this code was publicly attributed to me.
  • I fully understand what the code does and the impact of the changes I made.
  • I actually verified that it actually does what I said it does.

Here is an excellent example of cleaning up code with some of the above points in mind.

Block out time for process review

Reviewing sandwich making robot with clipboard

Just as we learn from reviewing our code, we refine our processes by reviewing them as well. Process review is most beneficial when visited at regular intervals throughout the project, not just after the project’s completion. For short-term projects, a good starting point for scheduling process reviews is at each half-mark - once midway through, and again after completion. Long-term projects may have reviews at each quarter-mark.

Process review questions

Process review can be as simple as a short list of questions:

  1. What were my top-level goals for this period? Did I meet them?
  2. What were my mid-level and low-level goals for this period? Did I meet them?
  3. Would I have been better served with different or more specific goals? Why?
  4. Did I successfully remove or automate obstacles?
  5. Did I stick to my code review schedule? Why or why not?
  6. How might I remove obstacles next time?

Setting aside dedicated time for our process review can help us to answer questions like these thoughtfully and honestly. This allows us to squeeze out every bit of learning we can from our review, helping to shorten our feedback loop.

Update your goals and processes with the results of your reviews

Adding additional arms to robot while envisioning a multi-layer PB&amp;J

All the performance data in the world is no good to us if we don’t put it into practice. With each successive code review, we can refine and add to our checklist. With what we learn from each process review, we can fine tune and improve our processes. The more we can invent concrete and observable ways to implement our learning, the more success we’ll have.

Making a conscious effort to utilize and practice the things we’ve learned is the final, vital, component of our feedback loop. The more often we incorporate new learning, the shorter our loop becomes, allowing us to improve that much faster.

Adorable bookmarklets want to help delete your social media data, 14 Jun 2018 13:12:02 (Victoria Drake) you can use in your browser to help clean up your social media data. <img src="" width="640" height="320"/>A little while ago I wrote about a Lambda function I called ephemeral for deleting my old tweets. While it’s a great project for someone familiar with or wanting to learn to use Lambda, it isn’t simple for a non-technical person to set up. There are services out there that will delete your tweets for you, but require your access credentials. There didn’t seem to be anything that provided convenience without also requiring authentication.

So, I went oldschool and created the ephemeral bookmarklet.

If that didn’t make you instantly nostalgic, a bookmarklet is a little application that lives as a bookmark in your web browser. You “install” it by dragging the link to your bookmarks toolbar, or right-clicking on the link and choosing “Bookmark this link” (Firefox). You click it to execute the program on the current page.

Here’s what the ephemeral bookmarklet will do:

The ephemeral bookmarklet is part of a new suite of tools for personal data management that I’m co-creating with Adam Drake. You can get all the bookmarklets on this page, and they’re also open source on GitHub.

There are currently bookmarklets for managing your data on LinkedIn and Twitter. We’re looking for testers and contributors to help make this a comprehensive toolset for your social media data management. If you write code, I invite you to contribute and help this toolset grow.

∩{｡◕‿◕｡}∩ – Bookmarklet says hi!

A coffee-break introduction to time complexity of algorithms, 30 May 2018 14:08:28 (Victoria Drake) groundwork understanding of algorithm time complexity and Big O notation in about fifteen minutes with Go code examples and silly food cartoons. <img src="" width="640" height="320"/>Just like writing your very first for loop, understanding time complexity is an integral milestone to learning how to write efficient complex programs. Think of it as having a superpower that allows you to know exactly what type of program might be the most efficient in a particular situation - before even running a single line of code.

The fundamental concepts of complexity analysis are well worth studying. You’ll be able to better understand how the code you’re writing will interact with the program’s input, and as a result, you’ll spend a lot less wasted time writing slow and problematic code. It won’t take long to go over all you need to know in order to start writing more efficient programs - in fact, we can do it in about fifteen minutes. You can go grab a coffee right now (or tea, if that’s your thing) and I’ll take you through it before your coffee break is over. Go ahead, I’ll wait.

All set? Let’s do it!

What is “time complexity” anyway

The time complexity of an algorithm is an approximation of how long that algorithm will take to process some input. It describes the efficiency of the algorithm by the magnitude of its operations. This is different than the number of times an operation repeats; I’ll expand on that later. Generally, the fewer operations the algorithm has, the faster it will be.

We write about time complexity using Big O notation, which looks something like O(n). There’s rather a lot of math involved in its formal definition, but informally we can say that Big O notation gives us our algorithm’s approximate run time in the worst case, or in other words, its upper bound.[2] It is inherently relative and comparative.[3] We’re describing the algorithm’s efficiency relative to the increasing size of its input data, n. If the input is a string, then n is the length of the string. If it’s a list of integers, n is the length of the list.

It’s easiest to picture what Big O notation represents with a graph:

A graph showing different classes of time complexity

Lines made with the very excellent Desmos graph calculator. You can play with this graph here.

Here are the main important points to remember as you read the rest of this article:

  • Time complexity is an approximation
  • An algorithm’s time complexity approximates its worst case run time

Determining time complexity

There are different classes of complexity that we can use to quickly understand an algorithm. I’ll illustrate some of these classes using nested loops and other examples.

Polynomial time complexity

A polynomial, from the Greek poly meaning “many,” and Latin nomen meaning “name,” describes an expression comprised of constant variables, and addition, multiplication, and exponentiation to a non-negative integer power.[4] That’s a super math-y way to say that it contains variables usually denoted by letters and symbols that look like these:

A polynomial example

The below classes describe polynomial algorithms. Some have food examples.


A constant time algorithm doesn’t change its running time in response to the input data. No matter the size of the data it receives, the algorithm takes the same amount of time to run. We denote this as a time complexity of O(1).

A graph showing constant time complexity.

Here’s one example of a constant algorithm that takes the first item in a slice.

func takeCupcake(cupcakes []int) int {
    return cupcakes[0]
Types of cupcakes

Choice of flavours are: vanilla cupcake, strawberry cupcake, mint chocolate cupcake, lemon cupcake, and wibbly wobbly, timey wimey cupcake.

With this constant-time algorithm, no matter how many cupcakes are on offer, you just get the first one. Oh well. Flavours are overrated anyway.


The running duration of a linear algorithm is constant. It will process the input in n number of operations. This is often the best possible (most efficient) case for time complexity where all the data must be examined.

A graph showing linear time complexity.

Here’s an example of code with time complexity of O(n):

func eatChips(bowlOfChips int) {
 for chip := 0; chip <= bowlOfChips; chip++ {
  // dip chip

Here’s another example of code with time complexity of O(n):

func eatChips(bowlOfChips int) {
  // double dip chip

It doesn’t matter whether the code inside the loop executes once, twice, or any number of times. Both these loops process the input by a constant factor of n, and thus can be described as linear.

Lifeguard MIQ the chip says no double dipping

Don’t double dip in a shared bowl.


A graph showing quadratic time complexity

Now here’s an example of code with time complexity of O(n2):

func pizzaDelivery(pizzas int) {
 for pizza := 0; pizza <= pizzas; pizza++ {
  // slice pizza
  for slice := 0; slice <= pizza; slice++ {
   // eat slice of pizza

Because there are two nested loops, or nested linear operations, the algorithm process the input n2 times.


A graph showing cubic time complexity

Extending on the previous example, this code with three nested loops has time complexity of O(n3):

func pizzaDelivery(boxesDelivered int) {
 for pizzaBox := 0; pizzaBox <= boxesDelivered; pizzaBox++ {
  // open box
  for pizza := 0; pizza <= pizzaBox; pizza++ {
   // slice pizza
    // eat slice of pizza
A pizza pie in a box with a pizza slicer dependency

Seriously though, who delivers unsliced pizza??


A logarithmic algorithm is one that reduces the size of the input at every step. We denote this time complexity as O(log n), where log, the logarithm function, is this shape:

A graph showing logarithmic time complexity

One example of this is a binary search algorithm that finds the position of an element within a sorted array. Here’s how it would work, assuming we’re trying to find the element x:

  1. If x matches the middle element m of the array, return the position of m
  2. If x doesn’t match m, see if m is larger or smaller than x
    • If larger, discard all array items greater than m
    • If smaller, discard all array items smaller than m
  3. Continue by repeating steps 1 and 2 on the remaining array until x is found

I find the clearest analogy for understanding binary search is imagining the process of locating a book in a bookstore aisle. If the books are organized by author’s last name and you want to find “Terry Pratchett,” you know you need to look for the “P” section.

You can approach the shelf at any point along the aisle and look at the author’s last name there. If you’re looking at a book by Neil Gaiman, you know you can ignore all the rest of the books to your left, since no letters that come before “G” in the alphabet happen to be “P.” You would then move down the aisle to the right any amount, and repeat this process until you’ve found the Terry Pratchett section, which should be rather sizable if you’re at any decent bookstore because wow did he write a lot of books.


A graph showing quasilinear time complexity

Often seen with sorting algorithms, the time complexity O(n log n) can describe a data structure where each operation takes O(log n) time. One example of this is quick sort, a divide-and-conquer algorithm.

Quick sort works by dividing up an unsorted array into smaller chunks that are easier to process. It sorts the sub-arrays, and thus the whole array. Think about it like trying to put a deck of cards in order. It’s faster if you split up the cards and get five friends to help you.

Non-polynomial time complexity

The below classes of algorithms are non-polynomial.


A graph showing factorial time complexity

An algorithm with time complexity O(n!) often iterates through all permutations of the input elements. One common example is a brute-force search seen in the travelling salesman problem. It tries to find the least costly path between a number of points by enumerating all possible permutations and finding the ones with the lowest cost.


An exponential algorithm often also iterates through all subsets of the input elements. It is denoted O(2n) and is often seen in brute-force algorithms. It is similar to factorial time except in its rate of growth, which as you may not be surprised to hear, is exponential. The larger the data set, the more steep the curve becomes.

A graph showing exponential time complexity

In cryptography, a brute-force attack may systematically check all possible elements of a password by iterating through subsets. Using an exponential algorithm to do this, it becomes incredibly resource-expensive to brute-force crack a long password versus a shorter one. This is one reason that a long password is considered more secure than a shorter one.

There are further time complexity classes less commonly seen that I won’t cover here, but you can read about these and find examples in this handy table.

Recursion time complexity

As I described in my article explaining recursion using apple pie, a recursive function calls itself under specified conditions. Its time complexity depends on how many times the function is called and the time complexity of a single function call. In other words, it’s the product of the number of times the function runs and a single execution’s time complexity.

Here’s a recursive function that eats pies until no pies are left:

func eatPies(pies int) int {
 if pies == 0 {
  return pies
 return eatPies(pies - 1)

The time complexity of a single execution is constant. No matter how many pies are input, the program will do the same thing: check to see if the input is 0. If so, return, and if not, call itself with one fewer pie.

The initial number of pies could be any number, and we need to process all of them, so we can describe the input as n. Thus, the time complexity of this recursive function is the product O(n).

A pile of pizza boxes with pies to be eaten

This function’s return value is zero, plus some indigestion.

Worst case time complexity

So far, we’ve talked about the time complexity of a few nested loops and some code examples. Most algorithms, however, are built from many combinations of these. How do we determine the time complexity of an algorithm containing many of these elements strung together?

Easy. We can describe the total time complexity of the algorithm by finding the largest complexity among all of its parts. This is because the slowest part of the code is the bottleneck, and time complexity is concerned with describing the worst case for the algorithm’s run time.

Say we have a program for an office party. If our program looks like this:

package main

import "fmt"

func takeCupcake(cupcakes []int) int {
 fmt.Println("Have cupcake number",cupcakes[0])
 return cupcakes[0]

func eatChips(bowlOfChips int) {
 fmt.Println("Have some chips!")
  // dip chip
 fmt.Println("No more chips.")

func pizzaDelivery(boxesDelivered int) {
 fmt.Println("Pizza is here!")
  // open box
   // slice pizza
    // eat slice of pizza
 fmt.Println("Pizza is gone.")

func eatPies(pies int) int {
 if pies == 0 {
  fmt.Println("Someone ate all the pies!")
  return pies
 fmt.Println("Eating pie...")
 return eatPies(pies - 1)

func main() {
 takeCupcake([]int{1, 2, 3})
 fmt.Println("Food gone. Back to work!")

We can describe the time complexity of all the code by the complexity of its most complex part. This program is made up of functions we’ve already seen, with the following time complexity classes:

Function Class Big O
takeCupcake constant O(1)
eatChips linear O(n)
pizzaDelivery cubic O(n3)
eatPies linear (recursive) O(n)

To describe the time complexity of the entire office party program, we choose the worst case. This program would have the time complexity O(n3).

Here’s the office party soundtrack, just for fun.

Have cupcake number 1
Have some chips!
No more chips.
Pizza is here!
Pizza is gone.
Eating pie...
Eating pie...
Eating pie...
Someone ate all the pies!
Food gone. Back to work!

P vs NP, NP-complete, and NP-hard

You may come across these terms in your explorations of time complexity. Informally, P (for Polynomial time), is a class of problems that is quick to solve. NP, for Nondeterministic Polynomial time, is a class of problems where the answer can be quickly verified in polynomial time. NP encompasses P, but also another class of problems called NP-complete, for which no fast solution is known.[5] Outside of NP but still including NP-complete is yet another class called NP-hard, which includes problems that no one has been able to verifiably solve with polynomial algorithms.[6]

Euler diagram

P vs NP Euler diagram, by Behnam Esfahbod, CC BY-SA 3.0

P versus NP is an unsolved, open question in computer science.

Anyway, you don’t generally need to know about NP and NP-hard problems to begin taking advantage of understanding time complexity. They’re a whole other Pandora’s box.

Approximate the efficiency of an algorithm before you write the code

So far, we’ve identified some different time complexity classes and how we might determine which one an algorithm falls into. So how does this help us before we’ve written any code to evaluate?

By combining a little knowledge of time complexity with an awareness of the size of our input data, we can take a guess at an efficient algorithm for processing our data within a given time constraint. We can base our estimation on the fact that a modern computer can perform some hundreds of millions of operations in a second.[1] The following table from the Competitive Programmer’s Handbook offers some estimates on required time complexity to process the respective input size in a time limit of one second.

Input size Required time complexity for 1s processing time
n ≤ 10 O(n!)
n ≤ 20 O(2n)
n ≤ 500 O(n3)
n ≤ 5000 O(n2)
n ≤ 106 O(n log n) or O(n)
n is large O(1) or O(log n)

Keep in mind that time complexity is an approximation, and not a guarantee. We can save a lot of time and effort by immediately ruling out algorithm designs that are unlikely to suit our constraints, but we must also consider that Big O notation doesn’t account for constant factors. Here’s some code to illustrate.

The following two algorithms both have O(n) time complexity.

func makeCoffee(scoops int) {
 for scoop := 0; scoop <= scoops; scoop++ {
  // add instant coffee
func makeStrongCoffee(scoops int) {
 for scoop := 0; scoop <= 3*scoops; scoop++ {
  // add instant coffee

The first function makes a cup of coffee with the number of scoops we ask for. The second function also makes a cup of coffee, but it triples the number of scoops we ask for. To see an illustrative example, let’s ask both these functions for a cup of coffee with a million scoops.

Here’s the output of the Go test:

Benchmark_makeCoffee-4          1000000000               0.29 ns/op
Benchmark_makeStrongCoffee-4    1000000000               0.86 ns/op

Our first function, makeCoffee, completed in an average 0.29 nanoseconds. Our second function, makeStrongCoffee, completed in an average of 0.86 nanoseconds. While those may both seem like pretty small numbers, consider that the stronger coffee took near three times longer to make. This should make sense intuitively, since we asked it to triple the scoops. Big O notation alone wouldn’t tell you this, since the constant factor of the tripled scoops isn’t accounted for.

Improve time complexity of existing code

Becoming familiar with time complexity gives us the opportunity to write code, or refactor code, to be more efficient. To illustrate, I’ll give a concrete example of one way we can refactor a bit of code to improve its time complexity.

Let’s say a bunch of people at the office want some pie. Some people want pie more than others. The amount that everyone wants some pie is represented by an int > 0:

diners := []int{2, 88, 87, 16, 42, 10, 34, 1, 43, 56}

Unfortunately, we’re bootstrapped and there are only three forks to go around. Since we’re a cooperative bunch, the three people who want pie the most will receive the forks to eat it with. Even though they’ve all agreed on this, no one seems to want to sort themselves out and line up in an orderly fashion, so we’ll have to make do with everybody jumbled about.

Without sorting the list of diners, return the three largest integers in the slice.

Here’s a function that solves this problem and has O(n2) time complexity:

func giveForks(diners []int) []int {
 // make a slice to store diners who will receive forks
 var withForks []int
 // loop over three forks
 for i := 1; i <= 3; i++ {
  // variables to keep track of the highest integer and where it is
  var max, maxIndex int
  // loop over the diners slice
  for n := range diners {
   // if this integer is higher than max, update max and maxIndex
   if diners[n] > max {
    max = diners[n]
    maxIndex = n
  // remove the highest integer from the diners slice for the next loop
  diners = append(diners[:maxIndex], diners[maxIndex+1:]...)
  // keep track of who gets a fork
  withForks = append(withForks, max)
 return withForks

This program works, and eventually returns diners [88 87 56]. Everyone gets a little impatient while it’s running though, since it takes rather a long time (about 120 nanoseconds) just to hand out three forks, and the pie’s getting cold. How could we improve it?

By thinking about our approach in a slightly different way, we can refactor this program to have O(n) time complexity:

func giveForks(diners []int) []int {
 // make a slice to store diners who will receive forks
 var withForks []int
 // create variables for each fork
 var first, second, third int
 // loop over the diners
 for i := range diners {
  // assign the forks
  if diners[i] > first {
   third = second
   second = first
   first = diners[i]
  } else if diners[i] > second {
   third = second
   second = diners[i]
  } else if diners[i] > third {
   third = diners[i]
 // list the final result of who gets a fork
 withForks = append(withForks, first, second, third)
 return withForks

Here’s how the new program works:

Initially, diner 2 (the first in the list) is assigned the first fork. The other forks remain unassigned.

Then, diner 88 is assigned the first fork instead. Diner 2 gets the second one.

Diner 87 isn’t greater than first which is currently 88, but it is greater than 2 who has the second fork. So, the second fork goes to 87. Diner 2 gets the third fork.

Continuing in this violent and rapid fork exchange, diner 16 is then assigned the third fork instead of 2, and so on.

We can add a print statement in the loop to see how the fork assignments play out:

0 0 0
2 0 0
88 2 0
88 87 2
88 87 16
88 87 42
88 87 42
88 87 42
88 87 42
88 87 43
[88 87 56]

This program is much faster, and the whole epic struggle for fork domination is over in 47 nanoseconds.

As you can see, with a little change in perspective and some refactoring, we’ve made this simple bit of code faster and more efficient.

Well, it looks like our fifteen minute coffee break is up! I hope I’ve given you a comprehensive introduction to calculating time complexity. Time to get back to work, hopefully applying your new knowledge to write more effective code! Or maybe just sound smart at your next office party. :)


“If I have seen further it is by standing on the shoulders of Giants.” –Isaac Newton, 1675

  1. Antti Laaksonen. Competitive Programmer’s Handbook (pdf), 2017
  2. Wikipedia: Big O notation
  3. StackOverflow: What is a plain English explanation of “Big O” notation?
  4. Wikipedia: Polynomial
  5. Wikipedia: NP-completeness
  6. Wikipedia: NP-hardness
  7. Desmos graph calculator
Knapsack problem algorithms for my real-life carry-on knapsack, 09 May 2018 21:00:35 (Victoria Drake) a greedy algorithm and dynamic programming to pack my full-time nomad travel bag. <img src="" width="640" height="320"/>The knapsack problem

I’m a nomad and live out of one carry-on bag. This means that the total weight of all my worldly possessions must fall under airline cabin baggage weight limits - usually 10kg. On some smaller airlines, however, this weight limit drops to 7kg. Occasionally, I have to decide not to bring something with me to adjust to the smaller weight limit.

As a practical exercise, deciding what to leave behind (or get rid of altogether) entails laying out all my things and choosing which ones to keep. That decision is based on the item’s usefulness to me (its worth) and its weight.

All my stuff.

This is all my stuff, and my Minaal Carry-on bag.

Being a programmer, I’m aware that decisions like this could be made more efficiently by a computer. It’s done so frequently and so ubiquitously, in fact, that many will recognize this scenario as the classic packing problem or knapsack problem. How do I go about telling a computer to put as many important items in my bag as possible while coming in at or under a weight limit of 7kg? With algorithms! Yay!

I’ll discuss two common approaches to solving the knapsack problem: one called a greedy algorithm, and another called dynamic programming (a little harder, but better, faster, stronger…).

Let’s get to it.

The set up

I prepared my data in the form of a CSV file with three columns: the item’s name (a string), a representation of its worth (an integer), and its weight in grams (an integer). There are 40 items in total. I represented worth by ranking each item from 40 to 1, with 40 being the most important and 1 equating with something like “why do I even have this again?” (If you’ve never listed out all your possessions and ranked them by order of how useful they are to you, I highly recommend you try it. It can be a very revealing exercise.)

Total weight of all items and bag: 9003g

Bag weight: 1415g

Airline limit: 7000g

Maximum weight of items I can pack: 5585g

Total possible worth of items: 820

The challenge: Pack as many items as the limit allows while maximizing the total worth.

Data structures

Reading in a file

Before we can begin thinking about how to solve the knapsack problem, we have to solve the problem of reading in and storing our data. Thankfully, the Go standard library’s io/ioutil package makes the first part straightforward.

package main

import (

func check(e error) {
    if e != nil {

func readItems(path string) {
    dat, err := ioutil.ReadFile(path)

The ReadFile() function takes a file path and returns the file’s contents and an error (nil if the call is successful) so we’ve also created a check() function to handle any errors that might be returned. In a real-world application we probably would want to do something more sophisticated than panic, but that’s not important right now.

Creating a struct

Now that we’ve got our data, we should probably do something with it. Since we’re working with real-life items and a real-life bag, let’s create some types to represent them and make it easier to conceptualize our program. A struct in Go is a typed collection of fields. Here are our two types:

type item struct {
    name          string
    worth, weight int

type bag struct {
    bagWeight, currItemsWeight, maxItemsWeight, totalWeight int
    items                                                   []item

It is helpful to use field names that are very descriptive. You can see that the structs are set up just as we’ve described the things they represent. An item has a name (string), and a worth and weight (integers). A bag has several fields of type int representing its attributes, and also has the ability to hold items, represented in the struct as a slice of item type thingamabobbers.

Parsing and storing our data

Several comprehensive Go packages exist that we could use to parse our CSV data… but where’s the fun in that? Let’s go basic with some string splitting and a for loop. Here’s our updated readItems() function:

func readItems(path string) []item {

    dat, err := ioutil.ReadFile(path)

    lines := strings.Split(string(dat), "\n")

    itemList := make([]item, 0)

    for i, v := range lines {
        if i == 0 {
        s := strings.Split(v, ",")
        newItemWorth, _ := strconv.Atoi(s[1])
        newItemWeight, _ := strconv.Atoi(s[2])
        newItem := item{name: s[0], worth: newItemWorth, weight: newItemWeight}
        itemList = append(itemList, newItem)
    return itemList

Using strings.Split, we split our dat on newlines. We then create an empty itemList to hold our items.

In our for loop, we skip the first line of our CSV file (the headers) then iterate over each line. We use strconv.Atoi (read “A to i”) to convert the values for each item’s worth and weight into integers. We then create a newItem with these field values and append it to the itemList. Finally, we return itemList.

Here’s what our set up looks like so far:

package main

import (

type item struct {
    name          string
    worth, weight int

type bag struct {
    bagWeight, currItemsWeight, maxItemsWeight, totalWeight, totalWorth int
    items                                                               []item

func check(e error) {
    if e != nil {

func readItems(path string) []item {

    dat, err := ioutil.ReadFile(path)


    itemList := make([]item, 0)

    for i, v := range lines {
        if i == 0 {
            continue // skip the headers on the first line
        s := strings.Split(v, ",")
        newItemWorth, _ := strconv.Atoi(s[1])
        newItemWeight, _ := strconv.Atoi(s[2])
        itemList = append(itemList, newItem)
    return itemList

Now that we’ve got our data structures set up, let’s get packing (🥁) on the first approach.

Greedy algorithm

A greedy algorithm is the most straightforward approach to solving the knapsack problem, in that it is a one-pass algorithm that constructs a single final solution. At each stage of the problem, the greedy algorithm picks the option that is locally optimal, meaning it looks like the most suitable option right now. It does not revise its previous choices as it progresses through our data set.

Building our greedy algorithm

The steps of the algorithm we’ll use to solve our knapsack problem are:

  1. Sort items by worth, in descending order.
  2. Start with the highest worth item. Put items into the bag until the next item on the list cannot fit.
  3. Try to fill any remaining capacity with the next item on the list that can fit.

If you read my article about solving problems and making paella, you’ll know that I always start by figuring out what the next most important question is. In this case, there are three main operations we need to figure out how to do:

  • Sort items by worth.
  • Put an item in the bag.
  • Check to see if the bag is full.

The first one is just a docs lookup away. Here’s how we sort a slice in Go:

sort.Slice(is, func(i, j int) bool {
    return is[i].worth > is[j].worth

The sort.Slice() function orders our items according to the less function we provide. In this case, it will order the highest worth items before the lowest worth items.

Given that we don’t want to put an item in the bag if it doesn’t fit, we’ll complete the last two tasks in reverse. First, we’ll check to see if the item fits. If so, it goes in the bag.

func (b *bag) addItem(i item) error {
    if b.currItemsWeight+i.weight <= b.maxItemsWeight {
        b.currItemsWeight += i.weight
        b.items = append(b.items, i)
        return nil
    return errors.New("could not fit item")

Notice the * in our first line there. That indicates that bag is a pointer receiver (as opposed to a value receiver). It’s a concept that can be slightly confusing if you’re new to Go. Here are some things to consider that might help you decide when to use a value receiver and when to use a pointer receiver. For the purposes of our addItem() function, this case applies:

If the method needs to mutate the receiver, the receiver must be a pointer.

Our use of a pointer receiver tells our function we want to operate on this specific bag in particular, not a new bag. It’s important because without it, every item would always fit in a newly created bag! A little detail like this can make the difference between code that works and code that keeps you up until 4am chugging Red Bull and muttering to yourself. (Go to bed on time even if your code doesn’t work - you’ll thank me later.)

Now that we’ve got our components, let’s put together our greedy algorithm:

func greedy(is []item, b bag) {

    for i := range is {

    b.totalWeight = b.bagWeight + b.currItemsWeight

    for _, v := range b.items {
        b.totalWorth += v.worth

Then in our main() function, we’ll create our bag, read in our data, and call our greedy algorithm. Here’s what it looks like, all set up and ready to go:

func main() {

    minaal := bag{bagWeight: 1415, currItemsWeight: 0, maxItemsWeight: 5585}
    itemList := readItems("objects.csv")

    greedy(itemList, minaal)

Greedy algorithm results

So how does this algorithm do when it comes to efficiently packing our bag to maximize its total worth? Here’s the result:

Total weight of bag and items: 6987g

Total worth of packed items: 716

Here are the items our greedy algorithm chose, sorted by worth:

Item Worth Weight
Lenovo X1 Carbon (5th Gen) 40 112
10 pairs thongs 39 80
5 Underarmour Strappy 38 305
1 pair Uniqlo leggings 37 185
2 Lululemon Cool Racerback 36 174
Chargers and cables in Mini Bomber Travel Kit 35 665
The Roost Stand 34 170
ThinkPad Compact Bluetooth Keyboard with trackpoint 33 460
Seagate Backup PlusSlim 32 159
1 pair black denim shorts 31 197
2 pairs Nike Pro shorts 30 112
2 pairs Lululemon shorts 29 184
Isabella T-Strap Croc sandals 28 200
2 Underarmour HeatGear CoolSwitch tank tops 27 138
5 pairs black socks 26 95
2 pairs Injinji Women’s Run Lightweight No-Show Toe Socks 25 54
1 fancy tank top 24 71
1 light and stretchylong-sleeve shirt (Gap Fit) 23 147
Uniqlo Ultralight Down insulating jacket 22 235
Patagonia Torrentshell 21 301
Lightweight Merino Wool Buff 20 50
1 LBD (H&M) 19 174
Field Notes Pitch Black Memo Book Dot-Graph 18 68
Innergie PocketCell USB-C 6000mAh power bank 17 14
JBL Reflect Mini Bluetooth Sport Headphones 13 14
Oakley Latch Sunglasses 11 30
Petzl E+LITE Emergency Headlamp 8 27

It’s clear that the greedy algorithm is a straightforward way to quickly find a feasible solution. For small data sets, it will probably be close to the optimal solution. The algorithm packed a total item worth of 716 (104 points less than the maximum possible value), while filling the bag with just 13g left over.

As we learned earlier, the greedy algorithm doesn’t improve upon the solution it returns. It simply adds the next highest worth item it can to the bag.

Let’s look at another method for solving the knapsack problem that will give us the optimal solution - the highest possible total worth under the weight limit.

Dynamic programming

The name “dynamic programming” can be a bit misleading. It’s not a style of programming, as the name might cause you to infer, but simply another approach.

Dynamic programming differs from the straightforward greedy algorithm in a few key ways. Firstly, a dynamic programming bag packing solution enumerates the entire solution space with all possibilities of item combinations that could be used to pack our bag. Where a greedy algorithm chooses the most optimal local solution, dynamic programming algorithms are able to find the most optimal global solution.

Secondly, dynamic programming uses memoization to store the results of previously computed operations and returns the cached result when the operation occurs again. This allows it to “remember” previous combinations. This takes less time than it would to re-compute the answer again.

Building our dynamic programming algorithm

To use dynamic programming to find the optimal recipe for packing our bag, we’ll need to:

  1. Create a matrix representing all subsets of the items (the solution space) with rows representing items and columns representing the bag’s remaining weight capacity
  2. Loop through the matrix and calculate the worth that can be obtained by each combination of items at each stage of the bag’s capacity
  3. Examine the completed matrix to determine which items to add to the bag in order to produce the maximum possible worth for the bag in total

It will be most helpful to visualize our solution space. Here’s a representation of what we’re building with our code:

A sketch of the matrix with rows for items and columns for grams of weight.

The empty knapsackian multiverse.

In Go, we can create this matrix as a slice of slices.

matrix := make([][]int, numItems+1) // rows representing items
for i := range matrix {
    matrix[i] = make([]int, capacity+1) // columns representing grams of weight

We’ve padded the rows and columns by 1 so that the indicies match the item and weight numbers.

Now that we’ve created our matrix, we’ll fill it by looping over the rows and the columns:

// loop through table rows
for i := 1; i <= numItems; i++ {
    // loop through table columns
    for w := 1; w <= capacity; w++ {
        // do stuff in each element

Then for each element, we’ll calculate the worth value to ascribe to it. We do this with code that represents the following:

If the item at the index matching the current row fits within the weight capacity represented by the current column, take the maximum of either:

  1. The total worth of the items already in the bag or,
  2. The total worth of all the items in the bag except the item at the previous row index, plus the new item’s worth

In other words, as our algorithm considers one of the items, we’re asking it to decide whether this item added to the bag would produce a higher total worth than the last item it added to the bag, at the bag’s current total weight. If this current item is a better choice, put it in - if not, leave it out.

Here’s the code that accomplishes this:

// if weight of item matching this index can fit at the current capacity column...
if is[i-1].weight <= w {
    // worth of this subset without this item
    valueOne := float64(matrix[i-1][w])
    // worth of this subset without the previous item, and this item instead
    valueTwo := float64(is[i-1].worth + matrix[i-1][w-is[i-1].weight])
    // take maximum of either valueOne or valueTwo
    matrix[i][w] = int(math.Max(valueOne, valueTwo))
// if the new worth is not more, carry over the previous worth
} else {
    matrix[i][w] = matrix[i-1][w]

This process of comparing item combinations will continue until every item has been considered at every possible stage of the bag’s increasing total weight. When all the above have been considered, we’ll have enumerated the solution space - filled the matrix - with all possible total worth values.

We’ll have a big chart of numbers, and in the last column at the last row we’ll have our highest possible value.

A strictly representative representation of the filled matrix.

A strictly representative representation of the filled matrix.

That’s great, but how do we find out which combination of items were put in the bag to achieve that worth?

Getting our optimized item list

To see which items combine to create our optimal packing list, we’ll need to examine our matrix in reverse to the way we created it. Since we know the highest possible value is in the last row in the last column, we’ll start there. To find the items, we:

  1. Get the value of the current cell
  2. Compare the value of the current cell to the value in the cell directly above it
  3. If the values differ, there was a change to the bag items; find the next cell to examine by moving backwards through the columns according to the current item’s weight (find the value of the bag before this current item was added)
  4. If the values match, there was no change to the bag items; move up to the cell in the row above and repeat

The nature of the action we’re trying to achieve lends itself well to a recursive function. If you recall from my previous article about making apple pie, recursive functions are simply functions that call themselves under certain conditions. Here’s what it looks like:

func checkItem(b *bag, i int, w int, is []item, matrix [][]int) {
    if i <= 0 || w <= 0 {

    pick := matrix[i][w]
    if pick != matrix[i-1][w] {
        checkItem(b, i-1, w-is[i-1].weight, is, matrix)
    } else {
        checkItem(b, i-1, w, is, matrix)

Our checkItem() function calls itself if the condition we described in step 4 is true. If step 3 is true, it also calls itself, but with different arguments.

Recursive functions require a base case. In this example, we want the function to stop once we run out of values of worth to compare. Thus our base case is when either i or w are 0.

Here’s how the dynamic programming approach looks when it’s all put together:

    if i <= 0 || w <= 0 {

    pick := matrix[i][w]
    } else {

func dynamic(is []item, b *bag) *bag {
    numItems := len(is)          // number of items in knapsack
    capacity := b.maxItemsWeight // capacity of knapsack

    // create the empty matrix
    for i := range matrix {

    // loop through table rows
        // loop through table columns

                // worth of this subset without this item
                // take maximum of either valueOne or valueTwo
            } else {

    checkItem(b, numItems, capacity, is, matrix)

    // add other statistics to the bag
    b.totalWorth = matrix[numItems][capacity]
    b.totalWeight = b.bagWeight + b.currItemsWeight

    return b

Dynamic programming results

We expect that the dynamic programming approach will give us a more optimized solution than the greedy algorithm. So did it? Here are the results:

Total weight of bag and items: 6982g

Total worth of packed items: 757

Here are the items our dynamic programming algorithm chose, sorted by worth:

Item Worth Weight
10 pairs thongs 39 80
5 Underarmour Strappy 38 305
1 pair Uniqlo leggings 37 185
2 Lululemon Cool Racerback 36 174
Chargers and cables in Mini Bomber Travel Kit 35 665
The Roost Stand 34 170
ThinkPad Compact Bluetooth Keyboard with trackpoint 33 460
Seagate Backup Plus Slim 32 159
1 pair black denim shorts 31 197
2 pairs Nike Pro shorts 30 112
2 pairs Lululemon shorts 29 184
Isabella T-Strap Croc sandals 28 200
2 Underarmour HeatGear CoolSwitch tank tops 27 138
5 pairs black socks 26 95
1 fancy tank top 24 71
1 light and stretchy long-sleeve shirt (Gap Fit) 23 147
Uniqlo Ultralight Down insulating jacket 22 235
Patagonia Torrentshell 21 301
Lightweight Merino Wool Buff 20 50
1 LBD (H&M) 19 174
Field Notes Pitch Black Memo Book Dot-Graph 18 68
Innergie PocketCell USB-C 6000mAh power bank 17 148
Important papers 16 228
Deuter First Aid Kit Active 15 144
Stanley Classic Vacuum Camp Mug 16oz 14 454
JBL Reflect Mini Bluetooth Sport Headphones 13 14
Anker SoundCore nano Bluetooth Speaker 12 80
Oakley Latch Sunglasses 11 30
Ray Ban Wayfarer Classic 10 45
Petzl E+LITE Emergency Headlamp 8 27
Peak Design Cuff Camera Wrist Strap 6 26
Travelon Micro Scale 5 125
Humangear GoBites Duo 3 22

There’s an obvious improvement to our dynamic programming solution over what the greedy algorithm gave us. Our total worth of 757 is 41 points greater than the greedy algorithm’s solution of 716, and for a few grams less weight too!

Input sort order

While testing my dynamic programming solution, I implemented the Fisher-Yates shuffle algorithm on the input before passing it into my function, just to ensure that the answer wasn’t somehow dependent on the sort order of the input. Here’s what the shuffle looks like in Go:


for i := range itemList {
    j := rand.Intn(i + 1)
    itemList[i], itemList[j] = itemList[j], itemList[i]

Of course I then realized that Go 1.10 now has a built-in shuffle… it works precisely the same way and looks like this:

rand.Shuffle(len(itemList), func(i, j int) {

So did the order in which the items were processed affect the outcome? Well…

Suddenly… a rogue weight appears!

As it turns out, in a way, the answer did depend on the order of the input. When I ran my dynamic programming algorithm several times, I sometimes saw a different total weight for the bag, though the total worth remained at 757. I initially thought this was a bug before examining the two sets of items that accompanied the two different total weight values. Everything was the same except for a few changes that collectively added up to a different item subset accounting for 14 of the 757 worth points.

In this case, there were two equally optimal solutions based only on the success metric of the highest total possible worth. Shuffling the input seemed to affect the placement of the items in the matrix and thus, the path that the checkItem() function took as it went through the matrix to find the chosen items. Since the success metric of having the highest possible worth was the same in both item sets, we don’t have a single unique solution - there’s two!

As an academic exercise, both these sets of items are correct answers. We may choose to optimize further by another metric, say, the total weight of all the items. The highest possible worth at the least possible weight could be seen as an ideal solution.

Here’s the second, lighter, dynamic programming result:

Total weight of bag and items: 6955g

Total worth of packed items: 757

Item Worth Weight
10 pairs thongs 39 80
5 Underarmour Strappy 38 305
1 pair Uniqlo leggings 37 185
2 Lululemon Cool Racerback 36 174
Chargers and cables in Mini Bomber Travel Kit 35 665
The Roost Stand 34 170
ThinkPad Compact Bluetooth Keyboard with trackpoint 33 460
Seagate Backup Plus Slim 32 159
1 pair black denim shorts 31 197
2 pairs Nike Pro shorts 30 112
2 pairs Lululemon shorts 29 184
Isabella T-Strap Croc sandals 28 200
2 Underarmour HeatGear CoolSwitch tank tops 27 138
5 pairs black socks 26 95
1 fancy tank top 24 71
Uniqlo Ultralight Down insulating jacket 22 235
Patagonia Torrentshell 21 301
Lightweight Merino Wool Buff 20 50
1 LBD (H&M) 19 174
Field Notes Pitch Black Memo Book Dot-Graph 18 68
Innergie PocketCell USB-C 6000mAh power bank 17 148
Important papers 16 228
Deuter First Aid Kit Active 15 144
JBL Reflect Mini Bluetooth Sport Headphones 13 14
Anker SoundCore nano Bluetooth Speaker 12 80
Oakley Latch Sunglasses 11 30
Ray Ban Wayfarer Classic 10 45
Zip bag of toiletries 9 236
Petzl E+LITE Emergency Headlamp 8 27
Peak Design Cuff Camera Wrist Strap 6 26
Travelon Micro Scale 5 125
BlitzWolf Bluetooth Tripod/Monopod 4 150
Humangear GoBites Duo 3 22
Vapur Bottle 1L 1 41

Which approach is better?

Go benchmarking

The Go standard library’s testing package makes it straightforward for us to benchmark these two approaches. We can find out how long it takes each algorithm to run, and how much memory each uses. Here’s a simple main_test.go file:

package main

import (

func Benchmark_greedy(b *testing.B) {
    itemList := readItems("objects.csv")
        greedy(itemList, minaal)

func Benchmark_dynamic(b *testing.B) {
    itemList := readItems("objects.csv")
        dynamic(itemList, &minaal)

We can run go test -bench=. -benchmem to see these results:

Benchmark_greedy-4       1000000              1619 ns/op            2128 B/op          9 allocs/op
Benchmark_dynamic-4         1000           1545322 ns/op         2020332 B/op         49 allocs/op

Greedy algorithm performance

After running the greedy algorithm 1,000,000 times, the speed of the algorithm was reliably measured to be 0.001619 milliseconds (translation: very fast). It required 2128 Bytes or 2-ish kilobytes of memory and 9 distinct memory allocations per iteration.

Dynamic programming performance

The dynamic programming algorithm was run 1,000 times. Its speed was measured to be 1.545322 milliseconds or 0.001545322 seconds (translation: still pretty fast). It required 2,020,332 Bytes or 2-ish Megabytes, and 49 distinct memory allocations per iteration.

The verdict

Part of choosing the right approach to solving any programming problem is taking into account the size of the input data set. In this case, it’s a small one. In this scenario, a one-pass greedy algorithm will always be faster and less resource-needy than dynamic programming, simply because it has fewer steps. Our greedy algorithm was almost two orders of magnitude faster and less memory-hungry than our dynamic programming algorithm.

Not having those extra steps, however, means that getting the best possible solution from the greedy algorithm is unlikely.

It’s clear that the dynamic programming algorithm gave us better numbers: a lower weight, and higher overall worth.

Greedy algorithm Dynamic programming
Total weight: 6987g 6955g
Total worth: 716 757

Where dynamic programming on small data sets lacks in performance, it makes up in optimization. The question then becomes whether that additional optimization is worth the performance cost.

“Better,” of course, is a subjective judgement. If speed and low resource usage is our success metric, then the greedy algorithm is clearly better. If the total worth of items in the bag is our success metric, then dynamic programming is clearly better. However, our scenario is a practical one, and only one of these algorithm designs returned an answer I’d choose. In optimizing for the overall greatest possible total worth of the items in the bag, the dynamic programming algorithm left out my highest-worth, but also heaviest, item: my laptop. The chargers and cables, Roost stand, and keyboard that were included aren’t much use without it.

Better algorithm design

There’s a simple way to alter the dynamic programming approach so that the laptop is always included: we can modify the data so that the worth of the laptop is greater than the sum of the worth of all the other items. (Try it out!)

Perhaps in re-designing the dynamic programming algorithm to be more practical, we might choose another success metric that better reflects an item’s importance, instead of a subjective worth value. There are many possible metrics we can use to represent the value of an item. Here are a few examples of a good proxy:

  • Amount of time spent using the item
  • Initial cost of purchasing the item
  • Cost of replacement if the item were lost today
  • Dollar value of the product of using the item

By the same token, the greedy algorithm’s results might be improved with the use of one of these alternate metrics.

On top of choosing an appropriate approach to solving the knapsack problem in general, it is helpful to design our algorithm in a way that translates the practicalities of a scenario into code.

There are many considerations for better algorithm design beyond the scope of this introductory post. One of these is time complexity, and I’ve written about it here. A future algorithm may very well decide my bag’s contents on the next trip, but we’re not quite there yet. Stay tuned!

Why I'm automatically deleting my old tweets using AWS Lambda, 12 Apr 2018 10:51:15 (Victoria Drake) now on, my tweets are ephemeral. Here’s why I’m deleting all my old tweets, and the AWS Lambda function I’m using to do all this for free. <img src="" width="640" height="320"/>From now on, my tweets are ephemeral. Here’s why I’m deleting all my old tweets, and the AWS Lambda function I’m using to do all this for free.

Click here to skip to the code part.

Stuff and opinions

I’ve only been a one-bag nomad for a little over a year and a half. Before that, I lived as most people do in an apartment or a house. I owned furniture, more clothing than I strictly needed, and enough “stuff” to fill at least a few moving boxes. If I went to live somewhere else, moving for school or family or work, I packed up all my things and brought them with me. Over the years, I accumulated more and more stuff.

Adopting what many would call a minimalist lifestyle has rapidly changed a lot of my longstanding views. Giving away all my stuff (an idea I once thought to be interesting in principle but practically a little bit ridiculous) has become normal. It’s normal for me, now, to not own things that I don’t use on a regular basis. I don’t keep wall shelves packed with old books or dishes or clothing or childhood toys because those items aren’t relevant to me anymore. I just keep fond memories, instead.

Imagine, for a moment, that I still lived in a house. Imagine that in that house, on the fridge, is a drawing I made when I was six-years-old. In the bottom right corner of that drawing scribbled in green crayon are the words “broccoli is dumb - Victoria, Age 6.”

If you were in my house and saw that drawing on the fridge, would you assume that the statement “broccoli is dumb” comprised an accurate and current account of my opinions on broccoli? Of course not. I was six when I wrote that. I’ve had plenty of time to change my mind.

Social media isn’t social

I have a friend whom I’ve known since we were both in kindergarten. We went through grade school together, then spoke to and saw each other on infrequent occasions across the years. We’re both adults now. Sometimes when we chat, we’ll recall some amusing memory from when we were younger. The nature of memory being what it is, I have no illusion that what we recall is recounted with much accuracy. Our impressions of things that happened - mistakes we made and moments of victory alike - are coloured by the experiences we’ve had since then, and all the things we’ve learned. An awkward moment at a school colleague’s birthday party becomes an example of a child learning to socialize, instead of the world-ending moment of embarrassment it probably felt like at the time.

This is how memory works. In a sense, it gets updated, as well it should. People living in small communities remember things that their neighbour did many years ago, but recall them in the context of who their neighbour is now, and what their current relationship is like. This re-colouring of history is an important part of how people heal, make good decisions, and socialize.

Social media does not do this. Your perfectly preserved tweet from five days or five years ago can be recalled with absolute accuracy. For most people, this is not particularly worrying. We tend to tweet about pretty mundane things - things that pop into mind when we’re bored and want someone to notice us. Individually, usually, our old tweets are pretty insignificant. In aggregate, however, they paint a pretty complete picture of a person’s random, unintentionally telling thoughts. This is the problem.

The assumption made of things written in social media and on Twitter specifically is a very different assumption than you might make about someone’s notepad scribble from last week. I’m not endeavoring to speculate why - I’ve just seen enough cases of someone getting publicly flogged for something they posted years ago to know that it does happen. This is weird. If you wouldn’t assume that a notepad scribble from last week or a crayon drawing from decades ago reflects the essence of who someone is now, why would you assume that an old tweet does?

You are not the same person you were last month - you’ve seen things, read things, understood and learned things that have, in some small way, changed you. While a person may have the same sense of self and identity through most of their life, even this grows and changes over the years. We change our opinions, our desires, our habits. We are not stagnant beings, and we should not let ourselves be represented as such, however unintentionally.

Ephemeral tweets

If you look at my Twitter profile page today, you’ll see fewer tweets there than you have fingers (I hope). I’m using ephemeral - a lightweight utility I wrote for use on AWS Lambda - to delete all my tweets older than a few days. I’m doing this for the same reason that I don’t hang on to stuff that I no longer use - that stuff isn’t relevant to me anymore. It doesn’t represent me, either.

The code that makes up ephemeral is written in Go. AWS Lambda creates an environment for each Lambda function, so ephemeral utilizes environment variables for your private Twitter API keys and the maximum age of the tweets you want to keep, represented in hours, like 72h.

var (
 consumerKey       = getenv("TWITTER_CONSUMER_KEY")
 consumerSecret    = getenv("TWITTER_CONSUMER_SECRET")
 accessToken       = getenv("TWITTER_ACCESS_TOKEN")
 accessTokenSecret = getenv("TWITTER_ACCESS_TOKEN_SECRET")
 maxTweetAge       = getenv("MAX_TWEET_AGE")
 logger            = log.New()

func getenv(name string) string {
 v := os.Getenv(name)
 if v == "" {
  panic("missing required environment variable " + name)
 return v

The program uses the anaconda library. It fetches your timeline up to the Twitter API’s limit of 200 tweets per request, then compares each tweet’s date of creation to your MAX_TWEET_AGE variable to decide whether it’s old enough to be deleted. After deleting all the expired tweets, the Lambda function terminates.

func deleteFromTimeline(api *anaconda.TwitterApi, ageLimit time.Duration) {
 timeline, err := getTimeline(api)

 if err != nil {
  log.Error("Could not get timeline")
 for _, t := range timeline {
  createdTime, err := t.CreatedAtTime()
  if err != nil {
   log.Error("Couldn't parse time ", err)
  } else {
   if time.Since(createdTime) > ageLimit {
    _, err := api.DeleteTweet(t.Id, true)
    log.Info("DELETED: Age - ", time.Since(createdTime).Round(1*time.Minute), " - ", t.Text)
    if err != nil {
     log.Error("Failed to delete! ", err)
 log.Info("No more tweets to delete.")


Read the full code here.

For a use case like this, AWS Lambda has a free tier that costs nothing. If you’re any level of developer, it’s an extremely useful tool to become familiar with. For a full walkthrough with screenshots of how to set up a Lambda function that tweets for you, you can read this article. The set up for ephemeral is the same, it just has an opposite function. :)

I forked ephemeral from Adam Drake’s Harold, a Twitter tool that has many useful functions beyond keeping your timeline trimmed. If you have more than 200 tweets to delete at first pass, please use Harold to do that first. You can run Harold with the deletetimeline flag from your terminal.

You may like to first download all your tweets before deleting them for sentimental value.

Why use Twitter at all

In anticipation of the question, let me say that yes, I do use Twitter besides just as a bucket for my Lambda functions to fill and empty. It has its benefits, most related to what I perceive to be its original intended purpose: to be a means of near-instant communication for short, digestible pieces of information reaching a widespread pool of people.

I use it as a way to keep tabs on what’s happening right now. I use it to comment on, joke about, and commiserate with things tweeted by the people I follow right now. By keeping my timeline restricted to only the most recent few days, I feel like I’m using Twitter more like it was meant to be used: a way to join the conversation and see what’s happening in the world right now - instead of just another place to amass more “stuff.”

Running a free Twitter bot on AWS Lambda, 05 Mar 2018 10:29:15 (Victoria Drake) to save some time with automated link sharing on Twitter - for free! <img src="" width="640" height="320"/>If you read About time, you’ll know that I’m a big believer in spending time now on building things that save time in the future. To this end I built a simple Twitter bot in Go that would occasionally post links to my articles and keep my account interesting even when I’m too busy to use it. The tweets help drive traffic to my sites, and I don’t have to lift a finger.

I ran the bot on an Amazon EC2 instance for about a month. My AWS usage has historically been pretty inexpensive (less than the price of a coffee in most of North America), so I was surprised when the little instance I was using racked up a bill 90% bigger than the month before. I don’t think AWS is expensive, to be clear, but still… I’m cheap. I want my Twitter bot, and I want it for less.

I’d been meaning to explore AWS Lamda, and figured this was a good opportunity. Unlike an EC2 instance that is constantly running (and charging you for it), Lambda charges you per request and according to the duration of time your function takes to run. There’s a free tier, too, and the first 1 million requests, plus a certain amount of compute time, are free. Roughly translated to running a Twitter bot that posts for you, say, twice a day, your monthly cost for using Lambda would total… carry the one… nothing. I’ve been running my Lambda function for a couple weeks now, completely free.

When recently it came to me to take the reigns of the @freeCodeCampTO Twitter, I decided to employ a similar strategy, and also use this opportunity to document the process for you, dear reader.

So if you’re currently using a full-time running instance for a task that could be served by a cron job, this is the article for you. I’ll cover how to write your function for Lambda, how to get it set up to run automatically, and as a sweet little bonus, a handy bash script that updates your function from the command line whenever you need to make a change. Let’s do it!

Is Lambda right for you

When I wrote the code for my Twitter bot in Go, I intended to have it run on an AWS instance and borrowed heavily from Francesc’s awesome Just for Func episode. Some time later I modified it to randomly choose an article from my RSS feeds and tweet the link, twice a day. I wanted to do something similar for the @freeCodeCampTO bot, and have it tweet an inspiring quote about programming every morning.

This is a good use case for Lambda because:

  • The program should execute once
  • It runs on a regular schedule, using time as a trigger
  • It doesn’t need to run constantly

The important thing to keep in mind is that Lambda runs a function once in response to an event that you define. The most widely applicable trigger is a simple cron expression, but there are many other trigger events you can hook up. You can get an overview here.

Write a Lambda function

I found this really straightforward to do in Go. First, grab the aws-lambda-go library:

go get

Then make this your func main():

func main() {

Where tweetFeed is the name of the function that makes everything happen. While I won’t go into writing the whole Twitter bot here, you can view my code on GitHub.

Setting up AWS Lambda

I’m assuming you already have an AWS account. If not, first things first here:

1. Create your function

Find AWS Lambda in the list of services, then look for this shiny button:

Create function

We’re going to author a function from scratch. Name your function, then under Runtime choose “Go 1.x”.

Under Role name write any name you like. It’s a required field but irrelevant for this use case.

Click Create function.

Author from scratch

2. Configure your function

You’ll see a screen for configuring your new function. Under Handler enter the name of your Go program.

Configure your function

If you scroll down, you’ll see a spot to enter environment variables. This is a great place to enter the Twitter API tokens and secrets, using the variable names that your program expects. The AWS Lambda function will create the environment for you using the variables you provide here.

Environment variables

No further settings are necessary for this use case. Click Save at the top of the page.

3. Upload your code

You can upload your function code as a zip file on the configuration screen. Since we’re using Go, you’ll want to go build first, then zip the resulting executable before uploading that to Lambda.

…Of course I’m not going to do that manually every time I want to tweak my function. That’s what awscli and this bash script is for!

go build && \
zip fcc-tweet && \
rm fcc-tweet && \
aws lambda update-function-code --function-name fcc-tweet --zip-file fileb:// && \

Now whenever I make a tweak, I just run bash

If you’re not already using AWS Command Line Interface, do pip install awscli and thank me later. Find instructions for getting set up and configured in a few minutes here under Quick Configuration.

4. Test your function

Wanna see it go? Of course you do! Click “Configure test events” in the dropdown at the top.

Configure test events

Since you’ll use a time-based trigger for this function, you don’t need to enter any code to define test events in the popup window. Simply write any name under Event name and empty the JSON in the field below. Then click Create.

Configuring an empty test event

Click Test at the top of the page, and if everything is working correctly you should see…

Test success notification

5. Set up CloudWatch Events

To run our function as we would a cron job - as a regularly scheduled time-based event - we’ll use CloudWatch. Click CloudWatch Events in the Designer sidebar.

CloudWatch Events trigger

Under Configure triggers, you’ll create a new rule. Choose a descriptive name for your rule without spaces or punctuation, and ensure Schedule expression is selected. Then input the time you want your program to run as a rate expression, or cron expression.

A cron expression looks like this: cron(0 12 * * ? *)

Minutes Hours Month Day of week Year In English
0 12 * ? * Run at noon (UTC) every day

For more on how to write your cron expressions, read this.

See the current time in UTC.

If you want your program to run twice a day, say once at 10am and again at 3pm, you’ll need to set two separate CloudWatch Events triggers and cron expression rules.

Click Add.

Set cron expression rule

Watch it go

That’s all you need to get your Lambda function up and running! Now you can sit back, relax, and do more important things than share your RSS links on Twitter.

Moving to a new domain without breaking old links with AWS & Disqus, 10 Jan 2018 08:56:20 (Victoria Drake) moved my site's blog to its own domain without breaking old links or losing comments. Here's how. <img src="" width="640" height="320"/>I started blogging about my nomadic travels last year, and so far the habit has stuck. Like all side projects, I won’t typically invest heavily in setting up web properties before I can be reasonably certain that such an investment is worth my time or enjoyment. In other words: don’t buy the domain until you’ve proven to yourself that you’ll stick with it!

After some months of regular posting I felt I was ready to commit (short courtship, I know, but we’re all adults here) and I bought a dedicated domain,

Up until recently, my #NomadLyfe blog was just a subdirectory of my main personal site. Now it’s all grown up and ready to strike out into the world alone! Here’s the setup for the site:

  • Static site in Amazon Web Services S3 bucket
  • Route 53 handling the DNS
  • CloudFront for distribution and a custom SSL certificate
  • Disqus for comments

If you’d like a walk-through for how to set up a new domain with this structure, it’s over here: Hosting your static site with AWS S3, Route 53, and CloudFront. In this post, I’ll just detail how I managed to move my blog to the new site without breaking the