text:
Cloud Computing Journey – Part 1
So, you decided to migrate a system to the cloud. It may be business or IT initiative, but what does it really mean switching between on premise and the cloud?
For ages, we used to manage our IT infrastructure by ourselves, on our own data centers (or network communication cabinets, for small companies…), using our purchased hardware, while maintaining and troubleshooting every software/hardware/network problem.
In the cloud, things change. In the cloud, we are one of the many customers sharing compute resources in multi-tenant environment. We have no control of the hardware or the chosen platform technology (from the servers' hardware vendor to the storage vendor), we barely control the virtualization layer, and don't even get me started talking about troubleshooting the network layer.
There are 3 cloud service models:
- IaaS (Infrastructure as a service) – In this service model, the customer controls (almost) everything from the virtual servers operating system, through the application layer, up until the data itself.
- PaaS (Platform as a service) – In this service model, the customer controls the application layer, up until the data itself.
- SaaS (Software as a service) – In this service model, the customer has access to a close application, but the customer is the data owner and can control permissions (and auditing) of the data.
Once we understood those basic rules, let's analyze what does it really means migrating to the cloud.
The word that differentiates a mature cloud service provider from a rookie is transparency.
Mature cloud service provider won't hesitate to answer tough questions such as "Can I have a copy of your last external audit report?", "Do you have a business continuity plan?", "Do you have an SDLC (Software development lifecycle) process?", etc.
When engaging with cloud service provider, it is important to know as much details about the provider as you can, until you are comfortable enough to sign a contract.
A solid contract will give you assurance about the cloud service provider's ability to fulfill his obligations and will higher the chances of project success.
In the next couple of articles, I will try to pinpoint important tips for a successful cloud project.
Stay tuned for my next article.
Here are some recommended articles:
Getting the Most Out of Your PHP Log Files: A Practical Guide
It could rightfully be said that logs are one of the most underestimated and underutilized tools at a freelance php developer's disposal. Despite the wealth of information they can offer, it is not uncommon for logs to be the last place a developer looks when trying to resolve a problem.
In truth, PHP log files should in many cases be the first place to look for clues when problems occur. Often, the information they contain could significantly reduce the amount of time spent pulling out your hair trying to track down a gnarly bug.
But perhaps even more importantly, with a bit of creativity and forethought, your logs files can be leveraged to serve as a valuable source of usage information and analytics. Creative use of log files can help answer questions such as: What browsers are most commonly being used to visit my site? What's the average response time from my server? What was the percentage of requests to the site root? How has usage changed since we deployed the latest updates? And much, much more.
This article provides a number of tips on how to configure your log files, as well as how to process the information that they contain, in order to maximize the benefit that they provide.
Although this article focuses technically on logging for PHP developers, much of the information presented herein is fairly "technology agnostic" and is relevant to other languages and technology stacks as well.
Note: This article presumes basic familiarity with the Unix shell. For those lacking this knowledge, an Appendix is provided that introduces some of the commands needed for accessing and reading log files on a Unix system.
Our PHP Log File Example Project
As an example project for discussion purposes in this article, we will take Symfony Standard as a working project and we'll set it up on Debian 7 Wheezy with
nginx, and
composer create-project symfony/framework-standard-edition my "2.6.*"
This quickly gives us a working test project with a nice UI.
Tips for Configuring Your Log Files
Here are some pointers on how to configure your log files to help maximize their value.
Error Log Confguration
Error logs represent the most basic form of logging; i.e., capturing additional information and detail when problems occur. So, in an ideal world, you would want there to be no errors and for your error logs to be empty. But when problems do occur (as they invariably do), your error logs should be one of the first stops you make on your debugging trail.
Error logs are typically quite easy to configure.
For one thing, all error and crash messages can be logged in the error log in exactly the same format in which they would otherwise be presented to a user. With some simple configuration, the end user will never need to see those ugly error traces on your site, while devops will be still able to monitor the system and review these error messages in all their gory detail. Here's how to setup this kind of logging in PHP:
log_errors = On
error_reporting = E_ALL
error_log = /path/to/my/error/log
Another two lines that are important to include in a log file for a live site, to preclude gory levels of error detail from being to presented to users, are:
display_errors = Off
display_startup_errors = Off
System Log (
syslog) Confguration
There are many generally compatible implementations of the
syslog daemon in the open source world including:
sysklogd– most often seen on BSD family systems, CentOS, Mac OS X, and others
syslog-ng– default for modern Gentoo and SuSE builds
rsyslogd– widely used on the Debian and Fedora families of operating systems
(Note: In this article, we'll be using
rsyslogd for our examples.)
The basic syslog configuration is generally adequate for capturing your log messages in a system-wide log file (normally
/var/log/syslog; might also be
/var/log/messages or
/var/log/system.log depending on the distro you're using).
The system log provides several log facilities, eight of which (
LOG_LOCAL0 through
LOG_LOCAL7) are reserved for user-deployed projects. Here, for example, is how you might setup
LOG_LOCAL0 to write to 4 separate log files, based on logging level (i.e., error, warning, info, debug):
# /etc/rsyslog.d/my.conf
local0.err /var/log/my/err.log
local0.warning /var/log/my/warning.log
local0.info -/var/log/my/info.log
local0.debug -/var/log/my/debug.log
Now, whenever you write a log message to
LOG_LOCAL0 facility, the error messages will go to
/var/log/my/err.log, warning messages will go to
/var/log/my/warning.log, and so on. Note, though, that the syslog daemon filters messages for each file based on the rule of "this level and higher". So, in the example above, all error messages will appear in all four configured files, warning messages will appear in all but the error log, info messages will appear in the info and debug logs, and debug messages will only go to
One additional important note; The
- signs before the info and debug level files in the above configuration file example indicate that writes to those files should be perfomed asynchronously (since these operations are non-blocking). This is typically fine (and even recommended in most situations) for info and debug logs, but it's best to have writes to the error log (and most prpobably the warning log as well) be synchronous.
In order to shut down a less important level of logging (e.g., on a production server), you may simply redirect related messages to
/dev/null (i.e., to nowhere):
local0.debug /dev/null # -/var/log/my/debug.log
One specific customization that is useful, especially to support some of the PHP log file parsing we'll be discussing later in this article, is to use tab as the delimiter character in log messages. This can easily be done by adding the following file in
# /etc/rsyslog.d/fixtab.conf
$EscapeControlCharactersOnReceive off
And finally, don't forget to restart the syslog daemon after you make any configuration changes in order for them to take effect:
service rsyslog restart
Server Log Confguration
Unlike application logs and error logs that you can write to, server logs are exclusively written to by the corresponding server daemons (e.g., web server, database server, etc.) on each request. The only "control" you have over these logs is to the extent that the server allows you to configure its logging functionality. Though there can be a lot to sift through in these files, they are often the only way to get a clear sense of what's going on "under the hood" with your server.
Let's deploy our Symfony Standard example application on nginx environment with MySQL storage backend. Here's the nginx host config we will be using:
server {
server_name my.log-sandbox;
root /var/www/my/web;
location / {
# try to serve file directly, fallback to app.php
try_files $uri /app.php$is_args$args;
# This rule should only be placed on your development environment
# In production, don't include this and don't deploy app_dev.php or config.php
location ~ ^/(app_dev|config)\.php(/|$) {
fastcgi_pass unix:/var/run/php5-fpm.sock;
fastcgi_split_path_info ^(.+\.php)(/.*)$;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param HTTPS off;
location ~ ^/app\.php(/|$) {
fastcgi_pass unix:/var/run/php5-fpm.sock;
fastcgi_split_path_info ^(.+\.php)(/.*)$;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param HTTPS off;
# Prevents URIs that include the front controller. This will 404:
# http://domain.tld/app.php/some-path
# Remove the internal directive to allow URIs like this
error_log /var/log/nginx/my_error.log;
access_log /var/log/nginx/my_access.log;
With regard to the last two directives above:
access_log represents the general requests log, while
error_log is for errors, and, as with application error logs, it's worth setting up extra monitoring to be alerted to problems so you can react quickly.
Note: This is an intentionally oversimplified nginx config file that is provided for example purposes only. It pays almost no attention to security and performance and shouldn't be used as-is in any "real" environment.
This is what we get in
/var/log/nginx/my_access.log after typing
http://my.log-sandbox/app_dev.php/ in browser and hitting
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /app_dev.php/ HTTP/1.1" 200 6715 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /bundles/framework/css/body.css HTTP/1.1" 200 6657 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /bundles/framework/css/structure.css HTTP/1.1" 200 1191 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /bundles/acmedemo/css/demo.css HTTP/1.1" 200 2204 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /bundles/acmedemo/images/welcome-quick-tour.gif HTTP/1.1" 200 4770 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /bundles/acmedemo/images/welcome-demo.gif HTTP/1.1" 200 4053 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /bundles/acmedemo/images/welcome-configure.gif HTTP/1.1" 200 3530 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:28 +0300] "GET /favicon.ico HTTP/1.1" 200 6518 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
192.168.56.1 - - [26/Apr/2015:16:13:30 +0300] "GET /app_dev.php/_wdt/e50d73 HTTP/1.1" 200 13265 "http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
This shows that, for serving one page, the browser actually performs 9 HTTP calls. 7 of those, however, are requests to static content, which are plain and lightweight. However, they still take network resources and this is what can be optimized by using various sprites and minification techniques.
While those optimisations are to be discussed in another article, what's relavant here is that we can log requests to static contents separately by using another
location directive for them:
location ~ \.(jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js)$ {
access_log /var/log/nginx/my_access-static.log;
Remember that nginx
location performs simple regular expression matching, so you can include as many static contents extensions as you expect to dispatch on your site.
Parsing such logs is no different than parsing application logs.
Other Logs Worth Mentioning
Two other PHP logs worth mentioning are the debug log and data storage log.
The Debug Log
Another convenient thing about nginx logs is the debug log. We can turn it on by replacing the
error_log line of the config with the following (requires that the nginx debug module be installed):
error_log /var/log/nginx/my_error.log debug;
The same setting applies for Apache or whatever other webserver you use.
And incidentally, debug logs are not related to error logs, even though they are configured in the
Although the debug log can indeed be verbose (a single nginx request, for example, generated 127KB of log data!), it can still be very useful. Wading through a log file may be cumbersome and tedious, but it can often quickly provide clues and information that greatly help accelerate the debugging process.
In particular, the debug log can really help with debugging nginx configurations, especially the most complicated parts, like
location matching and
rewrite chains.
Of course, debug logs should never be enabled in a production environment. The amount of space they use also and the amount of information that they store means a lot of I/O load on your server, which can degrade the whole system's performance significantly.
Data Storage Logs
Another type of server log (useful for debugging) is data storage logs. In MySQL, you can turn them on by adding these lines:
general_log = 1
general_log_file = /var/log/mysql/query.log
These logs simply contain a list of queries run by the system while serving database requests in chronological order, which can be helpful for various debugging and tracing needs. However, they should not stay enabled on production systems, since they will generate extra unnecessary I/O load, which affects performance.
Writing to Your Log Files
PHP itself provides functions for opening, writing to, and closing log files (
syslog(), and
closelog(), respectively).
There are also numerous logging libraries for the PHP developer, such as Monolog (popular among Symfonyand Laravel users), as well as various framework-specific implementations, such as the logging capabilities incorporated into CakePHP. Generally, libraries like Monolog not only wrap
syslog() calls, but also allow using other backend functionality and tools.
Here's a simple example of how to write to the log:
openlog(uniqid(), LOG_ODELAY, LOG_LOCAL0);
syslog(LOG_INFO, 'It works!');
Our call here to
- configures PHP to prepend a unique identifier to each system log message within the script's lifetime
- sets it to delay opening the syslog connection until the first
syslog()call has occurred
- sets
LOG_LOCAL0as the default logging facility
Here's what the contents of the log file would look like after running the above code:
# cat /var/log/my/info.log
Mar 2 00:23:29 log-sandbox 54f39161a2e55: It works!
Maximizing the Value of Your PHP Log Files
Now that we're all good with theory and basics, let's see how much we can get from logs making as few changes as possible to our sample Symfony Standard project.
First, let's create the scripts
src/log-begin.php (to properly open and configure our logs) and
src/log-end.php(to log information about successful completion). Note that, for simplicity, we'll just write all messages to the info log.
# src/log-begin.php
define('START_TIME', microtime(true));
openlog(uniqid(), LOG_ODELAY, LOG_LOCAL0);
syslog(LOG_INFO, 'BEGIN');
syslog(LOG_INFO, "URI\t{$_SERVER['REQUEST_URI']}");
$browserHash = substr(md5($_SERVER['HTTP_USER_AGENT']), 0, 7);
syslog(LOG_INFO, "CLIENT\t{$_SERVER['REMOTE_ADDR']}\t{$browserHash}"); <br />
# src/log-end.php
syslog(LOG_INFO, "DISPATCH TIME\t" . round(microtime(true) - START_TIME, 2));
syslog(LOG_INFO, 'END');
And let's require these scripts in
require_once(dirname(__DIR__) . '/src/log-begin.php');
syslog(LOG_INFO, "MODE\tPROD");
# original app.php contents
require_once(dirname(__DIR__) . '/src/log-end.php');
For the development environment, we want to require these scripts in
app_dev.php as well. The code to do so would be the same as above, except we would set the
MODE to
DEV rather than
We also want to track what controllers are being invoked, so let's add one more line in
Acme\DemoBundle\EventListener\ControllerListener, right at the beginning of the
ControllerListener::onKernelController() method:
syslog(LOG_INFO, "CONTROLLER\t" . get_class($event->getController()[0]));
Note that these changes total a mere 15 extra lines of code, but can collectively yield a wealth of information.
Analyzing the Data in Your Log Files
For starters, let's see how many HTTP requests are required to serve each page load.
Here's the info in the logs for one request, based on the way we've configured our logging:
Mar 3 12:04:20 log-sandbox 54f58724b1ccc: BEGIN
Mar 3 12:04:20 log-sandbox 54f58724b1ccc: URI /app_dev.php/
Mar 3 12:04:20 log-sandbox 54f58724b1ccc: CLIENT 192.168.56.1 1b101cd
Mar 3 12:04:20 log-sandbox 54f58724b1ccc: MODE DEV
Mar 3 12:04:23 log-sandbox 54f58724b1ccc: CONTROLLER Acme\DemoBundle\Controller\WelcomeController
Mar 3 12:04:25 log-sandbox 54f58724b1ccc: DISPATCH TIME 4.51
Mar 3 12:04:25 log-sandbox 54f58724b1ccc: END
Mar 3 12:04:25 log-sandbox 54f5872967dea: BEGIN
Mar 3 12:04:25 log-sandbox 54f5872967dea: URI /app_dev.php/_wdt/59b8b6
Mar 3 12:04:25 log-sandbox 54f5872967dea: CLIENT 192.168.56.1 1b101cd
Mar 3 12:04:25 log-sandbox 54f5872967dea: MODE DEV
Mar 3 12:04:28 log-sandbox 54f5872967dea: CONTROLLER Symfony\Bundle\WebProfilerBundle\Controller\ProfilerController
Mar 3 12:04:29 log-sandbox 54f5872967dea: DISPATCH TIME 4.17
Mar 3 12:04:29 log-sandbox 54f5872967dea: END
So now we know that each page load is actually served with two HTTP requests.
Actually there are two points worth mentioning here. First, the two requests per page load is for using Symfony in dev mode (which I have done throughout this article). You can identify dev mode calls by searching for
/app-dev.php/ URL chunks. Second, let's say each page load is served with two subsequent requests to the Symfony app. As we saw earlier in the nginx access logs, there are actually more HTTP calls, some of which are for static content.
OK, now let's surf a bit on the demo site (to build up the data in the log files) and let's see what else we can learn from these logs.
How many requests were served in total since the beginning of the logfile?
# grep -c BEGIN info.log
Did any of them fail (did the script shut down without reaching the end)?
# grep -c END info.log
We see that the number of
BEGIN and
END records match, so this tells us that all of the calls were successful. (If the PHP script had not completed successfully, it would not have reached execution of the
What was the percentage of requests to the site root?
# `grep -cE "\s/app_dev.php/$" info.log`
This tells us that there were 2 page loads of the site root. Since we previously learned that (a) there are 2 requests to the app per page load and (b) there were a total of 10 HTTP requests, the percentage of requests to the site root was 40% (i.e., 2×2/10).
Which controller class is responsible for serving requests to site root?
# grep -E "\s/$|\s/app_dev.php/$" info.log | head -n1
Mar 3 12:04:20 log-sandbox 54f58724b1ccc: URI /app_dev.php/
# grep 54f58724b1ccc info.log | grep CONTROLLER
Mar 3 12:04:23 log-sandbox 54f58724b1ccc: CONTROLLER Acme\DemoBundle\Controller\WelcomeController
Here we used the unique ID of a request to check all log messages related to that single request. We thereby were able to determine that the controller class responsible for serving requests to site root is
Which clients with IPs of subnet
192.168.0.0/16have accessed the site?
# grep CLIENT info.log | cut -d":" -f4 | cut -f2 | sort | uniq
As expected in this simple test case, only my host computer has accessed the site. This is of course a very simplistic example, but the capability that it demonstrates (of being able to analyse the sources of the traffic to your site) is obviously quite powerful and important.
How much of the traffic to my site has been from FireFox?
1b101cd as the hash of my Firefox User-Agent, I can answer this question as follows:
# grep -c 1b101cd info.log
# grep -c CLIENT info.log
Answer: 80% (i.e., 8/10)
What is the percentage of requests that yielded a "slow" response?
For purposes of this example, we'll define "slow" as taking more than 5 seconds to provide a response. Accordingly:
# grep "DISPATCH TIME" info.log | grep -cE "\s[0-9]{2,}\.|\s[5-9]\."
Answer: 20% (i.e., 2/10)
Did anyone ever supply GET parameters?
# grep URI info.log | grep \?
No, Symfony standard uses only URL slugs, so this also tells us here that no one has attempted to hack the site.
These are just a handful of relatively rudimentary examples of the ways in which logs files can be creatively leveraged to yield valuable usage information and even basic analytics.
Other Things to Keep in Mind
Keeping Things Secure
Another heads-up is for security. You might think that logging requests is a good idea, in most cases it indeed is. However, it's important to be extremely careful about removing any potentially sensitive user information before storing it in the log.
Fighting Log File Bloat
Since log files are text files to which you always append information, they are constantly growing. Since this is a well-known issue, there are some fairly standard approaches to controlling log file growth.
The easiest is to rotate the logs. Rotating logs means:
- Periodically replacing the log with a new empty file for further writing
- Storing the old file for history
- Removing files that have "aged" sufficiently to free up disk space
- Making sure the application can write to the logs uniterrupted when these file changes occur
The most common solution for this is
logrotate, which ships pre-installed with most *nix distributions. Let's see a simple configuration file for rotating our logs:
rotate 7
invoke-rc.d rsyslog rotate > /dev/null
Another, more advanced approach is to make
rsyslogd itself write messages into files, dynamically created based on current date and time. This would still require a custom solution for removal of older files, but lets devops manage timeframes for each log file precisely. For our example:
$template DynaLocal0Err, "/var/log/my/error-%$NOW%-%$HOUR%.log"
$template DynaLocal0Info, "/var/log/my/info-%$NOW%-%$HOUR%.log"
$template DynaLocal0Warning, "/var/log/my/warning-%$NOW%-%$HOUR%.log"
$template DynaLocal0Debug, "/var/log/my/debug-%$NOW%-%$HOUR%.log"
local1.err -?DynaLocal0Err
local1.info -?DynaLocal0Info
local1.warning -?DynaLocal0Warning
local1.debug -?DynaLocal0Debug
This way,
rsyslog will create an individual log file each hour, and there won't be any need for rotating them and restarting the daemon. Here's how log files older than 5 days can be removed to accomplish this solution:
find /var/log/my/ -mtime +5 -print0 | xargs -0 rm
Remote Logs
As the project grows, parsing information from logs gets more and more resource hungry. This not only means creating extra server load; it also means creating peak load on the CPU and disk drives at the times when you parse logs, which can degrade server response time for users (or in a worst case can even bring the site down).
To solve this, consider setting up a centralized logging server. All you need for this is another box with UDP port 514 (default) open. To make
rsyslogd listen to connections, add the following line to its config file:
$UDPServerRun 514
Having this, setting up the client is then as easy as:
HOSTNAME is the host name of your remote logging server).
While this article has demonstrated some of the creative ways in which log files can offer way more valuable information than you may have previously imagined, it's important to emphasize that we've only scratched the surface of what's possible. The extent, scope, and format of what you can log is almost limitless. This means that – if there's usage or analytics data you want to extract from your logs – you simply need to log it in a way that will make it easy to subsequently parse and analyze. Moreover, that analysis can often be performed with standard Linux command line tools like
sed, or
Indeed, PHP log files are a most powerful tool that can be of tremendous benefit.
Code on GitHub: https://github.com/isanosyan/toptal-blog-logs-post-example
Appendix: Reading and Manipulating Log Files in the Unix Shell
Here is a brief intro to some of the more common *nix command line tools that you'll want to be familiar with for reading and manipulating your log files.
catis perhaps the most simple one. It prints the whole file to the output stream. For example, the following command will print
logfile1to the console:
cat logfile1
>character allows user to redirect output, for example into another file. Opens target stream in write mode (which means wiping target contents). Here's how we replace contents of
tmpfilewith contents of
cat logfile1 > tmpfile
>>redirects output and opens target stream in append mode. Current contents of target file will be preserved, new lines will be added to the bottom. This will append
logfile1contents to
cat logfile1 >> tmpfile
grepfilters file by some pattern and prints only matching lines. Command below will only print lines of
grep Bingo logfile1
cutprints contents of a single column (by number starting from 1). By default searches for tab characters as delimiters between column. For example, if you have file full of timestamps in format
YYYY-MM-DD HH:MM:SS, this will allow you to print only years:
cut -d"-" -f1 logfile1
headdisplays only the first lines of a file
taildisplays only the last lines of a file
sortsorts lines in the output
uniqfilters out duplicate lines
wccounts words (or lines when used with the
|(i.e., the "pipe" symbol) supplies output from one command as input to the next. Pipe is very convenient for combining commands. For example, here's how we can find months of 2014 that occur within a set of timestamps:
grep -E "^2014" logfile1 | cut -d"-" -f2 | sort | uniq
Here we first match lines against regular expression "starts with 2014", then cut months. Finally, we use combination of
sort and
uniq to print occurrences only once.
How to Use E-mail PGP Encryption
Every day internet users send more than 200 billion emails and this statistics makes anonymous e-mail a number one feature to use in day-to-day communication.
Below here is a step by step guide to set up PGP encryption and communicate securely online. The tutorial is developed using Privatoria E-mail service, though one can use any e-mail client supporting this feature. The device used in this installation wizard is MacBook Air, OS X Yosemite. Pay attention that you can set up PGP on any device, regardless of OS.
Step 1
Please login to your Control Panel and go to Services /Anonymous E-Mail / Go to your inbox. You will be redirected to your mail inbox.
Step 2
Go to Settings section in the right top corner of your page. Click Open PGP – Enable OpenPGP – Save.
To start sending encrypted emails you need to first of all generate a KEY and send it to the recipient.
Click on Generate new key. It is recommended to set own password on this step and use it each and every time you want to send/receive encrypted emails. It provides additional security meaning that your PGP email can never be decrypted and/or your signature is really yours! Please note that this password cannot be reset, but you can create a new one in case you forget it 🙂
Click Generate and wait for about half a minute until it is being generated. Once ready, you will see the following screen
To send an encrypted email you will need to use Public Key. Click View and then choose Send.
Step 3
You will be redirected to email interface ready to send your email. Send this email to the recipient first and wait until they send you their key. Once received click Import. Now, once the key is received you can proceed with sending the actual both-end encrypted email. Type in the recipient's email, add subject and body of your email.
To finalize the encryption click on PGP Sign/Encrypt and proceed further. You will receive a message "OpenPGP supports plain text only. Click OK to remove all the formatting and continue." Click OK.
NB! Private key is your own key that encrypts the data and is being generated for you only.
Email recipient should use any email client supporting PGP encryption function.
Cold War Tech: It's Still Here, And Still Being Used
I'm a Cold War kid. I grew up watching news of Pershing II and SS-20 deployments in Europe, the Soviet war in Afghanistan, with some Terminator and Top Gun VHS action on the side. Yugoslavia was trying to play both sides, and for a while it worked like a charm. It all crashed a couple of years after the Berlin Wall came tumbling down, rendering our unaligned prowess pointless.
I admit this is an odd intro for a tech blog, but bear with me; it will start to make sense. Unlike most Europeans, we had good relations with both blocs. We sold tanks to Kuwait and rocket artillery to Saddam, we bought cheap fuel and MiGs from the Soviets, and in return we exported some stuff they couldn't get directly from the West. I know people who would stay in East Berlin hotels because they were cheaper, then cross the border into West Berlin to work, play and shop, only to cross back via virtually unused border crossings like Checkpoint Charlie, all in a matter of hours.
On one such trip, my dad got me a Commodore C64, which was pressed into service as our Cold War gaming machine. Most 80s video games, and indeed a lot of music and films, were inspired by countless proxy wars and the threat of a nuclear apocalypse. As the Wall came down, a lot of people assumed that would be the end of runaway defence spending and that the world would be a safer place. It didn't exactly work out that way, did it?
However, the long-term effect of the Cold War on science and technology is more profound than Nena's 99 Luftbalons, or any Oliver Stone Vietnam flick.
Minuteman: A Cold War Tech Case Study
If you are reading this, you're already using a technology developed for cold warriors; The Internet. That's not all. A lot of tech and infrastructure we take for granted was developed, or at least conceived, during these tumultuous decades.
That constellation of GPS satellites orbiting Earth? It wasn't put up there to geotag selfies or get an Uber ride; it was designed to help the US Strategic Air Command deliver hundreds of megatons worth of instant sunshine on Soviet targets with pinpoint accuracy. Integrated circuits, transistors, solid-state computing? Yep, all developed for the armed forces and paid for by the US taxpayer.
Here is just one example: the sleek and unfathomably deadly LGM-30 Minuteman intercontinental ballistic missile (ICBM). It wasn't the first ICBM out there, but when it appeared, it was revolutionary. It was a solid fuel missile, which meant it could respond to a threat and launch in a minute without having to be fuelled, hence the name. But solid fuel was only part of the story: Solid-state was a lot more interesting from a geek perspective. Prior to Minuteman, ICBMs relied on analogue computers with mechanical gyros and primitive sensors. Since they were wired to a specific target, the target package could not be changed easily. Minuteman was the first mass implementation of a general purpose digital computer; it integrated an autopilot and missile guidance system in one package, with reliable storage that could take the stress of a silo launch. The computer was also capable of storing multiple targets, and was reprogrammable.
Transistors were nothing new at that point; they were developed years before by Bell Labs. Yes, these primitive transistors were almost exclusively reserved for the military-industrial complex. Uncle Sam was the sole customer for virtually all early computers and chips, burning heaps of money. These early transistors offered a quantum leap over vacuum tubes, but they weren't perfect. By today's standards, they were rubbish. The reliability simply wasn't there, and if you needed to launch a few hundred thermonuclear warheads halfway across the planet, you sort of needed a guidance system that wouldn't fail as soon as the candle was lit.
So what do you do when you come across a technical problem you can't solve with money? Simple: You throw more money at it, and that's exactly what the US Air Force did. They burned millions to make the damn things reliable enough to be used in harsh environments and survive the stress of a high-G ascent to space. This was known as the Minuteman High Reliability (Hi-Rel) programme.
It worked, but the USAF got a bit more than they bargained for. In trying to improve a single weapons system, the USAF ended up giving a huge boost to the tech industry in general. Eventually, the Minuteman was upgraded to include a new microchip-based guidance system, with a primitive form of solid-state storage. This Cold War relic has been in service since the Kennedy administration, and the current incarnation has been around for 45 years, receiving multiple hardware and software updates over the years.
So, in outlining the development and evolution of a single strategic weapon delivery system, I have touched on a number of vital technologies we take for granted: reliable transistors, chips, solid-state storage, mass-produced programmable computers and so on. The Minuteman was also the first mobile digital computer.
Some may argue that the legacy of such weapons is that Mutually Assured Destruction (MAD), guaranteed by the nuclear triad, kept superpowers from going to all-out war. It probably did, but in doing so, it also allowed engineers around the world to develop technologies and concepts applicable in various industries and fields of study.
Their real legacy lies in every integrated circuit on the planet.
Capitalist Pioneers Try To Cash In
What could be more capitalist than monetizing instruments of mass murder? The taxpayers paid for their development, not venture capitalists!
Joking aside, it can be argued that the Red Scare of the fifties created Silicon Valley. Most of the money really did come from taxpayers, and most companies that got lucrative defence contracts were quick to make a buck on dual-use technology developed for the military. Remember Bell Labs? A few of their brightest people went on to co-found Fairchild Semiconductor, and eventually created Intel a decade later. The updated Minuteman guidance computer was based on chips from another semiconductor giant: Texas Instruments.
I am not disputing the brilliance of people like Intel co-founders Robert Noyce and Gordon Moore. I have no doubt they would have made their mark on the tech industry even without the biggest arms race in history, but it's also hard to dispute that the tech industry wouldn't have developed at nearly the same pace had there been no government funding. Yes, the taxpayers effectively subsidised the tech industry for decades, but in the long run, they're probably better off. Westinghouse did not need subsidies to develop washing machines and refrigerators, because consumer demand was strong, but in the early days of computing, there was virtually no consumer demand. That's why governments had to step in.
But what did the taxpayer get?
The space and arms race spawned a number of technologies that in turn created countless business opportunities. Even primitive computers had a profound impact on industry. They made energy grids and transportation infrastructure more efficient, helped improve safety in industrial facilities, including sensitive chemical and nuclear facilities, they changed the face of banking, communications, entertainment and so on.
Best of all, we somehow managed not to blow ourselves up with the weapons these technologies made possible, yet at the same time, we turned swords into ploughshares. Back in the fifties, the US and USSR launched initiatives designed to examine civilian uses of nuclear power (including civil engineering nuclear explosives schemes, which went terribly wrong), but they amounted to nothing. It wasn't the might of the atom that changed the world, it was the humble microchip and ancillary technologies developed for countless defence programmes.
Before they made their mark in science and beat Gary Kasparov at the chess table, supercomputers and their analogue predecessors were used to simulate physical processes vital in the development of thermonuclear weapons. An advantage in sheer computing power could yield advances in countless fields. Computer simulations allowed western navies to develop quieter submarines with new screws, digitally optimised to avoid cavitation. Digital signal processors (DSPs) made sonars far more sensitive, and a couple of decades later, advanced DSPs made music sound better. Computer aided design wasn't just used to reduce the radar cross-section of airplanes, it also made our buildings and cars cheaper, safer and more energy efficient.
Some of these efforts resulted in a technological dead-end, but most did not. One of my favourite tech duds was Blue Peacock, a British nuclear landmine (yes, landmine, not bomb), weighing in at 7.2 tons. Since it relied on early 50s technology and had to be buried in the German countryside, the engineers quickly realised the cold could kill the electronics inside, so they tried to figure out how to keep circuits warm. Their solution was so outlandish that it was mistaken for an April Fool's Day joke when the design was declassified on April 1, 2004.
A chicken was to be sealed inside the casing, with enough food and water to stay alive for a week. Its body heat would keep the bomb's electronics operational.
As civilian industries started implementing these cutting edge technologies en masse, our quality of life and productivity shot up exponentially. Our TVs, cars, phones, the clothes we wear, and just about any consumer product we buy: They're all better thanks to the biggest waste of money in history. Granted, we all have trace amounts of Strontium 90 in our bones, but in the big scheme of things, it's a small price to pay for the high-tech world we enjoy so much.
Oh yes, we also got video games out of it. Loads and loads of video games.
Kickstarting Game Development
Video games were pioneered on the earliest digital computers (and some analogue ones as well). In fact,Tennis for Two, arguably the first game to use a graphical display, was developed for an analogue computer in 1958. However, not even Bond villains had computers at that point, so the rise of the video game industry had to wait for hardware to mature.
By the mid to late seventies, microchips became cheap enough for mass market applications. Now that we had the hardware, we just needed some software developers and a use-case for cheap chips. Since the average consumer was not interested in expensive and complicated computers that were designed for big business, attention shifted to gaming; arcades, game consoles and inexpensive computers like the ZX and C64.
These humble machines brought programmable computers to millions of households, hooking a generation of kids on digital entertainment, and creating opportunities for game developers. Consoles and cheap computers brought the arcade to the living room, ushering in a new era of video gaming, and creating countless jobs in the industry. Even the Soviets got in on it with Tetris, the first game from behind the iron curtain.
It wasn't just entertainment. Unlike consoles, the ZX and C64 were proper computers, and geeky kids quickly found new uses for them. They started making demos, they started coding. Chances are you know a lot of these kids, and if you're reading this, you probably work with some of them.
If you're interested in the development of early video games, and what the Cold War had to do with them, I suggest you check out Nuclear Fruit; a new documentary that's a must see for all geeks and gamers born in the 70s and early 80s.
These guys and gals went on to develop a new breed of video games, build successful online businesses, create new technologies and revolutionise the digital world, all in the space of a decade. A generation that grew up with the constant threat of nuclear war, enjoying dystopian science fiction, helped make the world a better place. They didn't develop Skynet, they developed millions of mobile and web apps instead.
So, no Terminators. At least, not yet.
Cold War 2.0 And The Emergence Of New Threats
This is not a geopolitical blog, but if you happen to follow the news, you probably know the world is a messed up place. No, the end of the Cold War didn't bring an era of peace and stability, and there's already talk of a "Second Cold War," or worse, a "hot" war. While most of these worries are nothing more than hype and sensationalism, a number of serious threats remain. The threat of nuclear annihilation is all but gone, but the technology we love so much has created a host of potential threats and issues, ranging from privacy and security, to ethical concerns.
Thankfully, we aren't likely to see an arms race to rival the one we witnessed in the 20th Century, but we don't have to. The same technology that makes our lives easier and more productive can also be used against us. The digital infrastructure we rely on for work and play is fragile and can be targeted by criminals, foreign governments, non-state actors, and even lone nutjobs with a grudge.
These new threats include, but are not limited to:
- Cybercrime
- State-sponsored cyber warfare
- Misuse of autonomous vehicle technology
- Privacy breaches
- Mass surveillance abuses
- Use of secure communications for criminal/terrorist activities
All pose a serious challenge and the industry is having trouble keeping up. My argument is simple: We no longer have to develop ground-breaking technology to get an edge in geopolitical struggles, but we will continue to develop technologies and methods of tackling new threats and problems. It's a vicious circle since these new threats are made possible by our reliance on digital communications and the wide availability of various technologies that can be employed by hostile organisations and individuals.
Cybercrime is usually associated with identity theft and credit card fraud, but it's no longer limited to these fields. The advent of secure communication channels has allowed criminals to expand into new niches. The scene has come a long way since the romanticised exploits of phone phreaks like Steve Wozniak. Some offer hacking for hire, others are willing to host all sorts of illicit content, no questions asked. Some groups specialise in money laundering, darknet drug bazaars, and so on. The biggest threat with this new generation of cybercrime is that you no longer have to possess many skills to get involved. As cybercrime matures, different groups specialise in different activities, and they can be hired.
State-sponsored cyberwarfare poses a serious threat to infrastructure, financial systems, and national security. However, there is really not much an individual can do in the face of these threats, so there's no point in wasting time on them in this post. Another form of economic warfare could be to deprive a nation or region of Internet access. It has happened before, sometimes by accident, sometimes by government decree and enemy action.
Commercial drones don't have much in common with their military counterparts. Their range and payload are very limited, and while a military drone can usually loiter over an area for hours on end, the endurance of hobbyist drones is limited to minutes rather than hours. This does not mean they cannot be used for crime; they can still invade someone's privacy, smuggle drugs across a border, or even carry explosives. Autonomous cars are still in their infancy, so I don't feel the need to discuss the myriad of questions they will raise.
Privacy remains one of the biggest Internet-related concerns expressed by the average person. This is understandable; we have moved so much of our daily lives to the digital sphere, placing our privacy at risk. People don't even have to be specifically targeted to have their privacy and personal integrity compromised. Most data that makes its way online is released in the form of massive dumps following a security breach affecting many, if not all, users of a particular online service. People will continue to demand more privacy, and in turn clients will demand more security from software engineers (who aren't miracle workers and can't guarantee absolute security and privacy).
Mass surveillance is usually performed by governments and should not represent a threat to the average citizen or business. However, it's still a potential threat as it can be abused by disgruntled workers, foreign governments, or by way of data breaches. The other problem is the sheer cost to the taxpayer; mass surveillance doesn't come cheap and we will continue to see more of it.
Most governments wouldn't bother with mass surveillance and metadata programmes if they weren't facing very real threats. The same technology developed to keep our communications and online activities private can be abused by all sorts of individuals we wouldn't like to meet in a dark alley. The list includes multinational crime syndicates, terrorists and insurgents. However, not all of this communication needs to be encrypted and secure. The point of propaganda is to make it widely available to anyone, and the Internet has given every crackpot with a smartphone the biggest megaphone in history, with global reach, free of charge. You can use the Internet to rally a million people around a good cause in a matter of days, but the same principles can be applied to a bad cause. If the target audience is people willing to join a death cult with a penchant for black flags, you don't need a million people; you just need a few dozen.
The Difference Between Science And Science Fiction
For all their brilliance, the science fiction authors who helped shape popular culture in the 20th Century didn't see the real future coming. They didn't exactly envision the Internet, let alone its profound impact on society.
Sorry to burst your bubble, but Terminators and Artificial Intelligence (AI) aren't a threat yet, and won't be anytime soon. The real threats are more down to earth, but that does not mean we can afford to ignore them. You don't need a Terminator to create havoc, all you need is a few lines of really nasty code that can disrupt the infrastructure, causing all sorts of problems. You don't need a super-intelligent automaton from the future to cause damage. Since eBay doesn't carry Terminators, it's a lot easier to use an off-the-shelf drone, programmed to deliver a payload to a specific target: drugs to a trafficker, or an explosive charge to a VIP.
But these aren't the biggest threats, they're just potential threats: something for a Hollywood script, not a tech blog.
The real threats are criminal in nature, but they tend to stay in the cyber realm. You don't have to physically move anything to move dirty money and information online. Law enforcement is already having a hard time keeping up with cybercrime, which seems to be getting worse. While it's true that the crime rate in developed countries is going down, these statistics don't paint the full picture. A few weeks ago, the British Office for National Statistics (ONS) reported a twofold increase in the crime rate for England and Wales, totalling more than 11.6 million offences. The traditional crime rate continued to fall, but the statistics included 5.1 million online fraud incidents.
The cost of physical crime is going down, but the cost of cybercrime is starting to catch up. I strongly believe the industry will have to do more to bolster security, and governments will have to invest in online security and crime prevention as well.
Just in case you are into dystopian fiction and don't find criminal threats exciting, another frightening development would be data monopolisation: A process in which industry giants would command such a competitive lead, made possible by their vast user base, as to render competition pointless, thus stifling innovation.
Yes, I am aware that Terminators would make for a more eventful future and interesting blog post, but we're not there yet.
Source: Toptal
What To Look Out For In Software Development NDAs
The demand for technical talent, and the ease with which information can be shared, has increased entrepreneurs' reliance on business relationships with outsiders. It has never been easier for an entrepreneur to find, meet, communicate and eventually enter into some sort of business relationship with an individual or company that is otherwise not associated with the business.
Moreover, sky-high valuations and fairytale overnight success stories have fueled the notion that even a basic idea can be worth millions, if not billions, in a relatively short time. In light of these factors, you might presume that Non-Disclosure Agreements (NDAs) have been widely accepted in the tech world as a means to protect sensitive and potentially valuable information from theft and abuse. Not so fast.
Before jumping into the debate, though, it helps to have a quick understanding of what an NDA is, what one looks like and, eventually, what to look out for if you're asked to sign one as a freelance software developer.
What Is A Non-Disclosure Agreement?
An NDA is exactly what its name implies — a legal agreement between two or more parties that (i) defines certain confidential information that will be disclosed and (ii) imposes a legal obligation on the receiving party to keep that information confidential. NDAs are most commonly used when a business relationship between two companies or individuals requires the sharing of confidential information.
For example:
Company A, a local retailer, has hired ABC IT Co. to build an online inventory and order management system. To build the system, Company A must provide ABC IT Co. with a list of Company A's suppliers and certain pricing information. Before disclosing its supplier list and pricing information, Company A asks ABC IT Co. to sign an NDA forbidding ABC IT Co. from disclosing or using Company A's confidential information.
If a party to an NDA breaches the agreement, by disclosing or using confidential information for example, the other party to the NDA may sue the breaching party for monetary damages (compensation for lost profits or business), injunctive relief (a court order requiring the breaching party to refrain from taking some action) or specific performance (a court order requiring that the breaching party take some specified action).
So What Do Software Development NDAs Look Like?
NDAs are negotiated legal agreements that can be as simple or as complex as the parties desire. An NDA can be a one page fill-in-the-blank form or a lengthy document drafted from scratch to reflect the unique circumstances of the parties' relationship, the different negotiating leverage of each party, and the nature of the information that will be disclosed.
Although there is no such thing as a one-size-fits-all software NDA, for purposes of this overview, and to understand generally how NDAs work, it's important to appreciate the three "main-event" provisions that are common to all NDAs.
(a) The Definition of Confidential Information:
The definition of "Confidential Information" will set forth the type of disclosed information that is subject to the limitations on use and disclosure and, importantly, the type of disclosed information that is not subject to such limitations.
(b) The Term of the Recipient's Obligations
The term of an NDA sets forth the time limit on the parties' obligations. The term of an NDA may be measured in days, weeks, months or years depending on the circumstances of the relationship and the nature of the disclosed information.
(c) The Limitation on Use and Disclosure:
This provision will describe what a recipient party may do and what a recipient party may not do with disclosed information that falls within the definition of Confidential Information. This provision will almost certainly forbid disclosure of Confidential Information, but may also limit the use of Confidential Information and, in some cases, require that the recipient take certain affirmative steps to protect the confidentiality of Confidential Information.
The NDA Debate: Should You Ask For An NDA? Should You Sign One?
Although NDAs have been around for as long as there has been information worth protecting, the high-tech startup boom has thrust their use into the limelight and sparked a debate as to their value. As an industry that is highly dependent on data and constantly evolving technology, one would think that the high-tech startup world would embrace the use of software development NDAs. To understand why that isn't the case, and to better gauge whether you should ask for an NDA or sign one presented to you, consider the following:
NDAs Are Often Unilateral
NDAs are unilateral when the business relationship requires that only one party disclose confidential information (rather than a mutual exchange of information by each party).
A startup seeks to hire an engineer to build its mobile app and has asked the engineer to sign an NDA. The startup will disclose information to the engineer, but the relationship does not require the engineer to provide confidential information to the startup. The NDA will be unilateral and will impose legal obligations, and potential liability, on the engineer only.
Because only one party is exchanging confidential information, only one party (the recipient party) has a legal obligation to comply with and, as such, only the recipient party is subject to potential liability. What an entrepreneur might view as a means by which to protect an idea, an NDA recipient might view as a one-sided contract.
Entrepreneurs Often Overstate the Need for an NDA
There are surely circumstances where NDAs make sense. Customer lists, pricing information, proprietary formulas and algorithms might have intrinsic value that is best protected by an NDA. Many argue, however, that some entrepreneurs are NDA trigger happy and think that every idea is worthy of legal protection. Ideas though, it is argued, are rarely new and, moreover, often have no value without execution.
NDAs should be asked for only when there is something worth protecting, and many argue that an idea alone does not warrant asking for an NDA. Finally, those most often asked to sign software NDAs – investors and engineers – rarely have any interest in stealing an idea when doing so would likely ruin any professional goodwill and reputation they've earned in their respective professional communities.
NDAs Indicate Mistrust
In a perfect world, business would be business and would never be personal. In reality, though, business is often about perception. What might be "just a contract" to an entrepreneur asking for a software development NDA, may be perceived as an indication of mistrust and a questioning of personal integrity by the person being asked to sign one. NDAs are most often requested at the outset of a business relationship, signaling mistrust and calling into question one's professional integrity may start the relationship off on the wrong foot—even if that wasn't the intention…perception is powerful.
This issue is less of a concern for business relationships where both parties will be disclosing confidential information and, thus, an NDA will be bilateral and both parties subject to legal obligation. Outside of strategic joint ventures, partnerships, mergers and similar arrangements, however, bilateral exchanges are rare and unilateral NDAs are much more common.
NDAs Can Limit An Information Recipient's Ability To Earn A Living
As discussed earlier, an NDA defines a set of information that is to be considered "Confidential Information" and then specifies what a recipient may and may not do with that information during the term of the NDA. Whether an NDA is three pages or three-hundred pages, no contract can predict and plan for every possible circumstance and this limitation often works against the recipient of disclosed information.
What if, after signing an NDA, an engineer is asked to build a similar product or to execute a similar but technically different idea? Is using similar code on a different application a violation of the NDA's non-use provision? What if the engineer learned new skills during the engagement? Can the engineer use those skills for another client? Can the engineer list the client on his or her resume?
There is a real concern that signing even one NDA, whether as an engineer, an investor or otherwise, can drastically shrink one's pool of potential business. At worst, signing an NDA might foreclose a person's ability to work on even slightly related projects. At best, signing an NDA complicates future business development efforts as every new opportunity requires a time consuming analysis of conflicts and liability under each and every NDA that the person may be subject to.
Enforcement Isn't Cheap
The whole point of entering into an NDA is to have some legal remedy if the recipient party discloses confidential information in violation of the agreement. An NDA gives a disclosing party a basis to file a lawsuit seeking money damages and/or a court order against the breaching party. What many NDA proponents don't fully appreciate, however, is the cost of enforcement.
Filing a lawsuit can be extremely costly and time consuming. A lawsuit for breach of contract will very likely require hiring a lawyer to gather evidence, assess possible legal claims, file the initial complaint and supporting documents, depose the allegedly breaching party and any witnesses and related parties, and argue the case before a judge. Lawsuits can take years, and lawyers typically charge by the hour. Before asking for an NDA, one should assess whether the information to be protected is more valuable than the potential cost of enforcement.
Though the above factors have contributed to a move away from NDAs in the startup world, NDAs are not without their value. Whether you should ask for an NDA before disclosing information, or agree to sign one if you're on the receiving end of the equation, depends on the particular circumstances of the intended business relationship and each party's motivation to enter into the relationship. The more valuable the relationship is to a party, the less leverage that party has to negotiate for or against the use of an NDA. The less valuable the relationship is to a party, the more leverage that party has to get its way or walk away. This push and pull is at the heart of all negotiations, the party with the better "Best Alternative to a Negotiated Agreement (BATNA)" has the upper hand.
If You Must Have An NDA…
So what if you're an engineer and the opportunity to work on a particular project outweighs the risk of signing a software development NDA? What if you're a startup and the intrinsic value of your information justifies the need for an NDA, despite the difficulty of finding an engineer that will sign one? If you have to sign an NDA, or if you must ask for one, what are some of the things to look out for and consider?
As is always the case, I strongly suggest seeking the guidance of a competent and licensed attorney.Contracts can get complex quickly and legal rights and obligations shouldn't be left to "winging it." As you're finding an attorney, though, you can start by reviewing the some of the NDA's main operative provisions. The following are a few preliminary things you might consider when presented with or requesting an NDA:
1. Definition of Confidential Information
Recall that this provision defines the type of disclosed information that is subject to the confidentiality obligations of the NDA and, as such, it should reflect the nature of the business relationship and that of the information to be disclosed.
If you're a disclosing party, you'll likely ask for a broad definition of Confidential Information to cover everything that might be disclosed to the receiving party during the course of the relationship. If you're a receiving party, however, you might resist this request and seek instead to narrow the definition to include only specifically designated information such as, for example, written information that is marked "Confidential." Regardless of where the negotiations come out, the parties should think carefully about striking the right balance between a definition of Confidential Information that is too broad (and thus extremely restrictive to the recipient party), on the one hand, and, on the other hand, too narrow (thus minimizing the protective effect to the disclosing party).
Though it's important to determine the information that is to be held in confidence, it is equally important to "carve-out" certain information that is not subject to the confidentiality provisions. Common examples of such carve-outs include information that is or becomes publicly available and information that is lawfully known before entry into the business relationship.
2. Term of Confidentiality
The term of an NDA should reflect the nature of the parties' business relationship and the nature of the information to be disclosed. If the relationship is limited to a one-year engagement, it might not make sense for the term of the NDA to extend too far after termination of the relationship. Similarly, certain types of information become less valuable or sensitive over time. Financial statements, for example, may be particularly valuable at and immediately after the time they are prepared, but probably don't accurately reflect a company's financial health months or years after their preparation. If information is of a type that decreases in value or sensitivity over time, a long term is likely not necessary.
3. Disclosure to Representatives
As discussed throughout this article, NDAs are typically signed by a single disclosing party and a single recipient party. The problem, though, is that a recipient party may not always work alone and, rather, may from time to time need to disclose information protected by an NDA to such recipient party's colleagues, employees or representatives in order to carry out the terms of the business relationship.
David Developer has signed an NDA with BigCo to create a mobile app for BigCo.
During the project, David needs to enlist the help of his colleague, Peter Programmer, to write some code in a language that David is less familiar with. Peter has not signed an NDA, can David disclose information to Peter so that Peter can assist with the project?
Rather than go through the hassle of signing a new NDA for each new person to whom information needs to be disclosed during the course of a project, or trying to predict ahead of time every person to whom information may need to be disclosed, the parties to an NDA may include a representatives provision addressing permitted disclosures to certain defined persons.
The representatives provision is straightforward from a drafting perspective and is simply a definition of "Representatives" that specifies the persons or classes of persons to whom confidential information may be disclosed. A recipient party will likely want the definition to be broad and inclusive of any person with whom the recipient party may collaborate. The disclosing party, of course, will likely want to keep the definition of Representatives as narrow as possible to permit the project to move forward, on the one hand, while maintaining the protections of the software development NDA, on the other. Finally, the disclosing party will very likely wish to include a clause providing that, prior to any disclosure of confidential information to a Representative, the recipient party inform such Representative of the confidential nature of the information and of the terms of the NDA. A representatives clause may look something like the following:
During the Term of this Agreement, the Recipient Party will not disclose the Confidential Information to any person other than the Representatives, provided that, prior to any such disclosure to a Representative, the Recipient Party informs such Representative of the confidential nature of the information and the terms of this Agreement. "Representatives" shall include the employees, independent contractors, partners, agents and other third parties that are or may be engaged by the Recipient Party for purposes of the Project.
4. Non-Disclosure v. Non-Use
This is a big one. As mentioned earlier, NDAs will almost always include a prohibition on disclosure of Confidential Information. Some software NDAs, however, will also prohibit or limit use of Confidential Information. For example:
The Recipient Party agrees that, during the Term of this Agreement, the Recipient Party will not (i) disclose the Confidential Information to any person other than its Representatives and (ii) will not use the Confidential Information for any purpose other than for those purposes directly related to the Project.
Depending on the term of the NDA and the type of information disclosed, restriction on use may not be an issue. If the term is particularly long, however, or the definition of Confidential Information particularly broad, the "use prohibition" may be extraordinarily restrictive on the recipient party. For example, consider the following definition of Confidential Information:
"Confidential Information" includes (i) all information furnished by the Disclosing Party to the Recipient Party, whether furnished before or after the date of this Agreement, whether oral or written, and regardless of the manner in which it was furnished, and (ii) all analyses, compilations, forecasts, studies, interpretations, documents, code and similar work product prepared by the Recipient Party or its Representatives in connection with the Project.
What this means is that, for as long as the NDA is in effect, the Recipient Party cannot disclose or use anyinformation that the Disclosing Party made available to the Recipient Party or any information prepared in connection with the particular Project. Without any carve-outs or qualifications, these clauses could be incredibly limiting.
An engineer signs an NDA which includes the two provisions set out above. During the course of the Project, the engineer learns a new way of putting together common strings of code. The new method could be considered work product that was prepared in connection with the Project and, as such, the engineer may be prohibited from using the method in future projects during the term of the NDA.
Before hearing Startup A's pitch, an investor signs an NDA which includes provisions similar to those set out above. During the pitch, Startup A reveals its most recent financial statements and its strategy for growth. The investor does not invest. A few months later, the investor is approached by a similar startup, Startup B, and asked to attend a pitch. The investor may be precluded from investing in Startup B as doing so might involve use of information learned during Startup A's pitch, even if only remembered by the investor.
The above examples are admittedly extreme, but are used to stress the point that the combination of a broad definition of Confidential Information, an unnecessarily long term, and restrictions on use can be paralyzing. Additionally, these are by no means the only red-flags that can sneak into an NDA and what might be a red-flag for one NDA may be perfectly tolerable for a different business relationship.
So What Do I Do…Specifically?
Though you might now have a better understanding of what an NDA is, what a software development NDA might look like, and why many in the tech world are reluctant to sign them, you might still be wondering what, specifically, you should do when on the receiving end of an NDA. There is no substitute for the advice of a competent attorney, but, with an understanding of the concepts discussed in this article, you can approach the first read of an NDA armed with some knowledge as to what is most important to watch for:
- Is this a bilateral or unilateral NDA? Will both parties be disclosing information? If so, are the parties subject to identical limitations and requirements?
- How broad, or narrow, is the definition of Confidential Information?
- How long are the obligations in effect? Does the term of the NDA match the nature of the business relationship and the information to be disclosed?
- Am I only prohibited from disclosing the Confidential Information, or disclosing and using the Confidential Information?
- Am I permitted to disclose the information to my employees and colleagues who may assist with the project?
- Is this relationship valuable enough to assume a legal obligation that can be enforced in a court?
Finally, the above considerations, and this write-up generally, are not solely for the benefit of those who may be asked to sign an NDA. Certainly, a recipient party should consider very carefully an NDA's provisions before signing, but a party considering asking for an NDA, too, would be wise to consider these factors.
NDAs, like most contracts, have the most value, and are therefore most likely to be signed, when both parties are comfortable with the balance of risks managed by the NDA and the benefit to be realized by the underlying contractual relationship. By considering the perspective of the recipient party, a party asking for an NDA may be better able to tailor the scope of an NDA to match the business relationship and present to the recipient party a fair and balanced agreement.
Though the information in this write-up should give you a good starting point, there is a lot to consider when asking for or presented with an NDA. A competent attorney can work with both parties to draft an NDA that is protective to the disclosing party, without being overly restrictive to the recipient party, and help move the parties towards a mutually beneficial business relationship.
If you want to learn more about legal issues faced by startups and developers, I suggest you check outStartup Law Hacks as well.
Disclaimer: the contents of this article were written and are made available solely as general information and for educational purposes and not to provide specific legal advice of any kind or to establish an attorney-client relationship. This article should not be used as a substitute for competent legal advice from an attorney licensed in your jurisdiction. This article has been written by Bret Stancil in his individual capacity and the views and opinions expressed herein are his own.
Source: Toptal
Integrating Facebook Authentication in AngularJS App with Satellizer
With the arrival of feature-rich front-end frameworks such as AngularJS, more and more logic is being implemented on the front-end, such as data manipulation/validation, authentication, and more. Satellizer, an easy to use token-based authentication module for AngularJS, simplifies the process of implementing authentication mechanism in AngularJS, The library comes with built-in support for Google, Facebook, LinkedIn, Twitter, Instagram, GitHub, Bitbucket, Yahoo, Twitch, and Microsoft (Windows Live) accounts.
In this article, we will build a very simple webapp similar to the one here which allows you to login and see current user's information.
Authentication vs Authorization
These are 2 scary words that you often encounter once your app starts integrating a user system. According to Wikipedia:
Authentication is the act of confirming the truth of an attribute of a single piece of data (a datum) claimed true by an entity.
Authorization is the function of specifying access rights to resources related to information security and computer security in general and to access control in particular.
In layman terms, let's take an example of a blog website with some people working on it. The bloggers write articles and the manager validates the content. Each person can authenticate (login) into the system but their rights (authorisation) are different, so the blogger cannot validate content whereas the manager can.
Why Satellizer
You can create your own authentication system in AngularJS by following some tutorials such as this very detailed one: JSON Web Token Tutorial: An Example in Laravel and AngularJS. I suggest reading this article as it explains JWT (JSON Web Token) very well, and shows a simple way to implement authentication in AngularJS using directly the local storage and HTTP interceptors.
So why Satellizer? The principal reason is that it supports a handful of social network logins such as Facebook, Twitter, etc. Nowadays, especially for websites used on mobile, typing username and password is quite cumbersome and users expect to be able to use your website with little hindrance by using social logins. As integrating the SDK of each social network and following their documentations is quite repetitive, it would be nice to support these social logins with minimal effort.
Moreover Satellizer is an active project on Github. Active is key here as these SDKs change quite frequently and you don't want to read their documentation every now and then (anyone working with Facebook SDK knows how annoying it is)
AngularJS App with Facebook Login
This is where things start to become interesting.
We will build a web app that has regular login/register (i.e. using username, password) mechanism and supports social logins as well. This webapp is very simple as it has only 3 pages:
- Home page: anyone can see
- Login page: to enter username/password
- Secret page: that only logged in users can see
For backend, we will use Python and Flask. Python and the framework Flask are quite expressive so I hope porting the code to other languages/frameworks will not be very hard. We will, of course, use AngularJS for front-end. And for the social logins, we will integrate with Facebook only as it is the most popular social network at this time.
Let's start!
Step #1: Bootstrap Project
Here is how we will structure our code:
- app.py
- static/
- index.html
- app.js
- bower.json
- partials/
- login.tpl.html
- home.tpl.html
- secret.tpl.html
All the back-end code is in app.py. The front-end code is put in static/ folder. By default, Flask will automatically serve the contents of static/ folder. All the partial views are in static/partials/ and handled by the ui.router module.
To start coding the back-end, we'll need Python 2.7.* and install the required libraries using pip. You can of course use virtualenv to isolate a Python environment. Below is the list of required Python modules to put in requirements.txt:
To install all these dependencies:
pip install -r requirements.txt
In app.py we have some initial code to bootstrap Flask (import statements are omitted for brevity):
app = Flask(__name__)
def index():
return flask.redirect('/static/index.html')
if __name__ == '__main__':
Next we init bower and install AngularJS and ui.router:
bower init # here you will need to answer some question. when in doubt, just hit enter :)
bower install angular angular-ui-router --save # install and save these dependencies into bower.json
Once these libraries are installed, we need to include AngularJS and ui-router in index.html and create routings for 3 pages: home, login, and secret.
<body ng-app="DemoApp">
<a ui-sref="home">Home</a>
<a ui-sref="login">Login</a>
<a ui-sref="secret">Secret</a>
<div ui-view></div>
<script src="bower_components/angular/angular.min.js"></script>
<script src="bower_components/angular-ui-router/release/angular-ui-router.min.js"></script>
<script src="main.js"></script>
Below is the code that we need in main.js to configure routing:
var app = angular.module('DemoApp', ['ui.router']);
app.config(function ($stateProvider, $urlRouterProvider) {
.state('home', {
url: '/home',
templateUrl: 'partials/home.tpl.html'
.state('secret', {
url: '/secret',
templateUrl: 'partials/secret.tpl.html',
.state('login', {
url: '/login',
templateUrl: 'partials/login.tpl.html'
At this point if you run the server python app.py, you should have this basic interface at http://localhost:5000
The links Home, Login, and Secret should work at this point and show the content of the corresponding templates.
Congratulation, you just finished setting up the skeleton! If you encounter any error, please check out thecode on GitHub
Step #2: Login and Register
At the end of this step, you'll have a webapp that you can register/login using email and password.
The first step is to configure the backend. We need a User model and a way to generate the JWT token for a given user. The User model shown below is really simplified and does not perform even any basic checks such as if field email contains "@", or if field password contains at least 6 characters, etc.
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
email = db.Column(db.String(100), nullable=False)
password = db.Column(db.String(100))
def token(self):
payload = {
'sub': self.id,
'iat': datetime.utcnow(),
'exp': datetime.utcnow() + timedelta(days=14)
token = jwt.encode(payload, app.config['TOKEN_SECRET'])
return token.decode('unicode_escape')
We use the jwt module in python to generate the payload part in JWT. The iat and exp part correspond to the timestamp that token is created and expired. In this code, the token will be expired in 2 weeks.
After the model User was created, we can add the "login" and "register" endpoints. The code for both are quite similar, so here I will just show the "register" part. Please note that by default, Satellizer will call the endpoints /auth/login and /auth/signup for the "login" and "register" respectively.
@app.route('/auth/signup', methods=['POST'])
def signup():
data = request.json
email = data["email"]
password = data["password"]
user = User(email=email, password=password)
return jsonify(token=user.token())
Let's check the endpoint using curl first:
curl localhost:5000/auth/signup -H "Content-Type: application/json" -X POST -d '{"email":"[email protected]","password":"xyz"}'
The result should look like this:
"token": "very long string…."
Now that the back-end part is ready, let's attack the front-end! First, we need to install satellizer and add it as a dependency in main.js:
bower install satellizer --save
Add satellizer as dependency:
var app = angular.module('DemoApp', ['ui.router', 'satellizer']);
Login and signup in satellizer is actually quite simple in comparison to all the setup until now:
$scope.signUp = function () {
.signup({email: $scope.email, password: $scope.password})
.then(function (response) {
// set the token received from server
// go to secret page
.catch(function (response) {
console.log("error response", response);
If you have any difficulty setting up the code, you can take a look at the code on GitHub.
Step #3: But Secret View Is Not Really Secret, Because Anyone Can See It!
Yes, that is correct! Until now, anyone can go to secret page without logging in.
It's time to add some interceptor in AngularJS to make sure that if someone goes to secret page and if this user is not logged in, they will be redirected to the login page.
First, we should add a flag requiredLogin to distinguish secret page from other ones.
.state('secret', {
url: '/secret',
templateUrl: 'partials/secret.tpl.html',
controller: 'SecretCtrl',
data: {requiredLogin: true}
The "data" part will be used in the $stateChangeStart event which is fired each time the routing changes:
app.run(function ($rootScope, $state, $auth) {
function (event, toState) {
var requiredLogin = false;
// check if this state need login
if (toState.data && toState.data.requiredLogin)
requiredLogin = true;
// if yes and if this user is not logged in, redirect him to login page
if (requiredLogin && !$auth.isAuthenticated()) {
Now, the user cannot go directly to the secret page without logging in. Hooray!
As usual, the code of this step can be found here.
Step #4: It's Time to Get Something Really Secret!
At this moment, there's nothing really secret in the secret page. Let's put something personal there.
This step starts by creating an endpoint in the back-end which is only accessible for an authenticated user, such as having a valid token. The endpoint /user below returns the user_id and email of the user corresponding to the token.
def user_info():
# the token is put in the Authorization header
if not request.headers.get('Authorization'):
return jsonify(error='Authorization header missing'), 401
# this header looks like this: "Authorization: Bearer {token}"
token = request.headers.get('Authorization').split()[1]
payload = jwt.decode(token, app.config['TOKEN_SECRET'])
except DecodeError:
return jsonify(error='Invalid token'), 401
except ExpiredSignature:
return jsonify(error='Expired token'), 401
user_id = payload['sub']
user = User.query.filter_by(id=user_id).first()
if user is None:
return jsonify(error='Should not happen ...'), 500
return jsonify(id=user.id, email=user.email), 200
return jsonify(error="never reach here..."), 500
Again, we make use of the module jwt to decode the JWT token included in the 'Authorization' header and to handle the case when the token is expired or not valid.
Let's test this endpoint using curl. First, we need to get a valid token:
curl localhost:5000/auth/signup -H "Content-Type: application/json" -X POST -d '{"email":"[email protected]","password":"xyz"}'
Then with this token:
curl localhost:5000/user -H "Authorization: Bearer {put the token here}"
Which gives this result:
"email": "[email protected]",
"id": 1
Now we need to include this endpoint in the Secret Controller. This is quite simple as we just need to call the endpoint using the regular $http module. The token is automatically inserted to the header by Satellizer, so we don't need to bother with all the details of saving the token and then putting it in the right header.
function getUserInfo() {
.then(function (response) {
$scope.user = response.data;
.catch(function (response) {
console.log("getUserInfo error", response);
Finally, we have something truly personal in the secret page!
The code of this step is on GitHub.
Step #5: Facebook Login with Satellizer
A nice thing about Satellizer, as mentioned at the beginning, is it makes integrating social login a lot easier. At the end of this step, users can login using their Facebook account!
First thing to do is to create an application on Facebook developers page in order to have an application_idand a secret code. Please follow developers.facebook.com/docs/apps/register to create a Facebook developer account if you don't have one already and create a website app. After that, you will have the application ID and application secret as in the screenshot below.
Once the user chooses to connect with Facebook, Satellizer will send an authorization code to the endpoint/auth/facebook. With this authorization code, the back-end can retrieve an access token from Facebook/oauth endpoint that allows the call to Facebook Graph API to get user information such as location, user_friends, user email, etc.
We also need to keep track of whether a user account is created with Facebook or through regular signup. To do so, we add facebook_id to our User model.
facebook_id = db.Column(db.String(100))
The facebook secret is configured via env variables FACEBOOK_SECRET that we add to app.config.
app.config['FACEBOOK_SECRET'] = os.environ.get('FACEBOOK_SECRET')
So to launch the app.py, you should set this env variable:
FACEBOOK_SECRET={your secret} python app.py
Here is the method which handles Facebook logins. By default Satellizer will call the endpoint /auth/facebook.
@app.route('/auth/facebook', methods=['POST'])
def auth_facebook():
access_token_url = 'https://graph.facebook.com/v2.3/oauth/access_token'
graph_api_url = 'https://graph.facebook.com/v2.5/me?fields=id,email'
params = {
'client_id': request.json['clientId'],
'redirect_uri': request.json['redirectUri'],
'client_secret': app.config['FACEBOOK_SECRET'],
'code': request.json['code']
# Exchange authorization code for access token.
r = requests.get(access_token_url, params=params)
# use json.loads instead of urlparse.parse_qsl
access_token = json.loads(r.text)
# Step 2. Retrieve information about the current user.
r = requests.get(graph_api_url, params=access_token)
profile = json.loads(r.text)
# Step 3. Create a new account or return an existing one.
user = User.query.filter_by(facebook_id=profile['id']).first()
if user:
return jsonify(token=user.token())
u = User(facebook_id=profile['id'], email=profile['email'])
return jsonify(token=u.token())
To send a request to the Facebook server, we use the handy module requests. Now the difficult part on the back-end is done. On the front-end, adding Facebook login is quite simple. First, we need to tell Satellizer ourfacebook_id by adding this code into app.config function:
clientId: {your facebook app id},
// by default, the redirect URI is http://localhost:5000
redirectUri: 'http://localhost:5000/static/index.html'
To login using Facebook, we can just call:
As usual, you can check the code on GitHub
At this time, the webapp is complete in terms of functionality. The user can login/register using regular email and password or by using Facebook. Once logged in, the user can see his secret page.
Make a Pretty Interface
The interface is not very pretty at this point, so let's add a little bit of Bootstrap for the layout and the angular toaster module to handle an error message nicely, such as when login fails.
The code for this beautifying part can be found here.
This article shows a step-by-step integration of Satellizer in a (simple) AngularJS webapp. With Satellizer, we can easily add other social logins such as Twitter, Linkedin, and more. The code on the front-end is quite the same as in the article. However, the back-end varies as social network SDKs have different endpoints with different protocols. You can take a look at https://github.com/sahat/satellizer/blob/master/examples/server/python/app.py which contains examples for Facebook, Github, Google, Linkedin, Twiter and Bitbucket. When in doubt, you should take a look at the documentation on https://github.com/sahat/satellizer.
This article was written by Son Nguyen Kim, a Toptal freelance developer.
9 Essential System Security Interview Questions
- What is a pentest?
"Pentest" is short for "penetration test", and involves having a trusted security expert attack a system for the purpose of discovering, and repairing, security vulnerabilities before malicious attackers can exploit them. This is a critical procedure for securing a system, as the alternative method for discovering vulnerabilities is to wait for unknown agents to exploit them. By this time it is, of course, too late to do anything about them.
In order to keep a system secure, it is advisable to conduct a pentest on a regular basis, especially when new technology is added to the stack, or vulnerabilities are exposed in your current stack.
2. What is social engineering?
"Social engineering" refers to the use of humans as an attack vector to compromise a system. It involves fooling or otherwise manipulating human personnel into revealing information or performing actions on the attacker's behalf. Social engineering is known to be a very effective attack strategy, since even the strongest security system can be compromised by a single poor decision. In some cases, highly secure systems that cannot be penetrated by computer or cryptographic means, can be compromised by simply calling a member of the target organization on the phone and impersonating a colleague or IT professional.
Common social engineering techniques include phishing, clickjacking, and baiting, although several other tricks are at an attacker's disposal. Baiting with foreign USB drives was famously used to introduce the Stuxnet worm into Iran's uranium enrichment facilities, damaging the nation's ability to produce nuclear material.
For more information, a good read is Christopher Hadnagy's book Social Engineering: The Art of Human Hacking.
3. You find PHP queries overtly in the URL, such as
/index.php=?page=userID. What would you then be looking to test?
This is an ideal situation for injection and querying. If we know that the server is using a database such as SQL with a PHP controller, it becomes quite easy. We would be looking to test how the server reacts to multiple different types of requests, and what it throws back, looking for anomalies and errors.
One example could be code injection. If the server is not using authentication and evaluating each user, one could simply try
/index.php?arg=1;system('id') and see if the host returns unintended data.
4. You find yourself in an airport in the depths of of a foreign superpower. You're out of mobile broadband and don't trust the WI-FI. What do you do? Further, what are the potential threats from open WI-FIs?
Ideally you want all of your data to pass through an encrypted connection. This would usually entail tunneling via SSH into whatever outside service you need, over a virtual private network (VPN). Otherwise, you're vulnerable to all manner of attacks, from man-in-the-middle, to captive portals exploitation, and so on.
5. What does it mean for a machine to have an "air gap"? Why are air gapped machines important?
An air gapped machine is simply one that cannot connect to any outside agents. From the highest level being the internet, to the lowest being an intranet or even bluetooth.
Air gapped machines are isolated from other computers, and are important for storing sensitive data or carrying out critical tasks that should be immune from outside interference. For example, a nuclear power plant should be operated from computers that are behind a full air gap. For the most part, real world air gapped computers are usually connected to some form of intranet in order to make data transfer and process execution easier. However, every connection increases the risk that outside actors will be able to penetrate the system.
6. You're tasked with setting up an email encryption system for certain employees of a company. What's the first thing you should be doing to set them up? How would you distribute the keys?
The first task is to do a full clean and make sure that the employees' machines aren't compromised in any way. This would usually involve something along the lines of a selective backup. One would take only the very necessary files from one computer and copy them to a clean replica of the new host. We give the replica an internet connection and watch for any suspicious outgoing or incoming activity. Then one would perform a full secure erase on the employee's original machine, to delete everything right down to the last data tick, before finally restoring the backed up files.
The keys should then be given out by transferring them over wire through a machine or device with no other connections, importing any necessary .p7s email certificate files into a trusted email client, then securely deleting any trace of the certificate on the originating computer.
The first step, cleaning the computers, may seem long and laborious. Theoretically, if you are 100% certain that the machine is in no way affected by any malicious scripts, then of course there is no need for such a process. However in most cases, you'll never know this for sure, and if any machine has been backdoored in any kind of way, this will usually mean that setting up secure email will be done in vain.
7. You manage to capture email packets from a sender that are encrypted through Pretty Good Privacy (PGP). What are the most viable options to circumvent this?
First, one should be considering whether to even attempt circumventing the encryption directly. Decryption is nearly impossible here unless you already happen to have the private key. Without this, your computer will be spending multiple lifetimes trying to decrypt a 2048-bit key. It's likely far easier to simply compromise an end node (i.e. the sender or receiver). This could involve phishing, exploiting the sending host to try and uncover the private key, or compromising the receiver to be able to view the emails as plain text.
8. What makes a script fully undetectable (FUD) to antivirus software? How would you go about writing a FUD script?
A script is FUD to an antivirus when it can infect a target machine and operate without being noticed on that machine by that AV. This usually entails a script that is simple, small, and precise
To know how to write a FUD script, one must understand what the targeted antivirus is actually looking for. If the script contains events such as
File_Delete(), or
File_Copy(), it's very likely it wil be picked up by antivirus scanners, so these events are not used. Further, FUD scripts will often mask function names with common names used in the industry, rather than naming them things like
fToPwn1337(). A talented attacker might even break up his or her files into smaller chunks, and then hex edit each individual file, thereby making it even more unlikely to be detected.
As antivirus software becomes more and more sophisticated, attackers become more sophisticated in response. Antivirus software such as McAfee is much harder to fool now than it was 10 years ago. However, there are talented hackers everywhere who are more than capable of writing fully undetectable scripts, and who will continue to do so. Virus protection is very much a cat and mouse game.
9. What is a "Man-in-the-Middle" attack?
A man-in-the-middle attack is one in which the attacker secretly relays and possibly alters the communication between two parties who believe they are directly communicating with each other. One example is active eavesdropping, in which the attacker makes independent connections with the victims and relays messages between them to make them believe they are talking directly to each other over a private connection, when in fact the entire conversation is controlled by the attacker, who even has the ability to modify the content of each message. Often abbreviated to MITM, MitM, or MITMA, and sometimes referred to as a session hijacking attack, it has a strong chance of success if the attacker can impersonate each party to the satisfaction of the other. MITM attacks pose a serious threat to online security because they give the attacker the ability to capture and manipulate sensitive information in real-time while posing as a trusted party during transactions, conversations, and the transfer of data. This is straightforward in many circumstances; for example, an attacker within reception range of an unencrypted WiFi access point, can insert himself as a man-in-the-middle.
This article is from Toptal.
Credit Card Hacks and How To Avoid Them
If you know me, or have read my previous post, you know that I worked for a very interesting company before joining Toptal. At this company, our payment provider processed transactions in the neighborhood of $500k per day. Part of my job was to make our provider PCI-DSS compliant—that is, compliant with the Payment Card Industry – Data Security Standard.
It's safe to say that this wasn't a job for the faint of heart. At this point, I'm pretty intimate with Credit Cards (CCs), Credit Card fraud and web security in general. After all, our job was to protect our users' data, to prevent it from being hacked, stolen or misused.
You could imagine my surprise when I saw Bennett Haselton's 2007 article on Slashdot: Why Are CC Numbers Still So Easy to Find?. In short, Haselton was able to find Credit Card numbers through Google, firstly by searching for a card's first eight digits in "nnnn nnnn" format, and later using some advanced queries built on number ranges. For example, he could use "4060000000000000..4060999999999999" to find all the 16 digit Primary Account Numbers (PANs) from CHASE (whose cards all begin with 4060). By the way: here's a full list of Issuer ID numbers.
At the time, I didn't think much of it, as Google immediately began to filter the types of queries that Bennett was using. When you tried to Google a range like that, Google would serve up a page that said something along the lines of "You're a bad person".
About six months ago, while reminiscing with an old friend, this credit card number hack came to mind again. Soon-after, I discovered something alarming. Not terribly alarming, but certainly alarming—so I notified Google, and waited. After a month without a response, I notified them again to no avail.
With a minor tweak on Haselton's old trick, I was able to Google Credit Card numbers, Social Security numbers, and any other sensitive information of interest.
Yesterday, some friends of mine (buhera.blog.hu and _2501) brought a more recent Slashdot post to my attention: Credit Card Numbers Still Google-able.
The article's author, again Bennett Haselton, who wrote the original article back in 2007, claims that credit card numbers can still be Googled. You can't use the number range query hack, but it still can be done. Instead of using simple ranges, you need to apply specific formatting to your query. Something like: "1234 5678" (notice the space in the middle). A lot of hits come up for this query, but very few are of actual interest. Among the contestants are phone numbers, zip-codes, and such. Not extremely alarming. But here comes the credit card hack twist.
The "Methodology"
I was curious if it was still possible to get credit card numbers online the way we could in 2007. As any good Engineer, I usually approach things using a properly construed and intelligent plan that needs to be perfectly executed with the utmost precision. If you have tried that method, you might know that it can fail really hard—in which case your careful planning and effort goes to waste.
In IT we have a tendency to over-intellectualize, even when it isn't exactly warranted. I have seen my friends and colleagues completely break applications using seemingly random inputs. Their success rate was stunning and the effort they put into it was close to zero. That's when I learned that to open a door, sometimes you just have to knock.
The Credit Card "Hack"
The previous paragraph was a cleverly disguised attempt to make me look like less of an idiot when I show off my "elite hacking skills". Oops.
First, I tried several range-query-based approaches. Then, I looked at advanced queries and pretty much anything you might come up with in an hour or so. None of them yielded significant results.
And then I had a crazy idea.
What if there was a mismatch between the filtering engine and the actual back-end? What if the message I got from Google ("You are a bad person") wasn't from the back-end itself, but instead from a designated filtering engine Google had implemented to censor queries like mine?
It would make a lot of sense from an architectural perspective. And bugs like that are pretty common—we see them in ITSEC all the time, particularly in IDS/IPS solutions, but also in common software. There's a filtering procedure that processes data and only gives it to the back-end if it thinks the data is acceptable/non-malicious. However, the back-end and the filtering server almost never parse the input in exactly the same way. Thus, a seemingly valid input can go through the filter and wreak havoc on the back-end, effectively bypassing the filter.
You can usually trigger this type of behavior by providing your input in various encodings. For example: instead of using decimal numbers (0-9), how about converting them to hexadecimal or octal or binary? Well, guess what…
Search for this and Google will tell you that you're a bad person: "4060000000000000..4060999999999999"
Search for this and Google will be happy to oblige: "0xe6c8c69c9c000..0xe6d753e6ecfff".
The only thing you need to do is to convert credit card numbers from decimal to hexadecimal. That's it.
The results include…
- Humongous CSV files filled with potentially sensitive information.
- Faulty e-commerce log files.
- Sensitive information shared on hacker sites (and even Facebook).
It's truly scary stuff.
I know this bug won't inspire any security research, but there you have it. Google made this boo-boo and neglected to even write me back. Well, it happens. I don't envy the security folks at the big G, though. They must have a lot of stuff to look out for. I'm posting about this credit card number hack here because:
- It's relatively low impact.
- Anyone who's interested and motivated will have figured this out by now.
- To quote Haselton, if the big players aren't taking responsibility and acting on these exploits, then "the right thing to do is to shine a light on the problem and insist that they fix it as soon as possible".
This trick can be used to look up phone numbers, SSNs, TFNs, and more. And, as Bennett wrote, these numbers are much much harder to change than your Credit Card, for which you can simply call your bank and cancel the card.
Sample Queries
WARNING: Do NOT Google your own credit card number in full!
Look for any CC PAN starting with 4060: 4060000000000000..4060999999999999 ? 0xe6c8c69c9c000..0xe6d753e6ecfff
Some Hungarian phone numbers from the provider 'Telenor'? No problem: 36200000000..36209999999 ? 0x86db02a00..0x86e48c07f
Look for SSNs. Thankfully, these don't return many meaningful results: 100000000..999999999 ? 0x5f5e100..0x3b9ac9ff
There are many, many more.
If you find anything very alarming, or if you're curious about credit card hacking, please leave it in the comments or contact me by email at [email protected] or on Twitter at @synsecblog. Calling the police is usually futile in these cases, but it might be worth a try. The given merchant or the card provider is usually more keen to address the issue.
Where to Go From Here
Well, Google obviously has to fix this, possibly with the help of the big players like Visa and Mastercard. In fact, Haselton provides a number of interesting suggestions in the two articles linked above.
What you need to do, however (and why I've written this post), is spread the word. Credit Card fraud is a big industry, and simple awareness can save you from becoming a victim. Further, if you have an e-commerce site or handle any credit card processing, please make sure that you're secure. PCI-DSS is a good guideline, but it is far from perfect. Plus, it is always a good idea to Google your site with the "site:mysite.com" advanced query, looking for sensitive numbers. There's a very, very slim chance that you'll find anything—but if you do, you must act on it immediately.
Also, a bit of friendly advice: You should never give out your credit card information to anyone. My advice would be to use PayPal or a similar service whenever possible. You can check out these links for further information:
And a few general tips: don't download things you didn't ask for, don't open spam emails, and remember that your bank will never ask for your password.
By the way: If you think there's no one stupid enough to fall for these credit card hacking techniques or give away their credit card information on the internet, have a look at @NeedADebitCard.
Stay safe people!
This article was written by Gergely Kalman, a Toptal Security Specialist.
Scaling Scala: How to Dockerize Using Kubernetes
Kubernetes is the new kid on the block, promising to help deploy applications into the cloud and scale them more quickly. Today, when developing for a microservices architecture, it's pretty standard to choose Scala for creating API servers.
If there is a Scala application in your plans and you want to scale it into a cloud, then you are at the right place. In this article I am going to show step-by-step how to take a generic Scala application and implement Kubernetes with Docker to launch multiple instances of the application. The final result will be a single application deployed as multiple instances, and load balanced by Kubernetes.
All of this will be implemented by simply importing the Kubernetes source kit in your Scala application. Please note, the kit hides a lot of complicated details related to installation and configuration, but it is small enough to be readable and easy to understand if you want to analyze what it does. For simplicity, we will deploy everything on your local machine. However, the same configuration is suitable for a real-world cloud deployment of Kubernetes.
What is Kubernetes?
Before going into the gory details of the implementation, let's discuss what Kubernetes is and why it's important.
You may have already heard of Docker. In a sense, it is a lightweight virtual machine.
For these reasons, it is already one of the more widely used tools for deploying applications in clouds. A Docker image is pretty easy and fast to build and duplicable, much easier than a traditional virtual machine like VMWare, VirtualBox, or XEN.
Kubernetes complements Docker, offering a complete environment for managing dockerized applications. By using Kubernetes, you can easily deploy, configure, orchestrate, manage, and monitor hundreds or even thousands of Docker applications.
Kubernetes is an open source tool developed by Google and has been adopted by many other vendors. Kubernetes is available natively on the Google cloud platform, but other vendors have adopted it for their OpenShift cloud services too. It can be found on Amazon AWS, Microsoft Azure, RedHat OpenShift, and even more cloud technologies. We can say it is well positioned to become a standard for deploying cloud applications.
Now that we covered the basics, let's check if you have all the prerequisite software installed. First of all, you need Docker. If you are using either Windows or Mac, you need the Docker Toolbox. If you are using Linux, you need to install the particular package provided by your distribution or simply follow the official directions.
We are going to code in Scala, which is a JVM language. You need, of course, the Java Development Kit and the scala SBT tool installed and available in the global path. If you are already a Scala programmer, chances are you have those tools already installed.
If you are using Windows or Mac, Docker will by default create a virtual machine named
default with only 1GB of memory, which can be too small for running Kubernetes. In my experience, I had issues with the default settings. I recommend that you open the VirtualBox GUI, select your virtual machine
default, and change the memory to at least to 2048MB.
The Application to Clusterize
The instructions in this tutorial can apply to any Scala application or project. For this article to have some "meat" to work on, I chose an example used very often to demonstrate a simple REST microservice in Scala, called Akka HTTP. I recommend you try to apply source kit to the suggested example before attempting to use it on your application. I have tested the kit against the demo application, but I cannot guarantee that there will be no conflicts with your code.
So first, we start by cloning the demo application:
git clone https://github.com/theiterators/akka-http-microservice
Next, test if everything works correctly:
cd akka-http-microservice
sbt run
Then, access to
http://localhost:9000/ip/220.127.116.11, and you should see something like in the following image:
Adding the Source Kit
Now, we can add the source kit with some Git magic:
git remote add ScalaGoodies https://github.com/sciabarra/ScalaGoodies
git fetch --all
git merge ScalaGoodies/kubernetes
With that, you have the demo including the source kit, and you are ready to try. Or you can even copy and paste the code from there into your application.
Once you have merged or copied the files in your projects, you are ready to start.
Starting Kubernetes
Once you have downloaded the kit, we need to download the necessary
kubectl binary, by running:
This installer is smart enough (hopefully) to download the correct
kubectl binary for OSX, Linux, or Windows, depending on your system. Note, the installer worked on the systems I own. Please do report any issues, so that I can fix the kit.
Once you have installed the
kubectl binary, you can start the whole Kubernetes in your local Docker. Just run:
The first time it is run, this command will download the images of the whole Kubernetes stack, and a local registry needed to store your images. It can take some time, so please be patient. Also note, it needs direct accesses to the internet. If you are behind a proxy, it will be a problem as the kit does not support proxies. To solve it, you have to configure the tools like Docker, curl, and so on to use the proxy. It is complicated enough that I recommend getting a temporary unrestricted access.
Assuming you were able to download everything successfully, to check if Kubernetes is running fine, you can type the following command:
bin/kubectl get nodes
The expected answer is:
127.0.0.1 Ready 2m
Note that age may vary, of course. Also, since starting Kubernetes can take some time, you may have to invoke the command a couple of times before you see the answer. If you do not get errors here, congratulations, you have Kubernetes up and running on your local machine.
Dockerizing Your Scala App
Now that you have Kubernetes up and running, you can deploy your application in it. In the old days, before Docker, you had to deploy an entire server for running your application. With Kubernetes, all you need to do to deploy your application is:
- Create a Docker image.
- Push it in a registry from where it can be launched.
- Launch the instance with Kubernetes, that will take the image from the registry.
Luckily, it is way less complicated that it looks, especially if you are using the SBT build tool like many do.
In the kit, I included two files containing all the necessary definitions to create an image able to run Scala applications, or at least what is needed to run the Akka HTTP demo. I cannot guarantee that it will work with any other Scala applications, but it is a good starting point, and should work for many different configurations. The files to look for building the Docker image are:
Let's have a look at what's in them. The file
project/docker.sbt contains the command to import the
sbt-docker plugin:
addSbtPlugin("se.marcuslonnberg" % "sbt-docker" % "1.4.0")
This plugin manages the building of the Docker image with SBT for you. The Docker definition is in the
docker.sbt file and looks like this:
imageNames in docker := Seq(ImageName("localhost:5000/akkahttp:latest"))
dockerfile in docker := {
val jarFile: File = sbt.Keys.`package`.in(Compile, packageBin).value
val classpath = (managedClasspath in Compile).value
val mainclass = mainClass.in(Compile, packageBin).value.getOrElse(sys.error("Expected exactly one main class"))
val jarTarget = s"/app/${jarFile.getName}"
val classpathString = classpath.files.map("/app/" + _.getName)
.mkString(":") + ":" + jarTarget
new Dockerfile {
add(classpath.files, "/app/")
add(jarFile, jarTarget)
entryPoint("java", "-cp", classpathString, mainclass)
To fully understand the meaning of this file, you need to know Docker well enough to understand this definition file. However, we are not going into the details of the Docker definition file, because you do not need to understand it thoroughly to build the image.
the SBT will take care of collecting all the files for you.
Note the classpath is automatically generated by the following command:
val classpath = (managedClasspath in Compile).value
In general, it is pretty complicated to gather all the JAR files to run an application. Using SBT, the Docker file will be generated with
add(classpath.files, "/app/"). This way, SBT collects all the JAR files for you and constructs a Dockerfile to run your application.
The other commands gather the missing pieces to create a Docker image. The image will be built using an existing image APT to run Java programs (
anapsix/alpine-java:8, available on the internet in the Docker Hub). Other instructions are adding the other files to run your application. Finally, by specifying an entry point, we can run it. Note also that the name starts with
localhost:5000 on purpose, because
localhost:5000 is where I installed the registry in the
start-kube-local.sh script.
Building the Docker Image with SBT
To build the Docker image, you can ignore all the details of the Dockerfile. You just need to type:
sbt dockerBuildAndPush
sbt-docker plugin will then build a Docker image for you, downloading from the internet all the necessary pieces, and then it will push to a Docker registry that was started before, together with the Kubernetes application in
localhost. So, all you need is to wait a little bit more to have your image cooked and ready.
Note, if you experience problems, the best thing to do is to reset everything to a known state by running the following commands:
Those commands should stop all the containers and restart them correctly to get your registry ready to receive the image built and pushed by
Starting the Service in Kubernetes
Now that the application is packaged in a container and pushed in a registry, we are ready to use it. Kubernetes uses the command line and configuration files to manage the cluster. Since command lines can become very long, and also be able to replicate the steps, I am using the configurations files here. All the samples in the source kit are in the folder
Our next step is to launch a single instance of the image. A running image is called, in the Kubernetes language, a pod. So let's create a pod by invoking the following command:
bin/kubectl create -f kube/akkahttp-pod.yml
You can now inspect the situation with the command:
bin/kubectl get pods
You should see:
akkahttp 1/1 Running 0 33s
k8s-etcd-127.0.0.1 1/1 Running 0 7d
k8s-master-127.0.0.1 4/4 Running 0 7d
k8s-proxy-127.0.0.1 1/1 Running 0 7d
Status actually can be different, for example, "ContainerCreating", it can take a few seconds before it becomes "Running". Also, you can get another status like "Error" if, for example, you forget to create the image before.
You can also check if your pod is running with the command:
bin/kubectl logs akkahttp
You should see an output ending with something like this:
[DEBUG] [05/30/2016 12:19:53.133] [default-akka.actor.default-dispatcher-5] [akka://default/system/IO-TCP/selectors/$a/0] Successfully bound to /0:0:0:0:0:0:0:0:9000
Now you have the service up and running inside the container. However, the service is not yet reachable. This behavior is part of the design of Kubernetes. Your pod is running, but you have to expose it explicitly. Otherwise, the service is meant to be internal.
Creating a Service
Creating a service and checking the result is a matter of executing:
bin/kubectl create -f kube/akkahttp-service.yaml
bin/kubectl get svc
You should see something like this:
akkahttp-service 10.0.0.54 9000/TCP 44s
kubernetes 10.0.0.1 <none> 443/TCP 3m
Note that the port can be different. Kubernetes allocated a port for the service and started it. If you are using Linux, you can directly open the browser and type
http://10.0.0.54:9000/ip/18.104.22.168 to see the result. If you are using Windows or Mac with Docker Toolbox, the IP is local to the virtual machine that is running Docker, and unfortunately it is still unreachable.
I want to stress here that this is not a problem of Kubernetes, rather it is a limitation of the Docker Toolbox, which in turn depends on the constraints imposed by virtual machines like VirtualBox, which act like a computer within another computer. To overcome this limitation, we need to create a tunnel. To make things easier, I included another script which opens a tunnel on an arbitrary port to reach any service we deployed. You can type the following command:
bin/forward-kube-local.sh akkahttp-service 9000
Note that the tunnel will not run in the background, you have to keep the terminal window open as long as you need it and close when you do not need the tunnel anymore. While the tunnel is running, you can open:
http://localhost:9000/ip/220.127.116.11 and finally see the application running in Kubernetes.
Final Touch: Scale
So far we have "simply" put our application in Kubernetes. While it is an exciting achievement, it does not add too much value to our deployment. We're saved from the effort of uploading and installing on a server and configuring a proxy server for it.
Where Kubernetes shines is in scaling. You can deploy two, ten, or one hundred instances of our application by only changing the number of replicas in the configuration file. So let's do it.
We are going to stop the single pod and start a deployment instead. So let's execute the following commands:
bin/kubectl delete -f kube/akkahttp-pod.yml
bin/kubectl create -f kube/akkahttp-deploy.yaml
Next, check the status. Again, you may try a couple of times because the deployment can take some time to be performed:
akkahttp-deployment-4229989632-mjp6u 1/1 Running 0 16s
akkahttp-deployment-4229989632-s822x 1/1 Running 0 16s
k8s-etcd-127.0.0.1 1/1 Running 0 6d
k8s-master-127.0.0.1 4/4 Running 0 6d
k8s-proxy-127.0.0.1 1/1 Running 0 6d
Now we have two pods, not one. This is because in the configuration file I provided, there is the value
replica: 2, with two different names generated by the system. I am not going into the details of the configuration files, because the scope of the article is simply an introduction for Scala programmers to jump-start into Kubernetes.
Anyhow, there are now two pods active. What is interesting is that the service is the same as before. We configured the service to load balance between all the pods labeled
akkahttp. This means we do not have to redeploy the service, but we can replace the single instance with a replicated one.
We can verify this by launching the proxy again (if you are on Windows and you have closed it):
bin/forward-kube-local.sh akkahttp-service 9000
Then, we can try to open two terminal windows and see the logs for each pod. For example, in the first type:
bin/kubectl logs -f akkahttp-deployment-4229989632-mjp6u
And in the second type:
bin/kubectl logs -f akkahttp-deployment-4229989632-s822x
Of course, edit the command line accordingly with the values you have in your system.
Now, try to access the service with two different browsers. You should expect to see the requests to be split between the multiple available servers, like in the following image:
Today we barely scratched the surface. Kubernetes offers a lot more possibilities, including automated scaling and restart, incremental deployments, and volumes. Furthermore, the application we used as an example is very simple, stateless with the various instances not needing to know each other. In the real world, distributed applications do need to know each other, and need to change configurations according to the availability of other servers. Indeed, Kubernetes offers a distributed keystore (
etcd) to allow different applications to communicate with each other when new instances are deployed. However, this example is purposefully small enough and simplified to help you get going, focusing on the core functionalities. If you follow the tutorial, you should be able to get a working environment for your Scala application on your machine without being confused by a large number of details and getting lost in the complexity.
This article was written by Michele Sciabarra, a Toptal Scala developer.