How to send Apple Push Notifications Across TONS of Accounts

We needed an APNS package for use with a many-tenant MDM platform. Specifically, we needed the ability to quickly push many notifications to devices spanning across many push certificates.

Traditionally with APN, you establish a connection to Apple, send as many notifications as you desire, and then disconnect. The problem we have is that we need to create a separate connection for each account, and simply connecting and disconnecting for each notification will look like a DDoS attack from the perspective of Apple.

The particular project in question was written in Ruby, so we started looking at existing gem solutions.


We started here and eventually forked and added MDM support. apn_sender keeps a persistent connection to Apple, which is great. It doesn’t handle multiple certificates though, so that means we’d have to have a separate daemon process running for every single push certificate.

In fact, most APNS packages were eliminated for this reason: They weren’t built with multiple-certificate handling in mind, meaning something costly would have to be instantiated for each certificate.


This gem sets out to solve multiple-certificate handling, but it fell short in two ways:

  1. It requires the compilation and usage of a fork of ZMQMachine. We don’t want to have to manually compile the gem and we don’t want to depend on someone keeping a fork of a project maintained.
  2. It assumes the bottleneck is in the building of the APNS message, not in the SSL connection setup/teardown with Apple. The gem instantiates many workers for the activity of building the message to be pushed, but by default only runs a single firehose, which is responsible for connecting to Apple. We found the opposite to be true: the connection to Apple is the slowest part.

Houston? Grocer?

These are both outstanding, well used, active projects. They unfortunately have the same problems as most gems in that they were not designed for many, many different APNS accounts.

Alas, we finally found a solution.

Finally: AppleShove!

In brief, AppleShove receives push requests from a Redis queue structure. These push requests include the APNS certificate and the payload to be pushed. A single thread called the demultiplexer reads from this Redis queue and also manages a pool of connection threads to Apple. When a request is received, the demultiplexer sends the request to the appropriate connection thread. If the connection thread doesn’t already exist, it’s created first. That’s it!

For you concurrency fans out there, we are using the Actor concurrency pattern via Celluloid.

This architecture accomplishes a few things:

  1. “Caches” connections to Apple. If we’ve sent a notification with a particular certificate recently, we get to reuse the connection instead of having to re-establish it.
  2. Allows notifications to be sent in parallel. We aren’t waiting for a series of connections and disconnections to take place before we can send notification #n.
  3. Simplifies our client implementation. Since each notification contains all of the information AppleShove needs to send it on it’s way, we can request notifications via a single static method.

To see if AppleShove works well for you, too, check out the project here!


The Importance of Business Dashboards

Shortly before or after publicly releasing a product, I inevitably fashion some form of a business dashboard. I’m amazed at how many products are launched without any instrumentation. To me, this results in a scenario similar to “flying blind”, or only having partial, imperfect information about the health of your product.

A business dashboard is a realtime or near-realtime information screen that provides information about key metrics around your product and business. When set up correctly, they provide numerous benefits.

1. A business dashboard provides motivation

After any product launch, it’s easy to lose steam. You’re working on a marketing campaign and it seems to be increasing website traffic, but is it really helping sales? What sort of ROI are you getting on your effort? Should you continue with it or try something different?

It’s hard to feel motivated when you don’t have a good idea of where you stand and where you’re heading. Having an indication that you’re not doing so great can motivate you to work harder. An indicator that an action you took is paying off can also motivate you to keep it up.

2. A business dashboard gives you metrics of your progress

Tied closely to the benefits of motivation, having a measure of the health of your product is absolutely necessary. These will be crucial when measuring the health of your product over time. They can provide small metrics like “how many people are using this new feature we added?” or larger ones, like “what is the average lifetime value of a customer?”.

3. A business dashboard guides decision making and focuses your effort

Building metrics around key aspects of your business, such as customer acquisition, onboarding, retention, and churn will help you better understand why you’re at the place you are today.

Without business metrics, it’s much easier to base a business decision on the latest article you’ve read on Medium. Having metrics around your business, instead, will scream at you when there’s a problem. Retention may be great, but your customers are getting stuck during the onboarding process. Acquisition is lower this month since that article you were mentioned in became less visible.  With each of these indications, obvious, logical reactions can follow. When you invest time addressing them, you know it’s a worthwhile endeavor as opposed to simply “trying something” and seeing if any sort of measurable result occurs.

They’re good for successes, too. Maybe something you changed has resulted in a spike in new customers. That’s good to know! Perhaps the newsletter you’ve been sending out has increased product engagement and decreased churn more than you imagined. Keep it up!

Build a Dashboard Today

If you don’t have one already, spec one out today. If you’re using third party services, like Stripe, run a google search on “Stripe Metrics”. We’re partial to, but encourage you to check out multiple offerings. Other metrics engines like Google Analytics and Mixpanel are both great but are narrowly scoped. Use them, but try to extract the data and use it to measure bigger metrics and inform the higher-level measures of your business.

Yes, It’s OK to use a Relational Database as a Queue

In software design, it’s not uncommon to come up with the need for a queue. Perhaps you have certain tasks that should be scheduled for background processing. Maybe a type of request needs to be handled in batches.

Why the Flack?

Regardless of the reason, we’ve seen customers and fellow engineers squeam at the suggestion of using a database table to handle the storage of queue items. Here’s an incomplete list of reasons we generally hear:

  1. “We should use a proper queueing technology, like AWS SQS, RabbitMQ, ActiveMQ, Kafka, or IronMQ. MySQL and PostgreSQL aren’t designed for this! This feels wrong!”
  2. “The performance will be horrible! We should at least using something that will be in-memory!”
  3. “If anyone sees a design like this, they’ll think we’re idiots!”
  4. “This won’t scale well when our growth curve starts double hockey sticking!”
  5. “Will it even be transaction-safe? What about handling dead letters? Why even bother?”

Some of these are good points. Performance may matter, eventually. A queueing service may allow for a better eventual architecture. Someone may think you’re an idiot. But lets look at the bigger picture.

A Case For The RDBs

Software architecture can be a challenging process. This is because most software built is to address a problem where the best solution will only be fully realized over a long span of time. Your understanding of what the problem is and how the problem will best be solved will be quite different when you start than when you end, even with adequate discovery efforts, planning, and experience.

You may start out thinking “Hey, I have a bunch of commands that are going to be consumed by a worker process. I should put them in a queue!”. So, you install a queue service on your server and configure a queue. It works great. Look at how fast it is since it’s all in memory! Grabbing an item off of the queue takes 2 ms!

Some time passes and you add more worker processes. Some of these worker processes exist on a server on the west coast and some are on the east coast, to increase uptime in case there’s some sort of locational outage. Great! But, you’re seeing that performance isn’t scaling linearly. Every time you add a worker, it seems that you’re only getting around 60% of the gain you would expect. It turns out that many of the commands in the queue process quicker when they are grouped together and handled by the same worker rather than being interspersed across workers at different locations. Perhaps it has to do with some sort of context/memory switching that the worker has to perform. OK, no problem, we’ll somehow group those items together, right?

Wrong. You’re using a queueing technology that only allows FIFO operations, so there’s no way for you to see further into the queue without popping items off of it. So, bad news.

Another issue arises. Some queue items aren’t processable when you pop them off the queue. You want an hour to pass before retrying them. But wait, you can’t have the worker just sit there and hold onto the item. You also don’t want to set up another queue just for these delayed items. Even worse, you may be using a technology like RabbitMQ that doesn’t really allow for delayed visibility of queue items without some strange dead-letter queue hacking. More bad news!

Yes, there are architectures that can be implemented to get around both of these situations. But… that’s not the point.

What’s the Point, Old Man?

The point is that a relational database, like MySQL for instance, doesn’t have these limitations. It’s not a high performance queueing technology, but it is a very general, flexible tool that will allow you to do just about whatever you need it to compared to a more specific technology like a queueing or messaging service.

You think you know what you want to accomplish, and you may be right, but you may not. I like the saying “build first with wood, then steel”. If you’re unsure, you’re still discovering the right solution, and you’re proving that your software is actually going to get used, work with malleable tools that allow you to make changes and adjustments with ease. If you see, down the road, that you are really being hurt performance wise, make the switch to a queueing system with confidence.

A Smart Thermostat with Geofencing for Under $100

Where We Started

We replaced our non-programmable thermostat some time ago with a Nest thermostat. Besides it’s beautiful aesthetics, we were primarily interested in lowering our heating and cooling utility bills.

Shortly after purchasing the Nest, we realized that the Nest way of reducing our HVAC usage didn’t make sense for our house occupancy patterns. We determined that we needed to control the Nest in a different manner to really make it save money.


Adding Geofencing to the Nest Learning Thermostat

I have what I call an anti-schedule. I don’t head to work and return home at the same times each day of the week. I may work at home until 11:00 AM before heading to the office, or may be home rather late if I have an engagement in the evening. Weekends are even less predictable. I may take off to the coast at an early hour to get a day of surfing in, or, I may sleep in, have a friend over for breakfast and coffee, and enjoy time at home.

The Nest Learning Thermostat is designed to learn your patterns without you having to teach it. How can the Nest learn your schedule when you don’t have one? You quickly end up needing to manually program in a schedule to the Nest which sort of works, but is wrong a good deal of the time.

Nest, in the product’s defense, tries to solve this problem with a feature called Auto-Away. It keeps track of activity near it with a motion sensor. If it doesn’t see motion for a span of time, it assumes you’ve left and sets itself to away, overriding the schedule. If it sees motion again, it sets itself back to home. The amount of time it takes to do this spans somewhere between 15-120 minutes, depending upon learned behavior, according to Nest.

The problem with Auto-Away is that it gets it wrong a good deal of the time. If I’m working from home in my office, Nest will think the house is empty. If I actually do leave the house, the Nest sometimes takes longer to realize I’m gone than I’m actually gone for.

This lead to the question: how can I passively keep the Nest updated on my location at all times? Other thermostat manufacturers achieve this with a technology called geofencing. Geofencing is a way of creating a virtual perimeter around a particular space. When you cross this line, some sort of event occurs. The answer for us was to build an app that could keep track of whether we were within a geofence around our home or not, and update Nest accordingly.

Out of this, we created Skylark, a mobile app that keeps track of whether anyone is home or not and updates your Nest when your home empties out and whenever someone comes home.


Unexpected Realizations

The first month of using Skylark in beta was great. We found that our Nest home/away setting changed instantly whenever we arrived home or left. Finally, our Nest status was based on our actual location, not on what the Nest guessed was our location.

We had a problem though. Sometimes our Nest would still set to away while we were home. Other times, it set to home even though we were gone. Auto-away was still wreaking havoc on the Nest status. We realized that we didn’t need it any longer, as Skylark did a better job of determining our status than it ever could. We disabled it.

With Auto-Away and the schedule learning feature disabled, our Nest was turning more and more into a normal thermostat. This got us to thinking: if we hadn’t already owned a Nest, what would we purchase now? Is it worth paying the Nest premium if we aren’t using most of the features?


Commodity Hardware Options?

We considered building our own thermostat. What would it need to have? A basic thermostat with a schedule would suffice. The only extra feature we’d need is WiFi. From there, we could use Skylark to add the magic. But how would we produce at a large enough volume to justify selling a hardware product like this? How would we compete with a company like Honeywell?

A routine visit to Home Depot turned this question on its head. Honeywell already makes a variety of WiFi enabled thermostats. Could we potentially integrate with one of them, adding the much desired geolocation feature to an otherwise unremarkable thermostat? Yes we could.


The sub $100 Smart Thermostat

After some additional engineering effort, we were able to add Honeywell thermostat support to Skylark. Using almost any Honeywell WiFi thermostat, such as this one ($92 at the time of writing this article), we had a geofenced thermostat solution that is every bit as good at saving energy costs as our Nest thermostat is.

Are the Honeywell WiFi thermostats as aesthetically sexy as the Nest thermostat? Not really. Does it save on energy costs just as effectively at less than half of the price of the Nest? Absolutely, and we think that’s pretty sexy, too.

Skylark for Nest & Honeywell Smart Thermostats is available in the Apple App Store.
Most Honeywell WiFi enabled thermostats should work with Skylark.