Leverage Secrets Management for Effective On-Call Support

In many ways, on-call duty and secrets management might seem to occupy pretty different parts of the IT universe. When you think about being on call, things like alerting systems and triaging strategies come to mind. You probably don’t give much thought to passwords, API keys, or other secrets.

But the fact is that managing secrets is an important part of being an effective on-call engineer. Here’s why that is, and how on-call teams can streamline secrets management in order to do their jobs more effectively.

Why Secrets Matter to On-call Teams

There are several reasons why on-call engineers need to be able to work seamlessly with the secrets that provide access to the IT systems they help support.

Accessing Systems

First and most obvious is the fact that in order to respond to an issue that arises during on-call duty, you typically need to access whichever system or systems are failing. And getting that access requires secrets.

Therefore, unless your organization does something utterly irresponsible, like giving on-call engineers a shared password that lets them into all systems, the on-call team needs a way to access secrets for various systems when the need arises.

Updating secrets

Sometimes, on-call engineers also need to update secrets as part of their incident response duties. They might be responding to the breach of a server and need to reset its passwords, or they may be troubleshooting a performance issue within an application in a way that requires them to change API keys or tokens.

Thus, it’s not only access to secrets that on-call teams need to do their jobs; they must also be able to updatesecrets.

Secrets Management Self-Service 

A third critical requirement for on-call teams is the ability to gain access to the secrets they need in a fast, efficient manner that doesn’t require them to ask off-duty engineers for help. If you’re in the middle of responding to a critical alert at 2 a.m., the last thing you want is to have to wake up someone else (and wait for a response, delaying your time-to-resolution) because you can’t get the credentials you need to fix whatever is broken.

In other words, on-call engineers must be able to help themselves when issues related to secrets arise.

Secrets Management Best Practices for On-call Teams

If you have an ad hoc approach to secrets management, you’re in a poor position to ensure that your on-call team can manage secrets effectively. To avoid that pitfall, consider the following strategies for making secrets management as seamless as possible for on-call engineers — even the ones who don’t typically spend much time working with secrets.

Use Centralized Secrets Management to Reduces Friction & Improve Security 

One basic best practice is to make sure that all of your organization’s secrets are stored in a central, secure place. That way, your on-call team knows where to look when it needs to sort out an access issue or get into a system.

And by “central, secure place,” I don’t mean a spreadsheet or other highly insecure secrets management solution. I mean a secure secrets manager like Conjur.

Another point to emphasize is that your secrets manager should house all of your secrets. That means not just passwords, but also keys, tokens, and any other sensitive data that is used for authentication and authorization.

Use RBAC to Control Access to Secrets

By choosing a secrets manager that supports RBAC, you can ensure that on-call engineers have access to the secrets they need when necessary, without having to share more sensitive information than required. RBAC policies can also be modified easily if access permissions must be changed in order to remediate an issue, then restored to their original settings once the issue is resolved.

Embrace Declarative Management

Declarative management means writing configuration files that specify how systems should behave, then applying them across your environments. It’s the opposite of imperative management, in which you manually configure your environments to try to match a desired state.

Declarative management is superior for many reasons, but one way it can help on-call teams is to reduce the amount of expertise they need to have in your secrets management policies. When those policies are set in code, on-call engineers can focus on simply applying the configurations in order to adhere to your organization’s secrets best practices. This is one way to empower engineers without extensive secrets management skills to treat secrets appropriately, even in the midst of troubleshooting an issue.

Automate Secrets Sharing

Last but not least, you can do much to streamline secrets management by adopting a secrets management tool that minimizes the extent to which secrets need to be shared manually. In other words, let your secrets manager automate the process of giving machines and applications the information that they need to authenticate and authorize requests.

When you use a secrets manager this way, you minimize the engagement that on-call engineers need to have with secrets in the first place. As long as your secrets manager is properly configured, the on-call team will rarely need to worry about accessing passwords, keys, or other important data in order to do its job, because the secrets manager will transfer the information securely and automatically.


When you’re on on-call duty, scanning your dashboards for new alerts, the last thing you are likely to be thinking about is secrets. But when an issue arises that requires you (or the system you are fixing) to have access to secrets, you’ll be thankful that your team took the time to streamline its secrets management process and ensure that secrets don’t get in the way of remediating issues.

Join the Conversation on the CyberArk Commons

If you’re interested in this and other open source content, join the conversation on the CyberArk Commons Community. Secretless Broker, Conjur, and other open source projects are a part of the CyberArk Commons Community, an open community dedicated to developers, engineers, cybersecurity researchers and other technically minded people. To discuss Kubernetes, Secretless Broker, Conjur, CyberArk Threat Research and related topics, join us on the CyberArk Commons discussion forum.