Keeping Secrets Out of Logs Posted on Aug 2, 2024 tl;dr: There's no silver bullet, but if we put some "lead" bullets in the right places, we have a good shot at keeping sensitive data out of logs. "This is the blog version of a talk I gave at LocoMocoSec 2024. It’s mostly a lightly edited transcript with some screenshots, so if you’d prefer, you can watch the video or just flip through the slides ." This post is about how to keep secrets out of logs, and my claim is that (like many things in security) there isn’t a singular action or silver bullet that lets you do this. I would go so far as to say that there’s not even an 80/20 rule, where one action fixes 80% of the problem. It’s not like preventing SQL injection with prepared statements or preventing buffer overflows by using memory-safe languages. What I will offer instead, are lead bullets, of which there are many. I’m going to talk about 10 of them. They are imperfect and sometimes unreliable things that, if put in the right places and with defense-in-depth, can still give us a real good chance at succeeding. My hope is that by the end, you’ll have a slightly better framework for how to reason about this problem and some new ideas to add to your kit. Table of contents: The Problem Causes 🤦 Direct logging 🚰 Kitchen sinks 🔧 Configuration changes 🥧 Embedded secrets 📡 Telemetry 🕺🏻 User input Fixes (lead bullets) 📐 Data architecture 🍞 Data transformations 🪨 Domain primitives Compile-time Run-time Run-time: part deux 🎁 Read-once objects 🔎 Taint checking Awesome Not awesome 🗃️ Log formatters 🧪 Unit tests 🕵️ Sensitive data scanners Sampling 🤖 Log pre-processors 🦸 People Recap Strategy 0. Lay the foundation 1. Understand the data flow 2. Protect at chokepoints 3. Apply defense-in-depth 4. Plan for response and recovery Conclusion The Problem With that, let’s dive in and set the table by talking about the problem with secrets in logs. So, there are some problems that are annoying. And there are some problems that are difficult. This is both. I’m gonna level with you: I absolutely hate this problem. But I’m not going to gaslight you and tell you that this is the most important thing to work on worry about, because it probably isn’t! You have somewhere between 5 and 50 other problems in your backlog that seem more important, 1 of which you found out about this morning. But I think it’s likely that none of those problems are nearly as annoying. While researching this topic, I interviewed about a dozen other engineers and, on this point, they unanimously agreed! Nobody likes dealing with secrets in logs because it is extraordinarily annoying . This is a problem that’s also difficult, but not even in the fun sense, like being technically complex or interesting. Once you catch sensitive data in logs, it’s usually pretty straightforward (at least in retrospect) to determine how they got there. But, it’s also surprisingly elusive to prevent, and it crops up in incredibly unexpected places and ways. Secrets could mean lots of different things to lots of different teams, but I’ll use it interchangeably with “sensitive data”: stuff that you want to keep confidential. What’s so frustrating when breaching confidentiality in logs is the full spectrum of potential impact. In the best case (left) , you might log an isolated, internal credential, like an API key, which (kudos!) you rotate right after fixing the source of leak. The impact is minimal, and you just move on. Of course, all the way on the other end of the spectrum (right) , you might log something that an attacker or inside threat could use to do some real harm . And then somewhere in-between, where I suspect most of the incidents lie. You might log secrets that you unfortunately, can’t rotate yourself. Things like PII or your customer’s passwords, which are reused on other sites, because of course they are. And, depending on your policies, threat model, or regulations, you might choose to issue a disclosure or notification. And it is painful. You could be doing so many good data security practices, like secure-by-design frameworks, database and field-level encryption, zero-touch production, access control… but logging bypasses all of that… and ultimately degrades trust, in your systems and in your company. It feels unfair because it’s only a fraction of your security story. And this is a problem that happens to companies of all sizes: Something about “plaintext” just kinda stings, especially as a security practitioner. It’s like… the most profane insult you can hurl at a security engineer. Imagine retorting with, “Oh yea? Well, you store your passwords in plaintext!” But logging passwords and storing them in plaintext are… kinda the same thing. Because while logs are rarely or purposefully public, they’re typically afforded broader access than direct access to your databases. Everyone knows by now that storing plaintext secrets in your database is a terrible idea. Logs, however, are still data-at-rest, and we should...