Watch and Variable Areas

VS Code separates the watch and variable areas. IntelliJ/IDEA uses a single area but uses different icons to signify the difference. Throughout this book, I will talk about the watch area as a single area to keep things simple. Keep in mind that this should apply to both.

Threads

There’s no thread section in the VS Code IDE. In this case, we’re debugging JavaScript code. As such, the list of threads isn’t applicable. Notice that if we were debugging Java code, such an area would be available there.

Line Breakpoint

The line breakpoint is the workhorse of the debugging process. It’s so common that we usually drop the word “line,” so you might know it by the name “breakpoint.” It’s the most basic type of breakpoint, and most of us should be familiar with it. It’s the breakpoint we saw in Figures 1-1 and 1-2. We see it when placing a breakpoint next to a line of code that isn’t a field or method declaration.

Method Breakpoint

We can add a method breakpoint by clicking the gutter area next to the method or function. Method breakpoints aren’t supported by every runtime or every IDE and for good reason.

The implementation of method breakpoints is often quite slow since the debug API stops for every method entry or exit. This includes the JVM implementation. As a result, modern debuggers, such as the one in IntelliJ/IDEA, place line breakpoints to simulate method breakpoints when possible. This is seamless to us as developers, but it’s important to know since some dogma states (correctly) that they are slow.

In Figure 1-3, you can see the different breakpoint type representations in IntelliJ/IDEA,¹ specifically watchpoint, method breakpoint, and a line breakpoint.

A screenshot of a few lines of code. It highlights lines 4, 6, and 8 as the breakpoints.

We can configure a method breakpoint to stop at the method entry/exit or both. A method breakpoint on entry is functionally equivalent to a line breakpoint. An exit breakpoint is harder to simulate as a method can have multiple exit points, and it might be impossible to place line breakpoints that simulate that behavior from within the IDE.

Ultimately, a method breakpoint doesn’t appear very useful. This is indeed the case, and very few developers make use of this feature. There’s an interesting use case that makes sense for method breakpoints, which we’ll discuss later in this chapter.

Field Watchpoint

A watchpoint is a breakpoint placed on a field. We can configure it to break when the field is read from, written to, or both. This is a remarkably useful tool to detect the code that changes a value or shifts it around.

This feature is even more useful when combined with tracepoints, which we’ll discuss soon enough.

Exception Breakpoint

Missing from Figure 1-3 is the exception breakpoint. This breakpoint type stops execution when an exception of a given type is thrown. We can define whether to stop on exceptions that are caught, uncaught, or both. We typically add an exception breakpoint in the breakpoint management UI. In IntelliJ/IDEA, we can open this UI via the Run ➤ View Breakpoints menu option, which you can see in Figure 1-4.

A screenshot highlights a checked option on the left navigation panel named, any exception. On the right, it represents the list of customizations under any exception.

In the breakpoint management UI, we also have the ability to break on all exceptions. This isn’t a popular feature for JVM languages. The problem is that the JVM throws exceptions as part of normal execution. The solution for this is “Catch class filters” that let us ignore exceptions that are caught by specific classes.

Conditional Breakpoints

We can attach a boolean expression to a breakpoint; when the expression evaluates to true, the breakpoint will stop. Otherwise, it will be ignored. This is an important tool that lets us narrow down specific nuanced failures.

If we can determine the invalid state of the application, we can use conditional breakpoints to stop only when that invalid state occurs, and we can litter our code with breakpoints.

A good example would be the binding of a method breakpoint to a wide range of methods, then using a condition to stop when the application state is invalid. You will probably stop shortly after the application becomes invalid. At that state, there might still be valuable information you can extract from the application.

Tracepoints and Suspend State

If VS Code did one thing right, it’s this: it brought tracepoints to the forefront by making them easily discoverable in the IDE UI. When right-clicking a breakpoint and selecting edit, “Log Message” is one of the options. We can enter a message directly on the following line as seen in Figure 1-5.

A screenshot of a few lines of code. it indicates the log message between lines 5 and 6. The message reads, this is the log message, + num.

Notice that the log message accepts code expressions and not a plain string. This lets us write an expression where we can invoke methods or list variables. This log message will be visible in the “Debug Console” view.

This feature isn’t as discoverable in IntelliJ/IDEA; we can reach it via the UI in several ways, such as shift-clicking on the gutter area. We can also uncheck the “suspend breakpoint” option to expose this feature. Tracepoints are mustard colored in IntelliJ/IDEA as seen in Figure 1-6.

A screenshot of a few lines of code highlights the eighth line. A dialog box under the line indicates a textbox under evaluate and log option. it text reads the following text, the log message, + num.

Once we do that, we need to check the “Evaluate and log” option and type our expression there. Notice that in IntelliJ/IDEA, we can create a conditional tracepoint. This means we can create a log that only appears if a specific condition is met. This can reduce potentially noisy logs to a manageable volume.

There are several other interesting capabilities we notice within the IntelliJ/IDEA advanced dialog. We can print a stack trace when we hit the breakpoint. This can help narrow down a message to a specific area of the code that printed it.

But the most interesting option is the suspend option. A typical breakpoint suspends all the threads when it’s hit. A tracepoint disables that behavior and prevents suspension entirely. IntelliJ/IDEA also includes another option of only suspending the current thread. This is very useful for multithreaded applications where we might need the background threads running while debugging.

Asynchronous Stack Trace

In the past decade, reactive frameworks and functional programming have gained tremendous popularity thanks to many potential benefits. However, they have one major drawback: debugging is harder.

An asynchronous operation disconnects the stack trace and removes our ability to follow the chain of causality. As a result, when we stop at a breakpoint in an asynchronous callback, we might get a very short stack trace that doesn’t help by much.

If you use a modern IDE to debug such APIs, this might not have been your experience. You might have seen a nontrivial stack trace for asynchronous calls. This is a feature of modern debuggers, which includes elaborate features to glue a stack trace. Some IDEs hide this “glued” aspect of the stack trace, while others, such as IntelliJ/IDEA, highlight it, as seen in Figure 1-7.

A screenshot represents a few lines of text grouped into invoke 1 and asynchronous stack trace.

The glued stack trace consists of two or more stack traces attached to one another. This creates the semblance of a single, detailed trace. On the surface, it might seem like semantics, but it matters since the global state might change. There could be a big difference between the state of the application when between the two traces.

This system works by tracking all asynchronous calls and keeping a “shot” of the trace in memory. Then, when a callback occurs, the debugger uses a heuristic to “glue” the right stack trace instances together. The heuristic used could be the callback instance reference. If the reference is identical in both stacks, that means they belong together. As a result, this can add a lot of performance overhead to the debugging processes. The IDE supports disabling this feature in the preferences to keep the debugger performant.

Return Immediately and Force Throw

Reproducing complex or edge case behaviors in the debugger is usually challenging. This is especially difficult when we need to debug things related to error recovery. To do that, we need to reproduce the error, then we need to go over the logic. With these tools, we can skip the “reproduce” part of the process. We can even simulate problems/behaviors that might be theoretical.

We can force throw an exception by right-clicking the stack and writing the exception allocation code into the dialog. This triggers a throw of that exception, and we can then follow the logic of the error handling code. In Figure 1-8, we can see the context menu for these operations.

A screenshot of a context menu containing the following options. drop frame, throw exception, force return, copy stack, export threads, customize threads, add stepping filter, async stack traces.

In some cases, the behavior might be related to a different return value. In those cases, we can follow the same process and force a return with a different value from any location in the method. This lets us return a different value by typing in a custom expression. The expression can be anything, a method invocation or just a hardcoded value.

A big benefit of both approaches is the ability to skip the execution of some lines within the method. If we have code in the method that’s currently failing, we can use one of these approaches to skip that code. This can happen in projects that are still a work in progress or at the edge of a long debugging session.

Drop Frame

I love undo. The undo shortcut is one of the most commonly used shortcuts on my keyboard. Undo in debugging is problematic though. How many times did we click the “step into” button by mistake and would want to undo the process?

Drop frame just removes the top stack frame and returns the application to the previous execution location. If you accidentally stepped into a method, you can use “drop frame” to return to the previous state. This typically works like force return by right-clicking the stack frame we wish to drop.

Jump to Line and Set Value

A common feature in the IDE is “Run to Cursor” where we can stand on a line and let execution reach that line. Jump to line lets us move the execution to a new line in an arbitrary location without executing the lines in between. We can move the execution pointer back to a previous location!

That is a remarkably useful feature. We can jump back and execute the same code over and over again while testing various scenarios. This works exceptionally well with the “set value” capability. Set value lets us modify variables dynamically and test different execution scenarios. Notice you can also use the “evaluate expression” capability to trigger a state change that would result in the same effect.

A great difficulty in software development is increasing the test coverage. The arguments passed to the unit test might be difficult to tune, so they will trigger the right branch/test. By tuning the values of variables and arguments to the method over and over again, we can verify instantly that a specific set of arguments will cover the right area of the code.

Not all runtimes and IDEs support this capability. To get this in IntelliJ/IDEA, we need to install a third-party plugin2. VS Code supports this only for some runtimes, such as C#, and at the time of this writing doesn’t support it for Node.js.

Rendering

IDEs display elements in the watch area using a renderer. Some debuggers provide deep rendering logic customizability. IntelliJ/IDEA provides the “Customize Data Views” UI, which you can access by right-clicking the watch area. In this dialog, which we can see in Figure 1-9, we can determine nuances on how elements are displayed in the watch area.

A screenshot of a dialog box represents a list of options under java with checkboxes alongside.

Custom Rendering

This works well enough for most standard cases. But the most interesting and complex objects are often quite problematic. Their toString() methods often include logic that makes more sense in a log than in the watch area. Their variable state might be too large and unreadable. So looking at them in the watch as we step over might not help.

Custom rendering lets us define how the object is presented as you can see in Figure 1-10. A good example is the JpaRepository from Spring Boot. The JpaRepository is an interface that to some degree represents a single SQL database table (this is a gross oversimplification). Its toString() method contains nothing of value.

A screenshot of a dialog box represents a list of options under java type renderers. The renderer name reads J P A repository and it contains a set of options when rendering and expanding a node.

A renderer applies to a class (or base class) name. As you can see from Figure 1-10, we bind this renderer to the JpaRepository class. “When rendering a node” lets us define custom code to execute whenever the IDE needs to “decide” how an entry is displayed in the UI.

Notice that the method invocation count() doesn’t include a qualifier. It’s executed in the scope of the object that’s rendered. The same is true for the expressions when expanding a node. The findAll() method returns all the rows (collection of entities) within the table. By placing it in the “When expanding a node” section, we’re indicating that it should be invoked to show the entries when the watched variable is expanded. This means a user can click the “+” sign next to the entry, and the results of the method will be shown, as you can see in Figure 1-11.

A screenshot represents a few lines of code. It highlights a line containing a dropdown and reads the following text. visit repository = open curly bracket, proxy 132at 12594, closed curly bracket, Repo has 4 elements.

The result of the rendering process is a large volume of SQL invocations. The count() SQL operation is invoked to show the preview when we look at an entry. A full select call is made when we expand the actual entry UI. This can be an expensive operation, especially when debugging against a remote database. This would probably be reasonably performant for local debugging on a smaller dataset.

Mute Renderers

I sometimes click the step over button and have to wait several seconds for a response from the IDE. This is a major problem. Performance can be remarkably slow sometimes, and that can sour us on using debuggers altogether.

Renderers can have a significant impact on the performance and responsiveness of a debugger. To make the experience more responsive and flexible, IntelliJ/IDEA has the option of “Mute Renderers” on the right-click menu. When enabled, a renderer isn’t shown by default, and instead we get a clickable area. Explicit clicking invokes the renderer on the entry.

Show Objects

When our program is running within a virtual machine, we have incredible access to the VM state. One such element of the VM state is the list of objects in memory. IntelliJ/IDEA lets you right-click any object in the watch and select the option “Show * Objects…”; for example, if you right-click an object of type String, you would see the option “Show String Objects…”. Clicking that option will present you with every instance of a String object within the JVM as you can see in Figure 1-12 – including JVM internal strings!

A screenshot represents a list of string objects. A text on the right-side panel reads, enable tracking for new instances.

We can filter this list with simple Java conditions so we can pinpoint a specific object of interest from memory. This is helpful for the common case where we wonder where a specific value came from.

Just use a dialog like this to find the right object and understand where the value is coming from. This is a remarkably useful tool for discovering duplication. A common source of bugs I encounter is one where we have multiple object instances with the same values. Using the filter, we can detect such duplication.

Memory Debugger

Memory inspection provides another way to get into the object list dialog and some amazing capabilities. We can enable this additional debugger view, and as we step over the code, we can see the differences from the previous allocations. Double-clicking of each entry seen in Figure 1-13 can show the list of VM object instances similar to the list in Figure 1-12.

A screenshot of a dialog box represents a table under memory containing the details of class, count, and differences.

We can see exactly how many objects we have of each type and the change for every step-over or between breakpoints. But we can take this much further…

On the right-hand side of Figure 1-12, you will see an option to enable tracking. Clicking this will cause the VM to track every allocation of this object type. The result is that we’ll know exactly which objects were allocated between these two steps and the exact stack traces for each object!

We can show only the new instances of a specific object by clicking the context menu option “Show new instances.” But an even more interesting feature is the ability to see the stack traces, as we can see in Figure 1-14.

A screenshot represents a box containing a list of strings. It highlights a string on the left that reads, string at 74395 = org dot apache dot Catalina dot web resources. A text box at the right represents a list of codes.

Object Marking

Two of the worst types of bugs relate to object identity. The first I mentioned before: having two object instances with the same values within them. This can trigger painful bugs down the road.

The second relates to multiple threads accessing one resource simultaneously. Both are remarkably hard to detect and track. I used to pull out a pen and paper, write the object ID (or pointer) of each object, and compare this manually every time we stopped on a breakpoint. It’s an error-prone and fragile process.

Object marking solves both problems in the most elegant way possible. We can select any object instance listed in the watch area and mark it from the context menu. Object marking creates a global label for the object which we can then use to reference it from anywhere. We can add a label to the watch, which is helpful, but we can do something far more powerful: we can add it as a conditional breakpoint as seen in Figure 1-15.

A screenshot represents a dialog box titled vet controller dot java colon 110. It highlights a text box containing a condition for the current thread.

To create Figure 1-15, I watched the Thread.currentThread() instance. Then on right-click, I selected the option “Mark Object” which I named: MyThread. Notice that IntelliJ/IDEA appends an “_DebugLabel” suffix to named objects. I added the label to the watch as well, so it will be easy to see that this stores the current instance and is not just a label around the watch.

As you can see, the values of the two thread objects are different; both have similar names, but one is thread 1 and the other is marked as 2. It means that the second invocation of this method used a different thread than the first invocation. This means there’s a theoretical chance for race conditions, deadlocks, etc., if we write risky code.

The amazing thing is that we can detect this situation almost seamlessly by using a conditional breakpoint and referencing the label object. We can do this within a tracepoint as well and log the information, which is a very useful tool for debugging thread-related issues.

This applies to the comparison of every object. APIs such as JPA can fail subtly when we invoke the API from outside the transactional context. This can implicitly create new object instances, which we didn’t expect an intelligently crafted breakpoint such as this can instantly verify such a bug.

Filtering Collections and Arrays

I almost didn’t include the following feature because I assumed everyone knew about it… After a debugging session with a relatively senior developer, I had some doubts. This is understandable, filtering on arrays and collections isn’t a universal feature for debuggers.

We can add a filter to a list (collection) or array through the right-click context menu. This can be any expression that’s performed on each element and should return true in order to show the element or not. Instead of sifting through a list of elements, let the watch condition do the work for you as you can see in Figure 1-16.

A screenshot represents a list of text containing the first and last names along with the i d and specialties. It highlights a line that reads, filtered by get first name dot length more than 5 clear.

Stream Debugger

The problem with this code is that it’s much harder to debug. We can easily step over a for loop and unwind it to understand the underlying failures, but a stream can be problematic.

To solve that, JetBrains introduced the stream debugger, which you can launch by clicking the button shown in Figure 1-17. Notice that this button is only visible when the debugger stack frame is pointing at a line of code with a stream within it. Also, keep in mind that the location of the button changes sometimes; it can appear directly in the debugger toolbar in some cases.

A screenshot of a context menu represents the options to force step over, force step into, smart step into, step out of code block, run to cursor, show execution point, drop frame, get thread dump, and settings. It highlights the option to trace current stream chain.

Within the stream debugger, we can follow each stage of the process and look at how the values mutate through the states. This works very well with object streams too, but the screenshots would be far more confusing as there are many details to sift through. As you can see in Figure 1-18, you can track the values as they move through the streams and follow the lines. The stream debugger takes inspiration from the realm of back-in-time (a.k.a. time travel) debugging.

A screenshot represents a dialog box titled stream trace. It contains a list of numeral values. It highlights the prime numbers by compiling the skip, limit, and filter commands.

Dependency

Custom Renderers in Code

The custom renderers I mentioned earlier are pretty amazing. If you have a similarly elaborate object graph, you might have such issues when rendering them in the watch. A “proper” renderer might make sense in this situation.

Stack Trace Hinting

As we discussed before, asynchronous code is one of the hardest debugging problems. We lose the context of the code. Stitching the stack trace to the original point where the binding was made is a great way to follow the causality of an operation.

IntelliJ/IDEA does that implicitly for common APIs, such as asynchronous calls in Sprint and similar APIs. However, if you implemented a custom API, such as a listener or an executor API, you might have a problem. You might get a callback with no context of execution. At that point, the original caller is gone, and we have no way of knowing how we reached this state. We can solve this by using custom annotations to indicate that we’re interested in such stack trace stitching to occur and how we would like that carried out.

The annotation triggers the storage of the reference pointer to the listener when we invoke the add method. Then when we invoke the fire method, the references are compared by the IDE, and the stack traces are stitched together seamlessly.

Why Is It Completely Wrong?

There are several inherent problems with sending us back to the documentation. I’ll start with the obvious: I never ran into useful documentation that could be read in five minutes. There’s no such beast other than for trivial tools which are probably not the source of your bug.

Next is the documentation itself. Which documentation? The documentation describing the system, OS, language, frameworks, tooling, etc.? Where do we stop? You should read all of those as much as possible, but it isn’t always practical or relevant at the scale we need.

Documentation doesn’t point you at a bug or even at a general direction of interest. If you could hold the entire application source code in your head and the full set of documentation, then parse the whole thing to reach the conclusion within five minutes… Then you might have a useful alternative to debugging.

But here’s the most ridiculous bit. I’ve wasted countless hours because of wrong and partial documentation. Back in the 1980s, reading the manual to understand a low-level architecture was essential. Modern systems move at a much faster pace and are far more fluid. Often, the only documentation we have is the source code. This isn’t a good thing. Unfortunately, it’s often the reality.

You Should Still Read the Documentation

Documentation isn’t an alternative to debugging. However, it’s still remarkably useful. Search engines revolutionized debugging, and with many issues, you can just search for the error message as a starting point. If you have documentation, search for the error and various keywords related to the error within the documentation; this can indeed save you some time.

Avoid the mistake of diving directly into an issue without first checking if someone else solved it first or if it’s a part of the troubleshooting section.

“It Works on My Machine”

Both approaches have their trade-offs: remote debugging and developer observability¹ can help with the latter. Replicating the environment might include obvious aspects (e.g., OS differences), but often there are minor differences that are hard to isolate.

Understanding the subtle differences in the replication environment is crucial. For this, we must have an open and clear communication channel with the person who can reproduce the problem. Some tools discussed in Chapter 3, such as strace, dtrace, etc., provide deep insight into a running application. Using these tools, we can notice the differences and misbehaviors when running the application. For example, the version on the client machine might access a configuration file you weren’t aware of in the user’s home directory.

The rise of container technology (e.g., Docker) removed most of these subtle differences as it normalized and simplified uniform environments. But this is often not enough. Networking differences, data source differences, and scale can significantly impact the environment even if everything else remains the same. Often, these are the hardest issues to debug. How do you reproduce an issue that only appears when you have 1000 requests per second in a large cluster?

Bug That’s Inconsistent

Logging is an important feature of most applications; it’s the exact tool we need to debug these sorts of edge cases. We will discuss logging in Chapter 4 and the best practices related to that.

Try to automate the process to get a reproduction. When running into issues like this, we often create a loop that runs a test case hundreds or even thousands of times. We do that with logging and try to find the problem within the logs.

Notice that if the problem is indeed an issue in concurrent code, the extra logging might impact the result significantly. In one case, I stored lists of strings in memory instead of writing them to the log. Then I dumped the complete list after execution finished. Using memory logging for debugging isn’t ideal, but it lets us avoid the overhead of the logger or even direct console output.

Give Up

Well, don’t “really” give up. Only give up on the ability to reproduce the issue consistently on your machine. We might end up spending all our time chasing the ideal case of reproducing them, like the great white whale. There’s a point where we need to cut our losses and move to the next step in the debugging process. This isn’t ideal, but it’s the only thing we can do.

We need to make assumptions about the potential causes and create test cases that reproduce the potential assumptions. Some bugs are resistant to reproducing but can be simulated using such means, and that might be enough. It’s important to add logging and assertions into the code, so if we don’t resolve the bug, we’d have more to go on next time around.

Before you decide to go with this route, I suggest looking at the “Tips and Tricks” section at the end of this chapter. Sometimes, a cliché, trivial advice is what we need to get the job done.

Everybody Lies

I love the show Dr. House. It’s a Sherlock Holmes version of a medical drama with a grumpy misanthrope as the lead. Perfect. Medicine isn’t like software debugging in most ways. We can replicate problems without hurting people. We can be very invasive without representations, and we can conduct our own tests and get results instantly.

But there’s still a lot to learn, and one of the core tenants of the show is that everybody lies. People tell partial stories and miss crucial information for various reasons. This is a very common problem for debugging. The hardest bugs are often the ones where we spend most of our time on a wild goose chase. The reason for that is usually a lie or a misdirection because of a faulty assumption.

I often joke during my talks that the most common gesture developers do when we finish debugging is the face-palm. This is because of the nature of the most difficult bugs. In most such cases, we find that we got the assumption stage wrong and made a “stupid” mistake in this stage.

Double Verification

The solution to wrong assumptions is double verification. For every assumption, no matter how basic, find another approach for proof. Let’s say we have a bug in code that depends on a result from a remote service. We assume the service works correctly, and we verified that by using the cURL program. To double verify, we also need to add a tracepoint in the code that shows we received the response.

As we slowly narrow our assumptions, we don’t need to double verify every assumption. But if we feel the process is stuck, we need to go back and make sure that we verified every stage along the way. It’s often the case that we miss something silly or simple. Make sure you verify the “low-hanging fruit” first.

Sometimes, double verification means stepping outside the debugger. In Chapter 3, I discuss some external tools such as strace, dtrace, wireshark, etc. These tools let us verify that the application is accessing the resources we expect on the operating system level.

Defining the Assumptions

We can look at debugging as a process of hypothesis, prediction, and forming an experiment to prove the hypothesis. Assumptions fit into the hypothesis stage where we form the boundaries around the bug. Another approach is one of hunting.

Imagine setting up a fence to trap a predator within the fence. The biggest mistake you can make is a fence that’s too small. Such a fence could leave the predator outside. You will be on the inside of the fence wasting time searching within the confines of an empty fence. The inverse is just as bad, when fencing off a large area, it will take a great deal of time to narrow down on the predator.

The same is true for our assumptions. We need to make basic assumptions, but we don’t need to verify every little thing. It’s hard to quantify the right amount, and this is where experience makes a difference. Awareness and methodology help boost that experience.

In Figure 2-2, I review the two initial layouts of my assumptions. Notice that this figure brings up an interesting situation. Our symptom, bug, and root cause might be in three completely different locations in the code. In those cases, our initial assumption circle might need to grow to fit in the bug.

An ellipse has the label wide assumption perimeter outside it. Inside the ellipse are following labels. Narrow assumption perimeter. 1. Symptom. 2. Root cause. 3. Bug.

Increasing the circle of assumptions isn’t necessarily a problem, but we need to be prepared for such an increase. There were many cases where I reviewed (and authored) bad fixes that addressed the symptom and not the root cause. Locating the bug and even fixing it aren’t our goals when debugging. Understanding why the program behaves the way it does and the root cause for that is the ultimate goal. That isn’t always an attainable goal as the track might run dry, but we can leave traps for next time and lay the long-term groundwork. I have more to say about defensive coding (fail-safe vs. fail-fast) in Chapter 4.

The Right Direction

When we see a symptom, we need to prioritize the assumptions we make to find the root cause. Normally, we use our intuition and experience to guide us toward that root cause. But is there a better way?

We often make a shortcut, trying to get from the system to the root cause. That’s often misleading. Our focus should be on the next stage, which is the bug itself, once we understand that the journey to the root cause is hard, but we have better tools for that.

When working with a wide assumption perimeter, intuition is the only guide. However, when circling the symptom, initial assumptions often become obvious. If a value is incorrect on the screen, we can start by verifying that the source returns the right value and work our way from there.

We will get into the nuts and bolts of addressing both of these in this chapter.

Assumption Jump

Sometimes, we’re left with a value that “shouldn’t be here.” This can be a variable value, a UI element, or any other artifact. It’s a value that doesn’t make any sense, and we don’t understand how that value is returned from this method. At this point, we might not have an immediate “move.” We know something is “clearly wrong,” but we don’t know how it got to that point or where the data came from.

If this is something you can reproduce and the data is in a field, then a well-placed conditional field watchpoint (from Chapter 1) can be illuminating. This helped me solve many such issues in the past. But sometimes it just isn’t enough. The issue might not be easily reproducible, and in this case, we might have a problematic state that’s hard to investigate. This is one of the best scenarios for time travel debugging (discussed in Chapter 5) which lets us traverse the state of the application after execution completed.

If you’re using a safe language (a.k.a. managed languages) that’s guarded against memory or stack corruption, then you have one option: add a tracepoint to every caller and slowly expand the assumption scope in that direction. For unsafe languages, you will need to check the debugger capability of memory guarding or use a memory guarded allocator.²

Figure Working Code

A common mistake we make is rushing to the line of code that contains the bug, since we already know where the problem is. We then get settled into a specific area around the bug and waste a great deal of time looking at that code. We’re trapped by the lure of a quick fix, which we incorrectly assume we know. Due to a distance between the symptom and the root cause, we might be misled by the lure of familiar code.

The scope should typically start at the next logical abstraction level. If the symptom seems to be related to a private method, start verifying the public methods. If a specific class is misbehaving, verify the classes that provide it with input. I usually give the analogy of a mechanic checking all the nearby screws are tight before dealing with the actual problem.

A great example here is the invalid value of a field. Our initial assumption would usually be that we need better validation controls to stop the invalid value. This is probably bad input from the user that propagates all the way into the variable. Before we leap to that assumption, we need to verify that our code didn’t propagate a bug onward.

Exceptions or Faults

Exception-related failures are relatively easy to detect. You usually get a stack trace directly to the error together with a description. We can place a breakpoint on all exceptions and verify that nothing “fishy” is happening behind the scenes in one of the methods (see Chapter 1). Exceptions are “loud” and “obvious,” so unless someone silently catches an exception (which happens), we can track the root issue relatively easily.

I would also strongly recommend a linter rule that checks against swallowed or unlogged exceptions. Linters are tools that review your application code for common bugs and pitfalls. I highly recommend adding one to your CI/CD (Continuous Integration/Continuous Delivery) process. For example, Checkstyle, which is a common cross-language linter, supports a check³ that blocks empty catch blocks. It still can’t block stupid code that does “nothing” in that block, but it’s a start.

Faults are standard in some languages but can also occur in a managed language where the VM crashes. They are usually coupled with a core dump that we can open in the debugger to investigate. Platforms like Java also include the full state of all threads in case of a fault; this can point us to a VM bug or an issue in a native library. It’s very important to collect this information and analyze these crashes. Sometimes, a small change, such as a minor JVM version upgrade, can resolve such crashes.

Unfortunately, this is often not enough. Getting core dumps working on all OSs is tedious, so I would recommend checking out Oracle’s documentation on the subject;⁴ notice that this is specific to the JVM, but many of the concepts apply universally. Once we have a core file, we can open it with most native OS debuggers and inspect the stack traces of the crash.

Once we do this, you should see a simple GUI that will let you investigate the causes of the crash. Chapter 3 discusses jhsdb in more detail as it’s a useful tool for debugging running VMs as well.

Bad Application State

Threads are a source of difficult bugs, yet most bugs result from bad application state. To track state-related issues, try separating the state elements that are modified concurrently and the state that’s read by the problematic block of code.

Assuming you can, try overriding the variable value within your breakpoint by setting it during debugging. You can set variable values in the watch area or via the expression option in the debugger. This is a great capability that most developers don’t utilize often enough. If you’re able to narrow down the value of a specific variable as the cause of the problem, you’re already well on your way to solving the problem.

If this isn’t helping, try identifying specific fields that might be problematic. You can use a field watchpoint (discussed in Chapter 1) to track changes to state and even value reads.

If the problem persists, a powerful approach is returning a hardcoded state or a state from a working case. I’m normally not a fan of techniques that require code changes for debugging since I consider the two distinct tasks. However, if you’re out of all options, this might be your only recourse.

The memory debugging capabilities and object inspection capabilities we discussed in Chapter 1 are also very helpful for these cases. A very common bug is a case where a copy of an object is made with slight modifications. Both objects appear identical to the casual observer, and as we debug, we might get subtle failures as a result.

Concurrency, Timing, and Races

Thread problems have the reputation of issues that are hard to solve. We’ll discuss these in greater detail in Chapter 14. Right now, we’ll only focus on finding and understanding the bug, and that’s a more manageable task.

The easiest way to check for threading issues is, as I mentioned before, logging your current thread and/or stack. Do that in the block of code that’s causing the issue. Then add a similar tracepoint on fields used by the block of code. Thread violations should be pretty clear in the logs. Using marker objects to keep track of the various threads is also a powerful tool!

With threading issues, the key challenge is reproducing them consistently. Creating a massively threaded unit test is often the most powerful debugging tool you can have in your corner. Notice that you should probably avoid Java’s “virtual threads” as they might skew the results.

If the problem is a deadlock, then you’re probably in luck, as those are usually pretty clear once reproduced. The app gets stuck; you press pause, and the debugger shows you which thread is stuck waiting for which monitor. Monitors are the OS-level primitives that block a thread from entering a code block, roughly mapping to Java’s synchronized keyword. You can then review the other threads and see who’s holding the monitor. Fixing this might be tricky, but the debugger “tells us” what’s going on.

With a livelock, we hold one monitor and need another. Another thread is holding the other monitor and needs the monitor we’re holding as shown in Figure 2-3. A livelock can result in a deadlock; unlike a deadlock, a livelock can occur while threads are still functioning. It can seem that both threads are working correctly and aren’t stuck. Because livelocks can occur with live threads, the code might appear fine on the surface without a clear monitor in the stack traces.

A cycle diagram of livelock. The circles are labeled as follows. 1. Thread A. 2. Resource 1. 3. Thread B. 4. Resource 2. The resources are held by the threads, and the threads require resources.

Debugging this requires stepping over the threads one at a time in the thread view and reviewing each one to see if it’s waiting for a potentially contested resource. It isn’t hard on a technical level, but it’s very tedious. That’s why I recommended enabling thread groups in the thread view. A typical application has many threads which can make this hard to follow. This produces a lot of noise which we can reduce by grouping the threads and focusing on the important parts. Again, logging using tracepoints can be very helpful in these situations.

This is one of those elusive bugs, such as race conditions, which are often classified as thread problems (which they are), but during a debugging session, it’s often easier to see them as a separate task.

Timing issues and race conditions often occur when your code has some unintentional reliance on performance or timing. The way I approach race conditions in threading code is this: “It’s a state bug.” It’s always a state bug. When we have a race condition, it means we either read from the state when it wasn’t ready or wrote to the state too late/early. Field watchpoints are your friends in this case and can really help you get the full picture of what’s going on. You can also simulate bad state situations by changing variable values.

I suggest creating field watchpoints with conditions related to the current thread. This way, they will trigger if the wrong thread tries to access a field that it shouldn’t.

Performance and Resource Starvation

Performance problems caused by monitor contention are harder to track with a debugger. I normally recommend randomly pausing the code and reviewing the running threads in your application. Is thread X constantly holding the monitor? Maybe there’s a problem there. There are some system tools such as strace and dtrace which we can use to track overuse of monitors. We’ll discuss these and many other tools in Chapter 3.

We can derive assumptions and prove them by logging the entry/exit point for a lock/synchronized block. Notice you can use a profiler and it sometimes helps, but it might lead you on the wrong path sometimes.

Resource starvation is an offshoot of performance issues. Often, you would see it as an extreme performance issue that usually only happens when you try to scale. In this case, a resource needed by a thread is always busy, and we just can’t get to it. For example, we have too many threads and too few database connections – or too few threads and too many incoming web requests. This problem often doesn’t need a debugger at all. Your environment usually indicates immediately that it ran out of resources. The problem is almost always plain and obvious.

The first case is trivial. You can benchmark and see if something is holding you back, for example, if IO is slow or a query takes too long. The second is the more common: a resource leak. We can miss this entirely when running in a GC (garbage collection) environment. But in a high throughput environment, the GC might be too slow for our needs. For example, a common mistake developers make is opening a file stream which they never close. The GC will do that for us, but it will take longer to do that, and the file lock might remain in place, blocking further progress.

This is where encapsulation comes in handy. We must encapsulate all of our resource usage (allocation/release). If we do that properly, adding logging for allocation and freeing should expose such problems quickly. This is harder to detect with DI frameworks like Spring where connections, etc., are injected for you. You can still use some tricks for injected data and resource tracking as well.⁵

Unit Testing and Mocking

Unit tests are a great way to debug, as they can often isolate the problem to its core elements. We will discuss unit testing in more depth in Chapter 4. The biggest benefit of debugging through a unit test is that we can leverage mocking frameworks, such as Mokito. These frameworks let us remove large swaths of code and narrow the problem space considerably. The benefits of avoiding regression and simplifying our test cases are a bonus.

Notice that there are various “best practices” that cover the “right amount” of mocking per test. When building a test case to debug a specific issue, these practices don’t apply as much. Mock everything you need to reduce the problem to its bare essence.

Eliminating in Flaky Environments

The process of elimination works great when we have a problem that’s easily reproducible. Flaky issues, or issues that turn out to be flaky as we eliminate code, are a major problem. The solution for this is to only work on negatives and never on positives when performing elimination. This can be confusing, but it’s an important rule.

If you eliminate a block and the problem doesn’t reproduce, this doesn’t mean that the problem is related to that block. It could be flakiness. However, if you eliminate a block and the problem reproduces, then this block is probably not at fault. So only rely on cases where the problem reproduces… I would also run checks repeatedly if you suspect flakiness to verify that the problem doesn’t occur on multiple runs.

Understand the Tongs Process

The tong motion is a crucial concept that we need to keep in mind when we’re debugging. Think of debugging as tongs that go through the buggy software to try and grab the bug. We need to position the tongs so they match the sides and then squeeze to get the bug we’re trying to grab. Figure 2-4 shows a sample process for slowly narrowing the surface area of a typical Java web application.

An illustration of 3 debugging stages. 1. Mock web tier. Replace database tier with mock data. 2. Invoke presentation tier method. Mock database objects. 3. Invoke business method. Mock business method dependencies.

A common mistake is to move only one prong of the tongs or not position the other tong. If you don’t position the tongs, your investigation might impact the other side and you wouldn’t know what’s going on. You don’t have to move both tongs for all cases, but if you’re stuck, then you can always flip the direction of motion and investigate from the other side. Then flip back when convenient.

Issue

Don’t work on a bug without an issue in your issue tracking system. Make sure they assign the issue to you and that it’s marked as active. This way, the project manager and teammates can keep track. Keep up to date on other issues to avoid conflicts. Even if you and a teammate are working on two separate issues, if the investigation leads to the same area, your fixes might collide.

Issues are often easier to locate than pull requests or specific commits. When reviewing a regression, the ability to connect a pull request to the source issue is essential. We need to keep the discussion within the issue tracker and not in volatile environments like group chats.

Daily Meetings

Teamwork and a daily meeting are essential for projects with multiple active developers who work on similar things. Don’t be a wallflower in such meetings. Update the team about your current theories and direction. If a colleague is describing something that sounds familiar, engage!

This is the reason to have these meetings, and if there’s a chance of mid-air collision while working on a bug, you should work as a team and not as individuals. At this point, the project manager can help guide you on aligning your work.

Test

Many developers want to start the debugging process by creating a unit test that reproduces the issue. This isn’t easy or even possible before you understand the problem. You should create a test before fixing the issue, and it should be a part of the pull request that fixes the issue. I would highly recommend requiring a 60% or higher coverage ratio per pull request, so changes that aren’t covered by tests will be rejected by default.

The alternative is constant regressions, where an issue that was fixed comes back to bite us repeatedly. We’ll discuss unit tests in depth in Chapter 4.

Notice that unit tests are usually ideal since they run fast, produce instant feedback, and can pinpoint the issue with razor sharp precision. Integration tests are wonderful and arguably even more important than unit tests. In some cases, they’re the only way to reproduce an issue and that’s OK.

Rubber Ducking

The term rubber ducking originates with a developer who supposedly brought with him a rubber duck to talk to when facing an issue. This illustrates a common developer observation: when we consult a colleague, we often find the problem before the person we’re talking to says a word…

During conversations, we often find that we’ve missed something in one of these stages.

Redefine the Bug

“Moving the goalposts” is awful when someone does that to you. It’s a great way to attack a bug, though. Bugs usually start with a “user-level” description, which is usually very symptomatic, for example, wrong value for a UI element. As we work on the bug, we might find that this results from the backend that’s wrong.

We can change the bug and redefine it (ideally on the issue tracker too), so our goal and scope of code are narrower. This is an iterative process of honing in on the bug, which we normally do intuitively. But we rarely go to the trouble of redefining the bug. This is an important mental exercise and also an important team communication tool. When you mention this in the dailies, it might trigger recognition in a team member and help lead to the correct fix.

Change the Angle

Every system has at least two directions. I discussed this in the tongs section earlier, but it bears repeating. If you’re stuck, flip the direction and try from a different angle. There are many angles to approach any bug and many techniques. I’m a big fan of “pair debugging,” whereas in a team we can share different ways and ideas to debug something unique.

Unlike pair programming, which can be a lengthy process, just five minutes with a teammate can help change your outlook on an issue and expose you to a completely different thought process.

Disrupt Your Environment

Hard-to-reproduce bugs are terribly frustrating. When you’re faced with such a situation, try using external limiting factors that “break your environment” that might trigger the bug more readily. Good examples for such disruptions would be systems like network throttling, slow motion modes, etc.

I often debug client code when connected to a tethered phone. This removes me from the company’s internal network, which is a major disruption. A common bug is inadvertently relying on company network topology. It also slows my connection significantly, which might impact the application. This can produce different error messages, reproduce an issue more consistently, fail to reproduce it entirely, or some other behavior. All of which are crucial clues in the bug hunting process.

Switch Environments

Similarly to the previous point, switching the OS entirely can sometimes expose interesting information about a hard-to-track bug. At Sun, we used to have Spark and Windows machines connected to a KVM (Keyboard Video Mouse switcher). We’d constantly toggle between the two and often found “weird” issues when moving code back and forth.

Switching the OS, browser, JVM, compiler, etc., can have a significant impact. The tooling of a specific environment might also provide a different angle. I prefer working with Firefox and its developer tooling when debugging frontend issues. Occasionally, I’ll look at a problem in Chrome DevTools to get a different perspective where I might notice something I missed in Firefox.

Use Developer Extension Tooling

All development environments have special debugging extensions and tools we can use to get more information and insight and also perform disruptive investigations. Web development tools in the browser include deep inspection capabilities, and phones have developer mode that you can activate to leverage some amazing tooling. Server APIs like Spring include aspects for logging automatically on every method entry/exit, actuator, and so many other tools.

I will discuss some of the low-level tools like JMX (Java Management eXtensions) in the next chapter. We can use these tools either as an investigative prong or as a tool to impact the application as a disrupting mechanism. But in order to do that, we need to be deeply familiar with the environment we’re running in and with the vendor tools for it. I was working deep into JetBrains plugin development before finding out about the developer mode of the IDE. That’s on me. I was running too quickly and didn’t spend the time to read a proper plugin development book before getting started.

Walk Away/Sleep

For many developers, sleep is the best debugging tool, as they wake up with the solution. I agree with one personal caveat: I’m a terrible sleeper. When I have a bug, I can’t sleep. I need to “try everything” first; otherwise, I’ll lie in bed frustrated. But I can get up and walk away from the computer and drink a cup of coffee. Have a conversation with someone about something else…

Disconnecting and then coming back to the problem puts us in a fresh mindset to tackle the problem. It’s important not to jump right back into the place you were before when doing that. Try to forget everything and approach the problem “fresh” to notice the things you missed the first time around. I find that most frustrating problems result from something small I missed when I rushed when working on this problem initially.

At our offices, we have a gaming room where the R&D team spends time playing console games after lunch. This helps team bonding and helps them disconnect for a while from whatever they’re working on.

It’s like a Game

Debugging should be fun/challenging, and if it isn’t, you should find a way to enjoy this puzzle/treasure hunt. I’m very competitive. As a result, I’m not a fan of most puzzles/games because (like many) I don’t enjoy losing. Debugging is a game I love because I’m good at it. I still enjoy it when I’m terrible at it (we all are sometimes) because like in any game, we have our good and bad days.

If you can’t find the pleasure in debugging, try debugging code that isn’t yours. Try debugging things that aren’t a part of your job. Do you enjoy doing those?

We often conflate stress about having a bug or work stress related to the project. Since the debugging process is so amorphous, it can drag on. This makes us feel stupid and incompetent and can promote impostor syndrome. These are universal feelings that are shared by experienced and fresh developers alike.

I would suggest isolating whether these are job-related tensions that are causing this frustration or if they’re related to personal embarrassment. If the latter, I suggest sitting down with the most senior developer you know and tell that person about your dumbest bug. I’m confident that they’ll share a similar story. If not, feel free to send me your dumbest bugs. I spent a couple of days tracking a less than operator facing the wrong way.

A NASA engineer confused metric and imperial measurements costing them 327 million dollars because of that mistake.⁶ I’m sure the engineers or the code reviewers are some of the smartest people out there. If they can have such a terrible outcome, then nothing you can do can rival it.

If you feel that this is all due to work-related stress, I suggest taking time to talk to your manager about the stress. It’s often counterproductive and could slow you down as you rush to debug, which is something that should be a more creative process. I suggest talking to a senior mentor and seeking help in such cases. Some developers have an issue with confrontation; a senior friend can help in such situations.

Running DTrace

DTrace is a risky “low-level” system service and should be treated as such.

On a Mac, DTrace conflicts with “System Integrity Protection” which is a security feature that blocks some interactions between processes (among other things). Under normal circumstances, this would be great. But if you want to run DTrace, this would be a problem.

The solution is booting to recovery mode on Intel Macs; this means holding the Command-R keys on boot. On an ARM Mac, just long press the power button. Then, in a recovery mode terminal, issue the command: csrutil disable.

Upon reboot, DTrace should work as expected. If you wish to reenable integrity protection, follow the same instructions, but instead of csrutil disable, use csrutil enable.

Basic Usage

The code snippet passed to the dtrace command listens to the sendto callback on the target process ID (9999 in this case). Then it prints out the information to the console, for example, 9999 text.

If this seems like a bit too much and too hard to get started with, you’re 100% right. It’s a powerful tool when you need it. But for most of our day-to-day usage, it’s just too powerful. What we want is to know a bit of basic stuff, and this is just too much.

Simple Usage

These are just the tip of the iceberg. I suggest checking out this old dtrace tutorial¹ from Oracle or the book.² Disclaimer: I didn’t read the book…

Running Strace

Nowadays, strace is commonly used in Linux; it’s my favorite system diagnostic tool on that platform. It’s remarkably convenient to work with since we can run it with no special privileges. Notice that unlike DTrace, you should keep strace away from production environments (unless the code is segregated). It carries a significant performance overhead and would bring a production system down.

Java tries to load the pthread library from the tls directory using a system call open to load the file. The exit code of the system call is -1, which means that the file isn’t there. Under normal circumstances, we should get back a file descriptor value from this API (positive nonzero integer). Looking in the directory, it seems the tls directory is missing. I’m guessing that this is because of a missing JCE (Java Cryptography Extensions) installation. This is probably OK but might have been interesting in some cases.

There are many system calls you can learn and use to track many behaviors such as these: connect, write, etc. This is only the tip of the iceberg of what you can do using strace. Julia Evans wrote some of the most exhaustive and entertaining posts on strace.³ If you want to learn more about it, there’s probably no better place (also check out her other stuff… Amazing resources!).

Strace and Java

As you saw before, strace works great with the JVM. Since strace predates Java and is a very low-level tool, it has no awareness of the JVM. The JVM works like most other platforms and invokes system calls which you can use to debug its behavior. However, because of its unique approach to some problems, some aspects might not be as visible with strace.

A good example is allocations. System tools use malloc, which maps to kernel allocation logic, but Java takes a different route. It manages its own memory for efficiency and easier garbage collection logic. As a result, some aspects of memory allocation will be hidden from strace output. This can be a blessing in disguise, as the output can be overwhelming sometimes.

At the time of this writing, threading works well with strace. But this might not be the case moving forward as project Loom might change the one-to-one mapping between Java threads and system threads.⁴ This might make strace output harder to pinpoint in heavily threaded applications.

In the Old Days

Back in the old days of SVN (or CVS, SourceSafe, etc.), we used to check out an older version of the repository and test on it. If it failed, we’d go further back, and if it succeeded, we’d go forward to hone in on the specific commit that failed. Those among us who were lucky enough to work with competent QA departments could often pass this task to them. Ultimately, the work was manual.

When zeroing in on the issue, we’d follow the sensible search strategy of dividing the number of revisions in half and going to the middle of the set instead of going one step at a time. This significantly shortened the time spent looking for the problematic revision. However, there are still many revisions to search through. At this point, you might wonder, why didn’t you automate these things?

We sometimes did, but since no versioning system was as dominant as git, these automations didn’t last, and I’m not aware of such an automation that made it into any version control system. But git bisect made it in and is the automation of this heuristic.

Find the Bug. Automatically!

Notice there were eight revisions to test, and suddenly there are only four. We don’t need to test all eight revisions since we take the divide and conquer approach of splitting the revisions in the middle. Notice that in this example, I only tested three revisions to find the bug.

Can We Automate This?

Going through every revision manually is a major pain! It’s better than randomly looking through revision logs and jumping through revisions while keeping our place in our head – but just barely better. Luckily, there’s a better way: run.

Bisect can run an arbitrary command for us on every revision it encounters. If the command returns zero (as a process exit code), the revision is good. If it returns something else, it’s bad. This way, the broken revision will be identified automatically for us with no human interaction.

Normally, this works great for a shell command, but as a Java developer, this is a bit of a pain. Typically, I would have a unit test that shows the failure. Unfortunately, that unit test doesn’t exist in the older version of the project. Then I also need to compile the project for this to work. So while git bisect seems cool with the run command, how do we use it for a compiled language like Java?

That’s it. Notice that if compilation will fail because of dependencies within the test class, you might end up with the wrong revision. So keep the test simple!

Git bisect is probably the simplest tool I will cover in this chapter. But it’s also one of the most important tools. Learning to use it effectively can save you days of tedious work and hunting around for an issue. Despite these huge benefits, it’s a relatively obscure feature. The reason for this is that for 98% of the time, we don’t need it. But in that 2%, we really need it. Hopefully, the next time you run into a regression you’ll remember that it’s there and use this post to hunt down the issue.

How Does JMX Work?

JMX exposes management “beans” (MBeans); these are objects that represent control points in the application. Your application can publish its own beans which lets you expose functionality for runtime monitoring and configuration. This is very cool as you can export information that an administrator can wire directly to a dashboard (APM, Prometheus, Grafana, etc.) and use that for decision making.

If your server has multiple users connected concurrently, you can expose that number in JMX, and it can appear in the company dashboard thanks to some wiring from DevOps. On your side, most of the work would be exposing a getter for the value of interest. You can also expose operations such as “purge users,” etc. An operation is a method you can invoke on a JMX bean.

Spring also supports exposing a lot of server details as management beans via actuator.⁶ It exposes very deep metrics about the application and helps you jump right into “production ready” status!

JMXTerm Basics

Usually, one controls and reads JMX via web interface tools or dedicated administration tooling. If you have access to any of them, I suggest you pick one of them up and use them as it would work pretty well. Some developers like JConsole; I didn’t use it much. I’ve used some of other tools, and I prefer them in some cases. I also enjoy using IntelliJ/IDEA Ultimate’s support for actuator which is a pretty powerful visualization tool that you can see in Figure 3-1.

A screenshot of the debug screen. It has the options of thread and variables, console, and actuator at the top. The option actuator is selected. The actuator has the options of application availability auto configuration, auto configuration, and conditions report endpoint auto configuration. The option conditions report endpoint is selected.

JMXTerm is just as powerful but doesn’t include the visualization aspect; in that sense, it’s exceptionally convenient when we need to understand something quickly on a server that might be alien. It’s also pretty useful for getting high-level insights from the server internals. We can get started by downloading JMXTerm from https://docs.cyclopsgroup.org/jmxterm.

Notice that JMX needs to be explicitly enabled on that JVM. When dealing with a production or containerized server, there will probably be some “ready-made” configuration. However, a vanilla JVM doesn’t have JMX enabled by default.⁷

JMX is a remarkable tool that we mostly use to wire management consoles. It’s remarkable for that, and you should very much export JMX settings for your projects. Having said that, you can take it to the next level by leveraging JMX as part of your debugging process.

Server applications run without a UI or with deep UI separation. JMX can often work as a form of user interface or even as a command-line interface as is the case in JMXTerm. In these cases, we can trigger situations for debugging or observe the results of a debugging session right within the management UI.

Basics of jhsdb

debugd

The debugd command isn’t as useful for most cases as you probably don’t want to run debugd in production. However, if you’re debugging a container locally and need to get information from within the container, it might make sense to use debugd. Unfortunately, because of a bug in the UI, you can’t currently connect to a remote server via the GUI debugger. I could only use this with command-line tools such as jstack (discussed later).

to connect to the server with the jstack operation. Notice that the --connect argument applies globally and should work for all commands.

jstack

This snapshot can help us infer many details about how the application acts in production. Is our code compiled? Is it waiting on a monitor? What other threads are running and what are they doing?

jmap

When you tune GC flags to get better performance and reduce GC thrashing (especially when collecting very large heaps). The values you see here can give you the right hints on whether something is used. The true benefit is the ability to do this on a “real process” without running in a debug environment. This way, you can tune your GC in production based on real environment conditions.

There are a few other capabilities in the jmap command, but they aren’t as useful as those two for day-to-day work.

jinfo

jsnap

GUI Debugger

This launches the debug UI that you can see in Figure 3-2.

A screenshot of the screen with files, tools, and windows at the top. The screen of stack memory for container 0 indicates the interpreter frame method, the interpreter locals area for the frame, and the interpreter expression stack. The screen inspector indicates young org or Apache and interpreter constant pool cache.

The amount of capabilities and information exposed in this tool is absolutely fantastic. A lot of it might be too low level if we’re debugging from the Java side. But if you’re working with JNI native code, you can see the addresses matching the JVM thread/methods in the stack. This can put in context some native code debugging where the symbols might not be exposed coherently.

You can get the stack information for a specific thread (similar to jstack). The inspector tool contains deep internal JVM state information per thread that can provide internal status. Most of that information isn’t as useful for debugging typical application-level bugs, but if you’re trying to understand the root cause of a JVM crash, it might come in handy, especially if you’re somewhat familiar with VM internal monikers.

There are many other features in the “Tools” menu such as Class Browser, Code Viewer, Memory Viewer, etc. All of those are very low-level access tools that might come in handy for tracking crashes or niche edge case issues.

jhsdb is a remarkably powerful and useful tool that I hope you will add into your toolbox. Its biggest fault is that it’s too powerful and too low level. As a result, most Java developers might find it intimidating. This is fair when dealing with some of its more esoteric features that only make sense when interacting with native code.

However, jhsdb also includes many high-level capabilities from deadlock detection to JVM information. Those are easily accessible and easy to use. They’re performant enough for production environments, and since they’re a part of the JDK, they should be available out of the box and in the command line. When you need to explore a JVM crash, often after the fact where all you have is a core file, this is the best tool for the job. It’s also a pretty great tool if you need to tune your GC, thread pools, or JVM configurations. If you’re debugging JNI code and misbehavior, this can be the tool that makes a difference.

Getting Started with Wireshark

You can download and install Wireshark from their website; it’s open source and free for use.¹⁰ Notice that there are separate steps to install command-line tools, etc., so I suggest reviewing the instructions. Once you install and launch the application, you would face the UI in Figure 3-3.

A screenshot of a screen titled, Welcome to Wireshark. The page has options to enter a capture filter, and all interfaces are shown. The user's guide, wiki, questions and answers, and mailing lists are given below under learn.

Here, we need to decide which network interface we wish to capture. This can be confusing if you don’t come from a networking background. You might debug a local server and assume that you need to debug the en0 interface. This isn’t the case. If the server is running locally, you should debug the Loopback.

Look at the activity graph on the right-hand side. This shows you instantly which interface is active and which one isn’t. You probably only care about active network interfaces, so it has to be one of those. But even when we pick the right interface, the challenges are still severe. Wireshark is a packet capture solution. It doesn’t distinguish between apps; we get a lot of noise when connecting to an interface, as you can see in Figure 3-4, which only shows a small portion of a constantly updating screen!

A screenshot of the Apply a Display Filter Screen page. It depicts a table with a number, time, source, destination, protocol, length, and info. The transmission control protocol, S r c Port 53228, D s t port 61452, Seq 0, Len 0 is highlighted below the table.

The volume of TCP traffic is pretty big, and I’m not doing anything yet… All of that is just background noise that’s interesting to dive into in our spare time. But probably not what we want to look at when we’re trying to narrow down a damn bug…

Luckily, we have display filters that let us remove the noise by focusing only on the specific port which we’re interested in with the condition tcp.port == 8080. Notice that my server is running on that port which makes it pretty easy to filter everything else. This brings down the noise considerably as you can see in Figure 3-5.

A screenshot of the t c p dot port equals 8080 screen page. It depicts a table with a number, time, source, destination, protocol, length, and info. The hypertext transfer protocol is highlighted below the table.

Still there’s a lot of noise because of the protocol. Notice that Wireshark is smart enough to recognize and even highlight the HTTP request I made to the server. Assuming I don’t care about the nuances of TCP, I can further reduce the noise by adding an HTTP condition as tcp.port == 8080 && http. This will limit everything to only HTTP requests and responses – again very useful stuff that can have a big impact if you’re dealing with many HTTP requests and connections. We can see the result in Figure 3-6.

A screenshot of the t c p dot port equals 8080 and h t t p screen page. It depicts a table with a number, time, source, destination, protocol, length, and info. The second row of the table is highlighted.

The display filter syntax is pretty powerful and supported by auto-completion within the filter bar; you can read about it in the user guide.¹¹ You can filter by IP address (not as useful for Loopback), source port, hostname, protocol, content, and so much more. This is just the start though. You’ll notice the second entry is the response from the HTTP server, and it’s marked as JSON. Wireshark detects the response type and displays it conveniently as shown in Figure 3-7.

A screenshot of the screen highlights the hypertext transfer protocol. The screen contains JavaScript object notation colon application forward slash J S O N, array, object, member i d, member first name, string value Sharon, and member last name.

On the bottom portion of the screen, we can see the raw packet display, which is typical for sniffer tools. In the bottom-left corner, you can see the hexadecimal values returned from the server. On the right, you can see the matching ASCII (or UTF-8) characters matching the hex. This is readable but might be a bit “unpleasant” for developers that aren’t accustomed to debugging network traffic. If you look one level up, you will see a tree marked as “Array"; this is the JSON array from the response.

Wireshark detected the HTTP protocol. Then it detected that the response type was JSON. It then parses the JSON and displays it as a tree for you to read conveniently even if the packet is minimized and obtuse. When I make a selection within the tree, it highlights the place in the packet so we can inspect the surroundings and see that everything is in order. This lets us instantly see what went on when connecting to a server/client and act accordingly.

I’d love to go into more details about Ethernet networks, decrypting HTTPS, debugging devices, and other complexities, but this is mostly the pain of configuring OS-specific settings, browsers, and a whole mess of things to look into the SSL traffic.¹² Personally, I try to disable security for debugging; it makes this much simpler. But sometimes there’s just no choice.

Using Wireshark As a Debugging Tool

When we debug, we need to verify our assumptions, and in many cases, that assumption relates to something that happens outside of our control area. A common issue I experienced more times than I care to admit is an invocation of a web service that failed because of something internal. This means the web server rejected my request even before my code was reached.

One way to debug this is to go into the server implementation code and look at what’s happening over there. This is something I did more than once. But often this isn’t trivial as the code is hard to read and overly general. Debuggers are sometimes out of sync with the code, and it’s a painful experience overall. Using Wireshark to review the request/response and understand why something failed is very helpful, especially for the cases where one client works and another fails.

I’ve had several cases where a new client implementation failed to connect to the same server which wasn’t under my control so no access to sources. Using Wireshark, we discovered that this relates to HTTP chunked mode that is initialized implicitly in the Java SE URL API. By changing the way we invoked the API, we could get the new client to work in the same way as the old client. I can’t imagine finding or fixing this bug without Wireshark.

I don’t reach for Wireshark often in my day-to-day work. There are usually simpler ways to solve these issues than inspecting network packets. But when I need Wireshark, it’s a fundamental change. The capture process is trivial when compared to far more complex tools. The capture filters are powerful and fast. As a general-purpose network protocol analyzer, it’s absolutely fantastic and has so many deep use cases I can’t even begin to get into (e.g., mobile devices).

Too Much of a “Good Thing”

When our applications fail in production, we usually reach for the application log. It’s our first “line of defense.” If it isn’t enough, we will need to reproduce the problem in some way, and that isn’t trivial. While logging does receive a lot of tooling attention and appreciation, the practice and standardization around it is lacking. Most companies don’t have a “best practices” guide or any significant standardization around it.

Logging is the very definition of debugging ahead of time. We write the log before there’s a bug and then rely on it to point us in the right direction. If we could, we’d log everything. Unfortunately, that’s a disaster waiting to happen.

Overlogging is a major problem. The cost of logging in production environments is often estimated as roughly one-third of the total cloud costs. That’s a tremendous amount to spend on a problem that might not happen. This doesn’t account for the performance overhead incurred due to logging. That can cascade into significant additional costs overall.

Cutting down on logging is a double-edged sword. If we run into a production issue and a log is missing, then we have no way of tracking down the issue. There were multiple attempts at working around this issue, such as log levels. These are good ideas, but the standardization around them is too general. To make matters worse, developers don’t apply standardization uniformly, even within the same project.

The common solution to this problem is a “Logging Best Practices” document, similar to the ones we have for coding standards.⁷ Such a document should be a part of onboarding in companies and should be referred to by developers during peer reviews.

Use MDC (Mapped Diagnostic Context)

I’m not religious about loggers. I will use the one we have in a project without complaining about implementation nuances or small performance differences. If your logger doesn’t support MDC, switch to a different logger. MDC isn’t a feature you should compromise about or glaze over. Still, for such an important feature, you’d expect to get some name recognition for MDC, but it’s still a feature that’s relatively unknown.

MDC provides a way to associate a process with context information. Let’s say you have a process that implements a REST API response. The REST endpoint that received that request would be interesting contextual information, right?

This is problematic because we might have a failure before the remove call is made. An exception might be thrown in the thread, and as a result, the MDC code might be missed. We need to guard against that and make sure that threads are clean. The strategies for that are too varied to include here, but there are usually best practices covered in the MDC documentation from your logging vendor.

Should I Log?

The biggest question in logging is this: What should I put into the log and where? This is a difficult question to answer in a general enough way, but I will try. For most cases, I recommend having one log per method before the return statement. This means we can log the return value, which is usually the most important aspect of the method.

This policy is valuable in the case of logging, but also makes debugging much easier in the debugger itself. We can inspect the result more easily and efficiently in the debugger. I advocate the second form even though it feels “inelegant” by comparison. The value outweighs the cost, and the result is still very readable. Arguably, it is even more readable than the terse alternative.

I suggest picking a random bug; as you step over the code, look at the log statements along the way. Do they include the variables you inspect in the watch area? Can you deduce the rest? After running through this exercise a few times, you should develop a sense for the “right amount of logging.”

The most important tip I can offer after this is to read the resulting logs regularly. This is an obvious task that most of us don’t undertake. Go through the logs in depth and familiarize yourself with them just as you would with the code. You should develop a feel for what makes up a “healthy log.” When you’re debugging an on-call issue at 2am, this training will pay off. It will also pay off when you find small issues before they turn into major issues. This should be a part of a weekly “health” task that everyone on the team should have scheduled to their calendars (ideally at different times). This task should also include a review process of other observability monitors.

Role of Tests in the Debug – Fix Cycle

The bug fixing cycle is illustrated in Figure 4-1 and includes the following stages:

• Find bug – This book is about this stage in the cycle. This stage is the debugging process itself. The debugging process through which we understand why the bug occurs.

• Create test – We must create a test that reproduces the problem we encounter. A unit test is often ideal for these situations since it’s faster and easier to debug.

• Fix bug – Notice that this should come after the creation of the test. You need to create a test that fails and verify the failure. This provides two important benefits. We now have double verification for the fix of the bug, the unit test, and the code. The second benefit is an automated test that will prevent the bug from recurring.

• Run test – Verifying the fix against the test is usually easier and faster than doing so against the entire application.

• Run code – This is the final goal. We need to verify that the fix works with the actual application code. If it doesn’t, we need to restart the cycle all over again.

A cycle diagram depicts the following steps. 1. Find bug. 2. Create test. 3. Fix bug. 4. Run test. 5. Run app.

In the interest of brevity, I omitted some stages from this cycle. Specifically, I chose to omit version control and issue tracker processes. These are important aspects, but they’re better served by a book dedicated to software quality.

Using the Debugger to Compose Unit Tests

Typically, unit tests are easy to write. We invoke a method using various parameters and define the success criteria. The devil is in the details though; tuning coverage numbers to higher values is challenging and very difficult. Testing a specific bug or behavior and mocking appropriately are all challenging too.

Luckily, the debugger can help with the task of composing effective tests with good coverage. We can compose good tests without a debugger, but it would mean stopping execution, recompiling the test, and running again. The basis for all of these is “jump to line” which we discussed in Chapter 1. We can move the current execution to a point before the method was executed and rerun the test over and over until we get the desired result. Think of it as a reset button that doesn’t require recompilation and rerunning and lets you create any imaginary state. We can traverse the testable method repeatedly while verifying coverage with “step-over” and tuning variables to suit our needs. Once we understand the set of variable values that produce the desired test constraints, we can code a unit test with those parameters.

A typical test coverage report will highlight a line or a condition that isn’t reached by a test. In Figure 4-2, we can see a highlighted if statement whose body is always skipped in unit tests, thus resulting in a test without full coverage. By using the set value capability of the debugger, we can verify that the code is reached. In this case, you will notice that the next line in the body is a rejectValue call which will throw an exception. I don’t want an exception thrown as I still want to test all the permutations of the method. I can drag the execution pointer (arrow on the left) and place it back at the start of the method.

A screenshot of a code highlights if the function of string utils has length, and name. The dialog box at the bottom titled actuator highlights the line, name = string at 15953, Bla Bla.

This example is relatively trivial, but when we’re faced with complex conditions within a method, we need all the tools we can get. We can detect the right objects that need mocking for an invocation by stepping over and inspecting. Using a debugger to review the implementation would make things easier.

Writing Good Tests

There’s a school of thought among TDD (test-driven developers) that somehow debugging is a bug. If you write the right tests, you will never need debugging. Typically, in a project with high test coverage, I find that the test code is at least as big as the project source code. When I double the source code, I introduce more bugs. Most of the bugs I have in such projects originate in the tests. Most of the time I spend debugging in these projects is spent debugging the test cases.

Tests must be written so they can be run in isolation using standard test harnesses. This lets us run them in the IDE and debug them under all conditions. The main problem is that even when we follow those guidelines, we still run into flakiness in tests. Good testing practices typically correlate with easy debugging and easier tracking of flaky tests.

Remove external dependencies from unit tests. If you rely on a web service, it must not be a part of the test. It will make debugging harder and can make the test flaky. Instead, mock the service. Arguably, this should be done for some integration tests as well (not all of them).

Clean up the environment completely when running tests. Some developers disable that intentionally so they can review test failures after the fact. This triggers hard-to-catch failures and perpetuates problems. Most test environments support executing in a clean container environment, which I would recommend.

Increase logging for flaky tests. This is an obvious tip. Still, it bears repeating. Flakiness often goes away when adding logging, which is typically a sign of a race condition. This can be verified by adding a short sleep period to the start of the test to check if this relates to the environment setup (or in some cases, a teardown of the previous test environment). These are remarkably hard to reproduce since they relate to edge case behavior of the underlying APIs. With logging, we can narrow the failure or state corruption area that can help us point at a specific dependency that might not be ready.

Running the test in a loop, possibly with random test ordering, can help reproduce such flaky tests repeatedly. This is helpful when we’re trying to narrow down flakiness and isolate the problem.

I highly recommend running the test in isolation with a tool such as strace (which we will discuss in Chapter 5). strace provides insights into the true environmental requirements of the test. We can review them to understand what’s happening under the hood and adapt our thought process accordingly.

Fail-Fast vs. Fail-Safe

In case you aren’t familiar with the terms, fail-fast means a system that would quickly fail in an unexpected condition. A fail-safe system will try to recover and proceed even with bad input. Historically, Java tries to take the fail-fast approach, whereas JavaScript leans a bit more toward the fail-safe approach. Note that this is a broad rule of thumb, and both approaches are taken within the respective ecosystems.

A good example of fail-fast behavior would be the respective approaches to null handling. In Java, a null produces a NullPointerException which fails instantly and clearly. JavaScript uses "undefined" which can propagate through the system.

Which One Should We Pick?

This is hard to tell. There’s very little research, and I can’t think of a way to apply the scientific method objectively to measure this sort of methodology. It has both technical aspects and relates to core business decisions. Fail-fast is typically easier to debug; a failure is immediate and usually obvious. A fail-safe implementation will be more robust and will survive in a hostile production environment (like my kitchen).

Companies using microservices are committed to some form of fail-safe. Resiliency is a common trait of microservices, and that is the definition of fail-safe. Modern approaches to fail-safe try to avoid some pitfalls of the approach by using thresholds to limit failure. A good example of this is a circuit breaker, both the physical one and software based. A circuit breaker disconnects functionality that fails so it doesn’t produce a cascading failure.

The fail-fast approach handles such bugs better since there’s a lower risk of cascading effect. A fail-safe environment can try to recover from an error and postpone it. As a result, the developer will see an error at a much later stage and might miss the root cause of the error.

My bias is toward fail-fast; I believe it makes systems more stable when we reach production. But this is anecdotal and very hard to prove empirically. I think a fail-fast system requires some appetite for risk, both from engineering and from the executives – possibly more so from the executives.

Notice that despite that opinion, I said that the Thermomix was smart to pick fail-safe. Thermomix is hardware running in an unknown and volatile environment. This means a fix in production would be nearly impossible and very expensive to deploy. Systems like that must survive the worst scenarios. Imagine a failure in mission-critical systems of an airplane or a landing module. We need to learn from previous failure. Successful companies use both approaches, so it’s very hard to pick the best approach.

Hybrid Environment in the Cloud

The core assumption behind this direction is that we can control our local environment and test it well. Businesses can’t rely on a random service in the cloud. They can build fault-tolerant systems by avoiding external risks but taking the calculated risks of a fail-fast system.

For any such strategy to work, we need to clearly outline these decisions on all levels. These decisions need to be present in coding processes, code reviews, and project management decisions. They should be enforced in tooling and tests. But first, we need to redefine failure.

Defining Failure

The discussion of failure is often very binary. Code fails or succeeds; there’s a crash or “it works.” This is a very problematic state of mind. A crash or downtime is bad. But by no means is it the only type of failure or even the worst type of failure. A crash usually marks a problem we can fix and even work around by spinning up a new server instance automatically. This is a failure we can handle relatively elegantly. The whole industry of orchestration was designed to handle just this type of failure.

Data corruption is a far more sinister failure. A bug can result in bad data, making its way into the database and potentially causing long-term problems. Such a corruption can have widespread impact on security and reliability due to corrupted data. These types of problems are much harder to fix as it requires reworking the data which might be unrecoverable. A fail-fast system can sometimes nip such issues in the bud.

With cloud computing, we’re seeing a rise in defensive programming such as circuit breakers, retries, etc. This is unavoidable, as the assumption behind these solutions is that everything in the cloud can fail. An important part of a good QA process is long-running tests that take hours to run and stress the system. When reviewing the logs of these tests, we can sometimes notice issues that didn’t fail but conflict with our assumptions about the system. This can help find the insidious bugs that went through.

Don’t Fix the Bug

Not right away. Well, unless it’s in production, obviously. We should understand bugs before we fix them. Why didn’t the testing process find it? Is it a cascading effect or is it missing test coverage? How did we miss that?

When developers resolve a bug, they should be able to answer that question on the issue tracker. Then comes the hard problem: finding the root cause of the failure and fixing the process so such issues won’t happen again. This is obviously an extreme approach to take on every bug, so we need to apply some discretion when we pick the bugs to focus on. But this must always apply to a bug in production. We must investigate bugs in production thoroughly since failure in the cloud can be very problematic for the business, especially when experiencing exponential growth.

Debugging Failure

Now that we have a general sense of the subject, let’s get into the more practical aspects of a book focused on debugging. There’s no special innovation here. Debugging a fail-fast system is pretty darn easy. But there are some gotchas, tips, and tricks we can use to promote fail-fast. There are other strategies we can use to debug a fail-safe system.

The core idea is to fail quickly. Let’s say we need to invoke a web service; a networking issue can trigger a failure. A fail-fast system will expect a failure and present an error to the user.

For fail-safe, the core idea isn’t so much to avoid failure as it’s unavoidable. The core idea is to soften the blow of a failure. For example, if we take the web service example again, a fail-safe environment could cache responses from the service and would try to show an older response. We can build a test case to validate that behavior.

The problem here is that users might get out-of-date information, and this might cause a cascading effect. It might mean it will take us longer to find the problem and fix it, since the system might seem in order. The obvious tip here is to log and alert on every failure and mitigation so we can address them. But there’s another hybrid approach that isn’t as common but might be interesting to some.

Hybrid Fail-Safe

A hybrid fail-safe environment starts as a fail-fast environment. This is also true for the testing environment and staging. The core innovation is wrappers that enclose individual components and provide a fail-safe layer. When a system is nearing production, we need to review the fault points within the system, focusing on external dependencies but also on internal components.

A fail-safe wrapper can retry the operation and can implement recovery policies. There’s some ready-made software tools that let us define fail-safe strategy after the fact, for example, Failsafe, Spring Retry, and many other such tools. Some of these tools are at the SaaS API levels and can mitigate availability or networking issues.

This has the downside of adding a production component that’s mostly missing in development and QA. But it includes many of the advantages of fail-fast and keeps the code relatively clean.

This approach has some drawbacks when it comes to debugging; fail-safe behavior is encapsulated by libraries and can be hidden. When something fails and then retries, we can lose some context information. This is further compounded by the differences between production and development environments. While this isn’t an ideal approach, I would pick it for a case where I need fail-safe support.

Their Usage Is Awkward

This is probably the main reason such debuggers didn’t pick up as much traction as they could. Our typical debugging process is very iterative. We make changes, step over, inspect, and repeat. TTDs expect you to use them for every execution, then review the results after the fact. This is similar to the process of inspecting logs.

For most of us, this requires breaking deeply seeded development habits. The benefit is pretty big. We can run our tests with the back-in-time debugger and then review a flaky execution after the fact. A QA engineer can run with the debugger turned on and let us debug a specific failure after the fact. We no longer need to reproduce the issue on our machine.

But these use cases require a different way of looking at the code. We’re also faced with the risk of impacting execution due to the usage of the debugger. Running with it “all the time” is a different kind of risk.

No One Ruled Them All

As I mentioned before, there are many time travel debugging tools. This is a problem. Borland and Microsoft popularized the modern-day IDE by creating the market-leading IDEs of their time. No equivalent market leader rose in this field.

Microsoft did publish their own TTD, but it’s a low-level tool designed by system developers. In that sense, its scope is relatively limited; it’s still very impressive as you can see in Figure 5-1. I do believe that as mainstream vendors promote these capabilities more aggressively, we’ll see a major shift in the market. I don’t think such debuggers will become the norm. But I do believe they are an amazing tool we should all wield on occasion.

A screenshot of a screen displaying the scripts for command and plug and play on the left pane. The right pane depicts the expectations, breakpoints, and function calls based on the timelines selected. The parameters and values are listed for the selected time slot.

The Different Types

The final problem is that we don’t know what TTDs are. There are so many types of debuggers and many ways to use them. High-level languages such as JavaScript express very different problems and solutions within a TTD when compared to C or Rust. As a result, the surrounding tooling can be vastly different to the processes we’re used to. Or it can be surprisingly familiar.

Some debuggers present the regular IDE debug interface combined with tools to go back and forth through “time” and threads. Others can express the data in charts and let you follow execution with sequence diagrams. The variety is significant.

Frontend Debugging with Replay

In Figure 5-2, we see the Replay debugger view. This presents some of the very cool aspects of TTD, especially when dealing with UI bugs. At the bottom of the screen, you can see a timeline that you can drag to position execution at a specific point in time. On the top right, you can see the web UI at that time. This means we can scroll through the execution timeline to see everything that’s going on – events, network requests, exception, etc. We can correlate all the operations with a full omnipotent view of the code and the application state.

A screenshot of a window titled, your first replay. A text field reads, what would you like to do? From a list of options given below, the open sources option is selected. Three tabs labeled, console, elements, and network are placed. The network tab is selected. It lists the data in 5 columns labeled, state, name, method, type, and domain.

This illustrates a few of the benefits and problems of these tools. If a debugging session grows too long, the user experience quickly deteriorates. Finding the needle in the haystack becomes a far more challenging process. The benefit is obvious; we can never miss a thing.

It might seem that we’re missing a staple of debugging: the venerable print statement. But this is not the case, as you can see in Figure 5-3. When we hover over a line in the source code, the tool indicates the number of times this line was reached in the execution (very useful information). We can then use a standard print statement that references variables, etc. This will get printed as we go back and forth in the timeline.

A screenshot of a window. The left pane displays the code from line 10 to line 34. In line 12, the timeline appears. Three tabs labeled, console, elements, and network are given. The console tab is selected. Errors and logs checkboxes are selected. The filter output option is viewed on the right.

Notice those arrows next to the print entry. When a line was reached more than once, you can use them to toggle between the occurrences of this line. It will update the timeline and all the variables accordingly.

Low-Level Debugging with rr

The rr project was started by the Mozilla Foundation to support debugging tools like Firefox. It’s a remarkably powerful tool for narrowing down hard-to-reproduce bugs. But it’s a very low-level, spartan tool. We can install it on Linux by downloading a release from the git website² or building from source.

There are a few other tricks available there, but my main point in this section is to give you a sense of rr. It’s a relatively simple tool for a simple use case. You run into a bug in Firefox and don’t know how to fix it? Just run it with rr and send the result as part of an issue attachment. The engineer at Mozilla would be able to debug it as if the debugger was running on your machine. This is spectacular, albeit somewhat spartan. There are some solutions for this such as Pernosco which lets you upload the recording directory and view the information using a convenient UI.

That’s a problem for most cloud environments where we don’t have access to the virtualization controls. This significantly limits the utility of this tool to Linux desktop machines. That’s a shame, but it’s understandable since inspection happens at a system level and requires deep access.

History

In the “old days” when we wanted to deploy a complex application in production, we had to put a machine somewhere. Developers would either physically set it up or telnet into it (this predated widespread use of SSH). We’d set everything up and it was all good, until we had to do that again on another site. For that, we built complex installation scripts that would often fail because of dependencies, versions, etc. It also made it very difficult to deal with elaborate settings and manage the resource.

Containers solved both problems. They used some kernel capabilities to isolate the current running code. This meant it wasn’t fully virtualized on the machine, but “elt“ like a separate machine in most ways that matter (to learn more, check out namespaces and cgroups on Linux). A container can run on top of a virtual machine without adding much overhead. The second part was the standardization of a container format by Docker. It gained traction and made it very easy to move container images around.

This let developers package an entire application as an image, but it also raised a problem. Say I have an image of my application and it depends on an image of a database. That works perfectly well, but it isn’t the end of it. We need a caching server and an authentication server, and we want to scale; so, we need monitoring, observability, etc. We might move to a microservice architecture where we have many servers. That’s the beauty of containers; they make image creation trivial. Because of that, we create many images.

Containers enable massive scale. They have very little overhead,¹ but we can further reduce this by creating containers that don’t include the full operating system (distroless, bare). This sounds like nitpicking in the age of massive storage/RAM. But an OS image with everything can have a massive impact when scaled. By reducing this overhead, we can fit more containers on the same hardware and reduce our costs considerably.

There are a few things containers don’t cover though: dynamic, failover, updates, scale, etc. These are solutions that sit above the container runtime. They make sure the service isn’t down or overloaded and can dynamically provision additional resources.

Enter Kubernetes

How do we deal with these containers and the dynamic nature of deployment? The solution is orchestration; Kubernetes (a.k.a. k8s²) is an orchestration solution. Container orchestration essentially moves you from “ost-centric“ to “ontainer-centric“ infrastructure. It lets you define all the different parts of your deployment, including where you want it to scale, how you want it to scale, etc. Figure 6-1 shows the winding road to scaling with Kubernetes.

A flow diagram of the road to scaling with Kubernetes. It starts with Hardware, Virtualization, Containers, and Orchestration.

The problem is it isn’t magic. For Kubernetes to work properly, everything needs to be set up correctly. Let’s imagine I logged in to a container and changed its state while we’re running. A good example would be adding a configuration option. If this fails and Kubernetes needs to replace the container (or scale it up), we might run into serious problems. As you might surmise, this makes debugging Kubernetes harder as we discuss as follows.

Check out Figure 6-2 to understand a small part of the responsibilities and positioning of Kubernetes. Kubernetes is massive! Developers only focus on the small area marked as “your code“ within Figure 6-2. Consider that this is a deeply simplified view of the overall structure of a Kubernetes deployment.

A diagram of the wheel pods connects 2 nodes of the kubelet with 2 pods. On the right, it depicts 3 terms and the text reads Your Code.

We can use commands like kubectl describe pod to get information about the pod and general details. Sometimes, this would be enough to understand what’s going on, but only if we’ve got a configuration problem.

An important command when working with Kubernetes locally is kubectl logs. It lets us list logs for a specific container and even for a crashed container with --previous. But why is it only important locally?

Typically, in production your logs would be ingested and piped to remote storage. In that case, the fact that you’re using Kubernetes becomes irrelevant. You should be able to see the logs for all containers in the cloud. There are special cases, but for the most part container logs aren’t as interesting in day-to-day production debugging. This assumes that you ingest logs (which you should) and that you are a developer. There is the case of the system logs which is important, but this book is aimed at developers who would typically be the second line of defense for these types of error messages. If a container crashes because of configuration or a system-level failure, then it requires a different skill set to track. I will cover some of those common things in this chapter, but very briefly.

It’s Not Just About Scale

I want to stop for a second and revisit the core concept of “why we need Kubernetes.” As one website⁴ put it quite correctly, we probably don’t. Most of us don’t need the type of scale Kubernetes offers, and the overhead is often not worth the extra cost. Many problems can be solved with a single VPS (Virtual Private Server – typically built on top of a virtualization solution); Kubernetes makes simple solutions much harder.

This is true. There’s no question that many Kubernetes adopters are suffering from RDD (Resume-Driven Development) and are picking it up to bolster their CVs – not due to business needs. But there are some advantages that go beyond scaling that make Kubernetes an attractive deployment option for many cases, even where scale isn’t a problem.

The pod vs. container abstraction is powerful. It lets us focus on building our application in a container while ignoring the external elements. In a case where we need to provision a container with access to a caching API, we can ignore the caching requirement in the code and spin up a container. We need a remote database server, ignore that requirement too, and spin a pod containing the database server. The database server needs to scale too; that’s a separate problem, and we can easily deal with it later without changing a single line of code. This is powerful stuff. It lets us treat applications and their surrounding requirements as if they were Lego bricks with which we can assemble a custom architecture to fit our deployment.

This means we can punt on multiple complexities and choose simplified, prepackaged solutions for all these things. Then update or replace them relatively easily because every piece is modular. Observability servers can be placed in the same pod as a separate container. This means they’ll be physically near our application but will still have a strong separation. We don’t need to concern ourselves with the needs of observability during the development process.

This value proposition is compounded by the massive third-party ecosystem that sprung around Kubernetes and the Cloud Native Computing Foundation (CNCF).⁵ This ecosystem lets us integrate solutions and monitor and leverage infrastructure like never before.

In our industry, being “cloud neutral” is somewhat of an “inside joke.” Vendors keep promising it but deliver only partial solutions. Companies that start completely neutral slowly bind themselves to a single vendor because costs such as egress are too steep. Kubernetes isn’t vendor neutral, especially if you use vendor support for it. But it’s a piece of the puzzle that lets you encapsulate individual problems. Those can be fixed individually and addressed moving forward.

Ephemeral Containers

The solution to these problems is ephemeral⁶ containers. Kubernetes has a higher abstraction on top of containers called pods. Pods group together related containers, and they have unique access to one another. Ephemeral containers are added to the same pod as the container you’re trying to debug, but they are temporary and separate. What you do in them doesn’t impact the overall application. You don’t need to restart anything, and you can still get special access to the container.

we’re not connecting to our container. We’re creating a new container from the given ubuntu base image. This is an ephemeral container, and it will disappear when the pod is destroyed. In it, we can check things about the actual container of interest. We can use an arbitrary image for that container. This means that if the original container crashes, our ephemeral container will still work and can still inspect it. If it has no shell, we can still log in to our debug container where we have an entire OS. We can install arbitrary tools on the fly since they won’t impact the debugged container, and we can use them to debug. It’s very cool!

After running that, I’ll have access to a dizzying list of JVM-related tools and many other general-purpose tools right on the command line. I reviewed some of these tools when talking about external debugging tools in Chapter 3. I would recommend checking out KoolKits and submitting a PR⁸ if your favorite tool is missing.

Using jdb

Let’s go over what we did here. The jdb command launches the command-line debugger and attaches to the specific process ID. I’m then faced with a prompt, the > character. At the prompt, I issue the stop command and give it a class name followed by a line of code. This is equivalent to setting a line breakpoint.

The breakpoint is hit, at which point I can issue a step command. This isn’t hard, but it’s very tedious when compared to the experience in the IDE.

Remote Debugging

It might seem like we took too long to reach the obvious “unchline,” but we didn’t. I needed to go through all of that to explain some of the core problems in remote debugging. Remote debugging works very similarly to our jdb process from the previous section.

Basics

Remote debugging is a feature of pretty much any programming language or platform, but it’s common in Java using the JDWP (Java Debug Wire Protocol), which I’ll discuss soon. It’s a wire-agnostic protocol that defines the connection between the IDE and the JVM when debugging.

Notice that the real value proposition of remote debugging is on local and not on a remote machine. The value is in connecting to a running process that’s outside of our development environment. It’s great for connecting to a container running on our machine. I’ll elaborate on this later in the chapter.

The java command we used to launch the preceding application will work almost perfectly for remote debugging if you’re running on a local machine without Kubernetes. For Kubernetes, it will fail. Kubernetes masks the networking ports by default. This lets us run servers that listen to network connections on the same port within the same machine, and they won’t collide with one another. In order to broadcast the port to external connectors, we need to expose that port; this is done via the manifest.

Don’t touch that manifest! We don’t expose JDWP to the external world; this is a huge security vulnerability (more on that later). The trick we used to do in the days before Kubernetes is SSH tunneling using a command like this on your local machine:

ssh remoteUser@remoteHost -L 5005:127.0.0.1:5005 -N

This command links the port on the remote machine to the local machine. That means that port 5005 on localhost will map to port 5005 on the remote host. I would be able to debug the remote machine as if I’m working on a process running locally, and the server won’t expose any port. All traffic will be encrypted over SSH!

This won’t work on Kubernetes, though. For that, we will need to port forward using a slightly different command with the same impact:

kubectl port-forward podname 5005:5005

Once we have that, we can connect our IDE to the remote server as if the code was running on our local machine. In IntelliJ/IDEA, all I need to do is open the Run/Debug configuration and add a “Remote JVM Debug” entry. The defaults should work as expected since it will point to the localhost on port 5005 as you can see in Figure 6-3.

A screenshot with a text read Debug remote with configuration and Logs. On the left, the Debug remote from Remote J V M Debug option is selected.

VS Code uses a JSON-based configuration, which is slightly more manual, but the concept is the same. We need to add a remote debugging configuration and provide the port as follows:

{

      "type": "java",

      "name": "Debug (Attach)",

      "request": "launch",

      "mainClass": "MyClass",

      "projectName": "MyProjectName",

      "request": "attach",

      "hostName": "localhost",

      "port": 5005

}

Once set, we can run this configuration and start debugging as if the application is completely local to the IDE. Everything should work as if it’s local. Breakpoints, tracepoints, exception filters, and most of the stuff we covered in Chapter 1 should work seamlessly. That sounds too good to be true, right? It is…

CrashLoopBackOff

This is a general-purpose issue with several causes. You will need to get further details by using kubectl describe pod pod-name and inspecting the output. A common failure is insufficient resources, which might mean you need to scale up the cluster to fit additional nodes. Volume mounting can fail if the storage is defined incorrectly for the pod.

Image Error

Errors that mention the word image such as ErrImagePull or ImagePullBackOff typically mean a container fetch failed. One container in the configuration file can’t be fetched. Check that docker pull works with the image name to make sure we typed it correctly and that there aren’t any “nvalid characters“ somewhere. Some failures in docker pull (and here) can occur because of problematic credentials, secrets, or authorization.

Node Not Ready

This often means there was a physical issue with a node host. You can see this issue by invoking the kubectl get nodes command and inspecting the nodes. A reboot of the problematic machine or VM might solve the issue in some cases, and Kubernetes should “elf-heal” within five minutes.

CreateContainerConfigError

We get this error due to missing configuration keys or secrets. Kubernetes supports a config map which lets us define values such as a database host. A secret is an API key or similar value which we don’t want to place in version control or expose. It’s important to separate that information outside the YAML or IaC files so we can change it later and for security purposes (in the case of secrets).

The problem is that because of that separation we can end up with missing information since the actual configuration process requires a config map. You should use kubectl describe pod pod-name to get further information about the problem.

Debugging the Dockerfile

A relatively new addition to the configuration troubleshooting process is buildg.¹¹ Buildg is an interactive debugger for docker configuration that lets us run a configuration file while stepping over the stages. We can set breakpoints like we can with any debugger. It integrates nicely with VS Code and other IDEs through the Debug Adapter Protocol (DAP).

While stepping over, we can inspect the state of the container to see the impact of the previous line. As of this writing, this is a very new tool.

Idempotency

One of the ways serverless providers have evolved over PaaS is through the adoption of functional programming concepts. An important one is idempotency, which is a concept rooted in mathematics, meaning a function will have the same results if we pass in the same values.

This seems obvious. If we have a function called add to which we pass the values 2 and 3, the result should always be 5. At first, when you hear about this concept, it seems that this is the way our code works. But it isn’t always the case. If your function relies on an external web service, the result might change; at that point, it will no longer be idempotent. A good example would be if the add function would handle currencies and then would return a different currency (e.g., adds Euro and returns USD). In that case, the result would fluctuate with currency conversion changes. But there’s a simple solution: accept a timestamp. By requiring a timestamp for the currency value, we can force consistency in the results.

This is functional programming 101, but it’s also good programming practice in general. It makes debugging and testing serverless functions easier. We can rely on consistent, verifiable results. A log that will include the input should (in theory) match the output. Then once we have the logs, we should be able to reproduce any problem.

Staging and Dev

A common best practice in development is using a separate environment; this is useful for testing. We would usually have a dev environment into which nightly builds are uploaded and staging, which is a step before production. These environments let us test that our application is working correctly in a pseudo-production environment without the risk.

These environments are one of the chief advantages of using Kubernetes. Setting up mirrors of the production environment becomes trivial, especially when using IaC (Infrastructure as Code) such as Terraform. With serverless deployments, this isn’t as trivial, and because of the nuances involved, it’s very hard to get these environments in sync.

The main challenge is maintaining consistent environments. A typical deployment is complex; it can include multiple dependencies and endpoints. IaC can help with this process as well. Unfortunately, this raises the complexity level which reduces some of the benefits in adopting a serverless architecture.

Obfuscation

Code obfuscation is very common in frontend code and for good reason. We’re publishing our source code. That code might contain proprietary information; obfuscation will make it harder to read. The second benefit is reducing code size, which will improve performance by reducing download size, RAM overhead, etc.

Obfuscation is great, but it makes debugging very difficult. That’s why we have the {} button in the bottom toolbar area in both debuggers. When you click it, a single line obfuscated file will become a “well-formed” debuggable file. Both Figures 8-3 and 8-4 were initially obfuscated. This is an important capability as during debugging we might want to step into obfuscated framework code.

For this to work in Chrome, you need to make sure “Enable JavaScript source maps” is enabled in the developer tools settings as shown in Figure 8-5.

A screenshot depicts a list of settings on the left and the preferences on appearance and sources on the right. It includes the theme, panel layout, color format, and language.

Launch Debugger

Instead of printing the stack trace, one of the neat tricks in JavaScript is instant debugging. Just use the debugger keyword to launch the debugger on that line. If you have an error handling method, it would be an ideal place for that call.

This is probably not something we’d like to leave in a production environment. But it can be very useful in a debug version of your site, assuming you have a preprocessor that can remove that call for production.

The debug(hello) statement adds a breakpoint into the hello function. When we invoke the function in the last line, we’re sent directly into the debugger.

DOM Breakpoints

Think of them as the field watchpoints of the browser. We can place a breakpoint that will be hit when a specific area of the DOM is mutated. This only works for Chrome and Firebug (which is a Firefox plugin), but this is easy to use. Just right-click the DOM and select “Break On” followed by the behavior that you’re interested in.

Figure 8-6 shows the DOM breakpoint context menu on the Chrome web browser. Notice that this is in addition to conditional breakpoint and line breakpoint capabilities. Both capabilities are supported by all major browser development tools.

A screenshot highlights subtree modifications in the break-on option. The background includes the program and console dialog box.

XHR Breakpoints

Browsers have some amazing HTTP traffic monitoring capabilities that let us inspect all the outgoing requests and responses in detail. These are fantastic capabilities that I use often. They show every request as a set of bars indicating the time, size, and all the technical information (body, headers, etc.).

They’re missing one crucial piece: the culprit. Who made the specific request? This is often trivial to find; we can search for the URI. But in some cases, the URI might be assembled dynamically or might arrive from multiple sources, and the flow is unclear. For these cases, we have the XHR breakpoint which we can set in Chrome or Firebug by defining a substring of the URI that we wish to break on. When a request arrives to fetch that URI, Chrome will hit that breakpoint and stop the execution of the script.

Figure 8-7 shows the XHR breakpoint area in Chrome. It’s located in the sources tab under the call stack. Notice that if we don’t fill in the URL filter, an XHR breakpoint is set for any XHR request which might get noisy.

A screenshot of the X H R or fetch breakpoints menu. The option any X H R or fetch is selected.

Faking Browser Settings

When debugging hard-to-reproduce issues, a common technique is to spoof or simulate the problematic environment. These are some techniques we can use to simulate such an environment. Since debugging on a mobile device is a bit more challenging, it’s often easier to simulate some behaviors that are more common there in our desktop browser.

Simulating a different user agent can help us isolate browser- or OS-specific issues that might be triggered because of server-side platform-specific content. This can be customized in Chrome in the settings section. For Firefox, you will need to install an extension to accomplish the same.¹

You can use a database to find the user agent you wish to spoof, but I find it simpler to look at the user agent submitted from the browser where the code is working. You can inspect headers received in a failed server request and then try using that same user agent in another browser.

Touch gestures might produce different behaviors when compared to typical events. You can select “Emulate Touch Events” in the override options. In Firefox, if you’re in the responsive design view, you could emulate a touch event.

Frontend Logging

We discussed logging in Chapter 4, but frontend logging is different in many regards. The first big difference is the quality of debugging via printing. As you may recall, I’m a big advocate of using tracepoints (see Chapter 1) instead of logging. Tracepoints (a.k.a. logpoints) are still superior to print statements in most regards, but in frontend programming, the difference isn’t as pronounced.

In a frontend flow, we constantly reload, refresh, and switch browsers, and as a result, the development is very fluid. A tracepoint can become a hassle as we test extensively. Features such as ingestion, logging to a file, etc., aren’t useful in the frontend, and thus we don’t have the same level of need for a logger. It’s still a “best practice” to use a logger rather than invoking console.log directly, but it isn’t as much of a problem as print could cause on the backend.

One of the big advantages console has over simple print statements is its support for log levels. There are five supported levels: log, debug, info, warn, and error. Log and debug are indistinguishable from one another, and info is sometimes rendered in the same way. Figure 8-8 demonstrates the output of each level. The browser lets us filter the output so we can focus on a specific logging level similarly to typical server-side logs.

A screenshot indicates the console log, console info, console warning, and console error.

We can create completely custom versions of logging with CSS thanks to the way JavaScript works. We can add an additional logger method into the console and use CSS to customize the output. For example, the following code prints out “Dazzle” with a pink background as seen inFigure 8-9. Notice we’re using %c which indicates substitution with CSS. After the comma, we see the CSS that’s applied. Notice we can use more than one substitution block to mix CSS styles; this can let an API print out text with multiple distinct styles.

console.customLog = function(msg) {

    console.log("%c" + msg,"color:black;background:pink;font-family:system-ui;font-size:4rem;-webkit-text-stroke: 1px black;font-weight:bold")

}

console.customLog("Dazzle")

A screenshot of the console code and the debugger evaluation code indicate the output, Dazzle.

The console.trace method can print the stack trace to the current location. This isn’t as useful due to the asynchronous nature of JavaScript code. But in some circumstances, that might prove useful.

Besides logging, we can use assertions in the code to implement “design by contract.” Assertions let us determine an assumption we make while coding. Assert is used to verify a statement that must be true. If it isn’t, our application will fail with a clear error. In the browser, this isn’t as dramatic and is closer to the effect of console.error as seen in Figure 8-10.

A screenshot includes a console assert function, debugger evaluation code, and an error message, assertion failed.

One of the most powerful features of asserts is the ability to strip them out in a production build. We can remove the performance overhead of testing everything while promoting a “fail-fast” (see Chapter 4) mindset. Some minifiers support the ability to remove asserts in production.²

We can apply deep formatting to printed information; one common example is printing a table of data. For example, code like console.table(["Simple Array", "With a few elements", "in line"]) can print a table with the array of items. We can create more sophisticated examples by printing an object as a table; the elements of the object become the columns and rows of the table. We can see samples of both approaches in Figure 8-11.

A screenshot includes the console table function, debugger evaluation codes, and 2 tables with indexes and values. The values in the second table are Shai, Almog, and Debugging.

Printing the content of an object to the console is possible in various ways. We can use the copy(object) method to copy the content of an object into the clipboard so we can paste it anywhere. The console.dir(object) API provides a far more powerful option of inspecting an object as we do in a debugger. This is an amazing way to explore an API and the state of the environment as we see in Figure 8-12. Similarly, we can use dirxml(object) to view an object as xml or use the inspect(object) method to open the inspect panel with the given element or object.

A screenshot of the code for the console directory on a window object includes a website named code name one. It uses crisp and C L I function.

Counters are an important debugging tool. We often want to know the number of times a method was invoked or a line of code was reached. This is such a common use case; that it’s coded into the console API. With console.count(), we can increment an arbitrary number every time that line is reached; the current count is printed to the console. We can also use console.countReset() to reset the counter back to zero. This lets us verify that an API isn’t reached too often and validate assumptions. Figure 8-13 demonstrates the simple usage of these APIs.

A code indicates the console count and console count reset functions for the word hello.

Grouping lets us collect related printouts together in a collapsible group to keep the output organized. We can place all the logs related to a specific method in a group to reduce verbosity. To start a group, use console.group("GroupName"); this opens a collapsible region with the given name. Every following printout will be a part of that group until you invoke console.groupEnd(), as seen in Figure 8-14.

A code for the collapsible code starts with the console group function and ends with the console group end. The collapsible region titled my group consists of the console log functions.

Monitor is a Chrome-specific feature that lets us monitor calls to a specific function. This is like adding a tracepoint (logpoint) on a method (function) breakpoint. We can invoke the function monitor(functionName) on any function. After that point, every invocation of the function will print its name to the log and the arguments it received. Invoking unmonitor(functionName) will remove the monitor and return us to the previous state. See Figure 8-15 for a simple example.

A code for function hello includes the following lines, hello Shai, monitor hello, hello readers, unmonitored hello, and hello again.

Similarly, Chrome supports the ability to monitor events using the monitorEvents(object [, events]) and unmonitorEvents(object[, events]) APIs. All events are logged to the console once we start monitoring events until we choose to stop with the unmonitorEvents call. The events argument is optional; it lets us reduce the volume of events we receive. I can monitor all window events using monitorEvents(window) or just specific mouseout events using monitorEvents(window, "mouseout") as you can see in Figure 8-16.

A code of monitor events for the window mouse out includes 11 mouse events and 4 resize events.

Chrome can list all the objects created with a specific constructor or of a specific object type by using the queryObjects function. When we invoke that function with any object type, such as Promise, as we see in Figure 8-17, we can see the objects in the VM listed. This can work with a constructor to get only the instances created with the specific constructor.

A code of query objects functions for the object type, promise. The array includes 3 promise prototypes numbered 0, 1, and 2. The promised state is mentioned as fulfilled. The length is indicated as 3.

Networking

In Chapter 3, we discussed Wireshark, which is an amazing tool for observing network traffic. The tools within the browser exceed it for most common use cases and make debugging network performance and bugs stupid simple. Many problems express themselves in the frontend, either via the console or the network monitor. I’ll skip the discussion of performance, since that is a subject so big it warrants its own book. In this case, I want to highlight tricks for debugging using these network consoles. The Firefox network monitor seen in Figure 8-18 is very similar conceptually to the Chrome network monitor in Figure 8-19. Both include nearly identical functionality and capabilities. Due to that, I won’t demonstrate most functionality in both browsers.

A screenshot of the network monitor highlights the tab labeled all. The table has 9 columns. The column headers are labeled status, method, domain, file, initiator, type, transferred, size, and 0 milliseconds.

A screenshot of the network monitor depicts a table with 7 columns. The column headers are name, status, type, initiator, size, time, and waterfall. The bottom panel includes 115 requests, 112 kilobytes transferred, and 8.2 megabytes of resources.

One of the biggest capabilities is the ability to re-issue a request. This is far more convenient than pulling out curl or postman since it allows us to modify an existing network operation and change some arguments. Firefox lets us right-click over any entry in the network monitor and select “Resend” to launch an edit window. We can now issue a custom request and change the parameters we pass to the request as shown in Figure 8-20. Both Firefox and Chrome support copying the request as a cURL command. You can find that capability under the copy menu in the right-click. This lets us reproduce an issue from the command line without going through the browser. You can also copy request headers and paste them into postman’s headers section if you prefer working with it.

A screenshot highlights the new request tab in the top panel. It includes 4 U R L parameters and 8 headers. The checkboxes in the U R L parameters are selected.

One of the most useful features in the networking toolchain is the throttling feature seen in Figure 8-21. With it, we can slow down the request speed to various predetermined speeds. This is wonderful for debugging performance-related issues, but it’s also an important tool in debugging race conditions and problems that only occur in slower connections.

A screenshot of a network monitor includes a list of no throttling options. G P R S, regular 2 G, good 2 G, regular 3 G, good 3 G, regular 4 G or L T E, D S L, and Wi Fi.

Storage

Both Chrome and Firefox include great support for managing storage in their respective developer tools. Firefox has a dedicated storage tab (see Figure 8-22), whereas Chrome includes everything in its application tab (see Figure 8-23).

A screenshot highlights the storage tab on the tap panel and the website codename-one under cookies in the left pane. The column headers of the table in the right pane are name, value, domain, path, empires or maximum age, size, and h t t p only.

A screenshot highlights the application tab in the top panel. The left pane indicates the application, storage, and cache. The right pane includes a key and its value.

The storage area lets us push debug information into storage to test specific edge cases. An end user can provide us with a local state from their browser which we can add locally. We can remove and edit individual elements from all storage types including cache.

DOM and CSS

Layout and style bugs typically stand out on their own and are typically separate from the fullstack issues. However, since we’re discussing frontends so extensively, it makes sense to cover these issues as well. The inspect element view of the developer tools is possibly the most important feature within them. It’s the default view and the starting point for most frontend work as seen in Figures 8-24 and 8-25. Notice that at first glance Firefox might seem more functional than Chrome. Chrome hides some of the features behind additional tabs, which has some benefits and some tediousness.

A screenshot highlights the inspector tab in the top panel. The code in data elementor device mode is on the left pane. The center pane includes the margin, padding, border, font, and box sizing. The right pane highlights the layout tab and contains the box model properties and a preview.

A screenshot highlights the elements tab. The h t m l code is on the left pane. The right pane includes the crisp client function with z index, font family, font size, position, display, and other properties. The styles tab is highlighted.

When we inspect an element, we can change everything about it and its CSS. The specific element is highlighted in the web view, and changes are applied instantly. This is an amazing tool for debugging CSS- and HTML-related bugs.

Every value within the inspector can be edited. We can also add new attributes and instantly examine their impact. This lets us experiment rapidly while tuning the actual code based on the results. One of the biggest problems developers run into is the specificity issue. The specificity issue is the situation in which changes we make to the CSS aren’t applied. This can happen if a more specific selector is overriding the attribute we’re trying to set.

We can detect this when inspecting an element’s CSS. We would see our setting in strikeout mode and could scroll up to see the attribute that overrode our setting (see Figure 8-26). Once we understand the specific cause, we can edit our CSS rules to address that problem.

A screenshot includes the following attributes in the element function. Border, clip, height, margin, overflow, padding, position, width, white space, and overflow wrap.

Database Logic Layering

Before we delve into the problems and the solutions that we need, let’s discuss general architectural choices that help debugging database issues. I’m a big fan of Object-Relational Mapping tools (ORMs), but I understand they aren’t everyone’s “cup of tea,” which is fine. The important part is the isolation of direct database access within an abstraction layer that lets you inspect the seam between the database and your application. Such a layer should be fully mockable for the purpose of isolating database issues from application issues.

Assuming you use database caching (which you should⁴), its usage should be encapsulated within that layer for the same reason. This lets us use the standard debugger to test most database-related issues and lets us unit-test effectively. The database layer itself can be tested using tools such as datasource-assert,⁵ which mocks the database.

Database Connectivity

The latter ones aren’t challenging. The former seems like an easy fix that doesn’t require debugging. But this is a misleading premise. One of the most common fixes is to increase the number of resources allocated to the database, more connections, larger cluster, etc. While this is something that would be needed as the business grows, it’s often done prematurely and leads to a vicious cycle of cloud spending.

A common source of resource depletion problems occurs because of bad cleanup code. When working with limited database resources, we must always handle failure and always clean up after a failure. This is a tedious work, and it’s hard to detect sometimes. In the debugger for JVM languages, we can enable exception breakpoints on the SQLException and make sure every such exception is handled correctly.

Many resource depletion problems are caused by an errant feature that takes too long to release a database resource. Detecting and fixing these edge cases will solve the problem and increase performance. Detecting them is different for every database type but follows a relatively standard pattern.

The latter can be understood using the SQL explain keyword which generates an execution plan for a query. This can help us understand what’s going on when the query runs. I use IntelliJ/IDEA Ultimate which provides a visual interface to analyze database logs and query performance. There are many other visualization tools you can use, for example, DBeaver, etc.

Making the connection between the SQL that’s executed in the database and a high-level ORM abstraction is challenging, especially since the database runs in a separate process and we can’t follow the logging in the application server to the database log. This can be understood when we’re running locally, but if we’re in a production environment with a large codebase, finding the code responsible for a problematic query can be a needle in a haystack. Most tools like JPA support a verbose mode where they print out the generated SQL. Even when JPA does that, it doesn’t connect the printed SQL to the line of code or even the thread that executes the SQL. This is a common problem for many SQL abstractions, and, unfortunately, there are very few solutions. One solution for JPA is JPlusOne,⁸ which adds additional context. However, it also adds an overhead, so it might not be practical in a production setting. It might be useful for integration tests, though.

Having said all that, slow queries might be a misleading metric. A large volume of fast queries can often be worse for performance than one slow query. Such queries won’t be detected by database logging since individually they would be fast. This is something we can detect through our application logging and debugger. If one operation produces multiple queries, it might be indicative of a problematic operation. Unfortunately, I have no magic bullet for this, only vigilance.

Invalid Data

Bad data in the database is the bane of programmers everywhere. Proper database constraints are often challenging as they lack the same level of sophistication that we often require in the business layer. This can create a situation where code used to import or insert data into the DB might add invalid data. This data won’t work when we try to view or edit it using another flow. Cleaning up the data in production is also a problem. Once the bad data is in the database, it’s much harder to get it out.

How do we detect if a bug we’re experiencing is due to bad data? Conditional breakpoints that verify assumptions can be very helpful here. However, since most such issues are specific to production, it won’t be as valuable there. For this, we must log effectively so we can track such issues. Unfortunately, that’s a problematic strategy since the log has different access restrictions when compared to the database, and we can’t expose all data in the log. We can’t query the log either. There are no simple solutions for this specific problem other than reviewing the data in the DB, logging extensively, and trying to get DB constraints right the first time around.

Consistent Databases

This isn’t a bug, but rather a problematic concept that’s prevalent when developing database applications. Kubernetes should have made deployment-specific issues a thing of the past. Unfortunately, even with Kubernetes, the difference between staging and production is still huge. Serverless deployments have made local environments and staging even harder to work with. In many regards, “modern systems” are harder to debug in comparison to systems defined a decade ago.

There isn’t much we can do about serverless. But the problem is the usage of a different database for development and production, for example, HSQLDB for development and PostgreSQL for production. This made sense before the advent of docker. Thanks to docker we can run the same database in production and development. This is important for practicing debugging techniques locally that would be applicable when we need to address similar issues in production.

Logs

We talked about logs in Chapter 4, but that was mostly from the developer perspective. When looking at logs from an observability perspective, we see a different picture of the same artifact. In this perspective, context, scale, retention, and proper metadata are the story.

It’s no longer just a log that shows how the application behaved at a given time. It’s now a collection of logs from all the instances of the app, connected to ongoing processes or events with associated information. In order to represent this effectively, common loggers support structured logging, which means the output is in JSON or XML format. That way, the observability tools can break down the log into its individual components.

Logs often encode within them a great deal of information. When we look at logs through developer tools, we see the information encoded for our convenience. As we can see in Figure 9-1, the structure of a single line can include many problematic aspects. The time value is very ambiguous in this specific log. Which year? Which timezone? The metadata typically appears mixed rather than segregated; we don’t necessarily need to know the user ID or server URL for every request.

A log statement indicates the date and time, server address, application name and process I D, metadata M D C, log level, and log text and variable values.

This is where structured logs make sense. Logs can use the less readable epoch time format for accuracy, while the user who views the log will see the more readable time format. The same applies for the other metadata. We might want to see logs dealing only with a specific user; we can use the metadata to filter that.

Monitoring systems can take that to a different level. By ingesting the logs, we can use the broken-down data to get insights on niche details without changes to our code. Log ingestion is the process of collecting the logs from multiple isolated processes and processing them for fast search. A monitor can measure error rates per user by reviewing the number of logs marked as ERROR per user ID.

Metrics

Metrics can be custom created to fit the business needs and are a crucial part of our system view. They’re an interesting diagnostic tool, but as a debugging tool, they’re typically a stepping stone. We can identify potential problems via metrics, but usually fixing them requires more.

Distributed Tracing

Tracing is the thread that connects the pieces of a distributed system. It lets us follow the process across the container boundary while maintaining context. When an issue crosses the boundary between processes or containers, it’s hard to follow along. Traces are comprised of nested Spans that together represent the entire request from start to finish.

The root Span contains an entire request within its hierarchy. Tracing tools can visualize the process as a flame graph or as a waterfall diagram that we can expand to follow through. This is similar to reviewing a stack trace for causality, but it’s more elaborate since Spans are distributed and concurrent. For example, a future² can be a Span that would still remain under the parent Span. This provides essential context even for asynchronous operations. We will discuss this in more practical terms as we discuss OpenTelemetry.

Coverage

Resource exhaustion and IO failure issues are probably covered by your DevOps team. If you have a good DevOps team, it’s worth spending time with them, learning that process and aligning your conventions. Our team needs to collect all such potential risk factors. Then, for each risk factor, we need to describe what we would want to know to fix it.

Deciding

Now that we understand the type of problems that we might face in production, we need to design the observability infrastructure that we would like to deal with when running into such an error. The best practice is to have two sources of observability information. Observability can be expensive in terms of performance, storage, and costs. As a result, we need to keep a balanced approach and not log too much or include too many metrics. That’s why I suggest keeping two observable elements. If it’s only one, it can fail. If it’s more than three, the overall costs can become prohibitive.

Developer Dashboard

Many companies have multiple dashboards with Grafana or similar tools covering everything from the key performance indicators (KPIs) to system health. These dashboards are usually tuned for the needs of management, the DevOps team, SRE, etc.

Developers need their own dashboard with the values mentioned earlier. This should be the home page that we see every time we launch a new browser tab. These are the numbers and statistics that should be front and center for every member of the team. We should constantly look for anomalies and changes. Making this into a habit for the entire team changes the way we approach bugs and coding.

Remote Debugging?

Basics

To set up a developer observability solution, we need to install an agent (very much like the OpenTelemetry setup we discussed in Chapter 9). Once that is done, we can use the client to issue commands such as adding a log.

One of the big problems with remote debugging is scale. We want to place a tracepoint on a line of code, but it might never be reached if we have a cluster with multiple instances. A load balancer might send the request to a different machine, and we might make the false assumption that the line was never reached. This is a serious problem.

Developer observability solves this problem by introducing the concept of tags. Agents can be associated with multiple tags; as such, we can add an action to a tag, and all the agents associated with the tag will handle the action appropriately. For example, we can add a log to the “Production” tag which will add it to all the production servers. We can customize the tags and create them per group. A good example is an “Ubuntu-20” tag, which can help us test a problem that only reproduces on a specific OS version for our servers. Figure 10-2 shows two agents and one tag listed. The action added to the tag is implicitly added to both agents.

A screenshot represents a box containing the details of tags and agents. The tag reads production. There are 2 agents containing the tag.

Conditions

Logging can be very “noisy,” for example, if a method is invoked frequently, we might end up with so many logs that our costs will balloon. It might also make the logs less readable. A good example is the case where a specific user is experiencing a problem with the system. These are the hardest bugs to reproduce locally since they usually depend on the production environment, specific production data, latency, and deployment.

We would want to add additional logging to debug this problem, but we don’t want to go over the data for every user or pay the cost of logging for every user out there. The solution when debugging locally is conditional breakpoints. We can add a condition to a breakpoint, and it will hit only when the condition is met. In developer observability, all actions can use conditions, and it’s a far more important feature than it is in local debugging. When debugging locally, we’re the only users of the system; we get a reasonable number of logs. In production, we might have thousands of servers with millions of users. Imagine the amount of output we could add in such a scenario. Even for smaller deployments, the additional data can be overwhelming.

We can set a condition for any action, which means it will work for snapshots and metrics (both of which we’ll cover later in this chapter). A very common condition is one that limits the action to a specific user account. That way, we can filter out the noise and see only the data that’s applicable to the bug we’re tracking. In Figure 10-4, we can see a conditional log that will only print out the data for a user matching the condition Security.getCurrentUser().getId().equals("ID").

A screenshot represents a few lines of code. It highlights line 22 along with a popup dialog box under it. The box contains the details of the agent, file, format, and condition.

Collaboration

When we debug locally, it’s a solo process. As we add actions in a developer observability tool, we make changes to the production, for example, by adding new logs. When a member of our team adds a log or any other action, it can be viewed by other members of the team. Some products support team segregation and privacy solutions, but when using features such as logging, we will inherently all see the changes from one developer.

Some tools support the ability to work as a team on an action. After a log is added, all members of the team can see it within the IDE integrated with the code. They can see the specific user that added the log and the details about that as we can see in Figure 10-5. Notice that the line count in Figure 10-5 skips over the log line since it isn’t an actual code change.

A screenshot represents a few lines of code. It highlights line 22 which reads, c n t + +. A text box above the line reads the value of c n t is open curly bracket c n t, closed curly bracket.

In some cases, we would want to add logs as we track a specific issue, without disturbing our team. As I debug, I often add lines such as "methodX reached with value Y". Logs like that can be problematic, in terms of noise. There is also a case where a value we want to log is private, and we might not want it in the log. A good example is the user token; this might be something that can be used to impersonate a user. We might have policies that prohibit us from logging such a value. For such cases, we can pipe the output of an action or agent to a different location.

We can send all the logs from the developer observability tool to the IDE instead of the main application log. That way, they can remain internal within the team and reduce noise. Some solutions support user-level granularity for log output as well. In Figure 10-6, we can see logs piped into the IDE UI.

A screenshot represents a few lines of code. It highlights line 21. There is another table at the bottom titled light run console that contains the list of the value of c n t. A button on the box reads only my logs.

Segregation

As we saw in Figure 10-8, the agent only communicates with the backend servers. Typically, an agent will send a request to the backend and receive a response. That means backend servers aren’t exposed and can keep their IP addresses hidden from client machines. This is a crucial security measure when hardening a deployment.

This means that even if the developer observability platform is compromised by a hack, the hackers would still have limitations on what they can do. Even the backend server can’t access the agents directly, only send them commands. Commands have limitations in scope.

To further enforce secure communication in all tiers, some vendors offer certificate pinning to harder communications. This is a feature that verifies the signature of the backend certificate to protect against a man-in-the-middle attack.

Accountability

According to IBM, 60% of security breaches in the organization are internal.¹ This is a major problem. Many developers have access to the logs, but very few have access to the database. If we log something bad or problematic (user passwords, credit card numbers, etc.), we might be in big trouble. This can be a problem whether someone tries to do this on purpose or accidentally. Luckily, some solutions exist.

Some developer observability tools support an audit log. This log tracks the operations that occur on the backend and the user that performed them. If a user tries to do something malicious, it will be logged. Such a malicious user will keep a digital trail behind.

A screenshot of the audit log represents a table containing the details of the date, user, and state. There is an option to customize the range of dates at the top.

Audit logs are great, and they work well for malicious intent, at least in retrospect. How do we prevent a mistake or even a malicious attempt? There are two common solutions. The first is Personally Identifiable Information (PII) reduction, which I mentioned briefly in Chapter 4. It’s a niche logging feature which has gained special prominence in the world of developer observability.

The core idea is that some things should never be logged – credit card numbers, user tokens, etc. Some of these elements can be identified since they have a fixed pattern that we can recognize, for example, a Mastercard credit card has the pattern \b5[1-5]\d\d([\-\ ]?)(?:\d{4}\1){2}\d{4}\b as a regular expression. PII reduction lets us define such patterns and eliminate them from the log. This is especially important for developer observability tools where we can add a log without going through the code review process. Figure 10-10 shows the PII reduction UI of a developer observability tool.

A screenshot represents the details of the data redaction. It reads the following text. Disable breakpoints evaluation and logging of sensitive data. This will evaluate Mastercard credit card as REDACTED. It displays the name and pattern of the component.

Usernames and passwords don’t follow a pattern. The doomsday scenario would be a malicious developer on our team adding a log to the user login code, then logging all usernames and passwords. Such a person might be able to pipe all the information to their own machine. A person committing a breach could be caught thanks to the audit log, but administrators would need to know that they should look in the audit log.

Blocklists are there to prevent that situation from ever happening. We can block a pattern for a class or file name, then when a user tries to add an action in such a class, they will get blocked. When setting up a developer observability solution, this should be one of the first stops in the process. Figure 10-11 shows the setup of a blocklist.

A screenshot represents the details of the block list and block list exception. It indicates block list will block insertion in the com dot sales dot ticket class and in all classes that are not in com dot sales package. block list exception block list the com dot sales package except com dot sales dot ticket class.

Agent

This is a valid statement, and it would work by transferring money to my account, at least in theory, assuming the transferCash method is that simple. But some developer observability solutions have features to combat this vulnerability. The first is forcing read-only statements. This is difficult since read-only is sometimes ambiguous. Solutions like that will block some valid code. In my opinion, the peace of mind in read-only is worth the price of some false positives. Since transferCash will require a state change (even a network call), these solutions will fail.

Another capability is the sandbox. Some solutions offer the ability to run the code in a sandbox. That means the code won’t be able to run malicious code. But this goes further; it means that code won’t be able to consume too much CPU. It gets throttled. This solves a major concern when logging too much by mistake.

Moving to a Debugger

These tools are wonderful for an initial lay of the land. They don’t properly convey how things work in practice. A good example would be a generic interface that’s invoked in the application flow. Which specific concrete implementation is invoked? It might be hard to tell without reading all the code and following the elaborate runtime flows. This is something we can easily determine in seconds with the debugger.

With a debugger, we can verify assumptions, see “real-world” usage, and step over a code block to understand the flow. We can place a breakpoint to see if a block of code was reached. If it’s reached too frequently and we can’t figure out what’s going on, we can use a conditional breakpoint to limit the scope.

I make it a habit to read the values of variables in the watch when using a debugger to study a codebase. I then ask myself if this value makes sense now. If it doesn’t, then I have something to study. Nothing teaches you more about source code than following a “mystery.” If the value makes sense, then we clearly understand the source code, and we “win” either way. With this debugger, I can quickly understand the semantics of the codebase. In the following sections, I’ll cover techniques for learning both in the debugger and in production.

Who Changes This Variable?

Many developers know about field watchpoints and just forget about them. “Who changed this value, and why?” is probably the most common question asked by developers. When we look through the code, there might be dozens of code flows that trigger a change. But placing a watchpoint on a field will tell you everything in seconds.

Understanding state mutation and propagation is probably the most important thing you can do when studying a codebase. A common question I have is “is this the only path that changes this field?”. There are two ways I can verify that. The first is to use a tracepoint¹ with a stack trace as we can see in Figure 11-1. Notice that the stack trace checkbox is marked, and we only print it on field modification to avoid the noise of field access. I also print out the value of the field so we will have a context value to go with the stack trace. This way, I can review the output and instantly see which line of code changed the value of the field to a specific value.

A screenshot of a dialog box represents a list of options under prime main dot c n t. It highlights a text box under evaluate and log option that reads the text, field value changed + c n t.

The second option is even more interesting. Say you have a field that’s constantly accessed by one or two methods in the top stack which produce most of the “noise.” You want to see if something else in the code might be triggering a chance to this field. For this, we can use the “Caller Filters” feature that you can see at the lower portion of Figure 11-1. Unfortunately, as of this writing, this functionality isn’t supported for field watchpoints. However, if you have a setter method that encapsulates the field, you can use it on a breakpoint there.

When enabled, we can define inclusive and exclusive filters on the method that will trigger the breakpoint. An inclusive filter will allow us to stop only when the filter is applicable. An exclusive filter will block these methods and is the more useful case for this specific feature. In Figure 11-2, we can see a caller filter on a breakpoint; notice you can edit it directly or in the helper dialog that pops up when clicking the button next to the text field.

A screenshot of a dialog box represents a list of java line breakpoints. It highlights the prime main dot java 7. There is another popup of the caller filter, which contains a table of include and exclude data.

The syntax for the filter is a bit “odd” and takes some getting used to even if you have a deep background in low-level VM programming. Hopefully, JetBrains will expose a friendlier UI for this important feature in a future revision. For now, the syntax works like this.

Why Do We Need This Code?

As I step over in the debugger, I’m often stumped by blocks of code. Is this code even necessary? Do we even need that? This is very hard to answer as there might be wide-ranging implications. I sometimes comment out the code to try and see the impact, but this means restarting the debugging session, rebuilding, etc. That can be tedious and disruptive to my learning flow. A simpler solution is placing a breakpoint in that code and doing a return immediately (force return) every time we reach that block.

We can similarly use the throw exception option to trigger an exception in such a case. This is only useful for understanding code that might trigger such an exception and following the logic in a case of failure.

In both cases, we would want to understand the wide-ranging implications. Some of these we can see in the behavior of the application’s functionality, but some are more subtle in the application state and in reachability. A field might be set to a value we don’t want, or logic that might be “problematic” might be reached. We need to place breakpoints and tracepoints in those relevant areas to monitor for such implications.

Keeping Track

As we debug the code, it’s hard to keep track of everything that’s going on in our head, especially when we’re learning and trying to keep everything organized. When we write code, we have multiple tools to keep track of everything: packages, classes, encapsulation, naming, etc. I might have one breakpoint that’s related to one thing I’m trying to understand and another that’s related to something else. I might have a variable value that I want to follow and another that I need to understand.

The debugger includes some tools that let us keep this all under control. Breakpoints can be collected into groups related to specific debugging sessions or processes as seen in Figure 11-3. We can enable or disable an entire group, so if we want to check something else for a while and get back to this session later, we can pick up right where we left off. When we create a group, we can give it a descriptive name that can help us understand why we created this breakpoint in the first place.

A screenshot of a dialog box represents a list of java line breakpoints. It indicates a popup to create a new group and move a list of breakpoints to the group.

We can go even further and give every breakpoint a descriptive name that indicates its purpose as seen in Figure 11-4. This might seem unnecessary when we have a dozen breakpoints in place, but for sophisticated debugging sessions, that can become valuable. I often need to pause a debugging session due to a high priority issue. In these cases, I usually spend the time grouping and describing the breakpoints so I will have the context saved when I get back to the session later.

A screenshot of a dialog box represents a list of java line breakpoints. it indicates an option in the popup to edit the description of a breakpoint.

I discussed object marking at length in the first chapter. They are one of my favorite features in IntelliJ/IDEA. We can use watch variables to keep track of values, but this only applies in the current scope; marking an object keeps track of it as we step through the entire application, so we don’t lose track of the data. This is something I used to do with pen and paper as I was stepping through the code. This is a fantastic feature for learning and keeping track of the data.

Renderers, which we also discussed in the first chapter, are a remarkable learning tool. In many projects, I’m flooded with redundant information as I step over and view the watch. Objects have too many fields, and I want to focus on the main point of interest. For the learning session, I can create a custom renderer that only shows the variables I care about from the given object. This lets me understand what’s going on immediately without expanding the object. If I do want to dig deeper, I can expand the object, but the quick view can tell me more by default. Figure 11-5 shows a simple renderer that shows the full name of the vet. I expanded the vet object on the right to show that I still have full access to the data if I desire that during the learning process.

A screenshot of a dialog box represents the name of the renderer along with a few settings to choose from when rendering a node. A box on the right side lists a set of data and highlights the full name of the vet.

What’s Going On?

Context is very important when we’re trying to learn. We have a global view of the classes through a UML diagram or just by looking at the code. But the flip side of that is the state. The variable values for instances in RAM tell us more about our understanding of the code than most other techniques. I discussed “Show Objects” in the first chapter; it’s a remarkable feature. I also discussed object tracking with “Show new instances” which lets us see the stack traces that allocated the objects and helps us verify our assumptions related to the formation of the state.

By inspecting all the objects in memory, we can create a proper model in our mind of the application’s runtime behavior. One of the cool things is that the object viewer respects the renderer settings, which lets us quickly scan a large set of objects. The main tool I use for object inspection is the filter field at the top of the dialog. It lets us narrow down the list to a more manageable size. Figure 11-6 shows a filtered list of Vet objects with a renderer applied to them.

A screenshot represents a text box containing a condition to give the output of names with the first name starting with the letter S. There is a filter button alongside the condition. It indicates 2 names at the bottom.

Dashboard Exploration

A great way to start is to set up a custom dashboard and start adding metrics. Once you see the values for the metric, try to find the code that generates that metric. Code that generates a custom metric represents an area that’s important. Otherwise, developers wouldn’t have bothered generating said metric. There are some exceptions, such as metrics that are no longer relevant, so you need to first verify that data is coming through in the dashboard.

The dashboard gives us a bird’s-eye view of the system and its high-level state. It’s a way to get a feel of the “important information” in the system we’re studying.

Log Pinpointing

Few things teach you more about a system than reading the log and reviewing the applicable code. Opening production logs and following through with the source code in hand is a powerful process. User flows are hard to understand in theory, but when we walk through with an actual user and understand a specific process, it clicks.

For this to work properly, additional metadata per log is very helpful. I would often need a way to isolate the logs related to a specific user and ideally metadata about the request. That way, I can reduce a specific flow, like a signup flow, to a readable set of logs that I can then use to understand the underlying code.

Learning with Developer Observability

I often stumble on an API call which baffles me. Do people even use this feature? This is something we can’t always answer. With developer observability, we can add a log or a counter to resolve this question quickly. This can instantly inform us on the importance of a specific API.

Another important feature is log piping which lets us inject logs seen only by us; they won’t be ingested by the observability stack. Make sure to send logging output to the IDE when debugging rather than keeping it with the rest of the logs. Otherwise, the experimentation and learning process might interfere with the regular maintenance of the system. When you add logs and things still aren’t clear, a snapshot helps fill in the gaps by providing a stack trace and variable context.

JMeter Overview

A load testing tool such as JMeter lets us invoke a feature repeatedly to simulate heavy load on a system. The typical setup for running a load testing tool is in the cloud, where load test servers send requests to a cloud deployment. This simulates real-world scenarios and doesn’t starve the system of resources. That’s the “right way” to use the tool, but it’s also problematic.

Before we proceed, I want to be clear: setting up a test in the cloud is the right way to properly test load on the system. You should use that approach to assess the real bottlenecks in the system.

In this case, I’ll run JMeter locally against a simple server; this will give me inaccurate results. That isn’t a bad thing. Performance is like traffic. When we solve a bottleneck, we move the traffic jam to a new location. By running locally, we ignore large swaths of the stack. But the part we don’t ignore is the code; the overhead we see in the code is often the most important overhead for us as developers and most applicable for this book.

JMeter lost some of its coolness factor to newer tools such as Gatling, but it’s still a remarkably sophisticated solution. It lets us create test scripts and even record a session which it can later play back to the server with high intensity. For example, if we want to test a very sophisticated load process, we can set up JMeter as a web proxy to our browser and record everything we do on the website. It will generate a script based on that which we can run over and over on the load test.

Working with JMeter

JMeter works with test plans in which we define the requests we want sent. Ideally, one would run JMeter from the command line as part of our continuous integration process. It can fail a test if performance degrades or error rates rise, thus preventing a regression. But this is a different subject. We want to make a simple load test that we can use in conjunction with a debugger.

There are many ways to create a JMeter test; for simplicity, I often just use the import from cURL option as seen in Figure 12-1. This option pops up a dialog where we can paste in standard cURL commands, and a test plan is created for us implicitly. There are many other options such as recording,³ which is another favorite of mine.

A screenshot represents the context menu of tools in J meter. It lists the options to create a head dump, thread dump, function helper dialog, generate H T M L report, and others. It highlights the option to import from c U R L at the end.

Once we do that, we can run the test against our servers; it will generate high-volume traffic, and we can debug the bottlenecks with this tool. It’s important to set the result file name for the test plan as seen in Figure 12-2. Without it, the test results are lost, and it’s harder to get insights on the execution from the JMeter side.

A screenshot represents the details of the view results tree. It exhibits the name, comments, and file name along with a list of H T T P requests and a code at the bottom.

With the Tools ➤ Generate HTML Report menu option, we can generate a visualization of this execution that we can use to understand what happened on the JMeter side as seen in Figure 12-3. I’ll deal with the debugger for the rest of this chapter and use JMeter only to generate the load which we can subsequently test in the debugger.

A graph of percentile value in milliseconds versus percentiles represents the response time in percentiles. The line grows with a little margin.

Basics of Profilers

I thought about adding a section about profilers to this chapter where I would have covered Java Flight Recorder (JFR) or a similar profiler. It’s an interesting subject with many nuances. I eventually decided to skip that. There are many resources on using profilers; I don’t have anything interesting to say about that.⁴

That isn’t helpful. Memory profiling is even more annoying. We get the type of the object (or primitive) that takes up RAM and need to find out where it’s held. The tool sends us in a good direction, but it’s missing the last mile. That’s OK if you have deep familiarity with the code, it isn’t ideal if you don’t.

A Hacker Profiler

The advantage in this approach is in the localized nature of the benchmark. It doesn’t have the impact of a profiler, and it measures just what we need. This is great when we’re measuring a few calls to a method. But when running something like a JMeter test, we get a tremendous amount of data. In those cases, the profiler shines since it can show us great statistics. JMeter also has statistics, but they include the full round trip, not the details about the performance of the method and its degradation.

It’s also inconvenient for iterative work. I usually move these a lot. If a method takes time, I move the top line inward to narrow down the problematic line or move this to a different method within the call chain to see if that method is problematic. This approach means I need to stop the server, add code, rebuild, and rerun everything. It’s possible but tedious.

We would also add the corresponding method exit. But this is problematic. Since we run the JMeter test with multiple threads, we might see two entries at once from two different threads. It would make things even harder. We also need the thread name in the tracepoint. Furthermore, even a short run will result in hundreds or maybe thousands of lines. How can we process such a large amount?

My solution is to use comma-separated values (CSV), which is a standard we use to represent tabular data. This means we can import the resulting data directly into any spreadsheet and run analysis on it. I can get the visualizations I want and instant calculations. The following code will generate CSV output when used in a tracepoint as shown in Figure 12-4.

"\"Enter\",\"" + Thread.currentThread().getName() + "\"," + System.currentTimeMillis()

A screenshot of a dialog box. The text at the top reads Vet Controller dot java, 94. It highlights a text box under evaluate and log.

Notice we need to use quotes to map the first two arguments so the spreadsheet will import the information correctly. Once this is done, I can import the resulting data into any spreadsheet and view it there.⁵ In Figure 12-5, we can see a chart generated in a spreadsheet.

A screenshot of a spreadsheet containing the data of state, thread, value, and time. It represents a histogram of time in the middle of the sheet.

If you aren’t used to spreadsheets, this might look a bit alien to you. Let’s break it down; the = sign indicates a formula. It’s followed by an if statement that tests if this is an entry or exit for the method. We only want the exit method. At the end of the if statement, you can see that a blank string is returned. This is indeed the case for the first row; since it will be an entry statement, the cell will remain blank.

The code in the middle is a bit more complex. This will only happen for rows that have the “Exit” label. In these rows, we subtract the current time from the last entry we have for the given thread. The XLOOKUP function searches the given cell range of cells listing the thread name for the last occurrence of the current thread. It then returns the value of the cell in the following column (the one with the time). This formula would be simpler without the threading aspect. But now we have time in milliseconds which we can easily plot on a graph or use any other data analysis capability in the spreadsheet to understand.

This generates a sorted view of the data that we can then plot to generate the cool graph we see in Figure 12-6. I even added a trendline to show the degradation; it nicely highlights the edge cases where performance didn’t keep up.

A screenshot of a spreadsheet. It exhibits a graph in the middle with 2 lines representing sorted and a trendline for sorted in increasing order.

Pinpoint Precision

Figure 12-6 shows a typical picture. Performance degrades in a “nice” clean fashion until it falls off a cliff. This is what we often see in a profiler, especially under load testing. Sometimes, this indicates a failure; usually, it indicates we crossed a threshold somewhere, but which one? What’s failing in this case?

Since no state is changed, you might be able to add this line and reload the app without restarting the running app. In IntelliJ/IDEA, you can do that using Reload Changed Classes as shown in Figure 12-7.

A screenshot represents a context menu under the run option. It highlights the debugging actions, followed by the reload changed classes option in the menu.

Once this is done, we can add either tracepoints or a breakpoint with a condition to match the outlier time value. For example, System.currentTimeMillis() - startTime > 200. This way, we can dynamically extract information about the slow performance and narrow down the impact area without restarting the execution.

Error Rates

A major reason for problematic performance peeks is simply due to errors. Exceptions are thrown, and as a result, buffers get flushed, and performance degrades. This is a perfect time to review the “Exception Breakpoint” section in Chapter 1.

We should define an exception breakpoint and review the errors that occur when we reach a high scale of requests. We can filter the exceptions to make sure we get only the relevant information.

Debugging Cache Misses

Caching is the most powerful performance-related tool in our quiver, whether it’s in the CPU itself or from a remote database. Caching is the mechanism that tips the performance scale. It’s also one of the hardest problems in computer science. I won’t go into the various failures we can debug with caching (pollution, races, etc.) since this is a performance chapter. What I want to discuss is cache utilization.

When we access an element in the cache, there are two options: hit or miss. A hit is great; we have an element in the cache that we can return. A miss might be legitimate but might be an opportunity to improve the logic for the caching system. Unfortunately, these things are very hard to debug. The typical approach we use is to log cache hits and misses and then generate a chart representing this. Once we have that, we try to tune the code to improve utilization.

Our cache has limited room. When we add an object to the cache, we often need to remove a different object. This is important to keep memory usage low but also to keep the size of the cache small. Searching through a smaller cache is faster. The value of the caching system is determined by our logic, the choice we make on which objects should be kept near. As you can see, there are many trade-offs we can take. We can remove the object whose accessCount is low, but that might not make sense if creation time is close; we might not have enough data. Maybe the object is accessed frequently for a short duration; in this case, lastAccess indicates the object is no longer used.

The last argument is the one that matters. We can use the value of the miss time to check whether our logic could have been improved. For example, if the delta between the creation time and the miss time exceeds a threshold, this might be a legitimate miss. However, if the gap is shorter, we can conditionally highlight the problematic lines and review them individually in the spreadsheet.

We can implement this logic with a logger as well or with simple print debugging. The value tracepoints bring to the table is in conditionality and the ability to dynamically apply them to multiple eviction points. The fact that it includes zero overhead in production is a bonus.

Debugging Memory Issues

Memory corruption is a hard problem to debug without code changes; luckily, this isn’t a common issue in Java. Our most common issue is memory leaks and overuse of memory. This is typically easy to profile as the tools explicitly list the objects in memory.

We discussed the memory capabilities of the debugger in the first chapter. It’s a remarkable tool for gaining insight related to specific allocations and the current state of our application. The filter capabilities of the object view let us narrow down the objects and pinpoint a problematic entry.

But the killer feature is memory tracking (seen in Figure 1-14). This feature shows the stack trace associated with individual object allocations. We can zoom in on a specific line of code related to a wayward object and address the root cause of a problem.

Developer Observability and Performance

When discussing developer observability in Chapter 10, we discussed metrics. They are an amazing tool for typical production performance benchmarks. I suggest using them and ideally connecting them to your Grafana dashboards. Notice that most of the techniques I mentioned here can be used with developer observability and piped logs.

In this way, you can extract fine-grained profiling information from a production environment. The main drawback is that some high-volume data might be removed due to the sandbox. As such, the spreadsheet mentioned earlier won’t work since it relies on a perfect match between the entry and the exit (there being an exit call for every method entry call).

An Example

This is all very theoretical. When I was a kid, we found out that the local 1-900 phone numbers had a corresponding local number you could dial for free. When they listed their numbers in the newspaper, they usually had a free local number for tech support. When they bought those phone numbers, they often bought many numbers in bulk. When buying in bulk, we usually get sequential numbers.

My friends and I inadvertently discovered the ID scanning vulnerability. We started dialing random numbers in the region of the tech support number, and we were able to dial into the 1-900 number for free. That’s the exploit.

This isn’t limited to 1980s enthusiastic teens. A few years ago, an upstart social network used numeric IDs to represent its users. They exposed this numeric ID in a URL to fetch the user details. Furthermore, they didn’t even secure that URL. That meant one could just request URLs with numbers one by one and get all the user details. No restrictions to “friends,” no authentication required!

A user ran an exploit and downloaded the entire list of users from that social network. Basic hardening would have verified that a user has the right to get the details of another user. It would have required authentication and would have limited the number of requests a single user could run. Security would have used a hash as the publicly exposed ID to stop a scanning attack like that.

Tips and Best Practices

Security is layered like an onion. A properly hardened secure environment will make an attacker work for the exploit every step of the way. Even if the attack succeeds, we will have more time to catch the attacker, and they will have less time within the system. For hardening, we need to first verify every identity and every permission. We need to limit what can be done, even by authorized users. We need to log everything in an administrator log; every operation and action must be trackable. If the worst thing happens and we are hacked, we need to understand exactly what was done to our system and how. An administrator log is the ideal way to keep track of everything that happens.

Restrictions on internal users are where even the biggest companies fail. Even when an attack is external, it would often impersonate an internal user when executing the attack. If internal users were limited in their abilities, the damage that the attack could do would be limited.

I strongly suggest using linting tools such as SonarCloud and ideally a security analysis tool² to track code issues and security vulnerabilities. There’s absolutely no way to keep track of all the latest discovered vulnerabilities and notice elaborate issues.

Injection

This code might seem innocent enough, but it’s a huge vulnerability that was common in the early days of the Internet and, sadly, not extinct to this day. The ID can be anything. If I could set ID to ; DROP table users, that would be pretty unpleasant. The solution is to use prepared statements, validate input, and optionally use APIs like JPA which do those things for us seamlessly.

What can the debugger do to test injection style vulnerabilities? If we have suspect code, we can use the “evaluate” capability of the debugger to try various injection values to see if they indeed fail. But often, we don’t have an immediate suspect. I would suggest using structural search³ to find suspicious statements.

This is a bit obtuse due to the use of the == operator, simpler code would use the contains() method. At this point, our unit tests can mock the database and invoke verify for every argument received in the database layer. Only whitelisted values will pass. But we can check that quickly in the debugger using a conditional breakpoint. We can place such a breakpoint in the database driver code with the condition !verify(value). The breakpoint will hit for every entry that didn’t pass through the validate method as seen in Figure 13-1.

A screenshot of a screen has the options of enabled, suspend, all, thread, and done. It has a dropdown box labelled condition: the text in it reads exclamatory sign verify value.

ID Scanning

ID scanning is one of my favorite vulnerabilities as you might have noticed from my prior discussion. The knee-jerk reaction might be to remove the ID from the database. This might not be feasible for many cases. Performance of numeric IDs is often faster, but more importantly, changing the primary key at scale is a tremendous undertaking. It’s entirely possible we can’t change the ID.

If we can’t replace numeric ID values with strings, we need to protect them. We should never return them to client code or alternatively block the client from accessing them at scale. These are things we need to detect in code. One of the most important concepts in security is restrictions on every layer.

A good example of that is a web UI that checks if you have permission to access a resource. A hacker would just circumvent the web UI. We need to check authority and permissions in every layer of the system. Otherwise, a bug or vulnerability in another layer can be used as part of an exploit. This might sound theoretical, but let’s give a concrete example. Say we have a REST endpoint that returns the user details: /userDetails?id=numericId. The implementation of this method may be secure. But a new developer added a new endpoint /userCurrentLocation?id=numericId for the tracking app. This new endpoint wasn’t properly secured.

A good implementation will still work since it will verify security in the implementation layer (business tier) and will fail there. We need to verify in tests that the business tier itself doesn’t let us fetch an arbitrary user. But these sorts of tests are sometimes difficult to compose in an authentic way since the process of authorization is so deeply woven into the system. Do you have such a test in your project? Testing this from the debugger is much easier.

In the debugger, we can place a breakpoint in the database business layer code that fetches a user by its ID and add a condition that verifies the rights to fetch that ID. We can then use an unrelated business call with the debugger’s evaluate feature to fetch a user for which we have no permission. We can see such a breakpoint in Figure 13-2.

A screenshot of a screen indicating the breakpoints. Movie service dot java, 37. Checkboxes labeled, enabled, suspend, and condition are selected. Below it is the dropdown box, the text field reads, has permission for user of i d.

This is great, but what if someone is actively trying to scan IDs in your system? There are various tripwire solutions to trap such cases. These solutions are typically high level and would stop a scanner, but might stop legitimate usage as well. With developer observability tools, we can detect such misuse and pinpoint it to a specific account. We can’t stop the actual scanning, but we can get all the details.

Detecting a scanning operation works in the same way as detecting a local breakpoint. We could add a conditional snapshot to the problematic line of code and increase the max hit count, so if multiple violations occur, we’ll get additional data. In Figure 13-3, we can see such a snapshot on the same line of code. We could use a log at this point, but it wouldn’t include as much data as we might want. This is valuable if we discover a scanning vulnerability and want to know if someone exploits it while we’re working on a patch.

A screenshot of a screen has a tab open in it, titled insert a snapshot. The text below it reads Captures all the local variables and expressions in the specified location. It has the option to choose agent, file, expression, condition, max hit count. Below are the buttons cancel, advanced, and O K.

Serialization Issues

I recently attended a talk by Brian Vermeer who labeled serialization exploits as “the gift that keeps giving.” This is a well-known problem in the Java serialization specification; there are multiple attempts to fix those problems, but programming convenience, compatibility, and security are often at odds.

This chain shows how a single read from an untrusted source can slowly escalate by nesting object types until we reach TemplateImpl which lets us load an arbitrary class file. Once we have that, we can load any class we want and own the system. This is an elaborate exploit, but since the related tooling is very powerful, even novices can construct elaborate chains and deliver an exploit payload.

There’s a simple solution to harden systems against serialization attacks: serialization filter.⁷ With a filter, we can whitelist the serializable classes and limit the surface of an attack vector. I recommend using a whitelist approach for any application. However, this isn’t always an option. Some legacy applications might use serialization; I wrote such applications 20 or so years ago. Removing or even limiting serialization in such deployments might be untenable.

We need to add this code to the entry point of an application,⁸ then place a breakpoint on the return statement. This will unveil classes that are serialized, but that’s something we can detect with a simple logger (which might be worth adding here). The main value in this approach is looking at the stack that leads to serialization calls. Go up the debugger stack and find the specific call. Is it possible that this call will load a class from an untrusted source? How does one reach this serialization call?

Don’t Open Holes

Not opening a security hole sounds easy enough. Then we pass around a single universal key to the SSH of an AWS machine so we can debug something. Every permission granted for debugging should have an expiry date. The smaller the team that has access to these compute resources, the better.

As developers, we need to avoid remote debugging into production. As I mentioned before, this is a glaring security hole. We need to review every endpoint we open and try to limit information exchange. We need to limit what can be passed and how we accept data.

If we use a tool like developer observability, we need to review every potential hack and use blocklists aggressively. For example, if you use Keycloak for authentication, I would recommend blocking the entire org.keycloak package by default. It would make sense to block entire packages and individually whitelist specific packages. This is crucial as a malicious developer could add a log into the authorization code and steal tokens.

Refine Logging

I discussed logging in Chapter 4 but didn’t discuss auditing. Auditing is the process of semantic transactional logging that tracks changes to a system. We can follow the audit log to understand the history of every change and associate it with the specific user that triggered it. An audit log isn’t modifiable and can lead us down a breadcrumb trail in the case of a hack. Access to the audit log should be heavily restricted to avoid the risk of tampering.

The regular log should be minimized though. I even go so far as to recommend only logging warnings and errors. While many organizations properly secure the database, very few have the same restrictions on the log. We would see users who have no database read privileges but have full access to the log. It’s very common to log user information and even tokens. A malicious person with access to the log can use such tokens for impersonation or even use private data to gain access in case tokens weren’t logged.

Lessons Learned

The initial project should have used a fail-fast policy as discussed in Chapter 4. This is a great defensive approach and is important for reliability. This is something that I didn’t have control over, but it was an important tool in discovery.

When we looked at the code, we quickly zoomed in on the store operation. I assumed it failed even though I knew it shouldn’t. The rule of thumb is this: when you know code “can’t do that,” it probably really can’t. There’s something else going on. We should have spent more time investigating potential influences.

There were no tools to debug production which left us rather blind. This was before developer observability or similar tooling were available so we couldn’t use that. But the company didn’t have much in terms of a general observability stack either, only logging.

The project should have had better test coverage for failure situations. I don’t recall the full setup in terms of integration tests, but I’m sure that good coverage would have revealed the problematic error handling before production.

Lessons Learned

Tooling is revolutionary and beats any theory. Work with facts and measurements. Do that iteratively. That company could have saved on my consulting fees by just running the profiler on a regular basis and structuring their project in a way that would make it easy to run the profiler. Don’t waste time in meetings about theory when you don’t have the facts in front of you.

Talk to your colleagues. Two guys were sitting side by side working in the same area of the code. Yet each assumed the other person was responsible for caching the results. Had they done proper code reviews, this is probably something they would have noticed. I would also suggest defining conventions for behaviors like caching, for example, fetchMyElementCached(x), to indicate whether the method will keep data.

Lessons Learned

We need observability into production. We can’t be constrained due to costs when it comes to that. We need a way to look at logs and debug. That’s one of my biggest problems with serverless; it leads to the same exact problem, and I see many similar experiences online.

There are “good ways” to degrade services. When we debug, running out of resources is a problem I can debug. I can look at the resource and understand what’s taking up the machine. Solutions that scale and charge based on that scale are impossible to debug. I can’t put a breakpoint on Google’s billing algorithm. The ability to debug locally or on staging is crucial; without those abilities, everything is at risk.

Lessons Learned

That system didn’t allow me to use a tool like strace (discussed in Chapter 3) to check the execution of the project. Had it allowed that, I might have seen the issue. The network operations were returning while the IO was still in progress. These days, I try to delegate more of that sort of debugging to customers.

There was a subtle reliance on the speed of the network in initialization code. It’s an assumption that I made consciously, that local IO is faster than networking. I didn’t give that assumption enough thought when writing the code, nor when I searched for the bug. During the assumption check phase (Chapter 2), I should have seen that.

Lessons Learned

We did everything wrong here. We had no dedicated memory debugging tools. When working with an unsafe language, those are essential. We had no linting for the project; back then, linters were much simpler than they are today. I believe a modern linter would have instantly found that.¹

We need to use tools that match the platforms we use. When we write native code, we need to use native unit tests to verify that code and need to use memory monitoring tools to fix issues in these languages.

Lessons Learned

Had I read the Scala code properly, I would have probably seen the problem before the logging. I should have put more effort into understanding what was going on.

Observability is always recommended; with microservices, it isn’t an option. The complexity of the deployment requires a sophisticated monitoring and observability stack. With proper spans, we could instantly see the problem and would have understood everything.

Lessons Learned

Long-run tests are hard to build; the pain during their development and execution pays back in dividends when we go to production. The lack of visibility in that environment is painful; we do have developer observability tools installed there, but in this particular case, I’m not sure if they would have helped.

Some time travel debugging tool vendors offer their products as solutions for such issues (learn about TTDs in Chapter 5). I haven’t tried that myself, but I think that might be one of the best use cases for time travel debugging. I did try developer observability tools for such cases, and they do show promise, but it’s still a bit challenging.