Mastering Lambdas: Java Programming in a Multicore World (2015)

CHAPTER 7. API Evolution with Default Methods

Default methods allow interfaces to define behavior. Why did we need this change, and what are its consequences? The short answer to the first question is: to support API evolution. The need for this has been felt for a long time, but became pressing with the requirement for stream support in collections. (The longer answer is that, once introduced, default methods have other uses as well, as we shall see.) The Java Collections Framework—like its extensions, for example Google Guava—is strongly interface-based: that is, the capabilities of a collection are (with some exceptions) defined by the Javadoc contract of its interface. For the existing collections library to be enhanced to support streams, interface methods like Collection.stream were needed. The alternative—complete replacement of the Java Collections Framework—would have presented extremely unattractive problems of compatibility and code maintenance.

So it was decided to change the situation in which adding methods to existing interfaces created unmanageable compatibility problems. Reviewing these problems will help us understand the choices that the designers made in implementing that decision. Consider, for example, adding an abstract method stream to Collection. (But note that the problem described here applies to all published interfaces, not just those in the platform library.) Prior to Java 8, Collection declared only abstract methods:

and a class implemented the interface only if it overrode every abstract method declaration with a concrete method:

Now consider the problem of making collections into stream sources. The interface Collection had to be extended with a method stream; without default methods, that could be done only by adding an abstract method declaration:

If the implementation customCoiiection were now compiled against this new version of Collection, the result would be a source incompatibility: compilation would fail because Collection now declares an abstract method not overridden by customCoiiection. Without default methods, the only fix would be to make CustomCoiiection match the new definition of Collection by adding a concrete overriding declaration of stream. But if the interface is part of a library—Collection is particularly widely used, but any published API could be an example—then it would be hardly practicable to accompany its every enhancement with a requirement for extensive source changes to all its implementations.

Note that, without recompilation, existing bytecode for CustomCoiiection could still link to and run against the new version of collection—interface and implementation would be binary compatible, according to the Java Language Specification (§13.2). But leaving the implementation unchanged would only transfer the problem to the client programmer: now, client code written to take advantage of the advertised new capability of the interface would compile without problems:

but would fail at run time with an AbstractMethodError.

The basic problem was that the traditional role of a Java interface was restricted to helping define a contract between client and implementation; the responsibility for providing the functionality promised by the contract lay entirely with the implementation. Naturally, if the requirements of a supply contract are extended, the supplier must be aware of the change and must fulfill the new part. Once an interface has been published, however, it becomes impracticable to insist on this for all implementations. As a result, published interfaces are never extended with abstract methods.

The introduction of default methods changed this situation by extending the role of the interface beyond the definition of a contract to include its partial implementation. Interfaces can now include code, in the bodies of two kinds of non-abstract methods: default methods and static methods. Default methods are now central to API evolution; for example, this is how Collection.stream is actually declared in Java 8:

For a client, the interface still has the same meaning: it defines a type and lists the declarations of the methods that a client can call on that type. For an implementation, the main element of its relationship with the interface is also unchanged: it is still required to override the abstract method declarations with concrete ones. A new element is added to the interface-implementation relationship, however: the implementation can now choose to override default method declarations—because these are virtual—and replace or modify their behaviors, just as in traditional Java instance method overriding.

So now an interface can be extended to allow client code to call new methods without requiring any immediate change in existing implementations. This removes a serious block on API evolution, present ever since JDK1.0. It is worth asking why this block was allowed to remain in place for so long.

Aside from the considerable practical difficulties in making a change of this size, attitudes to multiple inheritance had to change. Aversion to multiple inheritance was a strong element in the original design of Java because of the difficulties that it had brought to other languages, most prominently to C++ in the form of the “diamond problem” (p. 169). Even so, multiple inheritance of interfaces has always been considered acceptable because interfaces were essentially type definitions, and multiple inheritance of types did not involve the same problems. Introducing default methods extends the role of interfaces to provide behavior, so multiple inheritance of behavior would now have to be allowed. The Java 8 designers were able to devise rules that make the problems with behavior inheritance manageable; the classic problems of multiple inheritance are associated with inheritance of state, which Java does not and never will support.

7.1 Using Default Methods

After a change as big as this, it takes time for the best new idioms to emerge. Some use cases can be seen in the Java 8 platform classes already, however. Here are four of them:

• Methods intended to be overridden The previous section presented methods like Collection.stream as the central use case for default methods. The implementation of stream is delegated to another default method, spliterator (defined in iterabie, the superinterface ofCollection). The purpose of this method is to return a spliterator that takes advantage of the structure of the collection to partition its contents for processing by different threads, like the MappedByteBuffer spliterator of §5.4. Of course, the default implementation itself cannot fulfill that intention; it is suboptimal for most Collection classes because it has no knowledge of the structure of the object it is splitting. Virtual method dispatch allows Collection implementations to override spliterator with concrete method definitions that take advantage of their specific features.

• Methods that will commonly not be overridden Some interfaces contain method declarations that will be commonly implemented by the same concrete method. For example, the default implementation of comparator returned by Comparator.thenComparing(Comparator) is implemented very simply: first evaluating the receiver of thenComparing on its arguments, and, if it returns zero, then—as the name implies—evaluating the supplied comparator. It is hard to see how a particular implementation could improve on this.

A contrasting example is iterator.remove. It’s debatable whether the interface should have originally included this method, unsupported as it is by so many implementations. Nevertheless, the problem was at least mitigated in Java 8 by making remove into a default method:

taking away the requirement to reproduce concrete boilerplate code in every iterator implementation not supporting remove.

• Auxiliary methods Although a functional interface has exactly one abstract method to define its central purpose, there is no restriction on how many default methods it may have. These can be useful for adding extra capabilities. For example, we have seen Predicate and its specialized variants used to define truth-valued functions for filtering streams. You will often want to operate on the results of these truth-valued functions using the standard boolean operators AND, OR, AND NOT. So Predicate and its variants declare default methods implementing these operators:

• Convenience methods: Default methods provide opportunities to relocate ex isting functionality in more appropriate places. For example, to sort a List in reverse natural order before Java 8, you had to call two static Collections methods:

Collections.sort(integerList,Collections.reverseOrder());

Now, however, the default method List.sort (together with a static interface method Comparator.reverseOrder) allows this code to be rewritten in a more readable way:

integerList.sort(Comparator.reverseOrder());

In fact, List.sort is more than a convenience; it allows the List class to override sort with an efficient implementation, using knowledge of its internal representation that is unavailable to the static method Collections.sort.

These are examples of interface methods making existing functionality much more discoverable: a sort method for List instances and a factory method for creating Comparators are clearly much better placed in these interfaces than in the general-purpose collections utility classCollections.

7.2 What Role for Abstract Classes?

Adding implementation capability to interfaces brings their role nearer to that of abstract classes, which also mix abstract method declarations with implementation capability. It is natural to ask what role remains for abstract classes.

It is easiest to understand the answer to that question in relation to a conventional Java API, containing interface, abstract class, and concrete implementations. For example, suppose that the library system of earlier chapters is to be made more general, by adding ebooks and audiobooks: these have titles and authors, but lack some properties of physical books—for example, height and page count. A refactored domain design might look in part like this:

Now suppose that we want to expose a method on Bookintf that will return the names of its authors concatenated into a single string. That would fall into the category of convenience methods described in the previous section:

This works as a default method because it can be defined entirely in terms of another interface method, getAuthors. By contrast, to define getAuthors itself requires access to instance data, which cannot be declared in the interface. The data that is common to different subclasses, likeauthors in this example, is still held in the abstract class, which is therefore also the location for getters and setters:

The need for instance state, as in this example, is the main reason for continuing to use abstract classes. There are other limitations to default methods that mean that abstract classes are still needed: default methods can only be implemented in terms of methods on the same interface (together with accessible static methods on any type). Further, abstract methods can declare protected state and methods to share with their subclasses; this option is not available to interfaces, all of whose declarations are automatically public.

7.3 Default Method Syntax

We have already seen the form of default method declarations. They are very similar to concrete method declarations in classes. In interfaces, they are distinguished from abstract method declarations by the presence of the modifier default and by the fact that their body is represented by a block rather than a semicolon. Lively debate took place over whether the default keyword should be mandatory, given that it is not needed to make the syntax unambiguous—an interface method declaration with a block body can only be a default method. One advantage of making it mandatory is that it immediately prompts a reader’s understanding, in the same way as the modifier abstract (also not strictly necessary) does for abstract methods and classes.

The main syntactic difference between default methods and concrete instance methods is in the modifiers that they are allowed—or, in the case of default, required—to have. Obviously, default methods may not be declared abstract or static, since these keywords distinguish the other kinds of interface methods (we will explore static interface methods in §7.5). They may not be declared final, because they must always be overridable by instance methods, as we shall see in the next section. Like all interface methods, they may be declared public but are implicitly public anyway; no other accessibility is possible.

The keyword this has its usual meaning; it refers to the current object, referenced using the type of the interface. For example, the method Iterator.forEachRemaining has the following implementation (slightly simplified for presentation):

The keyword super is also allowed, but only when it is qualified by the name of a superinterface, as the next section explains.

Interfaces are not allowed to declare a default method with the same signature as any of the methods of Object. So, for example, you cannot redefine Object.toString in an interface. This is in line with the principles for default method inheritance that we will explore in the next section: the most important of these principles dictates that default methods can never override instance methods, whether inherited or not.

7.4 Default Methods and Inheritance

We have seen the uses and benefits of default methods; what are the drawbacks? The challenge facing the language designers in introducing them was to devise a system for method inheritance that would be simple and unambiguous while minimizing compatibility issues (and unwanted interaction with other features). The most important compatibility problem to solve concerned resolution of method calls when a choice of implementations is available, especially when these include both a concrete method inherited from a superclass and one or more default methods inherited from interfaces. The language specification rules for method call resolution are complex, but they are carefully designed so that they can be understood by reference to two simple principles.

The first principle ensures that instance methods are chosen in preference to default methods. This is sometimes stated as “classes win over interfaces.” For example, in the following code FooBar inherits the method hello from both the interface Foo and the superclass Bar:

When FooBar.main is run, the output is Hello from Bar. This principle holds whether the instance method is declared within the class or inherited from a superclass, and whether the instance method is abstract or concrete. The motivation for this rule is to prevent behavioral incompatibility:that is, the addition of a default method resulting in a change in the behavior of an implementing class. If classes always win, then a class calling a method inherited from a superclass will continue to call that method even when one of the interfaces it implements introduces a matching method declaration of its own.

The second principle ensures that if more than one competing default method is inherited by a class, the non-overridden default method* is selected. By “non-overridden method” (this is not a standard term) we mean a default method not overridden by any other that is also inherited by the class. For example:

Again, when FooBar.main is run, the output is Hello from Bar.

Of course, there may be no single non-overridden default method, if more than one default method inherited by the class is not overridden by any other. For example:

In this case, FooBar fails to compile, with the error message

That is a reasonable response; the compiler has no basis for choosing between the two inherited methods, so prompts you to disambiguate the call. You can do this by making FooBar itself override hello, using a syntactic form (already present in Java, but until now only used for a different purpose in inner classes) provided to allow selection of one of the competing methods:

The same syntax can also be used in the body of an interface default method. Note that it can only be used to resolve a conflict, not to override either of the two main principles. So you cannot use it to select a method that is not non-overridden:

Even if FooBar implements both Foo and Bar directly, Foo.hello is not a non-overridden method and cannot be selected using the super syntax.

How do these principles help to resolve the “diamond problem”? This is the situation in which a class inherits a declaration by two different routes:

The problem gets its name from the shape of the class diagram:

The principles of this section provide a straightforward interpretation of this scenario and its variants: in the code shown, the implementation of hello inherited by FooBar is the one defined by Apex; there is no other possibility. If, however, Foo or Bar is now changed to also declare a default implementation of hello, that method becomes the one inherited by FooBar, on the “non-overridden method” principle. Notice that the this principle, combined with virtual method dispatch, can give apparently surprising results in this situation. For example, if in the preceding code the declaration of Foo was changed to

then this code

would produce the output Hello from Foo. The static type of bar is unimportant; what counts is that it refers to an instance of FooBar, whose non-overridden method is inherited from Foo.

If both Foo and Bar declare default implementations, then they conflict, and FooBar must provide an overriding declaration.

7.4.1 Compatibility Problems

The principles of the preceding section cover the great majority of practical situations. Unfortunately, it is not always possible to avoid incompatibilities. This section describes three problematic situations.

In the following code, Intf represents a published interface, and Impl a library implementation. They are not necessarily in the same library, so the maintainers of Intf may well know nothing about its implementations, and cannot take their declarations into account in evolving the API.

In this version, running Client.main obviously results in the output Hello from Impl. We now consider two different changes to Intf, each of which adds a method to Intf that almost matches the declaration of Impl.hello. For example, Intf could declare a version of hello with the same signature but an incompatible return type:

Between overriding methods, classes always win; but this is not an overriding method. Compiling Impl against the new version of Intf produces this message:

A variation of this problem occurs if Impl’s declaration of hello is non-public; in this case, even if the signature of the new method in Intf matches exactly, it will—since interface methods are automatically public—cause the compiler to report that Impl’s override is attempting to narrow access to hello.

The second problem is more serious. Here, the interface adds a default method declaration with a parameter whose type is assignment compatible with a corresponding parameter in the existing declaration. In this example, the new declaration might be:

According to the rules for method overloading, this declares a new overload of hello, which, since it does not override the existing one, is inherited by Impl. So both the new and the old method overloads are available for the call of hello(3) from Client and, when Client is compiled, themost specific one is chosen, as defined by the Java Language Specification (§15.12). Since 3 is an int, not a long, the newly inherited overload is more specific, and the output will now be Hello from Intf. The change in behavior takes place without any change in Impl! This example illustrates the difficulty of making fully compatible language changes in the face of already complex semantic rules. (Method overloading is notoriously difficult in this respect, as we also saw in §2.8.)

A different kind of behavioral incompatibility is unrelated to syntax problems but is inherent to dynamic method dispatch. A newly-introduced supertype method may be unable to respect the invariants of an implementing class, because it has no knowledge of them. For a real-life case, consider Map.putIfAbsent, introduced in Java 8:

This method, if not overridden, will destroy the thread safety of any implementing Map: between the time at which a thread evaluates the test and the time at which it executes the action, the value of v could have been set by another thread. The current thread would then overwrite that value, contrary to the specification of the method. There is no true solution to this problem; in the worst case, as here, all implementations must be inspected to ensure that they override newly introduced default methods.

Notice that the problems of this section aren’t new: any of them could occur with class inheritance. What makes them more serious is that, whereas it is expected and understood that changes in a class hierarchy are liable to cause problems like these, it is new for Java that interface changes can have these effects. Fortunately, they do not often occur in practice.

7.5 Static Methods in Interfaces

Once the decision had been made to allow interfaces to provide behavior, it was natural to examine static methods to see if they also could, or should, be allowed into interfaces. With hindsight, it’s easy to see the advantages of permitting them: throughout this book we have been taking advantage of their presence with method calls like

With the perspective of the preceding section of this chapter, however, we can see the importance of minimizing compatibility problems. Actually, these can be eliminated altogether for static interface methods by restricting the syntax that can be used for referencing them to the specific form Declaringinterface.MethodName. (Obviously, this solution wasn’t available for default methods, since it’s not compatible with virtual method dispatch.) So one difference from static class methods is that static interface methods are not inherited:

For the same reason, you cannot refer to static interface methods by the syntax objectReference.MethodName.¹ In other respects, static interface methods are declared in the same way, and have essentially the same properties as static class methods. Like other interface methods, they may be declared public but are implicitly public anyway. It is not permitted to declare a static interface method final, since that would be a meaningless modifier for a method that cannot be inherited anyway.

7.5.1 Using Static Methods

Again, it is too soon to know what idioms in API design will emerge to make use of static interface methods. In Java 8, the platform classes expose them as factory methods—as with Stream.of, Collector.of, and Comparator.comparing—and as ways of locating functionality that are often an improvement on the traditional use of utility classes.

Utility classes can present major difficulties in locating functionality. The most extreme example in the platform library, java.util.Collections, contains more than 60 static methods, of which the majority are associated not with collections in general but with one of Set, List, Queue, orMap (the last of which is not even a subtype of Collection). A newcomer to the Java Collections Framework cannot deduce in any systematic way where to find a factory method for producing, say, a Map instance. Such arbitrary allocation of function is a major obstacle to learning an API. By contrast, now that interfaces can expose methods like Comparator.comparing and Stream.of, they can be where a developer would naturally expect to find them. This improvement is not restricted to new methods; for example, Java 8 introduces a factory method Comparator.reverseOrder, which is implemented simply by delegation to the existing (but poorly located) method Collections.reverseOrder.

That said, it is unlikely that utility classes like Collections will altogether disappear in new APIs. Consider the interface Collector; there are more than 40 factory methods for this interface in the Stream API. Only two have been declared as static methods of the interface itself, namely the two general-purpose overloads of Collector.of that were discussed in §4.3. The forty-odd other predefined factory methods, described in §4.1, deserve and get a class—Collectors—of their own; putting them all into the Collector interface would overwhelm its core function. This example leads us to expect that a balance of usage between static interface methods and utility classes will emerge over time.

7.6 Conclusion

It is no easy task to change a programming language that has been in widespread use for nearly for two decades. When the feature to be changed is as central to the language as interfaces are to Java, then the problem is even more difficult. Despite the problems described in this chapter, the addition of behavior to Java interfaces has, overall, caused remarkably few difficulties.

This is partly because the features have been strictly tailored to the purpose of enabling API evolution. Some of the controversy around the introduction of default methods was based on expectations that they would reproduce features like traits or mixins in other languages. The absence of state in interfaces prevents that, however. Similarly, as we have seen in this chapter, expectations that interfaces could now replace abstract classes, or make utility classes redundant, are exaggerated. But they do fulfil the purpose of their design, to enable API evolution; the opportunities that this opens up, initially benefiting the Stream API, will create possibilities for maintaining and enhancing other APIs far beyond what has been feasible until now.

____________

¹The two differences with static class methods—preventing inheritance, and disallowing calls through object references—are probably best seen as a refusal to repeat two mistakes.