Towards Functional Java: Method References

In the previous post of the Towards Functional Java series, we saw how to create a reference to a method implementation using lambda expressions and functional interfaces, passing it as an argument to methods in the same way we are used to pass object references.

Note: to try out the examples in this post, you need a Java 8 development environment. If you do not have one, you can follow this tutorial to set one up.

In the following JUnit test, we implement the functional interface Comparator so that it compares two strings by length (instead of alphabetical order) using a lambda expression, and assign the implementation to the variable comp. We then use the variable name to pass the implementation to the Collections.sort method to sort a List of strings. The API doc can be found here:

https://docs.oracle.com/javase/8/docs/api/java/util/Collections.html#sort-java.util.List-java.util.Comparator-

@Test
public void lambdaExpressionTest() {

   Comparator<String> comp = (str1, str2) -> 
       Integer.compare(str1.length(), str2.length());

   List<String> list = new ArrayList<String>();
   list.add("abcd");
   list.add("abcdef");
   list.add("ab");

   Collections.sort(list, comp);

   assertTrue(list.get(0).equals("ab"));
}

 
Now, suppose the above functionality was already provided by a method of an existing class, as in the following example:

public class AlternativeStringComparison {	
   public int compareByLength(String str1, String str2) {
      return Integer.compare( str1.length(), str2.length());
   }
}

 
You can refer to the compareByLength method directly using the following notation:

@Test
public void lengthComparisonTest() {	
   AlternativeStringComparison comp = new AlternativeStringComparison();
      
   List<String> list = new ArrayList<String>();
   list.add("abcd");
   list.add("abcdef");
   list.add("ab");

   Collections.sort(list, comp::compareByLength);

  assertTrue(list.get(0).equals("ab"));	
}

 
The expression comp::compareByLength is called a method reference. In this case we are creating a reference to an instance method, using the notation object::instanceMethodName.

If the instance method is defined by the same class of the receiver object, you can use the notation Class::instanceMethodName. For example, the String class defines an instance method to perform a case-insensitive string comparison. See the API doc:

https://docs.oracle.com/javase/8/docs/api/java/lang/String.html#compareToIgnoreCase-java.lang.String-

You can create a reference to it as follows:

@Test
public void caseInsensitiveComparisonTest() {

   List<String> list = new ArrayList<String>();
   list.add("bobby");
   list.add("Andrew");
   list.add("john");

   Collections.sort(list, String::compareToIgnoreCase);

   assertTrue(list.get(0).equals("Andrew"));
}

 
Likewise, it is possible to create a reference to a static method with the notation Class::staticMethodName. Consider the following class:

public class AlternativeStringComparison {	
   public static int compareByLen(String str1, String str2) {
      return Integer.compare( str1.length(), str2.length());
   }
}

 
You can create a reference to its static method as follows:

@Test
public void staticLengthComparisonTest() {
		
   List<String> list = new ArrayList<String>();
   list.add("abcd");
   list.add("abcdef");
   list.add("ab");

   Collections.sort(list, AlternativeStringComparison::compareByLen);

   assertTrue(list.get(0).equals("ab"));	
}

 
Constructor methods can be also referenced using the notation Class::new. We will look at practical applications of constructor references in a later post when discussing Java 8 streams.

In summary, four types of method references have been introduced in Java 8:

Type Notation Notes
Reference to a bound instance method object::instanceMethodName  
Reference to an unbound instance method Class::instanceMethodName  
Reference to a static method Class::staticMethodName  
Reference to a constructor Class::new  

In the next post we will look at more Java 8 functional features. Feel free to leave any comments or suggestions. Thank you for reading.

Defining DevOps

Research and industry surveys clearly show that DevOps is one of the most ill-defined terms in the IT industry today, with definitions ranging from “improved collaboration between development, QA and operations” to “the shifting of operation tasks to the development team”.

Surprisingly however, there is a lot more consensus on what the objectives of DevOps are, so perhaps this could be used as a reliable starting point in the search for a coherent definition. In fact, most people would agree that DevOps has three key objectives:

  • Frequent releases of software to production
  • Increased software quality
  • Reduced production costs

It is a tall order, and one that can only be achieved through automation at three levels: infrastructure, deployment and testing. These are all intertwined and interdependent. Testing automation enables continuous delivery, and infrastructure automation enables the timely provisioning of runtime environments into which the software product is deployed and tested.

This automation effort is therefore what DevOps is all about, so it could be redefined as:

“The process of automating infrastructure provisioning, deployment and testing with the aim to deliver software to production frequently, with fewer defects and utilising fewer resources”

From this perspective, DevOps should not be considered a methodology by itself, but rather an enabling set of tools and practices to achieve Agile software delivery, as defined by the principles in the Agile Manifesto. Unlike other Agile processes like Scrum and XP, which emphasizes the human factor in software development without providing the tools for the job – with mixed results – DevOps clearly focuses on the latter – the tools – to empower Agile teams.

Towards Functional Java: an Introduction to Lambda Expressions

A major feature of the Java 8 release is the language-level support for lambda expressions, which brings Java into the realm of functional-style programming. Other object-oriented programming languages such as Scala, JavaScript and Python had lambda expressions built in from the start, and now Java has joined the rank.

Why is this interesting? Functional programming (FP) is based on the recursive evaluation of pure functions, whose output depends solely on the input arguments to the function, avoiding side-effects (i.e. changing the state of objects). This immutability results in code that behaves more predictably and performs better in concurrent and parallel systems. This is the first of a series of posts that will explore functional programming in Java, and the starting point is to understand lambda expressions (also known as closures or first-class functions).

So, what is a lambda expression? Put simply, it is a concise syntax to create a reference to a method implementation for later execution. In Java, lambda expressions are used in conjunction with functional interfaces. A functional interface is a plain old interface that defines just one single abstract method, and is marked with the @FunctionalInterface annotation.

It works like this: you define a variable of a functional interface type, and assign a lambda expression to it. This variable can then be passed around just like any other reference, but instead of containing a reference to an object, it contains a reference to a method implementation.

Confusing? Do not worry; it will all become clear with a few practical examples. By the way, to follow the examples in this post, you need a development environment that supports Java 8. If you don’t have one, you can follow this tutorial to get it setup.

For our first example, we will use the Runnable functional interface, whose API documentation can be found here:

https://docs.oracle.com/javase/8/docs/api/java/lang/Runnable.html

The Runnable interface has been around for a long time, and if you wrote multi-threaded programs in Java you will be familiar with it; it is used to define tasks for later execution in a separate thread.

If you look at the source code of Runnable, you will see that it is annotated with @FunctionalInterface. It is not mandatory to use the annotation, in fact any interface that declares a single abstract method can be used as a functional interface; but it is good practice to annotate it so the compiler will check that it is a valid definition.

@FunctionalInterface
public interface Runnable {
    /**
     * When an object implementing interface <code>Runnable</code> is used
     * to create a thread, starting the thread causes the object's
     * <code>run</code> method to be called in that separately executing
     * thread.
     * <p>
     * The general contract of the method <code>run</code> is that it may
     * take any action whatsoever.
     *
     * @see     java.lang.Thread#run()
     */
    public abstract void run();
}

 
By the way, if you don’t know where to find the source code of the Java library classes, this post will show you:

http://robertovormittag.net/learn-from-the-experts-a-look-at-the-java-source-code/

Enough talk. Let’s write some code. Create a Java console application on your IDE, and place this code in main()

public static void main(String[] args) {
   Runnable r = () -> {
   System.out.println("Hello lambda!");
   };

   Thread t1 = new Thread(r);
   t1.start();
}    

 

Code analysis

First, we have declared a variable r of type Runnable, and assigned a lambda expression to it. This automagically becomes the implementation of the run() method declared by the interface.

Next, we have built a Thread object passing to the constructor our Runnable implementation, and started the thread. If you run the app, the thread will run and the string “Hello from a lambda expression!” will be printed on the console.

Syntax analysis

Let’s now take a closer look at the syntax of the lambda expression. All lambda expressions consist essentially of three parts:

  • a set of parameters (in parenthesis)
  • an arrow (or rocket) operator
  • a body (with the method implementation)

In this case, because the run() method takes no parameters, we have an empty parenthesis at the start of the expression.

Let’s look at another example, this time with parameters, and implement the java.util.Comparator functional interface, defined here:

https://docs.oracle.com/javase/8/docs/api/java/util/Comparator.html

This interface declares the abstract method

int compare(T o1, T o2)

which compares its two arguments for order, and returns a negative integer, zero, or a positive integer as the first argument is less than, equal to, or greater than the second.

Note from the API documentation that Comparator also declares other non-abstract, static and default methods but that is OK; it still is a valid functional interface, as it declares only a single abstract method.

We will now implement this interface so that it compares two String objects based on length. Here is the code:

public static void main(String[] args) {

   Comparator<String> comp = (String str1, String str2) -> {
         return Integer.compare( str1.length(), str2.length());
   };

   int result = comp.compare("ab", "abc");
   System.out.println(result);
}    

 
Here we declare the variable comp of type Comparator<String> and assign it to the lambda expression, starting with the two String parameters in parenthesis, followed by the rocket operator, followed by the function body which returns a value. We use the Integer class utility method compare() to compare the length of the String arguments.

https://docs.oracle.com/javase/8/docs/api/java/lang/Integer.html#compare-int-int-

The variable comp becomes a reference to our method implementation, and can be used in a way similar to the way object references are used. In the next line, we use comp to call our implementation passing two arguments “ab” and “abc”, capturing the return value in the variable result, which is printed to stdout.

If you run the application, you will see the value -1 printed, as expected, since the length of “ab” is less than that of “abc”.

An even more concise syntax

If the compiler can work out the type of the method parameters (in this case String), it is not even necessary to specify them (this feature is called type inference). And, if the method body is limited to a single expression, you can omit the angle brackets {} and the return keyword.

So our implementation can be written even more concisely like this:

public static void main(String[] args) {

   Comparator<String> comp = (str1, str2) -> 
      Integer.compare( str1.length(), str2.length());

   int result = comp.compare("ab", "abc");
   System.out.println(result);
}    

 
If you run the new version of the app, you will get the same result. How much concise you want to be is simply a matter of personal preference and programming style. Personally speaking, I value more much more readability and clarity in the code than conciseness, but the choice is there.

We can make a more interesting use of our Comparator implementation by applying it to the List sorting algorithm in the Collections utilities:

https://docs.oracle.com/javase/8/docs/api/java/util/Collections.html#sort-java.util.List-java.util.Comparator-

This utility sorts the specified list according to the order induced by the specified comparator. Try the following code:

public static void main(String[] args) {

   Comparator<String> comp = (String str1, String str2) -> {
      return Integer.compare( str1.length(), str2.length());
   };

   List<String> list = new ArrayList<String>();
   list.add("abcd");
   list.add("abcdef");
   list.add("ab");

   System.out.println(list);
   Collections.sort(list, comp);
   System.out.println(list);
}    

 
Here, we pass comp (the reference to our implementation of Comparator<String>) as the second parameter to the Collections.sort method.

If you run the app, you can check from the console output that, after the call to Collections.sort, the list has been sorted according to our Comparator implementation.

We covered a lot of ground, it is time to summarize. This is what we have learned so far:

  • What is a lambda expression
  • What is a functional interface
  • How lambda expressions and functional interfaces work together
  • The syntax of lambda expressions
  • A couple of practical examples using threads and collection algorithms

In the next post we will look at method references. Feel free to leave any comments / suggestions. Thank you for reading.

Learn from the Experts: A Look at the Java Source Code

The best way to learn and improve coding skills in any programming language is to look at code written by expert programmers. The Java Development Kit (JDK) comes with the source code of the complete Java Runtime Library; it is located in the src.zip archive under the JDK installation directory.

java-src

The location of the source file archive in the JDK 7 installation

Obviously your main reference when using the Java library classes should be always the API documentation, which can be found online or downloaded separately from the JDK downloads website. Here is the URL for the Java 8 API:

https://docs.oracle.com/javase/8/docs/api/

However, it can be very instructive to take a look at the source code of some of the key classes in the library, such as those in the java.lang and java.util packages.

Using Eclipse to view the Java Runtime Library Source Code

First, make sure you have the JDK selected as the default installed JRE. To do this select from the Eclipse menu: Window → Preferences. The following dialog pops up:

Setting up the JDK as the default JRE in Eclipse Mars

Setting up the JDK as the default JRE in Eclipse Mars

Select Java → Installed JREs

If the JDK is not there, click [Add…] and follow the prompts to select your JDK installation directory. Then make this JRE default by selecting the checkbox next to it. Now click [Edit…]. The following dialog pops up:

Setting up the runtime library source attachment

Setting up the runtime library source attachment

Expand the rt.jar library (this is the runtime binary) and click on [Source Attachment…]

The source attachment configuration dialog

The source attachment configuration dialog

Browse to the external location of your src.zip and click [OK]. Apply and close all parent windows.

Now to view the source file of a library class, e.g. LinkedList, either select from the menu:
Navigate → Open Type or use the shortcut [CTRL]+[SHIFT]+[T]

The Eclipse Open Type dialog

The Eclipse Open Type dialog

Type the name of the class, or select it from the list. The source code will open in the Editor.

the Eclipse editor showing the source code for LinkedList.java

The Eclipse editor showing the source code for LinkedList.java

Happy studying!!

From SQL to NewSQL via NoSQL: A Database Odyssey

We often read about the “data explosion” phenomenon, but in parallel with that, there has also been a “database explosion” in recent years. Cassandra… MongoDB… NoSQL… NewSQL… the database technology landscape in 2015 can be daunting and confusing. From Oracle to VoltDB it has been a 30+ year’s journey, which we will revisit with the intent to better understand how this technology has evolved over time and what are the choices available today when designing the persistence layer of an enterprise software system.

Let’s begin by rewinding back the clock to where it all started.

The Dawn of SQL

odyssey011970: at IBM San Jose Research Lab, Edgar F. Codd conceptualized a model for data access consisting of a set of tables containing rows, with each row representing an entity of interest. Data can be normalized to eliminate duplication, and relationships between tables can be enforced by foreign keys. The relational model was born.

Codd’s research started from a simple premise: to be able to ask the computer for information, and then let the computer figure out how to retrieve it. No low-level knowledge of data structures should be required. His idea spawned the creation of an industry-standard computer language for working with relational databases: the Standard Query Language, or SQL.

At that time, computer memory on commodity hardware was a fairly limited resource, so the only option was to design relational database software as disk-based systems. This forced choice will have a significant impact on performance, as accessing data on disk is several orders of magnitude slower than accessing data in memory.

The first commercially available relational database management system (RDBMS) was Oracle, released in 1979 by Relational Software (now Oracle Corporation). Other notable early RDBMS commercial releases were Informix (1981), IBM DB2 (1983) and Sybase (1987).

Apart from supporting SQL and the relational model, the commercial RDBM systems also provide support for transactions, multi-user concurrent access, and failure recovery.

Simply put, a transaction consists of a collection of data updates that the RDBMS treats as a unit of work: all the updates within the transaction either succeed (commit) or fail (rollback). For example, in a financial application, if you move money from account A to account B, you want both account balances to be updated if all is well, or in case of an exception, no updates should occur. With correct use of transactions, the database is always left in a consistent state, with minimal effort on the part of the programmer. Any application with integrity constraints greatly benefits from transaction support.

Transactions, concurrent access and failure recovery help to build applications that are robust, accurate and durable, but at a cost. A large chunk of CPU cycles is spent by RDBM systems to perform the locking, latching and logging required to support them, slowing down performance.

However, the flexibility of the relational model and SQL, the robustness provided by transaction support and an adequate performance for most business applications have made RDBM systems very successful and the unchallenged persistence technology for more than two decades.

But something was about to change.

The Data Explosion and the arrival of NoSQL

2000: The evolution of the World Wide Web has created global-reaching organizations in the fields of Internet search, social media, blogging, collaboration and online retail which had to cope with massive amounts of data, growing at an unprecedented scale.

This scenario represented a challenge for the traditional SQL relational databases. To start with, RDBM systems were originally designed to run as standalone servers. A single machine cannot scale beyond the physical hardware limits of disk space, memory and processing power. What happens when you reach the limit? The only solution is to combine the resources of several machines together in a cluster.

A cluster is a set of computers that perform the same task, controlled and coordinated by software in such a way that it can be viewed as a single system from a client’s perspective.

Some of the traditional SQL RDBM systems (e.g. Oracle RAC, Microsoft SQL Server) can be clustered using shared-disk architecture. In this configuration the database relies on disk storage that is shared by all the computers in the cluster and accessed via the network. This configuration does offer a degree of scalability, but the disk subsystem still represents a single point of failure which limits availability.

The level of scale and availability required by the Web 2.0 and other Big Data use cases demanded clusters with shared-nothing architecture. In shared-nothing clusters, each machine uses its own local hardware resources (processor, memory and disk) and, as the name implies, it shares nothing with the other machines in the cluster. Requests from clients are routed to the machine that owns the resource.

The relational model used by the traditional SQL RDBMS also was not ideally suited to be distributed in a cluster, without heavy management from the application code outside the database.

Because of these limitations, some Web giants, primarily Google and Amazon, started to work on a new class of databases that would become known under the umbrella term of NoSQL with the following design goals:

  • Shared-nothing clustering
  • Fast access to large amounts of data
  • Data models that are cluster-friendly

The NoSQL family of databases did not adopt a standard data model or a standard query language. The only common denominator is that they are non-relational. In practice, the majority of NoSQL implementations use either an aggregate-oriented or a graph data model.

An aggregate, as defined in domain-driven design, is a collection of objects that we wish to treat as a unit for data manipulation and consistency management. For example, an with its associated , and objects could be treated as a single aggregate. Aggregates are used in key-value, column family and document data models.

The graph data model, on the other hand, allows the user to store entities and the relationship between those entities. It is suitable for models with a large number of small records and complex relationships.

Aggregates allow the fast retrieval of large chunks of data. In key-value databases for example, a single lookup for a given key it is all that is needed to retrieve an aggregate of any size. In a relational model to achieve this would typically require several queries or joins, and a much longer execution time.

To make data access even faster, the NoSQL designers also decided that it was OK to sacrifice consistency in exchange for performance, and adopted an “eventual consistency” solution, which means, in practice, that it is OK for most applications to accept inconsistent data in some nodes in the cluster for a period of time following a database update.

The pioneering work by Google and Amazon spawned a number of NoSQL implementations; some examples (by data model) include:

Key-Value databases:

Riak (Basho Technologies), Oracle NoSQL Database (Oracle Corporation), Redis (Pivotal Software)

Document databases:

MongoDB (MongoDB Inc.), CouchDB (Apache Foundation)

Column-Family databases:

Cassandra (Apache Foundation), HBase (Apache Foundation)

Graph databases:

Neo4j (Neo Technology), InfinitGraph (Objectivity Inc.)

The NoSQL databases support highly scalable and highly available architectures. Some implementations can efficiently store unstructured data, such as video and audio files, which are not handled very well by the traditional RDBM systems. With NoSQL, a wider range of data models became available, catering for specialized use cases unsuitable for relational models and SQL.

Despite these advantages, there are downsides. The trade-off between consistency and latency cannot be accepted by all applications; the lack of support for transactions places a big burden on the developers to circumvent it; and the inexistence of a standard query language requires specialized knowledge to extract data.

The NewSQL Era

odyssey03First used in 2011, the NewSQL designation refers to a new class of contemporary relational database management systems that have been designed to offer the same scalability of NoSQL stores by providing shared-nothing cluster ability, whilst at the same time maintaining support for transactions and the consistency guarantees of the traditional RDBMS SQL systems. This new class of databases aim to yield better performance by adopting new architectures that vastly reduce the overhead required to support transactions, consistency and failure recovery. For example, most implementations use replication instead of transaction logs to achieve high availability and durability, which is much more efficient.

By now, the amount of memory available on commodity hardware means that most NewSQL stores are implemented as main-memory databases, which vastly reduces data access times.

All NewSQL databases support the relational data model, as well as the standard SQL query language. To make the relational model more easily distributable over a cluster, most implementations provide a middleware layer to automatically split relational data across multiple nodes; a technique called sharding.

Some implementations of this class of databases are:

  • MySQL Cluster (Oracle Corporation)
  • Spanner (Google)
  • VoltDB (VoltDB Inc)
  • MemSQL (MemSQL Inc)
  • SQLFire (Pivotal Labs)

What Next?

2015: as the amount of data produced in the digital world will continue to grow at an exponential rate, it is reasonable to assume that database technology will continue to evolve at the same pace.

We left out of this discussion third-party hosted alternatives such as cloud databases and Database as a Service (DaaS). There are also multi-model configurations, which comprises of two or more database types coordinated by a common manager, and hybrid solutions where a caching technology (such as Oracle Coherence) creates an in-memory key-value layer on top of a traditional disk based RDBMS.

My aim is to follow up and report developments in this space. In another post, we will look at what considerations should be applied when selecting a persistence technology for your project.