Translate

Thursday, March 23, 2017

Java HashMap Search and Sort

A HashMap has plenty of uses, so let's see how you can find keys and values, load data from a CSV file into one, and use it for sorting.

Introduction

Having explored HashMap in several previous articles (here and here), let us now learn how to search and sort a HashMap by key as well as value.

Finding Keys and Values in HashMap

Finding a key in a HashMap is quite simple. The HashMap API provides the containsKey() method, which tells you whether the key exists.
Map<String,Integer> namefreq = new HashMap<>();
namefreq.put("Petra", 14);
namefreq.put("Mario", 11);
namefreq.put("Kasandra", 23);
namefreq.put("Charity", 18);
namefreq.put("Minerva", 5);
if ( namefreq.containsKey("Charity") ) {
}

Finding a value is also easy given the method containsValue().
if ( namefreq.containsValue(10) ) {
}

Pretty simple, right? Well, what if you need to find not a particular value but search for a general expression, such as names starting with say “A”. Gets a bit more involved.

Loading CSV Into HashMap

The following search and sort examples make use of a name frequency table, which is loaded from CSV using the following code. It streams lines from a CSV file, splits the line into fields, selects values for the “CA” state, and stores the value as a Map of (year => count). (The data is from US Census Bureau which publishes name frequency counts for each year.)
The code creates a multi-HashMap with the structure:
(name => ((year => count),
          (year => count)),
...

Pattern pattern = Pattern.compile(",");
String csvFile = "StateNames.csv";
try (BufferedReader in = new BufferedReader(new FileReader(csvFile));){
    Map<String,Map<Integer,Integer>> namefreq = in
    .lines()
    .skip(1)
    .map(x -> pattern.split(x))
    .filter(x -> x[4].equals("CA"))
    .collect(HashMap::new, (map, x) ->
         map.compute(x[1], (k, v) -> {
             if ( v == null )
                 v = new HashMap<Integer,Integer>();
             v.put(Integer.parseInt(x[2]),
                   Integer.parseInt(x[5]));
             return v;
             }),
         Map::putAll);
}

Here is a snippet of the sample data being loaded.
Id,Name,Year,Gender,State,Count
1,Mary,1910,F,AK,14
2,Annie,1910,F,AK,12
3,Anna,1910,F,AK,10
4,Margaret,1910,F,AK,8
5,Helen,1910,F,AK,7
6,Elsie,1910,F,AK,6
7,Lucy,1910,F,AK,6
8,Dorothy,1910,F,AK,5
9,Mary,1911,F,AK,12
10,Margaret,1911,F,AK,7

Search Key in HashMap

The following code searches for a key ending with “x” and prints the matches.
namefreq
    .entrySet()
    .stream()
    .filter(e -> e.getKey().endsWith("x"))
    .forEach(e -> {
        System.out.println(e.getKey() + " :");
        e.getValue().forEach((kk, vv) -> {
            System.out.println(" " + kk + " => " + vv);
        });
    });
Rex :
 1959 => 86
Margaux :
 2003 => 8
Maxx :
 2010 => 28

Search Value in HashMap

Let us now search for a value in the HashMap using the same method. Before we do that, let us total the names for each year and store it under the key 0 for each name. Something like this:
Elza :
  0 => 10
  1993 => 5
  2014 => 5
...

We compute the total value with the following code. It computes a total of the existing mappings of (year => count) and stores it under key 0.
namefreq
    .entrySet()
    .stream()
    .forEach(e -> {
        Map<Integer,Integer> v = e.getValue();
        int tot = v
        .entrySet()
        .stream()
        .reduce(0, (x, ee) -> x + ee.getValue(),
            (x, y) -> x+y);
        v.put(0, tot);
    });

Let us now search the HashMap for names occurring more than 1000 times.
namefreq
    .entrySet()
    .stream()
    .filter(e -> e.getValue().getOrDefault(0, 0) > 1000)
    .forEach(e ->
         System.out.println(e.getKey()+" : "+e.getValue().get(0)));
...
Solomon : 1967
Javier : 28472
Esther : 16974
Lenora : 1261
Sam : 6971
Lenore : 1261
Rowan : 1297
Lukas : 2888
...

The actual search is being performed by the following segment:
.filter(e -> e.getValue().getOrDefault(0, 0) > 1000)

Here, you can use an expression of arbitrary complexity to search for exactly what you need. In the following example, we search for key containing the string “mon” and total value more than 1000. The results are stored in another HashMap (for further processing).
Map<String,Map<Integer,Integer>> subset = namefreq
    .entrySet()
    .stream()
    .filter(e ->  e.getKey().contains("mon")&&e.getValue().getOrDefault(0, 0) > 1000)
    .collect(HashMap::new,
         (m, e) -> m.put(e.getKey(), e.getValue()),
         Map::putAll);
subset.forEach((k, v) -> System.out.println(k + " : " + v.get(0)));
Simone : 3114
Desmond : 2498
Ramona : 8139
Raymond : 60506
...

Sort HashMap by Key

To sort a HashMap by key and print the mappings, we can do something like this.
namefreq
    .entrySet()
    .stream()
    .sorted((x, y) -> x.getKey().compareTo(y.getKey()))
    .forEach(e -> {
        System.out.println(e.getKey() + " :");
        e.getValue().forEach((kk, vv) -> {
            System.out.println(" " + kk + " => " + vv);
        });
    });
Lylia :
 0 => 6
 2009 => 6
Lyliana :
 0 => 15
 2007 => 5
 1998 => 5
 1999 => 5
...

To store the result (sorted map), we use a LinkedHashMap (which preserves the insertion order) as follows.
Map<String,Map<Integer,Integer>> subset = namefreq
    .entrySet()
    .stream()
    .sorted((x, y) -> x.getKey().compareTo(y.getKey()))
    .collect(LinkedHashMap::new,
         (m, e) -> m.put(e.getKey(), e.getValue()),
         Map::putAll);
subset.forEach((k, v) -> System.out.println(k + " : " + v.get(0)));
Byanca : 90
Byanka : 79
Byran : 65
Byron : 7254
Cache : 10
Cadance : 30
Cade : 1874
...

Sort HashMap by Value

Sorting the HashMap by value is very similar to the above example. The following code sorts the name-count HashMap by total-count and prints the top 20 names.
namefreq
    .entrySet()
    .stream()
    .sorted((x, y) ->
        y.getValue().get(0).compareTo(x.getValue().get(0)))
    .limit(20)
    .forEach(x -> System.out.println(x.getKey() + " => " +
                     x.getValue().get(0)));
Michael => 422157
David => 364853
Robert => 347637
John => 310120
James => 274168
Daniel => 244229
Richard => 222633
Christopher => 215728
William => 209173
Anthony => 174064
...

Summary

Searching for a key in a HashMap involves applying a filtering pipeline to the entries. The same method is also applicable to search for the HashMap values. In fact, arbitrarily complex expressions can be used with such a pipeline for search. We also learned how to sort the HashMap by keys or values and possibly store the results in a LinkedHashMap.

See Also



Source: Java HashMap Search and Sort 

The Programmer's Way to Convert Excel to CSV (With UTF-8)

Excel provides a way to save spreadsheets as a CSV file, but it seems to fail at handling UTF-8 characters. See how a real programmer deals with the problem.


· Java Zone

CSV stands for Comma-Separated-Values and is a very common format used for exchanging data between diverse applications. While the Excel Spreadsheet file format is complex (since it has to accommodate a lot more!), CSV is a simpler format representing just tabular data.
In this article, we show you a way of exporting the data from an Excel spreadsheet to CSV. We use the Apache POI library for the purpose.

But Why?

Excel directly provides for exporting CSV data using the Save As functionality in the File menu. While it gets the job done in most cases, it leaves something to be desired.
Specifically, if your spreadsheet contains Unicode data, you are out of luck. It appears that Excel uses Windows default character set Windows-1252 (or cp-1252) to perform the export. This character set is a very limited set and cannot represent characters from most foreign languages. This leaves your CSV output from Excel severely broken if it contains such characters.
The proper solution is to use Unicode (specifically UTF-8) for encoding the CSV file so all your data is preserved.
It is indeed a surprise that the Excel team has not yet figured out in 2017 that people may have Unicode data that need to be exported properly to CSV.
Maybe the Excel developers have not read this article by Joel Spolsky stressing the need for programmers to be aware of Unicode. Ironically, Joel was (until 1995?) the project manager for the Excel team. Joel penned the article in 2003, long after leaving the Excel group.

Using Apache POI

We use Apache POI to read the Excel spreadsheet. Building the program requires this dependency to be declared in pom.xml (assuming Maven for the build) as follows:
<dependency>
  <groupId>org.apache.poi</groupId>
  <artifactId>poi-ooxml</artifactId>
  <version>${poi.version}</version>
</dependency>

Take 1 – The Naive Approach

The first cut of the approach to convert Excel spreadsheet data to CSV is shown in the block below.
Workbook wb = new XSSFWorkbook(new File(xlsxFile));
DataFormatter formatter = new DataFormatter();
PrintStream out = new PrintStream(new FileOutputStream(csvFile),
                                  true, "UTF-8");
for (Sheet sheet : wb) {
    for (Row row : sheet) {
        boolean firstCell = true;
        for (Cell cell : row) {
            if ( ! firstCell ) out.print(',');
            String text = formatter.formatCellValue(cell);
            out.print(text);
            firstCell = false;
        }
        out.println();
    }
}

Use the UTF-8 BOM

There are several problems with the code above. While the code above correctly outputs UTF-8 and encodes characters properly, Excel cannot load the generated CSV file. The reason is that Excel needs the Byte-Order-Marker to indicate that the file is encoded in UTF-8. With that modification, the code is now:
PrintStream out = new PrintStream(new FileOutputStream(csvFile),
                                  true, "UTF-8");
byte[] bom = {(byte)0xEF, (byte)0xBB, (byte)0xBF};
out.write(bom);
for (Sheet sheet : wb) {
...

Exporting Formulas

When Excel exports CSV, it evaluates all the formulas and writes out the data. In the code below, we have an option of exporting the formula itself (starting with a "=") to CSV, or evaluating the formula and exporting the result.
Workbook wb = new XSSFWorkbook(new File(xlsxFile));
FormulaEvaluator fe = wb.getCreationHelper().createFormulaEvaluator();
...
if ( fe != null ) cell = fe.evaluateInCell(cell);
String value = formatter.formatCellValue(cell);
if ( cell.getCellTypeEnum() == CellType.FORMULA ) value = "=" + value;
...

Escape Quotes and Commas

When exporting a field value, it is necessary to properly escape certain characters such as double quotes, commas, and line separators. This is handled as follows:
static private Pattern rxquote = Pattern.compile("\"");
static private String encodeValue(String value) {
    boolean needQuotes = false;
    if ( value.indexOf(',') != -1 || value.indexOf('"') != -1 ||
         value.indexOf('\n') != -1 || value.indexOf('\r') != -1 )
        needQuotes = true;
    Matcher m = rxquote.matcher(value);
    if ( m.find() ) needQuotes = true; value = m.replaceAll("\"\"");
    if ( needQuotes ) return "\"" + value + "\"";
    else return value;
}

Including Empty Rows and Cells

When a formula is exported as-is, it needs the cell references to remain intact. We achieve this by exporting empty rows and cells to CSV to maintain these references.
To ensure that empty rows and cells are also output to CSV, we use row and cell numbers explicitly since the for-each loop shown above skips empty rows and cells.
Sheet sheet = wb.getSheetAt(sheetNo);
for (int r = 0, rn = sheet.getLastRowNum() ; r <= rn ; r++) {
    Row row = sheet.getRow(r);
    if ( row == null ) { out.println(','); continue; }
    for (int c = 0, cn = row.getLastCellNum() ; c < cn ; c++) {
        Cell cell = row.getCell(c,Row.MissingCellPolicy.RETURN_BLANK_AS_NULL);
    }
}

Export Specified Sheet Only

If the spreadsheet contains multiple sheets, a sheet number must be explicitly specified for export.
Sheet sheet = wb.getSheetAt(sheetNo);

The Final Cut

The core exporting program segment with all these updates now looks like this.
Workbook wb = new XSSFWorkbook(new File(xlsxFile));
int sheetNo = Integer.parseInt(args[index++]);
FormulaEvaluator fe = null;
if ( index < args.length ) {
    fe = wb.getCreationHelper().createFormulaEvaluator();
}
DataFormatter formatter = new DataFormatter();
PrintStream out = new PrintStream(new FileOutputStream(csvFile),
                                  true, "UTF-8");
byte[] bom = {(byte)0xEF, (byte)0xBB, (byte)0xBF};
out.write(bom);
{
    Sheet sheet = wb.getSheetAt(sheetNo);
    for (int r = 0, rn = sheet.getLastRowNum() ; r <= rn ; r++) {
        Row row = sheet.getRow(r);
        if ( row == null ) { out.println(','); continue; }
        boolean firstCell = true;
        for (int c = 0, cn = row.getLastCellNum() ; c < cn ; c++) {
            Cell cell = row.getCell(c, Row.MissingCellPolicy.RETURN_BLANK_AS_NULL);
            if ( ! firstCell ) out.print(',');
            if ( cell != null ) {
                if ( fe != null ) cell = fe.evaluateInCell(cell);
                String value = formatter.formatCellValue(cell);
                if ( cell.getCellTypeEnum() == CellType.FORMULA ) {
                    value = "=" + value;
                }
                out.print(encodeValue(value));
            }
            firstCell = false;
        }
        out.println();
    }
}

Summary

Exporting data in Excel spreadsheet to CSV is quite simple; it is also provided directly in Excel. However exporting Unicode data within the spreadsheet is not done correctly by Excel. In this article, we covered some issues arising from this process including exporting formulas, maintaining cell references and proper escaping of data.

Source: DZone The Programmer's Way to Convert Excel to CSV (With UTF-8)

Wednesday, March 22, 2017

Immutables in Java

This in-depth look at mutability examines the pros and cons of using immutable classes as well as how to protect your mutability.


· Java Zone

In this article from my free Java 8 course, I will be discussing immutables in Java.

The concept of immutability has always been important in all programming languages, including Java. With the release of Java 8 however, immutables have become even more important. This version introduced functional programming, as well as the new java.time API. Both rely heavily on immutables.

What Is an Immutable?

An immutable class is a class whose instances cannot be modified. Information stored in an immutable object is provided when the object is created, and after that, it is unchangeable and read-only forever. As we can’t modify immutable objects, we need to work around this. For instance, if we had a spaceship class, and we wanted to change its location, we’d have to return a new object with modified information.
public Spaceship exploreGalaxy() {
    return new Spaceship(name, Destination.OUTER_SPACE);
}
Example 1

Advantages of Immutables

At first glance, you’d think that immutables were useless, however, they provide many advantages.
Firstly, immutable classes greatly reduce the effort needed to implement a stable and fault tolerant system. The property of immutables that prevents them from being changed is extremely beneficial when creating this kind of system.

Imagine that we have a Bank class that we are making for a very big bank. After the financial crisis, the bank is afraid of allowing their users to have a negative balance. So they institute a new rule and add a validation method to throw an IllegalArgumentException whenever a function call results in a negative balance. This type of rule is called an invariant.
public class BankAccount{
[...]
    private void validate(long balance) {
        if (balance < 0) {
            throw new IllegalArgumentException("balance must not be negative:"+ balance);
        }
    }
}
Example 2
In a typical class, this validate() method would be called anytime a user’s balance is changed. If the user makes a withdrawl, pays their debt, or transfers money from their account we would have to call the validate method. However, with an immutable class we only have to call the validate method once, in the class constructor.
public BankAccount(long balance) {
    validate(balance);
    this.balance = balance;
}
Example 3
Since an immutable object will never change, this condition holds true for the entire lifetime of the object. Further validation will not be needed. Whenever a method that modifies the balance is called, a new object is returned, calling the constructor again and revalidating the object. This is extremely useful as it allows us to centralize all our invariants and guarantee that objects are consistent for their entire lifetime.
Similarly, immutables can be used to support a fault tolerant system. Imagine that you try to withdraw money from a bank, but between the time your money is withdrawn from your account and the money is released from the ATM, there is some error. In a normal class, your money would be gone forever. The account object was changed, too late. But in an immutable class, you can throw an error, preventing your account from losing money before you physically receive it.
public ImmutableAccount withdraw(long amount) {
    long newBalance = newBalance(amount);
    return new ImmutableAccount(newBalance);
}
private long newBalance(long amount) {
    // exception during balance calculation
}
Example 4
An immutable object can never get into an inconsistent state, even in the case of an exception. This stabilizes our system and removes the threat of an unforeseen error destabilizing an entire system. This stability comes at no cost, apart from the cost of the initial validation.
A second advantage of immutables is that they can be shared freely between objects. Let’s say we create a copy of our account object. When we copy the object, we leave both objects sharing the same balance object.

Example 5
However, since the objects are both immutable, if we change the balance in one of the objects it doesn’t affect the other object. The other object creates a new immutable instance of the class and the two objects are unaffiliated.
Example 6
For the same reason, an immutable does not need a copy constructor when copying an object. Immutables can even be shared freely when using a lock-free algorithm in a multithreaded environment, where multiple actions happen in parallel.
Finally, immutable objects are also a perfect option to use as Map keys and Set elements, since Map keys and Set elements must never change.

Disadvantages of Immutables

As you’ve probably realized, the rigidness of immutables can be a huge asset, but it can also be a disadvantage. The biggest weakness of immutables is their potential to cause performance problems. Every time you have a new state of the class, you need to create a new object. Because of this, you usually need to create a lot more immutable objects than you would need to create mutable objects. Logically, the more objects you create, the more system resources you use.
This might be a problem, or it might not, it depends on a variety of factors. What are you trying to do? What kind of hardware is your program running? Are you building a desktop or a web application? How big is your program? The combination of these factors determines whether or not making your class immutable will cause performance issues. Typically, you should try to utilize immutables as much as possible. Start by making every class immutable and facilitate their immutability by creating small classes with simple methods. Simplicity and clean code are key. If you have clean code, that facilitates immutability; and if you have immutables, your code is cleaner. Once you have your program, test it. See how it is performing and if performance is not satisfactory, gradually relax the immutability rules.

How to Make an Immutable

Now that I’ve shown you why immutables are valuable and when you should use them, I’m going to show you how to make an immutable class. Let’s go through turning our mutable class Spaceship into an immutable class:
public class Spaceship {
    public String name;
    public Destination destination;
    public Spaceship(String name) {
        this.name = name;
        this.destination = Destination.NONE;
    }
    public Spaceship(String name, Destination destination) {
        this.name = name;
        this.destination = destination;
    }
    public Destination currentDestination() {
        return destination;
    }
    public Spaceship exploreGalaxy() {
        destination = Destination.OUTER_SPACE;
    }
[…]
}
Example 7
To make a mutable class into an immutable class you should follow four steps:
  1. Make all fields private and final
  2. Don’t provide any methods that modify the object’s state
  3. Ensure that the class can’t be extended
  4. Ensure exclusive access to any mutable fields

Make All Fields Private and Final

The first step in making a mutable class immutable is changing all of its fields to be private and final. We make variables private so that they cannot be accessed from outside the class. If they were accessible from outside the class, they could be changed. We also make fields final to clearly convey that we never want them to be reassigned within the class. Should someone try to reassign a reference variable, a compiler error will occur.
private final String name;
private final Destination destination;
Example 8

Don't Provide Any Methods That Modify the Object's State

The next step is to project our object’s state from being modified. As we’ve talked about before, immutables, by definition, cannot have their object’s state modified. Whenever you have a method that would modify an object’s state, you instead have to return a new object.
public ImmutableSpaceship exploreGalaxy() {
    return new ImmutableSpaceship(name, Destination.OUTER_SPACE);
}
Example 9
Any fields that we aren’t changing, such as name, can be directly copied from our current object. This is because, as I explained before, immutable objects can share fields freely. Fields that we do change need to be initialized as new objects.

Ensure That the Class Can't Be Extended

To prevent our class from being changed, we also need to protect our class from being extended. If a class can be extended we could override methods in the class. An overridden method could modify our object, which violates the rules of immutability. Let’s look at an example in code:
public class EvilSpaceship extends Spaceship {
    [...]
    @Override
    public EvilSpaceship exploreGalaxy() {
        this.destination = Destination.OUTER_SPACE;
        return this;
    }
}
Example 10
To stop this EvilSpaceship from destroying our sacred immutability, make the class final.
public final class Spaceship
Example 11

Ensure Exclusive Access to Mutable Fields

The final step in ensuring immutability is to protect our access to mutable fields. Remember, immutable fields can be shared freely, so it doesn’t matter if we have exclusive access. Anyone who holds access to a mutable field can alter it, thereby modifying our immutable object. To prevent anyone from directly accessing a mutable field, we should never obtain or return a direct reference to a Destination object. Instead we have to create a deep copy of our mutable object and work with that instead. As long as the mutable object is never directly shared, a change inside an external object will not have any effect on our immutable object. To achieve exclusive access we have to check all public methods and constructors for any incoming or outgoing Destination references.
The public constructor does not receive any Destination reference. The Destination object it creates is safe, as it cannot be accessed from outside. So the public constructor is good as it is.
In our currentDestination() method, however, we return our Destination object, which is a problem. Instead of returning the real reference, create a deep copy of our Destination object and return a reference to the copy.
public Destination currentDestination() {
    return new Destination(destination);
}
Example 12
The final public method we still have is the newDestination() method. It receives a Destination reference and directly forwards it to our constructor. This means that it is referencing the same object as whatever called this method. To guard against this, we can either make a deep copy within this method, or make a deep copy in our constructor. I’m going to implement this change in the constructor:
private ImmutableSpaceship(String name, Destination destination) {
    this.name = name;
    this.destination = new Destination(destination);
}
Example 13
It is better to make this change inside the private constructor, because now, if we create other methods that modify the destination, they also will automatically make a deep copy of this mutable field.
Now, we’ve completely ensured the immutability of our class:
public final class ImmutableSpaceship {
    private final String name;
    private final Destination destination;
    public ImmutableSpaceship(String name) {
        this.name = name;
        this.destination = new Destination("NONE");
    }
    private ImmutableSpaceship(String name, Destination destination) {
        this.name = name;
        this.destination = new Destination(destination);
    }
    public Destination currentDestination() {
        return new Destination(destination);
    }
    public ImmutableSpaceship newDestination(Destination newDestination) {
        return new ImmutableSpaceship(this.name, newDestination);
    }
[…]




}
Example 14
I hope you now have a better understanding of immutables, and can see just how useful they are. Remember to keep your code simple and clean, by using immutables to the greatest possible extent.
If you enjoyed this article — check out my free Java 8 clean code course!

Source: Immutables in Java