New article: file system notifications

A new article has been added on file system notifications. The article explains how your Java app can ask the underlying O/S to be notified of modifications to files in particular directories, e.g. for monitoring log files, watching for files created by an external process, or files opened by your application.

The article also looks at some of the limitations and pitfalls of using Java's WatchService API.

Additional information on the 'final' keyword

The section on the Java final keyword has been expanded. Previously, we concentrated on the use of 'final' for thread-safety but did not give much information about a separate use of 'final' to indicate that a class or method cannot be overridden. The new material expands upon this use with a look at the performance implications of the 'final' keyword as a class or method modifier.

A common view that I have both heard among colleagues and seen in various Java textbooks is that 'final' is as much a performance hint to the compiler as a specification of design. In the new section, I present some data showing that this view is probably misguided: the timing of calls to methods to final vs non-final classes comes down to whether the JIT compiler can determine at runtime the precise subclass of an object rather than whether or not the class in question is or may be overridden.

New material: doing maths in Java

The section on mathematical operations in Java has been expanded over the last few days to include some new material that will be useful to those writing maths-related applications in Java. Notable inclusions include:

• The section on the performance of java.util.Math methods has been expanded somewhat to include more information about the timing and implementation of certain operations.
• A new section on BigDecimal and BigInteger looks at how to manipulate arbitrarily large numbers (or numbers of arbitrary precision) in Java. We include an analysis of the performance of multiplication.

As usual, suggestions for improvements to this section or suggestions for new material are always welcome, either on this blog or on the Javamex forum.

The performance impact of object orientation

Today's new Javamex article looks at the performance impact of using an object-oriented approach vs an optimised, non-object-oriented approach in a case where the OO approach requires high object throughput. Specifically, we look at the case of calculations with complex numbers.

The basic tenet is that the most intuitive program design is to have a complex number encapsulated as a Java object, and furthermore, as an immutable object. This means that at each individual step in a calculation, a brand new object is created. However, performance measurements suggest that using immutable objects is only around 28% slower than using mutable objects, whilst the codability and maintainability impact (threading issues; bugs introduced by forgetting to copy objects and hence making accidental modifications to an obejct) may be considerable.

In our example, a version of the algorithm that does away with encapsulation and "inlines" its calculations on raw double primitives is more than twice as fast, as perhaps might be expected. On the other hand, "only" being between twice and three times as fast means that a comparable improvement will be possible in some applications by making use of more processors, or potentially making other algorithm improvements, whilst the codability and maintainability implications of not using encapsulation become more severe for more complex applications.

When the simple gets complicated: XPath from applets

Those who have read my introduction to XML will be familiar with my mixed feelings on this "technology". In principle, the idea of a standardised, human-readable data format is a handy one. Just in practice, the XML frameworks that are supposed to take the work out of reading this data end up being fundamentally infuriating for some reason.

Of the irritating options available, the XPath API is about the least tedious. Essentially, the idea is that you can call an evaluate() method, passing in a "path" into the XML document by which you refer to elements (data) in the document. Java's standard implementation is stupidly faffy, making you mess around with document builders and configuration exceptions and god knows what other nonesense that you really shouldn't have to care about. But as I say, of the various frustrating options available, it's still about the least frustrating.

At least, until you come to use it in an applet. Picture the scene: my beautiful XML-consuming applet nearing completion and tested locally, I excitedly gmail my colleague to let him know I'll have it up on the web for testing shortly. Then imagine my dismay when the selfsame applet uploaded to the web fails to initialise. Half an hour or so of frustrating debuggery later, it turns out that the culprit is the XPath.evaluate() method, inside whose gubbinery a network fetch is being made for some spurious property on every single attempt to read a piece of data from an XML document. Vive la pluggable architecture.

As I explain on the page, the workaround I found for this was to explicitly set the system property being looked for to its default value via a call to System.setProperty(). For this, alas, the jar must be signed. But I decided that even signing the jar would be less frustrating than re-writing my code to avoid the XPath API. That's how desperate I was by that stage.

With the jar signed, my colleague is now free to accept the warning message about the apocalyptic consequences of running my applet and proceed with testing. Ginger programmer 1, frustrating API nil. For now.

Random numbers in Java

A few updates have been made to the site's section on generating random numbers in Java. Random numbers can crop up in all sorts of applications, and it's worth having a good understanding of them.

Traditionally, the bread-and-butter means of random number generation has been the java.util.Ranom class. The technique it uses (see the section on the java.util.Random algorithm for more details) is still suitable for some very casual applications of random number generation, such as some simple games. But in many cases, you should probably be thinking of moving away from java.util.Random towards a higher quality generator. As discussed in the section, problems with java.util.Random include its low period, biases in the different bits of the numbers generated, and its unsuitability for generating combinations of values. Using a weak random generator can have side effects such as the following:

• it can skew the results of a test harness that introduces subtle biases in the code path due to the random number generator;
• when testing the performance of data structures, and in various simulations, a generator such as java.util.Random will not produce a good range of possible combinations of values, giving false results;
• the series of numbers can be predictable, leading to disasterous results if used as the basis for security (e.g. generating a random encryption key, nonce or session ID).

One interesting technique, published in 2003 by Goerge Marsaglia, is the XORShift generator. This generates medium-quality random numbers in a few simple machine instructions. For example, given a Java long, x, we can generate the next random number as simply as follows (in fact, various combinations of shift values are possible, as discussed in Marsaglia's paper):

  x ^= (x << 21);
  x ^= (x >>> 35);
  x ^= (x << 4);

continually executing these three lines will cycle through all (2^64)-1 possible values of x (i.e. all values of a long except zero) in pseudorandom order. Initialising x with the value of System.nanoTime(), or some other "random" seed, gives us a fast, medium-quality generator suitable for, say, generating random game data. It will make an excellent choice in many J2ME games, for example.

We also consider a Java implementation of the combined generator suggested by the Numerical Recipes authors, which generates numbers within a similar order of execution time as java.util.Random, but with a higher period and quality.

For applications where security depends on the quality and unpredictability of the random numbers generated, Java's SecureRandom class provides cryptographic strength random numbers, though is some 20-30 times slower than the other techniques.

New article: how to choose a Java collection

A new addition to the Java Collections section of the Javamex site looks at a question that, perhaps unsurprisingly, crops up fairly frequently: which collections class should you use for a given task? The various collection classes provide a powerful means of managing objects and data in memory, but they're so powerful that choosing between them can sometimes be a bit daunting.

The approach that I take is to split the question firstly into two sub-questions. Firstly, what is the general type of structure that you need? That is, how is the data basically going to be organised? Usually, there is not too much confusion between a list and a map, especially if you consider whether or not you need to answer the question "for a given X, what is the Y"? But the difference between a set and a list is sometimes not well understood, or at least, not considered. With a little bit of thought, it is usually possible to decide in advance whether the purpose of your collection is to decide "is something there or not"? The trick is for programmers to remember to ask that question in the first place, and not simply plump for a list regardless.

Then, having decided on a list, map, set or queue, the next subquestion is obviously which particular flavour is required. In the case of a list, the cases where you wouldn't plump for an ArrayList are relatively uncommon and it should be clear in your head if you do choose something else that you have a "special case".

But in the case of maps, sets and queues, there are definitely more "horses for courses". But, especially in the first two, there are essentially two factors to consider: concurrency and ordering. As in the article, when the various choices are presented in tabular form according to these criteria, things become a little easier. As mentioned, a key rule of thumb is to choose the class that provides the minimum features that you require. If you don't need ordering, don't pay for it.

Java queues present a slightly complex set of choices, but in at least some cases-- especially a DelayQueue or SynchronousQueue-- it should be really clear that you need that class for a special case scenario. It should also be borne in mind that the executors framework means that in many common producer-consumer scenarios, you don't explicitly have to deal with the underlying job queue.

As usual, further questions and comments to any of the articles on Javamex are welcome on either this blog or the associated Java forum.

Java on your Kindle

It's excellent to see an array of Java programming books now available on the Kindle. Various of these books, which you may not have considered buying due to their cost, are available at a sometimes significantly reduced price on the Kindle.

Of notable interest to Javamex readers will be Brian Goetz's venerable Java Concurrency in Practice, which is really something of a bible for information on the Java Memory Model and the Java 5 concurrency library. As I say in the review, I can pretty much guarantee that if you're writing concurrent code (as many of us either are or will soon have to, not just on the server, but increasingly client-side), then Java Concurrency in Practice will allow you to fix various bugs in your code that you were possibly unaware of (it certainly fixed some in mine!), as well as help you make decisions about how to architecture your program around the Java 5 concurrency utilities.

For a more light-hearted read, but still an extremely enlightening one, Java Puzzlers remains excellent as ever. Josh Bloch's Effective Java got a whole new lease of life when its second edition was published, and its addition to the Kindle repertoire is most welcome.

Also worthy of note are the Core Java books. These, and in particular the second volume of Advanced Features, is not the most succinct of works, but covers various Java topics that you don't readily see explained in detail in other books.

Java for beginners turorial: what would you like to see?

Readers of the Javamex web site may or may not have seen the first pages of the site's Java for beginners tutorial, which covers a number of very basic Java topics such as variables, control structures (for loops, if/else), plus information on Java arrays.

Various topics are going to be added to the tutorial over the coming weeks. But how do you think it should be expanded? Are you just starting out in Java and having trouble with a particular topic? Or are you an experienced Java programmer, but remember having particular trouble with some aspect of learning Java with the tutorials that you used? Any feedback is welcome on this blog entry.

The secret life of the Java 'final' keyword

I'll freely admit that an omission that the Javamex site should have addressed sooner is a more explicit discussion of the Java final keyword. Its use for guaranteeing thread-safety is hinted at in some of the articles, but some more explicit information has been long overdue.

As a step towards addressing this gap, the aforementioned article looks at how, as of Java 5, the final keyword has acquired a very important characteristic for thread-safe programming. Most programmers think of final in terms of its impact on program design or presumed optimisation (which, as I'll mention in a moment, is probably a red herring). But its thread-safety guarantee is far more significant: any field declared as final can be safely accessed by another thread once the constructor completes, whereas, subtly, without this or any other thread-safety mechanism, that guarantee does not hold.

The final keyword and optimisation

An issue with the Java final keyword, also criticised by Josh Bloch and Neal Gafter in their excellent Java Puzzlers book, is that it means different things in different places. When applied to a class or method, it means that that class or method cannot be extended or overridden. This has led to a general (false) conception that "final is to do with optimisation", and its importance in thread-safety has been overlooked.

Even when applied to classes and methods, the notion that final is about optimisation is probably false in most cases. The argument appears to stem from languages such as C++, where "compilation" is a one-off process. In such languages, there are optimisations that you can make to method calls and field accesses if you know that the classes and fields in question will never be overridden. But when you're running in a VM, whether a class or method might "potentially" be overridden doens't really matter. What counts is whether it has been overridden at a given moment. It it hasn't at the point of JIT-compilation, and the JVM makes certain optimisations based on that observation, but then later the class/method is overridden, the JVM generally has the luxury (which a C++ compiler or linker generally doesn't) of being able to re-compile.

So when applied to classes and methods (and in fact, local variables inside methods), final is essentially a program design feature. When applied to instance and class variables, final is an important thread-safety mechanism.

CyclicBarrier

A new section of the Javamex web site looks at the CyclicBarrier class. In case you haven't come across it, CyclicBarrier is a construct that makes coordinating threads easier. Unlike CountDownLatch, which is designed for "one-off" coordinated actions, CyclicBarrier is designed to be re-used, so that it is suitable for iterative processes where the participating threads need to repeatedly perform a parallel operation, but periodically "converge" so that their results can be amalgamated.

In the article, we discuss the example of using a CyclicBarrier to coordinate a parallel sort algorithm. Essentially, the sort takes place in three stages. Each of the stages occurs in parallel, but at the end of each stage, a small single-threaded step must take place to amalgamate the results of the previous parallel operation.

Essentially, the code for a worker thread involved in the operation repeatedly carries out an operation and then calls the barrier's await() method. The latter method blocks until all participating threads have also called await() (we sometimes call this "arriving at the barrier"). At that point, CyclicBarrier executes our pre-determined "amalgamation" routine on one of the threads (the last one to arrive at the barrier, in fact), and then relases all the threads to move on to the next stage. Overall, the class takes out a lot of the actual thread coordination work, although, as we discuss in the article, we must still think about regular concurrency issues such as data synchronization and lock contention.

An additional feature of CyclicBarrier which we discuss is that it handles propagation of interruptions to all participating threads. In other words, if any thread involved in the operation is interrupted, then the whole operation will cease once all threads have called the await() method.

Whilst definitely one of the lesser used of the Java 5 concurrency utilities, CyclicBarrier is definitely a useful class that should not be overlooked.

New content

A few new pages have been added to the site that you may be interested in:

• The section on Java cryptography now considers password-based encryption, which we not too surprisingly conclude is fraught with difficulties! At present, we look at the PBE algorithms provided as standard in Sun's Java 6 implementation, although we unfortunately conclude that none of them are terribly great!
• The section on Swing User interfaces covers various common topics, including a "rogue's gallery" of common Swing components, with a list of common constructors and listeners for those components, plus some hints on adding listeners to your code.

Further pages are planned for both of these sections very shortly. Watch this space!

Arcmexer: a library for reading archive files in Java

A beta version of the Arcmexer library is now available from the Javamex site. Arcmexer is a library that allows you to read the contents of various types of archive file from Java. The idea is that the library could be useful in various data conversion and data recovery operations.

At present, the following are supported:

• ZIP files, including those with files encrypted with 128-bit AES encryption or the traditional (but insecure) PKZIP encryption scheme. Note that other encryption algorithms (including 256-bit AES) currently aren't supported but may be in the future if I'm told that people need them. (See the page on reading encrypted ZIP files.)
  
• Tar files, commonly used on UNIX platforms. Tar files may come directly off a tape drive (usually via the UNIX dd command), or are created on disk via the tar command and used to transport bundles of files.
• GZIP-compressed tar files, commonly with the ending .tar.gz or .tgz.

Other archive formats are likely to be added to future versions if people nag me about them. (That means, leave a comment on this blog entry asking for it!)

As well as reading files from the archive, a method is provided that can aid in ZIP file password recovery.

Please bear in mind that this should very much be considered a beta version. I've found it the routines it contains useful and thought they could be useful for other people. If you encounter problems, please let me know! Comments can be left on this blog post, or on the site's Java discussion forum.

Using strings in Java

Various questions and problems commonly arise with Java Strings. Especially for C programmers, strings in Java have certain quirks, notably the fact that they're immutable (once you've created a String object, you can't change its contents; if you need a mutable string, then you generally have to use some other CharSequence class such as StringBuilder).

At the aforementioned link, I show code examples of some common string functions in Java. Surely, there'll be things I've missed out/forgotton. So please let me know if there's some thing you find yourself needing to do with strings in Java that I haven't mentioned!

New section: Java cryptography

Several pages of the new section on Java Cryptography are now available for review and general criticism. Topics currently covered include:

• how to encrypt data in Java, looking at encryption with symmetric block ciphers as well as the RSA encryption scheme commonly used for passing session keys;
• different block modes, and how to use block modes in Java (including how to set an initialisation vector necessary in many cases via the IvParameterSpec class);
• how to choose and set the key size to use with encryption, and how to remove the key length restriction which is imposed by current versions of Sun's JDK by default.

Topics not currently covered, but planned for the near future include:

• comparison of block cipher algorithms: performance and security considerations;
• secure hash functions and authentication;
• cryptographic protocols;
• secure random number generation (note that the Java SecureRandom class is currently discussed in the site's section on random numbers in Java);
  
• digital signatures.

As usual, comments about the currently published sections are always welcome, as are suggestions for future additions to the cryptography section or to any other section of the web site. Please leave comments either on this blog or in the Java cryptography section of the Javamex discussion forum.

Update 12/04/09: The section now contains some information on secure hash functions in Java, plus a comparison of encryption algorithms.
19/04/09 As discussed in a separate blog entry, some information on password-based encryption is now included.

Enjoy...!

Java Certification Tips

A new page of Java Certification Tips gives a "cribsheet" of some of the "syntactic niggles" and other commonly overlooked features of Java that could trip up an otherwise moderately experienced programmer when taking the Sun Certified Java Programmer (SCJP) exam. Did you know, for example that:

• this is invalid Java syntax?
  float f = 2.5;
• goto is a Java keyword?
• this line does not assign the value 13 to the variable?:
  int i = 013;

See the page of tips for more information on these and other niggles.

Regular expression tutorial: new example

The Java regular expression tutorial section has been updated with a new example of using regular expressions. In this example, we look at how to perform what is sometimes referred to as HTML scraping: pulling out data from an HTML page (or indeed XML document).

The example is located here: HTML scraping with Java regular expressions. As explained in this tutorial, regular expressions are a good candidate for data scraping because they are flexible. Various libraries exist to parse an HTML or XML document and return an object representation of that document. But such libraries are often "too fussy": many if not most web pages actually do not conform strictly to HTML standards. Similarly, many XML parsing libraries are too fussy for real-life RSS feeds, which are often malformed, strictly speaking. Using regular expressions cuts through the fuss.

When should you scrape web pages?

Note that the article focusses on how technically to scrape HTML pages in Java. It doesn't deal with the "political" issue of whether the site in question wants its content scraped in the first place. In general, it is good practice to do the following:

• find out if he web site in question has an API to provide the data in a more convenient format (and in a format that it would prefer to provide it to you in!)
• be open about what you're doing: if the web site administrator things you're trying to "hide" something, they may block your IP address
• respect server resources: if you are retrieving multiple pages, consider putting a thread sleep between fetches
  

Information on memory usage of objects

The section on Java memory usage now contains the following additional articles:

• information on how to calculate the memory usage of a Java object in general, considering the memory used for "housekeeping" by the JVM
• calculating the memory usage of Strings, which can often the type of object to use up the biggest proportion of space in a Java application: this section actually considers the memory use of string-related objects such as StringBuffers and StringBuilders

A section on reducing the memory taken up by Strings looks at string canonicalisation, a fairly standard approach (but one which requires certain caveats), plus introduces the example of a CompactCharSequence class, that stores strings as 1 byte per character, thus taking up around half othe memory taken up by a regular Java String (at the expense of not supporting Unicode).

Comments on these articles welcome as usual.

Beta: Classmexer agent

The beta version of a simple instrumentation tool is available for download from the Javamex site. The Classmexer jar provides various calls for querying the memory usage of Java objects. Via the provided MemoryUtil class.deepMemoryUsageOf() method, it is possible to get an estimate from the JVM of the number of bytes taken up by an object and its "subobjects" (objects referred to by a non-public reference, or by references with other visibility criteria). The memory usage of subobjects is combined recursively (so subobjects of subobjects are considered etc), but without counting the same object more than once.

A variant of the call is also provided which gives the total memory usage of several objects at a time, without counting as duplicates objects referenced by more than one of the objects.

Updates to profiling section

Firstly, some minor corrections and additions to the Java profiling section. The corrections mainly concern a couple of typos that crept into the variable names of the examples. Readers should be reassured that the code, like that of the site in general, is copied and pasted from working, live profiling code. But things such as variable names are occasionally changed or shortened for the purposes of making it clearer on the site, and that seems to be where the errors crept in. I've also taken the opportunity to add a few links to other sections of the site (such as the section on threading, sleep() and yield()) that were added since the profiling tutorial was written.

Readers interested in Java profiling may also be interested in the first page of an upcoming section on Java and memory. This first page looks at how to find out the memory usage of a Java object. The technique involves using the Java Instrumentation framework introduced in version 5 of the language to query the JVM directly for the size of an object. Although slightly fiddly to set up, the technique has the advantage that there's less guesswork involved than if we were to just estimate an object's size (although future pages in the section will nonetheless look at estimation).