Javamex News and Development

Symantec advise users to disable pcAnywhere

2012-01-25T17:30:00.003-06:00

According to a white paper released by Symantec, the source code to various of its produces that hacker group Anonymous recently threatened to disclose was stolen in 2006, and users are advised to disable pcAnywhere until further notice. Specifically, the paper states:

"pcAnywhere is a product that allows for direct PC to PC communication and this does expose some risk if the compromised code is actually released."

This seems to imply that pcAnywhere is based on security through obscurity. (Presumably, the same security risk actually exists, albeit to a lesser extent, whether or not somebody releases the source code: whatever information is in the source code is in principle available by reverse engineering the compiled code.)

To me, the event underlines at least two lessons:

- this is precisely what may happen if you rely on security through obscurity
- if you have some security-sensitive source code stolen, "right now" would be a good time to review the stolen code, rather than 6 years later...

Or is there something about the content of the white paper and the incident in general that I'm misunderstanding?

A copyright-obsessed French government gets a taste of its own medicine

2011-12-29T09:54:00.001-06:00

As you might expect when your president is married to a singer, the French government takes a fairly totalitarian line on copyright. The infamous HADOPI law effectively enables citizens to be prohibited from contracting Internet access with an ISP on the basis of copyright infringement complaints, apparently with such complaints centering around access from a particular IP address.

So it would be slightly embarrassing if it turned out that IP addresses belonging to the president's official residence and a French government ministry turned up in a database of illegal downloads. Unfortunately, this is precisely what appears to have happened: records of apparently "illegal" downloads from the French Ministère de la Culture and Élysée (official presidential residence and offices) have turned up in the databases of YouHaveDownloaded.com, a site publishing records from (among other sources) various public BitTorrent servers.

So, should we conclude that a poverty-stricken Sarkozy has had to resort to using public resources to download illegal copies of his favourite flicks and tracks in these times of austerity? Should we now engage in month-long trial to determine whether we can prove beyond reasonable doubt that Mr Sarkozy did or did not download that dodgy low-quality MP4 of La Cage aux folles? Should the Élysée now spend public money on a lengthy witch-hunt to establish which petty office clerk or work experience temp is responsible for this shocking infringement of some random fat cat's right to stuff his coffers a little fuller?

Well, I would suggest not-- but that's the point. Hopefully this revelation may help the French government to understand people's concerns about the glib connection that they are insistent to draw between an IP address in a database and the download in question having definitely occurred under the actions of a particular person, and to weigh up the pros and cons of establishing totalitarian means in an attempt to enforce the practically unenforceable with arguable benefit to society.

Is copyright worth breaking the Internet over?

2011-12-19T23:40:00.002-06:00

The truth is that for some time now, the Internet can no longer be relied on to fulfil its simple infrastructural purpose of delivering bytes from A to B unhindered when requested to do so. The appetite of the powers that be for intruding on their citizens' privacy on the one hand and for succumbing to capitalist pressures on the other make any Internet connection an increasingly noisy channel.

Lemley, Levine and Post now outline and give an enlightening critique of some recent and alarming steps being taken in their essay Don't Break the Internet. As an author, I completely sympathise with concerns about copyright, and I would possibly agree that the current process of having infringing material removed is insufficient-- whilst also suspecting that the impact of the "copyright problem" is massively overexaggerated. But as with traffic shaping measures (among others), it is particularly concerning to see proposals to allow fundamental pieces of infrastructure to be undermined almost on a whim. Is this really the most intelligent counter-measure to copyright infringement that we can think of?

Can more indeed be done to disincentivise smartphone theft?

2011-12-04T00:45:00.001-06:00

San Francisco Chronicle columnist C.W.Nevius has an interesting article on how Australian mobile operators do far more than in the US to block stolen phones. On the surface, it is intriguing that more governments, manufacturers and operators can't bang their respective heads together to put more measures in place, both technologically and legislatively, to curb mobile phone theft, particularly that of high-end smartphones.

Direction from governments is probably required. As the aforementioned article mentions, if left to capitalism alone, there is little incentive for mobile operators and manufacturers to put such schemes in place: for them, theft simply equates to more sales. Another interesting thought is that various countries in Latin America do apparently have some measures in place to block the connection of stolen phones. If the US doesn't, then this situation presumably serves to make the US an even more prime target market for the criminals selling stolen devices.

However, such measures surely aren't a panacea. Any system is only as strong as its weakpoints. What measures would need to be put in place to ensure the accuracy of the database and eliminate "false positives"? Since (presumably) operator staff have the ability to override the system in order to deal with cases of error, how does one prevent them from becoming targets for blackmail, bribery and gang involvement, in effect simply shifting the problem? What methods could criminals fight back with (e.g. replacement of the chip containing the serial number), and what impact would this ultimately have in terms of where new criminal opportunities would spring up? And what more serious crimes would criminals then commit instead of stealing mobile phones?

Nonetheless, I haven't seen much debate of these issues and it does seem that potential measures at least deserve more consideration from governments and operators and a clearer explanation for why they cannot be put in place if indeed they can't.

The Carrier IQ debacle

2011-12-02T17:46:00.002-06:00

In case you've missed the commotion, concerns have been raised over the last few days about a component produced by a company called Carrier IQ and installed on many smartphones which (a) is set to run by default on some smartphones; (b) may be more covert than other processes running on the system; and (c) seems to have hooks to which it is passed various confidential data such as the identity of keypresses, the content of SMS messages and the cleartext version of data sent over HTTPS.

Such is the concern that some commentators are already describing the Carrier IQ software as a rootkit.

It is possible that the module in question is entirely innocent, and that the reason for confidential data being passed to its hooks is simply due to some slightly diabolical API design. Be that the case, I would then expect:

- smartphone manufacturers who have embedded the Carrier IQ component in their OS's to come forward with details about the close scrutiny that the software underwent on their part before being approved;
- Carrier IQ to come forward with some convincing and reassuring details about why this confidential data is apparently being passed to the process without constituting a breach of trust and confidentiality.

As I say, the Carrier IQ component may be entirely innocent and above board. But the longer the above two actions continue not to occur, the more concerning things appear. And sometimes, it isn't whether there is anything untoward that matters, but whether there appears to be...

Apple infuriates iOS 5 beta users by suddenly bricking their phones without warning

2011-10-21T11:28:00.000-05:00

A huge number of iOS 5 beta users have apparently been infuriated after their phones were suddenly bricked yesterday evening US time.

Oracle releases Java patches

2011-10-19T08:38:00.003-05:00

Oracle has released critical patch updates for Java runtimes.

Not news, but worth emphasising: SSL certificate authorities can be compromised

2011-10-18T23:39:00.003-05:00

I can't actually spot the piece of news in this "news" item by the BBC. However, the general point of the article still stands as a universal truth: our secure web communications generally depend, and always have depended, on the assumption that certificate authorities are not compromised. They also depend among other things on the assumption that client machines' certificate stores, software and operating systems are not compromised and don't have bugs. People do sometimes forget this and take the little padlock icon in their web browser as some kind of "guarantee"; it's worth reminding ourselves from time to time that SSL offers a degree of confidence that a communication is secure, but no absolute guarantee.

However, I repeat that this isn't really a piece of news to have come to light on 18 October 2011; it has been universally true since the inception of the secure web communications protocols that we so readily rely on.

Amazon's Kindle Fire: does it have something to offer other than cheapness?

2011-10-01T03:55:00.000-05:00

The latest company to jump on the tablet bandwagon-- in as much as such a bandwagon actually exists-- is Amazon with the launch of its new Android-based Kindle Fire.

Clearly, the distinguishing feature of buying Amazon's offering is that the device is coupled with Amazon's purchasing ecosystem, as opposed to, say, Apple's if you were to buy an iPad.

One thing that strikes me is that at present at least, the majority of media focus appears to be on the price, which at $199 does seem aggressive to the point of being a loss leader. When Apple originally launched the iPad, arguably the spark of the recent tablet frenzy among the rest of the market, it is interesting to compare Apple's approach of offering what they saw as a "fairly priced"-- but by no means "cheap" if you sounded opinion at the time-- marketed essentially on features. Amazon clearly have their ecosystem on their side compared to the average Miscellaneous Android Tablet Manufacturer X. But they do appear to be buying into the price-focussed strategy of the other manufacturers. It will be interesting to see how this plays out!

What do you think? I'd love to hear your opinion about the Kindle Fire.

Even 'random junk' may need encryption

2011-09-27T09:36:00.003-05:00

There appear to be some interesting legal cases springing up around the fact that software developers have been taking "raw" Unique Device IDs and using them to store data against a given user 'in the cloud'. UDIDs essentially resemble a long 'random number' rather than directly encoding any user-identifiable information, so you may be forgiven for wondering what the issue is. The problem comes when applications have then stored user-identifiable information (such as social network data) alongside the UDID and make this information available to other applications, thus leading to a 'leaking' of user-identifiable information on the basis of this shared UDID.

The solution is in principle simple: append the UDID to an application-specific code or 'salt' and then encode this combination using a secure hash scheme such as SHA-256. The application-specific salt doesn't need to be a secret. But doing this means that I cannot compare a hash for user X from one application with a hash for user X from another application and deduce that they are for the same user. Again, in principle.

However, this scheme does rely on UDIDs containing sufficient entropy. Since the number of devices of a particular model sold is maybe in the tens or at most hundreds of millions (iOS devices appear to have sold something in the order of 200 million so far), if it is possible to make a good prediction of which range of the theoretically possible UDIDs has actually been allocated, then I can simply pre-compute all the potential combinations of (allocated hashes, application ID) for two given applications and find all the matches.

I haven't yet looked into all the details, but it appears that iOS 5 will remove the ability to read device-wide UDIDs in favour of application-specific UDIDs, presumably generated using a scheme such as the above. Although this is arguably the scheme that should have been adopted from the outset, if true, it will interesting to see what incompatibilities this throws up.

So why didn't people think of this in the first place? I wonder if one of the issues is that to a human, UDIDs do just look like 'random junk': it's just a string of random numbers, right, so why would you bother encrypting it? It's a good example of how when deciding when and how to employ encryption, we have to think not only about the data itself but also about protocols and practices governing how that data is used and stored.

Some thoughts in response to Matthew Baxter-Reynolds

2011-09-17T16:38:00.006-05:00

Matthew Baxter-Reynolds wrote an interesting piece for the Guardian yesterday giving some points of view essentially on what Windows 8 will mean for businesses and IT careers. In particular, he makes the point that writing for a Windows 8 device means a more natural progression from the "C# in Visual Studio" type development that is the bread and butter of most business applications. And he makes the point that this time around, Microsoft will be fostering a tighter coupling of software and hardware platforms to move closer to the model of the iPad, part of whose success relies on it being more of a "self-contained ecosystem".

From this point of view, I think Matthew is probably correct: if the thing you want to concentrate on is writing boring old "bread-and-butter" business applications, then a platform that builds on the existing "bread-and-butter" platform of C# in Visual Studio will be a more attractive proposition for businesses to get a footing on the tablet bandwagon. And even it if wasn't more attractive from a development perspective, "Microsoft Windows tablet" may just sound a bit more 'serious and businessy' on a tender bid.

But, I think Matthew could have included a few other important observations (which don't necessarily contradict his point of view and if anything support it-- but which are nonetheless worth mentioning):

the iOS "ecosystem" may still present an attractive market to developers in the sense that Apple have done the job of (a) isolating the 100+ million people with sufficient income to splash out on fancy toy; (b) sold them that toy on the basis of it providing enjoyment: or in other words, persuaded them that it is to their benefit to spend money on this new gadget (and associated apps); and (c) built a system for developers to market fairly effortlessly to that income-to-spare-for-toys-and-apps sector;
games (and, apparently, knocking out games that you can sell for a buck or two a download) remain the predominant iOS market; the iOS development framework allows you to write virtually all of your application in a bog-standard C/OpenGL paradigm which will allow the creation/porting of a huge number of games with a minimal learning curve;
while slightly quirky, in a sense Objective-C is just "another C syntax-based language" and if you stick around in programming long enough, you generally end up learning a new C syntax-based language every 10 years or so; indeed, "Java" as it looks today, and certainly how it will look if currently discussed language features make it into Java 8, is almost a whole new language compared to how Java was when it first materialised back in 1735 (anyone remember when Java didn't have inner classes, let alone generics or closures?);
given that many business applications can and indeed ought to be written as web apps (a point which Matthew himself makes), for as long as HTML5/Javascript remains a standard enough development paradigm, I wonder if the Microsoft tablet will "C# in Visual Studio" actually become the paradigm of choice for tablet business applications anyway?

So, while I think Matthew is probably right that we could end up with an 80:20 split in one market versus a 20:80 split in the other, I don't know that that means that the "not-the-boring-bread-and-butter-database-application" market isn't viable.

Windows 8 soon to be upon us

2011-09-14T08:01:00.005-05:00

It hardly seems 5 minutes since Windows 7 was the Next Big Thing. But apparently Windows 8 will soon be upon us, the principal motivation apparently to allow Microsoft to have another crack at jumping on the tablet bandwagon. It will be interesting to see how many people are actually burning to have Windows on their tablet, in whatever touchy-tabletified guise.

I'm quite happy not to have the following features on my iPad, for example:

the need to grind the entire system down to the speed of a ZX Spectrum thanks to the latest antivirus software
irritating pop-ups every five seconds requesting permission for applications to do their basic job
having to battle with system that insists on saving documents in fundamentally stupid locations buried somewhere 5 million levels deep in the filing system
the need for every Application That Goes Ping to take up 7 squillion terabytes of disk space, delay startup by half a minute and require at least 2 resets before they will work

But of course, I am open minded and eagerly await the first demonstration of the wondrous crop of Windows 8 tablets which will not fail to ensue.

UK petition to get programming on the curriculum

2011-09-12T10:08:00.004-05:00

There's an interesting petition being made to the UK government to teach programming to children from primary school age rather than waiting until later in their school career. I'm not sure that the rationale specifically given by the petition's initiator-- that there would be "far less of a disparity between the sexes"-- is terribly convincing as it stands. It's not clear to me either that there's really some great barrier, preventing female students from learning program if they choose to, that will be broken down specifically if 9-year-olds are taught programming. And in any case, it's not clear to me that even if it were the case, that it would be the most compelling reason for teaching programming. However, whether that proposition proves true or false, I think there are other valid reasons for teaching programming at this early stage. And so I support the petition itself, even if not the exaggerated gender-gap mantra that seems to be its underlying motivation.

I think a more compelling rationale is to be found in the types of skills that programming develops and simply in the ubiquity of programmable devices. As has been expressed elsewhere, learning to be a good programmer seems to entail making certain problem-solving skills become intuitive. (As one fellow blogger has put it, learning to "algorithmate".) And in today's world where the average iPhone packs computing power that would have classed it as a mainframe not so many years ago, the potential for innovation through programming is practically beyond comprehension. I also suspect-- though I concede that I am also guilty of fluffy argument here and would be hard pushed to substantiate this view-- that the mindset that is made intuitive through learning to program gives people a more intuitive grasp of other concepts crucial to understanding our universe, be it genetics or the syntax of language. But in any case, the computer is now so engrained into our day-to-day devices that I would argue that "understanding our universe" really entails understanding how the microchip and computer work just as much as it does understanding the atom, the laws of thermodynamics etc.

Or put another way, there's really no reason nowadays to view programming as a uniquely specialist, "nerdy" subject.

However, I have some implementational concerns. I worry, if programming was brought earlier on to the curriculum, about how severely it would be dumbed down; educators need to understand that "programming" doesn't mean drawing inane patterns with "turtle graphics" or similar glorified spirograph imitations. I also have mild concerns-- though from a UK perspective, yes, they are mild-- that having a stronger computing component to the curriculum could widen existing digital divides if, for example, basic computer literacy was sidelined. (However, I do disagree with an apparent over-obsession with "teaching" skills such as browsing the Internet or word processing to those children for who such skills are as intuitive as operating their iPod.)

I look forward to seeing what success this and other similar petitions in other countries may have. And if you are based in the UK, I urge you to sign this petition and encourage your friends and colleagues to do likewise.

Speculated iPhone specifications... seem somewhat boring

2011-09-04T11:43:00.002-05:00

As we approach the predicted launch date of the iPhone 5, we unsurprisingly start to see more of the media speculation machine whirr into action. Key specifications that are pretty much on the cards are the dual-core A5 processor and an 8 megapixel camera. And the new version of iOS isn't a terribly big secret given that any iOS developer can download the beta version.

Now, perhaps I'm missing something, but it strikes me that if this is the big spec improvement to make a big fuss about, it seems pretty unexciting.

The dual-core processor will hopefully make, for example, video editing a little less klunky. And it may be put to good use in a few specialist applications such as to allow extra tracks or more complex virtual instruments in some of the excellent music making software that has appeared for iOS over the last couple of years. But for your bog-standard iPhone game, I suspect the extra capacity will go largely unused: the standard game programming paradigm simply doesn't used multithreading, or only to a very minimal extent (e.g. sound playback is handled by background threads). While I'm not sure whether to agree with Don Knuth's take that multi-core processors are essentially a passing gimmick while we work out how to get faster single cores, it's true that what game programmers generally want is a single core that runs as fast as possible.

As for an 8 megapixel camera instead of a 5 megapixel one. Well, I can't help shrugging my shoulders a little. So maybe we'll get a 2x digital zoom that's just about worthwhile if the extra megapixels actually add resolution rather than just being extra noise. But unless Apple is about to announce that they've just designed a revolutionary new lens, a few extra megapixels don't sound terribly exciting on the face of it.

Of course, I also eagerly wait to be corrected... :)

Encryption: it's the protocols that often go wrong

2011-09-03T01:29:00.003-05:00

Bruche Schneier has an interesting summary of the recent diplomatic cable disclosure due to (a) the encrypted file being (surreptitiously but) publicly available, and (b) the decruption key being publicly disclosed on a separate occasion. It's a good example of how what can often go wrong with cryptography is not the core technology or algorithm itself, but a failure in handling associated protocols (be it technical or social).

Google's use of the +1 button for search ranking

2011-08-30T08:52:00.002-05:00

So firstly, a confession: I had always assumed that Google used +1 ratings in some small way for search engine rankings. Perhaps I'm misreading their blurb, but it seemed that they were pretty much advertised as such to web publishers. Clearly the influence would have to be small, as +1 ratings are easily fakable, but I assumed they were used in some way.

According to a recent Wired article, it seems that this isn't yet the case, but that Google are still in the process of considering using tihis nformation.

A couple of problems I see:

as Ryan Singel notes, it's easy enough for web site authors to "fraudulently" arrange +1 clicks on their own site, so some kind of filtering would presumably be required, and the degree of influence that +1 ratings had would need to be quite carefully balanced and monitored;
it may be that certain more "socially oriented" sites are more susceptible to the +1 or "Like" concept; just because a site is more focussed on social networking doesn't necessarily mean that its content is more "worthy" or relevant in response to a given search.

The problem of spotting scams

2011-08-10T13:06:00.004-05:00

Ed Bott writes an interesting article on the subject of Why do people fall for trojans? As he explains, one of the problems is that the steps that one must go through to make even a legitimate purchase can often resemble the experience of a scam, including false alarms such as Windows warning us that our legitimately purchased item "might harm your computer".

This is also one reason why I think schemes such as the "verified by Visa" scheme for credit card purchases, in addition to being senselessly irritating, are actually a bad security measure because they imitate during a legitimate procedure precisely a process that an attacked might exploit or which might be the warning sign of an attack. Or in other words, in both this case and the scarey dialogs that can appear in Windows while you are downloading and installing a legitimate program, we are actually risking that what should be warning signs actually become ignored due to being "dulled by familiarity".

The Google files

2011-08-07T15:27:00.003-05:00

The current milestone in the Google-Oracle debacle has been reported this week to hing around two e-mails, one dating back to 2005 and the other a draft of an e-mail possibly constructed after Oracle's patent claim expressing a need to negotiate a licence for Java.

Whilst the first of these e-mails discusses strategies for what to do "If Sun doesn't want to work with [Google]", if will be interesting to see how much this weight carries in light of the subsequent 2007 blog posting in which Sun's CEO apparently praises and supports Google for their use of Java technology in Android.

The scond, apparently only a draft of an e-mail, but slightly dammingly indicating "We conclude that we need to negotiate a license for Java under the terms we need." poses other interesting questions. How admissible as evidence is a draft of an e-mail reported to never have actually been sent? And if it is established that this draft was composed after Oracle first raised its case (from the reports I've seen, the timing still does not appear clear), how damming is an e-mail saying "we need to negotiate a license" if it effectively in response to Oracle saying "You need a license".

I'm not a lawyer, but it will be interesting to see how this plays out.

I would still be interested to see a re-examination of the actual patents in question. As I've expressed before, my opinion of at least some of these is that they appear to essentially be fancy verbiage to describe procedures that are inherently obvious to an expert in the field, with little in the way of an actual invention.

"Lost Programming Skills"

2011-08-06T10:01:00.002-05:00

All too familiar with that "corporate IT solution" consisting of various inefficient black boxes haphazardly glued together using a plethora of acronyms?

This article entitled Lost Programming Skills looks at some of the once common knowledge which has not been passed on to generations of programmers. Interestingly, they highlight the inability to appreciate I/O issues (as much as, say, code optimisation).

Why Java 7's hotspot bug is a bad bug

2011-08-01T18:45:00.003-05:00

I'm responding partly to Alex Blewitt's article in which he downplays the severity of the Hotspot bug exhibited in Java 7.

First of all, I think that in principle he may have a good point: previous Java releases have exhibited bugs which are arguably just as severe if not more so, there has been a temporary workaround, and they have been fixed and the world has carried on ticking. It's also true that any critical application upgrading to Java 7, as with any major upgrade of a key system component, would be mad not to have a regression plan should the proverbial shit hit the fan.

Where I disagree with downplaying this bug is that I believe the problem is largely one of process rather than the substance of the actual bug:

this is the first major release of Java since Oracle took over stewardship of the Java language: eyes were on them to see how they would handle it;
this is the first major release since the departure of various key ex-Sun employees;
the bug(s) in question were reported before the release and were exhibited by code present in the experimental stage in releases of Java 6 (or to paraphrase: "if the fix for that known serious-sounding bug wasn't tested properly, then what else wasn't tested properly?");
the precise circumstances under which the bug(s) exhibited themselves were not clearly published, let alone by Oracle;
without knowing the details, a vague birds-eye view of the problem (a "bug with JIT-compiling nested loops") sounds like a bug in a fundamental function of the JVM that you would ideally want to work reliably 100% of the time.

Alex Blewitt takes the view that: "since Java 7 has only just been released it would be unlikely to be used in production environments yet". I don't buy this view. If this is an alpha release, then call it an alpha release. But if your position is that you are making a "release", then that quite reasonably means "this is a version that has been through rigorous testing which I believe is suitable for use in production environments". Let's not flannel our way out of this: it was a cock-up.

If I were using Oracle Database for any critical applications, I would certainly now be looking for some reassurance that they take the testing and release procedure of their database products more seriously than they do Java...

Today's spurious software patent is...

2011-07-29T09:41:00.002-05:00

Sadly, on any random day, we could probably pick a case like this. Today it appears to be Spotify's turn to face a lawsuit for "infringement" of a spurious patent that should never have been granted in the first place.

The abstract of the patent begins:

"In order to distribute music information from a central memory device (2) via a communications network (4) to a terminal (6), this information is organized in a digital music information object."

As you can probably guess from this opening line, and sadly like so many other "software" patents, the rest is pure flannel: a very broad overview of the blindingly obvious steps that one would take in designing an application, but without having actually invented any non-obvious implementational details. Pretty much every line of the patent leaves you coming away thinking "Sure, that's obviously true, but what's your actual invention?". It's another example of how the patent system urgently needs to be reformed with at least:

- patent examination which goes beyond the intelligence of a chimpanzee on hallucinogenics (or as a next best thing, to paraphrase Donald Knuth slightly: if you would expect a 1st year software engineering student to be able to solve a problem as a homework assignment, you shouldn't be able to patent the solution as an "invention");
- severe punishment for companies that attempt to file spurious patents;
- more severe punishment for false claims of infringement (such measures do already exist to discourage large companies from bullying small developers, but they clearly don't go far enough).

There's also an interesting article in this week's Pod Delusion on the subject of patent trolls, looking at another recent case of patent abuse (Lodsys-Apple).

There's no hiding from ngram analysis

2011-07-28T23:32:00.003-05:00

It's not often that a corpus linguistics paper makes it to Slashdot. A notable exception today came in the form of Burger et al's paper on Discriminating Gender on Twitter. Using surprisingly simple techniques, they present data suggesting that a machine algorithm can determine a tweeter's gender more reliably than human judges.

As far as I can see, their measure of reliability is of the % of correctly attributed tweets over the data set as a whole. It would be interesting to see more analysis on the nature of tweets that the system can't judge reliably vs human judges (and vice versa). To what extent do typical tweets fall into polarised categories of "readily categorisable" vs "highly ambiguous", or is there more gradience? And it will be interesting to see what other demographic data can be gleaned using similar techniques.

Estoppel

2011-07-26T13:09:00.002-05:00

Following on from my post yesterday about how a statement in Jonathan Schwartz's blog post indicated that Sun approved Google's use of Java in Android, I should perhaps have pointed out-- as others reporting on this have-- that there is indeed a judicial basis for Schwartz's statement being used in Google's defense. The principle is estoppel, defined by Black's Law Dictionary as:

"An affirmative defense alleging good-faith reliance on a misleading representation and an injury or detrimental change in position resulting from that reliance."

Will Jonathan Schwartz's blog post finally persuade Oracle to pick its toys up and put them back in the pram?

2011-07-25T11:19:00.009-05:00

In case you've missed all the commotion, or just found it a little bit dull, I remind you that in August 2010, Oracle filed suit against Google, claiming that their Android operating system violates various patents held by Oracle after they acquired Sun Microsystems.

The extent of damages that they can actually claim for appears to be in some doubt. After an initial claim of several billion dollars, and a judge effectively ruling "don't be so silly" last week, the claim may be reduced to as "little" as 100 million dollars. If I had 100 million dollars, I wouldn't be sitting round writing silly blog posts. But to Google, that's petty cash.

Now, to naive bystanding programmers such as myself who are not lawyers, a few things always seemed a little bit odd about this case:

would a company the size of Google with the legal resources it can afford really just go ahead and commit billions of dollars worth of patent infringement and hope that "nobody would notice"?
if there was a big problem with Android's use of Java technology, why didn't Sun raise this at the time?
some of the patents in question really boil down to some quite small, specific points of implementation (e.g. specific details of implementation of protection domains, the "pre-processing and packaging of class files", and some implementational details of the virtual machine), many of which could probably be worked around by using alternative implementations or (and hence) a reasonable licence fee negotiated with Sun if need be.

Or put another way, the whole case smells of somebody scratting around through their recently acquired arsenal of patents looking for an excuse to sue. Rather than learning to play happily with all the other children in the classroom, Oracle seems intent on shouting "But Miss, the red crayons are mine!".

Meanwhile, commentators had noticed the curious deletion of blog posts of previous Sun CEO Jonathan Schwartz. Unfortunately, such Orwellian attempts to rewrite history don't tend to work on the Internet. Thanks to nothing more sophisticated than a search in The Way Back Machine, a key post from "Jonathan's Blog" has been restored to its former glory. And in this post, Schwartz states: "I just wanted to add my voice to the chorus of others from Sun in offering my heartfelt congratulations to Google on the ennouncement of their new Java/Linux phone platform, Android". As commentator Steven Vaughan-Nichols has also pointed out, this doesn't exactly sound like Sun were saying "We disapprove of Android because it violates our patents and will be suing Google for billions of dollars".

It will be interesting to see if this evidence can persuade the two parties to finally bang their heads together and get on with more interesting things. Like it or lump it, Android is a key market for Java technology, with Android accounting for a 30% share of tablet shipments in April-June 2011, for example. I can't help feeling that it would be more conducive to the platform's development for Oracle and Google to have a more sensible working relationship.

Google's malware warning

2011-07-24T21:24:00.002-05:00

Google have given more details of the malware warning that they will be issuing at the top of their search engine. They have been able to identify around 2,000,000 infected machines on the basis of traffic sent to their servers.

A couple of things strike me about this:

I wonder how applicable this technique could be to other servers in general? Could we see other sites performing a similar civic duty of warning users about potential malware.
Given that one route of infection (which Google blames specifically as the culprit in this case) are those fake "Your computer has been infected" popups already littering the Internet, how do we give out a message to users that these popups should generally be ignored because they're likely to be fake, whilst at the same time issuing genuine warnings in an ostensibly similar manner?

I'm interested to hear your thoughts either here, or by tweeting to BitterCoffey.

Dear universe: please make your default settings sensible

2011-07-22T05:35:00.004-05:00

I'd occasionally noticed that alerts for my calendar events would sometimes go off at odd times on my iPad. For example, I'd wake up in the morning to find that the "5 minute" alert for a mid-morning meeting had already gone off. I suppose the first couple of times, I just assumed I must have entered something wrong when setting up the event. But, being the aspiring cross between Columbo and Paddington Bear that I am, gradually my suspicions were aroused.

Luckily, some other people more patient than I am with spuriously designed user interfaces were able to come to the rescue. In case you haven't found it, the offending setting is an option burried in the "Mail, Contacts, Calendars" section of your iPhone/iPad settings curiously entitled "Time Zone Support". With this set to a particular Time Zone (rathr than simply being switched off), then your device will not adjust calendar events to match your current time zone.

So you might be thinking "hey, what's your beef, this is a great feature to have, no?". Well, yes, it's a great optional feature. But the things that strike me are:

(a) to say that the iPhone/iPad are supposed to be the holy grail of PDAs (aren't they?), it doesn't seem like a terribly fine-grained feature, particular as there is a "Location" field in calendar events, and that thanks to the miracles of GPS, the device knows where you are at any given moment (or at least, knows where you are to within an approximation of a couple of skyscrapers/underground bunkers-- usually good enough to tell what time zone you're in);
(b) why give this option a silly, non-intuitive name and then turn it on by default? why by default would I tend to want events not to be in synch with where I am at the moment, given that the whole raison d'être of these devices is for travelling...?

Well, I'm sort of happy now I've discovered the option. But on the other hand, grrrrrr.

Rip-off for rip-off's sake

2011-07-21T14:18:00.004-05:00

In response to crackpot loon conspiracy theories that instead of landing on the moon in 1969, Nasa played a hoax that has then been kept under wraps for the past 40 odd years, my argument has always been that rather than stage and hide such an elaborate hoax, it's probably easier just to go to the moon. (The other main evidence for the falsity of the moon landing hoax being that Fox TV made a documentary about it...)

Well, my grip on reality may have been overturned today by reports and pictures of fake Apple stores in China. Somebody who can actually be bothered to go to these lengths of counterfeiting and then sustain the momentum of preventing the relevant authorities from finding out must be doing it out of some bizarre sense of national pride. I'm really not convinced it isn't easier to just franchise a genuine Apple store...

On the dangers of our reliance on opaque algorithms

2011-07-21T12:45:00.003-05:00

Kevin Slavin offers an illuminating TED presentation on the implications of our reliance on computer algorithms whose complexity has reached the point of being opaque to human understanding.

The fine lines of iPhone app rules

2011-07-02T22:01:00.004-05:00

"It's not whether there is anything wrong. It's whether or not there appears to be."

It appears that app developer Daniel Amitay has been finding out recently about the finely shaded grey areas of Apple's rules on what you can and can't do from an iPhone app.

One of Daniel's apps, Big Brother Camera Security, is designed to mimic the iPhone passcode screen and then photograph any user making an incorrect attempt to enter the passcode. He also decided to collect statistics on the most common passcodes chosen by users by having the app anonymously send those passcodes. Apple have since removed the app from the App Store.

Daniel argues on his blog that this data collection was legitimate, as the purpose was to warn users against choosing frequently chosen passcodes in a future update to the app, and the the App Store EULA informs users that apps may collect diagnostic data. Apple, on the other hand, appear to be arguing that application was "surreptitiously harvesting user paasswords".

It seems that whether that is strictly the case depends at least on one's interpretation of what "harvesting" constitutes, and how one defines the nature of the separation between the app's lock code and the actual iPhone passcode, given that the app is designed to simulate the iPhone lockup screen. A potentially complicating factor is that the data has also been used as the basis for a (somewhat interesting) article that Daniel has published giving an overview of the statistics obtained for the most common passcodes and most common digits in different positions. Publishing this article, though of interest to the security community, may be deemed to make the data something other than for diagnostic purposes of "facilitat[ing] the provision of software updates, product support and other services [...] related to the Licensed Application" as set out in the App Store EULA.

Some interesting questions to consider:

if secure hashes of the passcodes, rather than the actual passcodes, had been sent, would that have constituted "harvesting"? (Even though with only 1000 possibilities, the actual codes could of course be resolved from the hashes.)
which precise conditions would need to be changed to allow Apple to approve the app? (Is it ultimately the data itself or the surreptitiousness that is the problem? or the publishing of the data in an article not strictly necessary to updating and supporting the app?)

It will be ineteresting to see how this case evolves over the coming days/weeks and whether Daniel is ultimately successful in getting the app reinstated, and what changes are necessary for this to happen.

Have CAPTCHAs gone slightly too far...?

2011-06-21T07:02:00.004-05:00

I like a bit of security just like the next man. But I wonder if this piece of "security" is going slightly too far. It appears that in order to make a posting on Facebook, we are now required to fill in a CAPTCHA consisting of words in two different alphabets...?

Now, I was trying to decide if this was some kind of perverse general knowledge test, or the latest CIA "initiative" to track down terrorists. Mention the wrong keyword and show an ability to type Arabic, and you're suddenly on the list of America's most wanted...?

(Oh, and even if I could have read Arabic, I would have still got stuck with the mysterious blob on the left. Answers to that one a postcard...)

Why lawmakers' ignorance matters

2011-06-16T23:55:00.005-05:00

The other day I blogged on the apparent ignorance shown towards what is now basic technology infrastructure by members of a committee hearing in the current UK libel law reform case.

It appears that our lawmakers, possibly with similar knowledge of Internet infrastructure to those on the libel reform committee, have now allowed us to reach a stage where putting a link on a web site is an extraditable offence. I'm just going by what I've read in the media and cannot confirm any of the details. And the aforementioned article doesn't appear to mention What Law Has Actually Been Broken. But if true, I am slightly curious as to why the focus of the Powers That Be is on removing sites linking to the offending material rather than just removing that material from the sites hosting it. If I put a note on my web site saying "There's illegal stuff on the Internet and if you do a Google search for it you might find it", is that also illegal...?

The lawmaking process and technology

2011-06-14T14:49:00.008-05:00

A process is currently underway in the UK to reform the libel law. Before people outside the UK click their "back" button thniking this is just a piece of local trivia, I'd like to suggest that the procedure highlights some more general symptoms of how the Internet may be understood (or not) by those involved in the lawmaking process which are probably applicable in many countries.

In this session, various members of the science community who have in one way or another been on the receiving end of libel cases (notably Simon Singh and Ben Goldacre for their attempts to alert the public to instances of bogus medicine) give their opinion on changes that they believe should be included in the reform. Very telling are several instances where they appear to be educating the panel on some basic concepts about the Internet, blogs and ISPs. Some key points to look out for:

- at 17'50, where Simon Singh is practically describing what the Internet is
- at 18'10, where Ben Goldacre is pretty much explaining what a blog is, and making some very fundamental statements about how basic and prevalent they are to the Internet
- at 18'27, where it is asked what the witnesses think of a "Prior system whereby a claimant could write to a web host ... and the web host would be under the obligation to put up a notice alongside the story", and Ben is forced to explain some extremely basic information about the relationship between ISPs and their clients.

Now on the one hand, I should emphasise that I don't begrudge this level of transparency in a world where not all citizens are so fortunate in seeing their lawmaking process in action. On the other hand... I worry about how this demonstration of an apparent oblivion to basic technology and "social infrastructure" is going to be translated into a new law that truly fulfils the wishes of those calling for reform.

Making the difference with lego

2011-06-14T09:35:00.004-05:00

There are various methods for evaluating a polynomial. But perhaps one that may not have occurred to you is to build a difference engine out of lego. According to this story in Wired, that's the approach that Andrew Carol decided to take.

Whether further lego difference engines will be built is not yet clear. A difference engine is currently used in the banking sector as part of the processing of international transfers as, despite adding several days on to the time taken for transfers to complete in some cases, difference engines are deemed to be more reliable than conventional microprocessors. However, banking officials openly admit that the decision to use a solid gold difference engine has prohibited rolling out similar machines across Europe to speed up transactions. Now that this cheaper, lego version is available, industry leaders may be looking to see whether it has the reliability required to allow the rolling out of these cheaper machines and hence enable banks to speed up the processing of international transfers. However, it is unlikely that a decision will be reached before 2012. For the time being, those of us transferring money between accounts in different countries will have to continue to wait up to 5 days for transfers to complete.

Irritation, irritation, irritation...

2011-05-23T07:46:00.002-05:00

Over 1,000 users of the Javamex have responded to the site's survey in which they were asked to vote for their most irritating software. It seems that the topic has struck a nerve...

I have a put up an initial hit parade of irritating software based on the first 100 repsonses (a more in-depth analysis of all results to be published shortly).

In first place, perhaps unsurprisingly, are antivirus programs. I've used practically every major antivirus program on the market, and have found every single one to be completely infuriating. Whether it's popping up irritating paranoic messages every few milliseconds, making it difficult to access certain web sites properly or slowing down your eight-core CPU to the speed of a ZX Spectrum on tranquilizers, every single antivirus product appears to be designed to make you want to throw your PC off a very high balcony.

I was somewhat reassured to find iTunes in second place. Especially under Windows, I have also personally found it to be one of the most buggy and infuriating pieces of software ever written. But whether it's freezing every five seconds, launching into a lengthy "sync" process just because I accidentally wanted to upload an extra song or thinking at random intervals that my iPhone has mysteriously transformed into a new one never seen before in the history of humanity, I now feel less alone in the universe knowing that other users share my view.

Suggestions for new programming articles

2011-03-10T10:16:00.002-06:00

The Javamex site is open to suggestions! Please make a suggestion for a new programming topic at the top of the home page!

Update to Currency Quoter utility

2011-03-04T11:28:00.004-06:00

The latest update to the Currency Quoter utility available on this site, version 0.04a, fixes an issue which prevented currency data from being downloaded under some circumstances. It also introduces a parameter allowing you to control the number of months' worth of data that are taken in producing the forecast value of a given currency exchange. You can currently configure this between 1 and 12 months. The default is 3 months, which appears to be suitable in most cases. However, you may wish to take a greater period, for example when looking at a longer-range forecats of the value of a currency that has had a particular trend over a longer time.

Please report any issues with this program bugs@javamex.com

Additional information on the 'final' keyword

2011-02-17T20:36:00.003-06:00

The section on the Java final keyword has been expanded. Previously, we concentrated on the use of 'final' for thread-safety but did not give much information about a separate use of 'final' to indicate that a class or method cannot be overridden. The new material expands upon this use with a look at the performance implications of the 'final' keyword as a class or method modifier.

A common view that I have both heard among colleagues and seen in various Java textbooks is that 'final' is as much a performance hint to the compiler as a specification of design. In the new section, I present some data showing that this view is probably misguided: the timing of calls to methods to final vs non-final classes comes down to whether the JIT compiler can determine at runtime the precise subclass of an object rather than whether or not the class in question is or may be overridden.

Initial release of "Currency Quoter" utility

2011-02-16T16:06:00.003-06:00

A very initial release of a currency conversion tool is available from the Javamex site. The Currency Quoter utility allows you to exchange between various currencies using exchange rate data reported by the IMF, and also allows you to forecast the likely range of values that a particular exchange will have on future dates.

For news and updates, you may also wish to subscribe to the Currency Quoter project page on Freshmeat. You can report bugs at bugs@javamex.com

New material: doing maths in Java

2011-02-16T15:59:00.002-06:00

The section on mathematical operations in Java has been expanded over the last few days to include some new material that will be useful to those writing maths-related applications in Java. Notable inclusions include:

The section on the performance of java.util.Math methods has been expanded somewhat to include more information about the timing and implementation of certain operations.
A new section on BigDecimal and BigInteger looks at how to manipulate arbitrarily large numbers (or numbers of arbitrary precision) in Java. We include an analysis of the performance of multiplication.

As usual, suggestions for improvements to this section or suggestions for new material are always welcome, either on this blog or on the Javamex forum.

2011-02-04T08:49:00.002-06:00

I thought this was interesting, although the actual security impact is hard to assess. Various sites "leak" information not through the payload returned by a particular HTTP request, but simply by the response code. Thus, as this article illustrates, we can find out, for example, whether a user is logged on to sites like Facebook as follows:

- find a particular page that responds with an error code or not depending on whether or not the user is logged on;
- using a "script" tag, ask the browser to load that page as though it were a script;
- in the onload() and onerror() handlers, take action that assumes the user is logged on in the first place and not in the second.

The fact that a user is logged into, say, Facebook or GMail probably isn't a very exciting discovery: half the Internet population probably are at any given moment. But more controversial sites may want to think about what kind of information they accidentally leak in this way.

New articles in the 'Java collections' section

2011-01-15T20:30:00.002-06:00

Readers are invited to explore some new material that has been added to the Java collections section of the web site. By rights, this material would probably belong in a "Java algorithms" section, as they don't use the collections framework as such, but rather look at structures closely related to those used in the standard collections classes.

In the section on Advanced use of hash codes, some new material firstly looks at how we can implement a strong, 64-bit hash code in Java, and how we can use this to create a Java hash map implementation that saves memory by storing only the keys.

A further section on Bloom filters builds on this strong hash function further, allowing a set of objects to be represented very compactly in memory in return for a certain "false positive rate". This tradeoff is useful in certain situations where we want to "weed out" certain items that require further, resource-intensive processing from those that don't (for example, when wanting to reduce database hits), but where it doesn't matter too much if a small percentage of items "slip through the net". We look more closely at how the initialisation parameters of the Bloom filter can be configured to tune the false positive rate in return for more or less memory space being required.

Javamex moved to a new server

2010-12-27T12:19:00.002-06:00

The Javamex web site has been moved to a new server. Hopefully the move was fairly transparent to readers, but I of course welcome any feedback in case of any gremlins you may spot...!

iPad for programmers: new section and survey

2010-11-28T10:18:00.003-06:00

In preparation for a new section of the site dedicated to programming tools for iPad users, the site contains a survey to find out which tools people are currently using. If you have a few seconds to spare, you are invited to fill in this very quick survey.

What kind of tools are we interested in? Well, essentially any iPad app that you consider to be "programming-related" in some way: for example, code editors, tools to run scripts, calculators that ease with programming-related tasks etc. Even general tools such as VNC clients are also welcome if you use them for programming-related tasks.

Javamex survey

2010-08-26T13:30:00.002-05:00

Readers are invited to take 5 minutes out to answer the site's Java programmer survey. The survey asks you your opinion about various aspects of Java programming: how difficult or easy do you consider aspects of Java such as threading, the 'volatile' keyword, using databases in Java etc. The survey is completely anonymous and the results will be used to improve the focus of future articles published on the Javamex web site.

The cost of object orientation: an update

2010-07-09T15:35:00.003-05:00

Back in March, an article was added to the site on the performance cost of object orientation which showed that on a test system, "hand-cranking" computationally expensive complex number calculations to avoid the use of encapsulating objects gave a notable performance benefit. The results also showed, however, that the benefit of using mutable objects was limited.

In some work I've been doing more recently on a wider range of test machines and versions of Hotspot, at appears that this effect is not universal: on at least some systems, there is a benefit from using mutable rather than immutable objects. More details will be posted to the site in due course.

The performance impact of object orientation

2010-02-28T11:37:00.006-06:00

Today's new Javamex article looks at the performance impact of using an object-oriented approach vs an optimised, non-object-oriented approach in a case where the OO approach requires high object throughput. Specifically, we look at the case of calculations with complex numbers.

The basic tenet is that the most intuitive program design is to have a complex number encapsulated as a Java object, and furthermore, as an immutable object. This means that at each individual step in a calculation, a brand new object is created. However, performance measurements suggest that using immutable objects is only around 28% slower than using mutable objects, whilst the codability and maintainability impact (threading issues; bugs introduced by forgetting to copy objects and hence making accidental modifications to an obejct) may be considerable.

In our example, a version of the algorithm that does away with encapsulation and "inlines" its calculations on raw double primitives is more than twice as fast, as perhaps might be expected. On the other hand, "only" being between twice and three times as fast means that a comparable improvement will be possible in some applications by making use of more processors, or potentially making other algorithm improvements, whilst the codability and maintainability implications of not using encapsulation become more severe for more complex applications.

When your antivirus nags you worse than your mother: the power of bloatware

2009-11-04T21:56:00.000-06:00

If you're looking to rub salt into that wound we call life, few things can top the mighty blue screen of death followed by the severe ticking off that comes from allowing thy computer to be shut down in an improper fashion.

But if there's one thing that comes close, it's got to be Antivirus software. Few other inventions succeed in reducing a squillion-gigahertz processor to the power of a ZX Spectrum. And on the scale of severe tickings off, little can beat the scolding that I received today from the antivirus I had trustily installed not four score and ten days ago.

- "For the last 90 days thou hast willingly deterred from scanning thy device for evil malware," quoth ye antivirus.
- "Evil what?", quoth I.
- "You know-- bad stuff like cookies and worms and shit.".
- "But isn't that your job?"

Alas, my assumptions regarding the autonomy of the antivirus software had proven false. Despite years of research into artificial intelligence and the inordinate number of clock cycles that said software consumed, it would transpire that scanning for evil malware was a task upon which it preferred not to take the initiative.

Thanks be to pointless bloatware.

New Section: JNI (Java Native Interface)

2009-10-20T21:04:00.002-05:00

The new section on the Java Native Interface will be expanded over the next few weeks. For those not familiar, JNI is the standard framework for calling native code from Java. Thus, JNI effectively allows you to call Windows API functions from Java (or API calls from the operating system in general).

As explained in the section, one caveat of using native code is that the JNI overhead means that writing your code natively may not be as beneficial as anticipated.

Fragmentation

2009-10-20T21:01:00.003-05:00

Disk layout:

Blue: MyDocuments
Red: MS Office (Program Files)
Green: Windows\System32

When the simple gets complicated: XPath from applets

2009-10-06T17:46:00.004-05:00

Those who have read my introduction to XML will be familiar with my mixed feelings on this "technology". In principle, the idea of a standardised, human-readable data format is a handy one. Just in practice, the XML frameworks that are supposed to take the work out of reading this data end up being fundamentally infuriating for some reason.

Of the irritating options available, the XPath API is about the least tedious. Essentially, the idea is that you can call an evaluate() method, passing in a "path" into the XML document by which you refer to elements (data) in the document. Java's standard implementation is stupidly faffy, making you mess around with document builders and configuration exceptions and god knows what other nonesense that you really shouldn't have to care about. But as I say, of the various frustrating options available, it's still about the least frustrating.

At least, until you come to use it in an applet. Picture the scene: my beautiful XML-consuming applet nearing completion and tested locally, I excitedly gmail my colleague to let him know I'll have it up on the web for testing shortly. Then imagine my dismay when the selfsame applet uploaded to the web fails to initialise. Half an hour or so of frustrating debuggery later, it turns out that the culprit is the XPath.evaluate() method, inside whose gubbinery a network fetch is being made for some spurious property on every single attempt to read a piece of data from an XML document. Vive la pluggable architecture.

As I explain on the page, the workaround I found for this was to explicitly set the system property being looked for to its default value via a call to System.setProperty(). For this, alas, the jar must be signed. But I decided that even signing the jar would be less frustrating than re-writing my code to avoid the XPath API. That's how desperate I was by that stage.

With the jar signed, my colleague is now free to accept the warning message about the apocalyptic consequences of running my applet and proceed with testing. Ginger programmer 1, frustrating API nil. For now.

Security and public wireless networks

2009-07-10T23:53:00.003-05:00

The other day I wrote of the problem with Social Security Numbers being used as the basis of authentication. I mentioned that the underlying problem was an assumption that security lay somewhere that it didn't.

A recent Fox News article about rogue wireless networks set up by criminals in airports and other public places demonstrates a similar failure to understand where security lies. As pointed out in the article, criminals can easily set up "trojan" networks in places where we would expect a legitimate one to exist.

But from a security point of view, worrying exclusively about these fake networks kind of misses the point. In attempting to make your computer and computer use secure, you should always assume that any network is inherently insecure. It should not matter whether you're connecting via the "official" JFK Airport network or its fake counterpart. The problem isn't that you need to avoid sending confidential details encrypted over the hacker's network, or update your antivirus specifically for connecting to that network. You should always be taking such measures for any network. If you've got your security policy right, then connecting via the hacker's network should be completely safe! You're security should not be relying on a particular network being "safe"; no network carries such a guarantee-- and especially no publicly accessible network (note that even if you had to type in a password to access the airport's network, it's still a public network!).

As well as having a paid-for firewall and antivirus that you keep up to date, you should be taking measures such as always accessing e-mail via an encrypted service, ensuring any financial transactions are made via an encrypted service, heeding warnings from your browser about problems with certifiates, not installing software from untrusted web sites, making sure you e-mail service has built in antivirus, and in any case not opening e-mails from suspicious recipients.

Social Security Numbers and "security": a case of misguided assumption

2009-07-07T03:02:00.003-05:00

The research on predicting social security numbers published today by Alessandro Acquisti and Ralph Gross from Carnegie Mellon University unfortunately highlights a fairly classic case of something we do all too often: basing security on something that was never secure-- and never really intended to be secure-- in the first place.

Much of the information that we readily pretend is a valid authentication key (such as Mother's Maiden Name, Date of Birth and Post Code-- and indeed Social Security Number) has really always been publicly available information. The parameter that has changed is how financially viable it is for a criminal to access the public records necessary to deduce this "secret" information. The SSN allocation scheme is perfectly well documented, public information, and the scheme clearly has no element of security built into it whatsoever. The historical origins of the scheme are also documented: the scheme has no security now, never did and was never intended to.

So what do we need to do about this? We need to understand where security actually lies, and not pretend that it exists in places where it doesn't. In most cases, the "security" does not currently lie in whether somebody can guess your PIN number, forge your signature, find out your mother's maiden name or guess the last couple of digits of your SSN. Our measures for preventing discovery of these largely unsecret "secrets" are predictably diabolical, and they are thus extremely weak forms of authentication. A transaction that is "authenticated" by an SSN or signature is essentially unauthenticated and the security of that transaction relies on it being quickly reversed in the event of fraud. So long as users, banks and lawmakers all understand this, the situation isn't so dire. The big danger comes when we pretend that there is security and authentication where there really isn't.

Java, XML and XPath

2009-07-03T14:10:00.003-05:00

The Javamex site now includes a basic introduction to using XML and XPath in Java. Java provides various means to read XML, but XPath is generally the most practical for moderately-sized XML documents. XPath effectively allows you to treat a document as a file system and refer to elements by their "path" within the document.

Random numbers in Java

2009-05-31T13:10:00.004-05:00

A few updates have been made to the site's section on generating random numbers in Java. Random numbers can crop up in all sorts of applications, and it's worth having a good understanding of them.

Traditionally, the bread-and-butter means of random number generation has been the java.util.Ranom class. The technique it uses (see the section on the java.util.Random algorithm for more details) is still suitable for some very casual applications of random number generation, such as some simple games. But in many cases, you should probably be thinking of moving away from java.util.Random towards a higher quality generator. As discussed in the section, problems with java.util.Random include its low period, biases in the different bits of the numbers generated, and its unsuitability for generating combinations of values. Using a weak random generator can have side effects such as the following:

it can skew the results of a test harness that introduces subtle biases in the code path due to the random number generator;
when testing the performance of data structures, and in various simulations, a generator such as java.util.Random will not produce a good range of possible combinations of values, giving false results;
the series of numbers can be predictable, leading to disasterous results if used as the basis for security (e.g. generating a random encryption key, nonce or session ID).

One interesting technique, published in 2003 by Goerge Marsaglia, is the XORShift generator. This generates medium-quality random numbers in a few simple machine instructions. For example, given a Java long, x, we can generate the next random number as simply as follows (in fact, various combinations of shift values are possible, as discussed in Marsaglia's paper):


  x ^= (x << 21);
  x ^= (x >>> 35);
  x ^= (x << 4);

continually executing these three lines will cycle through all (2^64)-1 possible values of x (i.e. all values of a long except zero) in pseudorandom order. Initialising x with the value of System.nanoTime(), or some other "random" seed, gives us a fast, medium-quality generator suitable for, say, generating random game data. It will make an excellent choice in many J2ME games, for example.

We also consider a Java implementation of the combined generator suggested by the Numerical Recipes authors, which generates numbers within a similar order of execution time as java.util.Random, but with a higher period and quality.

For applications where security depends on the quality and unpredictability of the random numbers generated, Java's SecureRandom class provides cryptographic strength random numbers, though is some 20-30 times slower than the other techniques.

New article: how to choose a Java collection

2009-05-30T10:24:00.002-05:00

A new addition to the Java Collections section of the Javamex site looks at a question that, perhaps unsurprisingly, crops up fairly frequently: which collections class should you use for a given task? The various collection classes provide a powerful means of managing objects and data in memory, but they're so powerful that choosing between them can sometimes be a bit daunting.

The approach that I take is to split the question firstly into two sub-questions. Firstly, what is the general type of structure that you need? That is, how is the data basically going to be organised? Usually, there is not too much confusion between a list and a map, especially if you consider whether or not you need to answer the question "for a given X, what is the Y"? But the difference between a set and a list is sometimes not well understood, or at least, not considered. With a little bit of thought, it is usually possible to decide in advance whether the purpose of your collection is to decide "is something there or not"? The trick is for programmers to remember to ask that question in the first place, and not simply plump for a list regardless.

Then, having decided on a list, map, set or queue, the next subquestion is obviously which particular flavour is required. In the case of a list, the cases where you wouldn't plump for an ArrayList are relatively uncommon and it should be clear in your head if you do choose something else that you have a "special case".

But in the case of maps, sets and queues, there are definitely more "horses for courses". But, especially in the first two, there are essentially two factors to consider: concurrency and ordering. As in the article, when the various choices are presented in tabular form according to these criteria, things become a little easier. As mentioned, a key rule of thumb is to choose the class that provides the minimum features that you require. If you don't need ordering, don't pay for it.

Java queues present a slightly complex set of choices, but in at least some cases-- especially a DelayQueue or SynchronousQueue-- it should be really clear that you need that class for a special case scenario. It should also be borne in mind that the executors framework means that in many common producer-consumer scenarios, you don't explicitly have to deal with the underlying job queue.

As usual, further questions and comments to any of the articles on Javamex are welcome on either this blog or the associated Java forum.

Patently stupid...

2009-05-27T00:01:00.003-05:00

Like anyone, I have my fair share of stop-the-world-I-want-to-get-off moments. But I really did have to check the calendar to make sure it wasn't April 1st today when I read through IBM's patent application for their "solution for providing real-time validation of text input fields using regular expression evaluation during text entry" (20090132950). Wading through all the verbiage, it really does appear that IBM think they have "invented" evaluating a regular expression inside an onChange event handler.

In terms of this specific patent application, I don't think many programmers are too worried that they're suddenly going to be prosecuted for calling RegExp in the wrong place-- needless to say, plenty of prior art is being cited, in case it were needed.

But the case does leave some other more interesting questions. Why does IBM of all companies think it needs to stoop this low? And how can it help us ditinguish the "bleeding obvious" from the "possibly patentable"?

I'm really at a loss to answer the first question. Possibly we're simply looking at a clerical error on the part of an overenthusiastic junior employee. But for a company that is supposedly at the forefront of computing research and invention, claiming patent rights on what amounts to a chain of JavaScript API calls doesn't really help them uphold their reputation.

In answer to the second question, I'm reminded of a comment by Donald Knuth, that he regularly sees patents granted to "solutions" to problems that he sets as undergraduate homework questions. If "how do you validate input as the user types into a web form?" had been posted as a question on one of the various Internet programming forums (and probably it has...), it would almost certainly have been tagged as "smells like homework". Few programmers, I suspect, would have tagged it as "goodness me how novel-- I think we should patent this".

I really have no idea how in the field of software patents you can concretely separate "significant invention" from "doing the bleeding obvious" (though, as in this case, I can sometimes recognise the latter when I see it!). But in order for a software solution to be patentable, I would at least expect to see:

- evidence of significant empirical research necessary to find the solution
- an invention of an actual algorithm, with perhaps some guage of number of API calls per other instructions/lines of code in order to determine if a new algorithm had actually been invented.

I would also like to see severe financial penalities for cases such as this, where a company is clearly attempting to use its abundant resources to abuse the system. The small developers that the patent system was originally designed to protect really have to think twice before committing to the thousands of dollars per year that it costs to keep a patent going. A company such as IBM really has nothing to loose by continually paying for nonesensical patent applications out of petty cash on the off-chance that at some point, one of them will sucessfully fly over the cuckoo's nest.

In the case of this particular patent application, I really really hope for the sake of human sanity that common sense fianlly prevails...

Java on your Kindle

2009-05-23T21:31:00.003-05:00

It's excellent to see an array of Java programming books now available on the Kindle. Various of these books, which you may not have considered buying due to their cost, are available at a sometimes significantly reduced price on the Kindle.

Of notable interest to Javamex readers will be Brian Goetz's venerable Java Concurrency in Practice, which is really something of a bible for information on the Java Memory Model and the Java 5 concurrency library. As I say in the review, I can pretty much guarantee that if you're writing concurrent code (as many of us either are or will soon have to, not just on the server, but increasingly client-side), then Java Concurrency in Practice will allow you to fix various bugs in your code that you were possibly unaware of (it certainly fixed some in mine!), as well as help you make decisions about how to architecture your program around the Java 5 concurrency utilities.

For a more light-hearted read, but still an extremely enlightening one, Java Puzzlers remains excellent as ever. Josh Bloch's Effective Java got a whole new lease of life when its second edition was published, and its addition to the Kindle repertoire is most welcome.

Also worthy of note are the Core Java books. These, and in particular the second volume of Advanced Features, is not the most succinct of works, but covers various Java topics that you don't readily see explained in detail in other books.

Java for beginners turorial: what would you like to see?

2009-05-23T19:40:00.002-05:00

Readers of the Javamex web site may or may not have seen the first pages of the site's Java for beginners tutorial, which covers a number of very basic Java topics such as variables, control structures (for loops, if/else), plus information on Java arrays.

Various topics are going to be added to the tutorial over the coming weeks. But how do you think it should be expanded? Are you just starting out in Java and having trouble with a particular topic? Or are you an experienced Java programmer, but remember having particular trouble with some aspect of learning Java with the tutorials that you used? Any feedback is welcome on this blog entry.

The secret life of the Java 'final' keyword

2009-05-12T14:57:00.004-05:00

I'll freely admit that an omission that the Javamex site should have addressed sooner is a more explicit discussion of the Java final keyword. Its use for guaranteeing thread-safety is hinted at in some of the articles, but some more explicit information has been long overdue.

As a step towards addressing this gap, the aforementioned article looks at how, as of Java 5, the final keyword has acquired a very important characteristic for thread-safe programming. Most programmers think of final in terms of its impact on program design or presumed optimisation (which, as I'll mention in a moment, is probably a red herring). But its thread-safety guarantee is far more significant: any field declared as final can be safely accessed by another thread once the constructor completes, whereas, subtly, without this or any other thread-safety mechanism, that guarantee does not hold.

The final keyword and optimisation

An issue with the Java final keyword, also criticised by Josh Bloch and Neal Gafter in their excellent Java Puzzlers book, is that it means different things in different places. When applied to a class or method, it means that that class or method cannot be extended or overridden. This has led to a general (false) conception that "final is to do with optimisation", and its importance in thread-safety has been overlooked.

Even when applied to classes and methods, the notion that final is about optimisation is probably false in most cases. The argument appears to stem from languages such as C++, where "compilation" is a one-off process. In such languages, there are optimisations that you can make to method calls and field accesses if you know that the classes and fields in question will never be overridden. But when you're running in a VM, whether a class or method might "potentially" be overridden doens't really matter. What counts is whether it has been overridden at a given moment. It it hasn't at the point of JIT-compilation, and the JVM makes certain optimisations based on that observation, but then later the class/method is overridden, the JVM generally has the luxury (which a C++ compiler or linker generally doesn't) of being able to re-compile.

So when applied to classes and methods (and in fact, local variables inside methods), final is essentially a program design feature. When applied to instance and class variables, final is an important thread-safety mechanism.

CyclicBarrier

2009-05-11T23:23:00.002-05:00

A new section of the Javamex web site looks at the CyclicBarrier class. In case you haven't come across it, CyclicBarrier is a construct that makes coordinating threads easier. Unlike CountDownLatch, which is designed for "one-off" coordinated actions, CyclicBarrier is designed to be re-used, so that it is suitable for iterative processes where the participating threads need to repeatedly perform a parallel operation, but periodically "converge" so that their results can be amalgamated.

In the article, we discuss the example of using a CyclicBarrier to coordinate a parallel sort algorithm. Essentially, the sort takes place in three stages. Each of the stages occurs in parallel, but at the end of each stage, a small single-threaded step must take place to amalgamate the results of the previous parallel operation.

Essentially, the code for a worker thread involved in the operation repeatedly carries out an operation and then calls the barrier's await() method. The latter method blocks until all participating threads have also called await() (we sometimes call this "arriving at the barrier"). At that point, CyclicBarrier executes our pre-determined "amalgamation" routine on one of the threads (the last one to arrive at the barrier, in fact), and then relases all the threads to move on to the next stage. Overall, the class takes out a lot of the actual thread coordination work, although, as we discuss in the article, we must still think about regular concurrency issues such as data synchronization and lock contention.

An additional feature of CyclicBarrier which we discuss is that it handles propagation of interruptions to all participating threads. In other words, if any thread involved in the operation is interrupted, then the whole operation will cease once all threads have called the await() method.

Whilst definitely one of the lesser used of the Java 5 concurrency utilities, CyclicBarrier is definitely a useful class that should not be overlooked.

Security issue with Adobe Reader

2009-04-29T15:02:00.002-05:00

You may not even have realised it, but for some reason, Adobe Reader can run JavaScript. Why on earth is that?, you might be asking. Isn't the point of a PDF file to store a printable document, not to run programs? Well, you'd have thought so.

But it turns out that, for whatever reason, Adobe Reader can run JavaScript. Not only that, but it can run it really badly. So badly, in fact, that is has a vulnerability whereby "an attacker can exploit this issue to execute arbitrary code the the privileges of the user running the application or crash the application, denying service to legitimate users" (SecurityFocus).

To get round this vulnerability, load Adobe Reader (ideally, don't load it by double-clicking on a PDF file that has been sent to you in an e-mail from an unknown person in South Korea...). Then, go to the Edit menu and to the Preferences option. The option you need is hidden away in the section marked "Javascript". Click on this in the list on the left hand side of the preferences menu, then make sure that the option Enable Acrobat JavaScript is not enabled.

Finally, never re-enable JavaScript in Adobe Reader or any other PDF reader application. There are certain features that are necessary in a printable text document reader application such as, well, text and the ability to print. But you really don't need JavaScript in PDF documents!!!!

On a related note, Microsoft have just announced a vulnerability in Notepad which allows a maliciously formatted txt document to accelerate the mutation of swine flu. Users are advised to paint a white cross on their door before launching any text document more than 4 characters in length.

New content

2009-04-19T00:04:00.003-05:00

A few new pages have been added to the site that you may be interested in:

The section on Java cryptography now considers password-based encryption, which we not too surprisingly conclude is fraught with difficulties! At present, we look at the PBE algorithms provided as standard in Sun's Java 6 implementation, although we unfortunately conclude that none of them are terribly great!
The section on Swing User interfaces covers various common topics, including a "rogue's gallery" of common Swing components, with a list of common constructors and listeners for those components, plus some hints on adding listeners to your code.

Further pages are planned for both of these sections very shortly. Watch this space!

Arcmexer: a library for reading archive files in Java

2009-04-16T16:50:00.004-05:00

A beta version of the Arcmexer library is now available from the Javamex site. Arcmexer is a library that allows you to read the contents of various types of archive file from Java. The idea is that the library could be useful in various data conversion and data recovery operations.

At present, the following are supported:

ZIP files, including those with files encrypted with 128-bit AES encryption or the traditional (but insecure) PKZIP encryption scheme. Note that other encryption algorithms (including 256-bit AES) currently aren't supported but may be in the future if I'm told that people need them. (See the page on reading encrypted ZIP files.)
Tar files, commonly used on UNIX platforms. Tar files may come directly off a tape drive (usually via the UNIX dd command), or are created on disk via the tar command and used to transport bundles of files.
GZIP-compressed tar files, commonly with the ending .tar.gz or .tgz.

Other archive formats are likely to be added to future versions if people nag me about them. (That means, leave a comment on this blog entry asking for it!)

As well as reading files from the archive, a method is provided that can aid in ZIP file password recovery.

Please bear in mind that this should very much be considered a beta version. I've found it the routines it contains useful and thought they could be useful for other people. If you encounter problems, please let me know! Comments can be left on this blog post, or on the site's Java discussion forum.

StreamCorruptedException

2009-04-15T13:25:00.002-05:00

The How to fix... section, looking at common Java bugs and problems, has an additional section on a couple of common causes of StreamCorruptedException. This is a serialisation error that often occurs when the streams reading and writing serialised data get "out of kilter". It's surprisingly common for this to occur accidentally.

If you haven't seen the How to fix... section yet, then it's worth a look, as it deals with a couple of other common headaches such as OutOfMemoryError and StackOverflowError. If you've got another Java bête noir that you'd like to see covered there, please let us know by posting to the Javamex forum.

Using strings in Java

2009-04-11T13:49:00.003-05:00

Various questions and problems commonly arise with Java Strings. Especially for C programmers, strings in Java have certain quirks, notably the fact that they're immutable (once you've created a String object, you can't change its contents; if you need a mutable string, then you generally have to use some other CharSequence class such as StringBuilder).

At the aforementioned link, I show code examples of some common string functions in Java. Surely, there'll be things I've missed out/forgotton. So please let me know if there's some thing you find yourself needing to do with strings in Java that I haven't mentioned!

New section: Java cryptography

2009-04-11T00:33:00.008-05:00

Several pages of the new section on Java Cryptography are now available for review and general criticism. Topics currently covered include:

how to encrypt data in Java, looking at encryption with symmetric block ciphers as well as the RSA encryption scheme commonly used for passing session keys;
different block modes, and how to use block modes in Java (including how to set an initialisation vector necessary in many cases via the IvParameterSpec class);
how to choose and set the key size to use with encryption, and how to remove the key length restriction which is imposed by current versions of Sun's JDK by default.

Topics not currently covered, but planned for the near future include:

comparison of block cipher algorithms: performance and security considerations;
secure hash functions and authentication;
cryptographic protocols;
secure random number generation (note that the Java SecureRandom class is currently discussed in the site's section on random numbers in Java);
digital signatures.

As usual, comments about the currently published sections are always welcome, as are suggestions for future additions to the cryptography section or to any other section of the web site. Please leave comments either on this blog or in the Java cryptography section of the Javamex discussion forum.

Update 12/04/09: The section now contains some information on secure hash functions in Java, plus a comparison of encryption algorithms.
19/04/09 As discussed in a separate blog entry, some information on password-based encryption is now included.

Enjoy...!

PIFTS.EXE: Symantec finally own up

2009-03-10T16:12:00.007-05:00

So, the world can rest easy in their beds. A message tucked away on Symantec's forums-- the same forums from which all communication about the issue was previously banned-- in which they have finally owned up to what happened:

they released a patch to do some boring things that any old patch might have done
but they released the patch unsigned, causing it to hit the firewall when it otherwise wouldn't have done
because some of the posts on the Symantec forum were judged to be abusive, all posts were pulled down.

Whilst this seems to be an astonishing example of customer relations, and has brought the world's attention to the kind of behaviour that such patches may be conducting on a routine basis, it does at least appear that the Feds are not about to plunder our computers for illicit chocolate chip cookie recipes. We were spared... this time.

(And yes, I did back up my recipe collection... just in case.)

What is the mysterious PIFTS.EXE?

2009-03-10T00:04:00.006-05:00

Update 10/03/09 11:50am Luckily, it does appear that PIFTS.EXE is just a storm in a teacup. Symantec still appear to be saying about as much as the Queen did after Diana died.

So after a mysterious PIFTS.EXE program hits the Kaspersky firewall asking to connect out from one of our machines, I hit the Internet to find that nobody knows, but the world is wondering. According to Google Trends, it has been hovering between the 15th and 25th most frequent search for the last couple of hours. Various theories about PIFTS.EXE appear to be emerging: was it some component of Norton Antivirus that went wrong? Is it some mad terrorist plot to wipe the Internet off the face of the earth and thus prevent people from finding out about why Lil' Kim went to jail?

Update 10/03/09 Reading things so far, and just possibly maybe having had a quick look at a disassembly of the exe in IDAPro, the consensus seems to be that the file is essentially harmless, but was an attempt by Symantec to gather some statistics from users' machines about installed antivirus components. A user posting to Reddit also suggests that a code review of PIFTS.EXE does not reveal anything too nafarious, and that automated code analysers such as ThreatExpert don't pull anything up either (then again, a "well written" virus wouldn't do, would it?).

Sometimes in cases like this, it's not so much whether there is anything wrong but whether there appears to be. Pulling down all forum posts about the file when there is clear user anxiety without then making an official statement doesn't making it look as though everything's hunky-dory...

Java Certification Tips

2009-01-22T20:05:00.004-06:00

A new page of Java Certification Tips gives a "cribsheet" of some of the "syntactic niggles" and other commonly overlooked features of Java that could trip up an otherwise moderately experienced programmer when taking the Sun Certified Java Programmer (SCJP) exam. Did you know, for example that:

this is invalid Java syntax?
float f = 2.5;
goto is a Java keyword?
this line does not assign the value 13 to the variable?:
int i = 013;

See the page of tips for more information on these and other niggles.

Regular expression tutorial: new example

2009-01-21T16:25:00.002-06:00

The Java regular expression tutorial section has been updated with a new example of using regular expressions. In this example, we look at how to perform what is sometimes referred to as HTML scraping: pulling out data from an HTML page (or indeed XML document).

The example is located here: HTML scraping with Java regular expressions. As explained in this tutorial, regular expressions are a good candidate for data scraping because they are flexible. Various libraries exist to parse an HTML or XML document and return an object representation of that document. But such libraries are often "too fussy": many if not most web pages actually do not conform strictly to HTML standards. Similarly, many XML parsing libraries are too fussy for real-life RSS feeds, which are often malformed, strictly speaking. Using regular expressions cuts through the fuss.

When should you scrape web pages?

Note that the article focusses on how technically to scrape HTML pages in Java. It doesn't deal with the "political" issue of whether the site in question wants its content scraped in the first place. In general, it is good practice to do the following:

find out if he web site in question has an API to provide the data in a more convenient format (and in a format that it would prefer to provide it to you in!)
be open about what you're doing: if the web site administrator things you're trying to "hide" something, they may block your IP address
respect server resources: if you are retrieving multiple pages, consider putting a thread sleep between fetches

Updates to the Spanish-English glossary

2008-12-30T22:43:00.003-06:00

Various new entries have been added to the Javamex site's Spanish-English glossary of computing terms. For those not familiar with the glossary, it contains English translations of various Spanish computing terms, covering various IT topics such as programming, networking, the Internet, software and hardware, GUIs etc.

Information on memory usage of objects

2008-12-29T19:04:00.003-06:00

The section on Java memory usage now contains the following additional articles:

information on how to calculate the memory usage of a Java object in general, considering the memory used for "housekeeping" by the JVM
calculating the memory usage of Strings, which can often the type of object to use up the biggest proportion of space in a Java application: this section actually considers the memory use of string-related objects such as StringBuffers and StringBuilders

A section on reducing the memory taken up by Strings looks at string canonicalisation, a fairly standard approach (but one which requires certain caveats), plus introduces the example of a CompactCharSequence class, that stores strings as 1 byte per character, thus taking up around half othe memory taken up by a regular Java String (at the expense of not supporting Unicode).

Comments on these articles welcome as usual.

Beta: Classmexer agent

2008-12-27T22:57:00.003-06:00

The beta version of a simple instrumentation tool is available for download from the Javamex site. The Classmexer jar provides various calls for querying the memory usage of Java objects. Via the provided MemoryUtil class.deepMemoryUsageOf() method, it is possible to get an estimate from the JVM of the number of bytes taken up by an object and its "subobjects" (objects referred to by a non-public reference, or by references with other visibility criteria). The memory usage of subobjects is combined recursively (so subobjects of subobjects are considered etc), but without counting the same object more than once.

A variant of the call is also provided which gives the total memory usage of several objects at a time, without counting as duplicates objects referenced by more than one of the objects.

Updates to profiling section

2008-12-25T23:18:00.002-06:00

Firstly, some minor corrections and additions to the Java profiling section. The corrections mainly concern a couple of typos that crept into the variable names of the examples. Readers should be reassured that the code, like that of the site in general, is copied and pasted from working, live profiling code. But things such as variable names are occasionally changed or shortened for the purposes of making it clearer on the site, and that seems to be where the errors crept in. I've also taken the opportunity to add a few links to other sections of the site (such as the section on threading, sleep() and yield()) that were added since the profiling tutorial was written.

Readers interested in Java profiling may also be interested in the first page of an upcoming section on Java and memory. This first page looks at how to find out the memory usage of a Java object. The technique involves using the Java Instrumentation framework introduced in version 5 of the language to query the JVM directly for the size of an object. Although slightly fiddly to set up, the technique has the advantage that there's less guesswork involved than if we were to just estimate an object's size (although future pages in the section will nonetheless look at estimation).

RSS feeds of Java tutorials

2008-12-10T22:13:00.002-06:00

The Javamex web site now publishes various RSS feeds containing links to articles published recently or on particular topics of frequent interest. The available feeds are as follows:

Main RSS feed: all Java programming articles published in the past 3 months
Java threading feed: recent articles on multithreading and concurrency (including both "classical" thread programming and the Java 5+ concurrency libraries)
Java Web programming feed: articles on topics such as Servlets and AJAX programming
Java collections and algorithms feed: articles covering the Java collections framework itself, plus other algorithm-related frameworks such as compression
Java performance feed: performance and profiling-related articles
Java Regular expressions feed

Suggestions are welcome if you think there's a feed on another theme that you think would be useful.

New section: using threads in Java

2008-12-05T17:47:00.005-06:00

The first articles in this new section look primarily at how to use "raw" threads in Java. As well as the basics such as how to create a thread and stop or interrupt it, the section looks at more advanced threading topics such as:

How threads work

A look at threads "under the hood": an examination of some of the details of thread scheduling, and the implications of different scheduling algorithms for Java threads and the methods that control them.

Thread priorities

How they work (or rather, how they don't work...) on specific operating systems. Did you know, for example:

on Linux, setPriority() has no effect pre Java 6, and that even then you need to run as rootand use a special command-line flag?
on Windows, the implementation of setPriority() changed between Sun's Java 5 and Java 6 implementations?
on Windows, thread priorities have little effect on threads competing for CPU?

See the article for more information.

sleep() and yield()

Information on the limitations and behaviour of sleep() under different load, sleep granularity, and bugs in the Windows implementation. And finally, find out what yield() actually does...!

Suggestions welcome!

As with all sections of the Javamex site, suggestions are very welcome here for new topics or specific questions you'd like to see answered. Updates and additions will be added periodically. Corrections are also very welcome since, particularly in the case of the some of the threading topics, I have tried to pull together information which is elusive or described in contradictory ways in different sources!

What is the Java equivalent of...?

2008-11-19T22:29:00.004-06:00

If you're a C/C++ programmer who has more recently moved into Java, you may welcome a new section of the Javamex site loosely entitled What is the Java equivalent of...?

The section aims to examine Java equivalents of some of those awkward little features of C/C++ that people tend to miss or not know how to achieve when they migrate to Java. For example:

What is the Java equivalent of unsigned? In C/C++, you can tell the compiler to treat an integer variable as unsigned by adding the unsigned modifier to its declaration. In Java, integer primitives are generally unsigned. But as we see, bitwise operations generally only require the right shift operator to be modified (so that Java uses >>> where C/C++ uses >> to operate on unsigned values). And for arithmetic operations, judicious use of the AND operator can often get us round the problem.

What is the Java equivalent of const? In C/C++, this operator tells the compiler not to allow the variable in question (or the value it points to) to be modified. We discuss the nearest Java equivalents, which actually depend on the circumstances.

The section on memory management in Java vs C/C++ looks at equivalents of operators such as new and malloc(), and their opposites. We see that in general, C/C++ puts more burden on the programmer to deal with memory management issues. Possible C/C++ bugs relating to memory management include allocating memory on the stack and then letting it "escape" the function; such bugs are not possible in Java. But there are issues that do crop up in Java that we sometimes need to be aware of, such as how garbage collection object finalization works, or how to deal with a "raw" block of memory (one typical use of malloc() in C/C++).

If you have a suggestion for a new topic in the Java equivalents section, please leave an appropriate comment on this blog. Corrections or suggestions for the existing articles are always welcome too!

Section on Java Servlets

2008-10-28T14:06:00.003-05:00

The newly updated section on writing Java Servlets, looks at issues such as the following:

some of the "mechanics" of getting start with Servlets, such as getting the appropriate version of the JDK and some Servlet hosting tips such as the features and quotas that you should be looking out for in a Servlet hosting package
the anatomy of an example Servlet
dealing with HTTP sessions, and why you should use the Java Session API to handle them;
dealing with "raw" cookies when you need them;
deciding if you need to modify your Servlet to work with keep-alive connections: in some configurations, you may be beneficial to add some code to set the Content-Length header; we discuss how to find out if this is necessary.

Web programmers may also be interested in the section on Java AJAX programming currently in development.

Comments and suggestions for extra material are always welcome, of course!

Improvements to section on 'volatile'

2008-08-12T17:46:00.002-05:00

The site's section on the Java volatile keyword has been expanded to include information such as:

a tabular comparison of volatile and synchronized
the definition of volatile and its significante under Java 5
clearer guidance on when to use volatile.

New section on sorting

2008-08-09T11:35:00.002-05:00

A new section on sorting has been added to the Java Collections section of the web site. The section currently contains information on:

Intrudction on sorting
Simple sorting of strings (sorting a list or array of "naturally sortable" objects)
The Comparable interface, used to dictate the "natural sort order" of objects
The false economy of compreTo optimisations
Comparators, used to dicate the sort order for a particular sort, or for classes whose sort order can't otherwise be specified.
An analysis of Java's sort algorithm, including a look at its performance.

The latter section shows that, if your application makes repeated sorts of very small data sets, an alternative sort algorithm may be beneficial.

Advanced use of hash codes

2008-07-16T17:18:00.003-05:00

For many purposes, the plain old HashMap and related collections serve their purpose well and you don't need to worry too much about their gory details.

But in some cases, some more advanced hashing-related techniques can help us save memory. In a new section on advanced hashing techniques, we discuss using a BitSet to perform duplicate elimination with a certain degree of error but a significant memory saving. Knowing a little about the statistics of hash codes helps us work out the degree of error involved. For example, with a BitSet taking up around 120K, we can perform duplicate elimination on tens of thousands of strings with a 'false positive' rate of around 2%. This is a technique that I use, for example, in gathering web site statistics, and in similar cases where I want the functionality of a hash map or hash set but can accept a certain degree of error. The technique allows what would otherwise be quite a large amount of data to be 'held in memory', thus significantly reducing database hits, for example.

Have you been using hash codes in an interesting way that I haven't mentioned? If so, I'd be interested to hear.

The wait/notify mechanism in Java

2008-07-06T10:11:00.002-05:00

One of the most commonly viewed articles on the Javamex site is our discussion of the wait/notify mechanism. It appears that this is a particular topic of doubt among developers.

The wait/notify mechanism is essentially used for one thread to "signal" or pass information to another thread. As of Java 5, the new concurrency classes provide more convenient alternatives to many common uses of wait() and notify(). See in particular:

The CountDownLatch class used to co-ordinate threads;
The BlockingQueue classes uesd to implement a producer-consumer pattern.

Suggested additions to these discussions are welcome (on this blog), as well as feedback on how well these articles helped you solve your problem.

Welcome!

2008-07-06T00:03:00.000-05:00

Welcome to the news and development blog for the Javamex web site. The site contains various resources for new and experienced Java developers.

Announcements about new and upcoming content will be posted to this blog. The blog also serves as a placeholder for comments about particular pages on the site.