I took some time on this holiday break to start experimenting with Silverlight and Flex, neither of which I have any experience with. While I was toying around with them I started thinking about the dizzying number of the RIA framework options.
With Silverlight I really liked developing in Visual Studio, it was intuitive and fast. I also liked that fact that I could use XAML to markup the layout, then back it up with C# for the event handling.
Flex looks good too. I especially like the fact that you can mark up your presentation in an XML format (MXML), similar to XAML. I am not a fan of ActionScript though, but I will have to see how east the IDE makes editing the code when I get that far.
Then there is GWT. Well, I liked that so much that I wrote a book about it. I like the fact that I can used my editor of choice (Eclipse), code in Java, and it is open and free.
I once used Ext-JS on a large project, and although it looked great it was extremely painful. When I got done the code was completely unmaintainable.
I have also dealt with Scriptaculous, Prototype, DWR, and other tools over the years, but for smaller projects that probably can't be considered RIAs.
Then there is Wicket, Echo2, blah, blah, blah... so many to choose from!
So which is best? I am partial to GWT, mostly because I know it the best, and it is free and has a great community. On the other hand, I could get used to Silverlight (although the cost can be prohibitive).
If you have an opinion I would like to hear it.
Showing posts with label Java. Show all posts
Showing posts with label Java. Show all posts
Monday, December 29, 2008
Saturday, December 13, 2008
Spring Security + Splunk = Security Monitoring
For many web sites Spring Security provides the framework to secure the site from unauthorized personal. This is a good start, but it won’t let you know when someone is testing your security to see if maybe they can sneak in. That is where Splunk comes in. Splunk is a tool that will analyze your log files real time, allowing reports and alerts to be generated. So if Splunk it to be the proverbial security guard in your security system, how do you get the data from Spring Security into Splunk? That is the topic of this article.
The Goal
We need to have some instructions for our virtual security guard. First and foremost the guard must look out for unwanted behavior and alert us. Let’s set a threshold of 20 failed login attempts in a five minute period. And when the guard sees 20 such attacks, they should send us an email. Of course we can tweak this formula depending on the specific site needs. For example, we could change the number of failed login attempts to something higher for high traffic sites, or have Splunk execute some script instead of emailing, which might do something like send an IM message.
A second goal is to have our guard keep a record of the guests that enter our site. We can use this information later for various things. For example, if someone broke something at 2am, we can check out records to see who was using the site at that time. Or perhaps we want to see usage patterns. For example, we want to know when the sales staff uses the web site so that we can plan maintenance. Or maybe we just had some layoffs and we forgot to disable their web accounts. We could use this log to verify that they didn’t use the site.
The Setup
We need to add a few things to our Java application in order to log security traffic. We assume that you are already using Spring Security, but if not, this is covered by their online documentation. What we need to add are a few jar files.
If you are using Maven, add these dependencies.
If you are using Ivy, add these.
If you need to download the jars and add them to the project manually, you can get them here. Not that Maven and Ivy will automatically download the dependencies of these three jar files, so if you add them manually you will need to also add their dependencies per the instructions on their project sites.
Spring Downloads
http://www.springsource.org/download
AspectJ Downloads
http://www.eclipse.org/aspectj/downloads.php
The Strategy
Spring Security is a framework meant to make it make it very easy to set up security, and in many cases the entire website can be secured without any coding. The facade that Spring Security provides is a blessing, but makes it difficult to alter its behavior to add the logging we need. That is where Aspect-Oriented Programming (AOP) comes in. You don’t need to know anything about AOP to implement this solution, but if you would like to know more, there is a pretty good AOP introduction on Wikipedia.
Within Spring Security there is an interface AuthenticationProvider, and that interface defines a method that interests us named authenticate. When you use Spring Security, no matter what method you allow a user to login to your site, they need to pass through one of the classes that implement this interface. This is ideal for us, and by using AOP we will be able to intercept these method calls and log the result to a file.
For the actual logging we will use commons-logging, a generic logging API that acts as a facade for whatever logging API you already use. If you have not used commons-logging before, the online documentation will provide details for connecting to your preferred logging system (Log4J, JDK, etc). You should already have this jar in your project because it is a dependency of the Spring AOP jar.
The format which we use to log the data needs to be easy for Splunk to parse so that we can get the most benefit out of this data. Fortunately Splunk can parse name/value pairs out of the box, so that is what we will use. The format will begin with a timestamp, followed by semi-colon delimited name/value pairs so that it looks something like this.
Besides the timestamp it says "[AUTH]". This is the name of the logger we use, and having the logger show up in the logs allows us to filter for these lines when we create our Splunk search. It also includes the application name. This allows us to search for authentication attempts for a specific application, assuming you need to log authentication events across multiple applications.
Ok, enough setup, let’s get to the code.
Coding the Aspect
The entire code of the class that intercepts the authenticate() calls is below. Throughout I have added markers (e.g. /*[1]*/), which I explain following the code. So take a quick look at the code then read the explanation.
Our class has a constructor [1], that takes the application name as an argument. This allows us to set the name, the only variable that might change, in the Spring configuration file making it easy to reuse this class.
Next we define two methods [2] logAuthPass() and logAuthFail(). This is the code that actually logs to commons-logging. Both methods take the name of the user attempting authentication as the sole argument. If for some reason commons-logging is not appropriate for your project, this should be the only piece of code that you would need to change.
Last is the method that actually intercepts the authorization call [3], the logAuth method. It receives a join-point object as its sole argument. If you aren’t familiar with AOP speak, this is essentially the information required to call the method that was intercepted. What has happened is that the intercepted authentication() call was never made, instead Spring intercepts it and passes the call information to logAuth() instead.
In logAuth() we first need to [3a] inspect the argument the intercepted call, namely an Authentication object. We can get this from the join-point object, and cast it. We then extract the username from that object. Just in case something goes wrong, we catch any exception that might occur. This should not happen, but you know the Boy Scout motto.
Next we use the join-point object [3b] to execute the method that was intercepted to get the result of the call. The behavior of authenticate() is to throw an exception on an authentication failure. So we trap the exception and log a failure. On success we hold on to the resulting object so that we can return it as the result of the call.
The last thing we need to do is [3c] test for a null return value. A null result does not mean auth success, but it does not mean a failure either. Per the Spring Security documentation a null result is due to the AuthenticationProvider not being able to answer the question of if the user is authenticated. Spring Security may then call other AuthenticationProviders until it can find an answer.
Now that we know how it works, it is time that we configure Spring to use it. The configuration file below does just that.
It is hard to explain what all of this does without getting using AOP speak, but basically it says that Spring should intercept calls to AuthenticationProvider.authenticate(), and call AuthLogger.logAuth() instead. The only variable that you need to change is the constructor argument passed in the AuthLogger bean definition, which is the name of your application.
Notice that we didn’t change any controller code, or even change your existing Spring Security configuration. The whole point of AOP is to separate concerns, and to not mix things like logging code with your application code.
The only thing left is to configure logging.
Configuring Logging
I am not a master of all logging tools, so I am going to stick with Log4J, which is perhaps the most popular Java logging tool available today.
For the purposes of this article I am using Log4J’s XML-based configuration, but you could use the properties based-configuration if you like. And if you already use Log4J, you will want to merge this into your existing configuration file.
I don’t want to get into the line-by-line details, but there are a few things to point out. First of all, the conversion pattern uses "[%d]" at the beginning of the line, which is the timestamp. In testing I noticed that when you don’t have the timestamp at the beginning of the line Splunk will sometimes see the "event" as being multiple lines long instead of each line being a separate event, losing searchable data. This is followed by the logger name "%c" and the message "%m". I prefer to log security events to a separate file, which allows me to specify a source type of "security" in Splunk. This allows you to conduct searches against "security" files across all of your applications.
Searching and Charting
Once you start logging events and having them ingested into Splunk it should be fairlry straight-forward to report on them. I will provide a few examples of thing you can do.
This searches for security authentication failures in the last seven days. The query assumes that when you added the security log file you specified its source type as "security". As mentioned, this is useful if you want to search the security logs across multiple log file locations as the same time.
This is the same query, but also looks for the name of the logger "[AUTH]". This is useful if you want to use the security log for logging other events as well, like when an admin runs as another user, like "su name" in Unix. You could log these with the logger "[RUNAS]" in the same security log.
This query takes the original query and charts it. Best viewed as a bar chart it shows a count of authentication failures per day, splitting it by application. Adding this to a security dashboard that you review each day (hopefully) gives you visual indication if there is some hanky-panky going on.
All of this reporting should be supplemented with alerting. In Splunk take this search and create a saved search from it. On the Schedule and Alert tab (assuming you used the pop-up), have it run on a schedule every five minutes. Then specify that it should alert when the number of events is greater than 20, and enter your email address, or better yet your SMS email address. You may also want it to send the events, it will do this as a CSV file, but be cautions, because if someone is trying to break into your site, this file could get large.
This concludes the article. If you tried out using this and made your own tweaks, leave a comment, you may be helping out someone else. And as always, if you liked/hated/questions this article, constructive comments are always welcome.
Happy Splunking.
The Goal
We need to have some instructions for our virtual security guard. First and foremost the guard must look out for unwanted behavior and alert us. Let’s set a threshold of 20 failed login attempts in a five minute period. And when the guard sees 20 such attacks, they should send us an email. Of course we can tweak this formula depending on the specific site needs. For example, we could change the number of failed login attempts to something higher for high traffic sites, or have Splunk execute some script instead of emailing, which might do something like send an IM message.
A second goal is to have our guard keep a record of the guests that enter our site. We can use this information later for various things. For example, if someone broke something at 2am, we can check out records to see who was using the site at that time. Or perhaps we want to see usage patterns. For example, we want to know when the sales staff uses the web site so that we can plan maintenance. Or maybe we just had some layoffs and we forgot to disable their web accounts. We could use this log to verify that they didn’t use the site.
The Setup
We need to add a few things to our Java application in order to log security traffic. We assume that you are already using Spring Security, but if not, this is covered by their online documentation. What we need to add are a few jar files.
If you are using Maven, add these dependencies.
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-aop</artifactId>
<version>2.5.6</version>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjrt</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>1.6.1</version>
</dependency>
If you are using Ivy, add these.
<dependency org="org.springframework" name="spring-aop" rev="2.5.6"/>
<dependency org="org.aspectj" name="aspectjrt" rev="1.6.1"/>
<dependency org="org.aspectj" name="aspectweaver" rev="1.6.1"/>
If you need to download the jars and add them to the project manually, you can get them here. Not that Maven and Ivy will automatically download the dependencies of these three jar files, so if you add them manually you will need to also add their dependencies per the instructions on their project sites.
Spring Downloads
http://www.springsource.org/download
AspectJ Downloads
http://www.eclipse.org/aspectj/downloads.php
The Strategy
Spring Security is a framework meant to make it make it very easy to set up security, and in many cases the entire website can be secured without any coding. The facade that Spring Security provides is a blessing, but makes it difficult to alter its behavior to add the logging we need. That is where Aspect-Oriented Programming (AOP) comes in. You don’t need to know anything about AOP to implement this solution, but if you would like to know more, there is a pretty good AOP introduction on Wikipedia.
Within Spring Security there is an interface AuthenticationProvider, and that interface defines a method that interests us named authenticate. When you use Spring Security, no matter what method you allow a user to login to your site, they need to pass through one of the classes that implement this interface. This is ideal for us, and by using AOP we will be able to intercept these method calls and log the result to a file.
For the actual logging we will use commons-logging, a generic logging API that acts as a facade for whatever logging API you already use. If you have not used commons-logging before, the online documentation will provide details for connecting to your preferred logging system (Log4J, JDK, etc). You should already have this jar in your project because it is a dependency of the Spring AOP jar.
The format which we use to log the data needs to be easy for Splunk to parse so that we can get the most benefit out of this data. Fortunately Splunk can parse name/value pairs out of the box, so that is what we will use. The format will begin with a timestamp, followed by semi-colon delimited name/value pairs so that it looks something like this.
[2008-12-10 12:34:08,526] [AUTH] app=my-website;auth=pass;user=rhanson
[2008-12-10 17:52:31,846] [AUTH] app=my-website;auth=fail;user=joehacker
Besides the timestamp it says "[AUTH]". This is the name of the logger we use, and having the logger show up in the logs allows us to filter for these lines when we create our Splunk search. It also includes the application name. This allows us to search for authentication attempts for a specific application, assuming you need to log authentication events across multiple applications.
Ok, enough setup, let’s get to the code.
Coding the Aspect
The entire code of the class that intercepts the authenticate() calls is below. Throughout I have added markers (e.g. /*[1]*/), which I explain following the code. So take a quick look at the code then read the explanation.
package org.roberthanson.loggers;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.aspectj.lang.ProceedingJoinPoint;
import org.springframework.security.Authentication;
public class AuthLogger
{
private String appName;
private Log authLog = LogFactory.getLog("AUTH");
/*[1]*/
/**
* Constructor.
* @param appName name of the application
*/
public AuthLogger (String appName)
{
this.appName = appName;
}
/*[2]*/
private void logAuthPass (String userName)
{
if (authLog.isInfoEnabled()) {
authLog.info("app=" + appName + ";auth=pass;user=" + userName);
}
}
private void logAuthFail (String userName)
{
if (authLog.isInfoEnabled()) {
authLog.info("app=" + appName + ";auth=fail;user=" + userName);
}
}
/*[3]*/
/**
* Wraps a call to Spring Security's
* AuthenticationProvider.authenticate().
*/
public Object logAuth (ProceedingJoinPoint call) throws Throwable
{
Authentication result;
String user = "UNKNOWN";
try { /*[3a]*/
Authentication auth = (Authentication) call.getArgs()[0];
user = auth.getName();
}
catch (Exception e) {
// ignore
}
try { /*[3b]*/
result = (Authentication) call.proceed();
}
catch (Exception e) {
logAuthFail(user);
throw e;
}
if (result != null) { /*[3c]*/
logAuthPass(user);
}
return result;
}
}
Our class has a constructor [1], that takes the application name as an argument. This allows us to set the name, the only variable that might change, in the Spring configuration file making it easy to reuse this class.
Next we define two methods [2] logAuthPass() and logAuthFail(). This is the code that actually logs to commons-logging. Both methods take the name of the user attempting authentication as the sole argument. If for some reason commons-logging is not appropriate for your project, this should be the only piece of code that you would need to change.
Last is the method that actually intercepts the authorization call [3], the logAuth method. It receives a join-point object as its sole argument. If you aren’t familiar with AOP speak, this is essentially the information required to call the method that was intercepted. What has happened is that the intercepted authentication() call was never made, instead Spring intercepts it and passes the call information to logAuth() instead.
In logAuth() we first need to [3a] inspect the argument the intercepted call, namely an Authentication object. We can get this from the join-point object, and cast it. We then extract the username from that object. Just in case something goes wrong, we catch any exception that might occur. This should not happen, but you know the Boy Scout motto.
Next we use the join-point object [3b] to execute the method that was intercepted to get the result of the call. The behavior of authenticate() is to throw an exception on an authentication failure. So we trap the exception and log a failure. On success we hold on to the resulting object so that we can return it as the result of the call.
The last thing we need to do is [3c] test for a null return value. A null result does not mean auth success, but it does not mean a failure either. Per the Spring Security documentation a null result is due to the AuthenticationProvider not being able to answer the question of if the user is authenticated. Spring Security may then call other AuthenticationProviders until it can find an answer.
Now that we know how it works, it is time that we configure Spring to use it. The configuration file below does just that.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:aop="http://www.springframework.org/schema/aop"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://www.springframework.org/schema/aop
http://www.springframework.org/schema/aop/spring-aop-2.5.xsd">
<bean id="authLogger" class="org.roberthanson.loggers.AuthLogger">
<constructor-arg value="my-website" />
</bean>
<aop:config>
<aop:aspect id="authLoggerAspect" ref="authLogger">
<aop:pointcut id="authPointcut" expression="execution(public * org.springframework.security.providers.AuthenticationProvider.authenticate(..))" />
<aop:around pointcut-ref="authPointcut" method="logAuth" />
</aop:aspect>
</aop:config>
</beans>
It is hard to explain what all of this does without getting using AOP speak, but basically it says that Spring should intercept calls to AuthenticationProvider.authenticate(), and call AuthLogger.logAuth() instead. The only variable that you need to change is the constructor argument passed in the AuthLogger bean definition, which is the name of your application.
Notice that we didn’t change any controller code, or even change your existing Spring Security configuration. The whole point of AOP is to separate concerns, and to not mix things like logging code with your application code.
The only thing left is to configure logging.
Configuring Logging
I am not a master of all logging tools, so I am going to stick with Log4J, which is perhaps the most popular Java logging tool available today.
For the purposes of this article I am using Log4J’s XML-based configuration, but you could use the properties based-configuration if you like. And if you already use Log4J, you will want to merge this into your existing configuration file.
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="security-file" class="org.apache.log4j.RollingFileAppender">
<param name="File" value="/usr/local/tomcat/logs/security.log"/>
<param name="Append" value="true"/>
<param name="MaxFileSize" value="2MB"/>
<param name="MaxBackupIndex" value="5"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="[%d] [%c] %m%n"/>
</layout>
</appender>
<category name="AUTH" additivity="false">
<level value="info"/>
<appender-ref ref="security-file"/>
</category>
</log4j:configuration>
I don’t want to get into the line-by-line details, but there are a few things to point out. First of all, the conversion pattern uses "[%d]" at the beginning of the line, which is the timestamp. In testing I noticed that when you don’t have the timestamp at the beginning of the line Splunk will sometimes see the "event" as being multiple lines long instead of each line being a separate event, losing searchable data. This is followed by the logger name "%c" and the message "%m". I prefer to log security events to a separate file, which allows me to specify a source type of "security" in Splunk. This allows you to conduct searches against "security" files across all of your applications.
Searching and Charting
Once you start logging events and having them ingested into Splunk it should be fairlry straight-forward to report on them. I will provide a few examples of thing you can do.
sourcetype="security" daysago=7 auth=fail
This searches for security authentication failures in the last seven days. The query assumes that when you added the security log file you specified its source type as "security". As mentioned, this is useful if you want to search the security logs across multiple log file locations as the same time.
"[AUTH]" sourcetype="security" daysago=7 auth=fail
This is the same query, but also looks for the name of the logger "[AUTH]". This is useful if you want to use the security log for logging other events as well, like when an admin runs as another user, like "su name" in Unix. You could log these with the logger "[RUNAS]" in the same security log.
sourcetype="security" daysago=7 auth=fail | timechart count(_raw) by app
This query takes the original query and charts it. Best viewed as a bar chart it shows a count of authentication failures per day, splitting it by application. Adding this to a security dashboard that you review each day (hopefully) gives you visual indication if there is some hanky-panky going on.
sourcetype="security" auth=fail minutesago=5
All of this reporting should be supplemented with alerting. In Splunk take this search and create a saved search from it. On the Schedule and Alert tab (assuming you used the pop-up), have it run on a schedule every five minutes. Then specify that it should alert when the number of events is greater than 20, and enter your email address, or better yet your SMS email address. You may also want it to send the events, it will do this as a CSV file, but be cautions, because if someone is trying to break into your site, this file could get large.
This concludes the article. If you tried out using this and made your own tweaks, leave a comment, you may be helping out someone else. And as always, if you liked/hated/questions this article, constructive comments are always welcome.
Happy Splunking.
Thursday, December 04, 2008
Log Analytics: Splunk
Sure, I use logs to troubleshoot problems, but I never gave it much thought as an analytics device since the rise of JavaScript based traffic analysis tools like Omniture and Google Analytics. I must say that I have sorely underestimated the power of logging.
I say this because of a tool that I have begun using called Splunk. If I had to characterize Splunk, I would say that it is a data analysis tool, with the ability to generate reports and alerts based on events.
The data could be an Apache log file, the output of the PS command, or a configuration file that you need to watch for changes. Input sources can come via file, via TCP/UDP port, and other mechanism.
The analysis of the data is done via a search, but the language is quite complex. The search allows you to target a specific data source, or class of sources (like all of your access logs), extract fields from the log entries using a regex (or just split key/val pairs), transform the data, and them chart it as a line, pie, bar, or other type of chart.
Here are some specific examples of what I have been doing with it.
Response Times - With a little aspect, I was able to wrap all of my controller methods (and some repository methods too), and have them dump response time data to a file. The data looks like this.
All of this data is logged using commons-logging, and put into a separate performance.log file to separate it out from other logging. The result is that I can create a search for any class/method, and generate a chart of min/avg/max response times for that method. This allows me to spot any degradation that might occur over time (as the DB gets larger), or during certain parts of the day (when the network is congested). I can then set up an alert so that if the response times exceed a threshold, Splunk will send me an email (or execute a script).
Change Control - Simply put, I can have Splunk monitor a configuration file, and it will log any changes to that file. I can then use Splunk's diff command to see the actual change. Again, I can have it email me when a change occurs.
Service Unavailability - Because all of my Java application servers sit behind an Apache proxy, I can monitor the Apache logs and look for proxy errors (501, 503, 504). Splunk is able to alert me when these events occur because it can parse the logs and look specifically for these error codes. With Apache specifically this is super each because Splunk already knows how to parse an Apache log using a standard format. This can be more effective that monitoring that uses polling because polling will only let you know if the service is done when the polling occurred, when Splunk can alert me even if only a single hit against the site caused a proxy error.
There is much more to Splunk, and I still have much to learn, but I hope this helps provide some insight as to its capabilities.
I say this because of a tool that I have begun using called Splunk. If I had to characterize Splunk, I would say that it is a data analysis tool, with the ability to generate reports and alerts based on events.
The data could be an Apache log file, the output of the PS command, or a configuration file that you need to watch for changes. Input sources can come via file, via TCP/UDP port, and other mechanism.
The analysis of the data is done via a search, but the language is quite complex. The search allows you to target a specific data source, or class of sources (like all of your access logs), extract fields from the log entries using a regex (or just split key/val pairs), transform the data, and them chart it as a line, pie, bar, or other type of chart.
Here are some specific examples of what I have been doing with it.
Response Times - With a little aspect, I was able to wrap all of my controller methods (and some repository methods too), and have them dump response time data to a file. The data looks like this.
app=my-app;class=org.example.Demo;method=doSearch;response=348
app=my-app;class=org.example.Demo;method=doSearch;response=654
app=my-app;class=org.example.Demo;method=doSearch;response=439
All of this data is logged using commons-logging, and put into a separate performance.log file to separate it out from other logging. The result is that I can create a search for any class/method, and generate a chart of min/avg/max response times for that method. This allows me to spot any degradation that might occur over time (as the DB gets larger), or during certain parts of the day (when the network is congested). I can then set up an alert so that if the response times exceed a threshold, Splunk will send me an email (or execute a script).
Change Control - Simply put, I can have Splunk monitor a configuration file, and it will log any changes to that file. I can then use Splunk's diff command to see the actual change. Again, I can have it email me when a change occurs.
Service Unavailability - Because all of my Java application servers sit behind an Apache proxy, I can monitor the Apache logs and look for proxy errors (501, 503, 504). Splunk is able to alert me when these events occur because it can parse the logs and look specifically for these error codes. With Apache specifically this is super each because Splunk already knows how to parse an Apache log using a standard format. This can be more effective that monitoring that uses polling because polling will only let you know if the service is done when the polling occurred, when Splunk can alert me even if only a single hit against the site caused a proxy error.
There is much more to Splunk, and I still have much to learn, but I hope this helps provide some insight as to its capabilities.
Saturday, July 05, 2008
Launched Penlets.com, for Pulse Pen Users
<backstory>
Back in May I was at JavaOne and had a great time. Among other things I picked up a Pulse pen from Livescribe. If you haven't heard about the Pulse pen, here is the short version... The pulse pen is a computer in a pen, allowing you to record writing and audio as well as write penlet applications in Java. Pre-installed demos include a piano that your draw on the paper then tap to play, and a translator that can translate written words into several other languages.
</backstory>
A colleague and myself were really blown away by the possibilities, and picked up a few of the pens at JavaOne so that we could play around with the Java API. In doing so we learned some things that weren't completely documented, and required some trial and error. So we thought that if we were already doing the research that we might as well publish it on the web. In doing so we created Penlets.com. The site has tutorials, applications that you can install on your pen, and Pulse pen related resources.
So, if you picked of one of these cool pens, we hope that you will visit Penlets.com
Cheers.
Back in May I was at JavaOne and had a great time. Among other things I picked up a Pulse pen from Livescribe. If you haven't heard about the Pulse pen, here is the short version... The pulse pen is a computer in a pen, allowing you to record writing and audio as well as write penlet applications in Java. Pre-installed demos include a piano that your draw on the paper then tap to play, and a translator that can translate written words into several other languages.
</backstory>
A colleague and myself were really blown away by the possibilities, and picked up a few of the pens at JavaOne so that we could play around with the Java API. In doing so we learned some things that weren't completely documented, and required some trial and error. So we thought that if we were already doing the research that we might as well publish it on the web. In doing so we created Penlets.com. The site has tutorials, applications that you can install on your pen, and Pulse pen related resources.
So, if you picked of one of these cool pens, we hope that you will visit Penlets.com
Cheers.
Tuesday, April 29, 2008
Countdown to JavaOne 2008 (and Twittering)
This will be my first year going to JavaOne, and I am expecting to have a lot of fun. In creating my schedule I was a little surprised by the number of sessions on scripting languages. I guess I shouldn't be though, there has been a lot of news regarding scripting languages on the JVM in the last year. Here is a sampling of the sessions.
Add to this a strong sampling of JavaFX sessions, Ajax sessions, and a session on Sun's new Fortress language provides a strong sense of what we can expect in the next ten years. Frankly I was hoping that my scripting days were behind me, but it looks like I need to get my hands dirty again. I just wish that there was a JPerl!
By the way, if you didn't notice, I decided to try out this Twitter thingy. I am thinking that I will Twitter from JavaOne in between sessions. Are there any other programmer type Twitterers out there? Anyone else Twittering from JavaOne?
- JRuby: Why, What, How...Do It Now
- JavaScript Programming Language: The Language Everybody Loves to Hate
- Comparing JRuby and Groovy
- Programming with Functional Objects in Scala
Add to this a strong sampling of JavaFX sessions, Ajax sessions, and a session on Sun's new Fortress language provides a strong sense of what we can expect in the next ten years. Frankly I was hoping that my scripting days were behind me, but it looks like I need to get my hands dirty again. I just wish that there was a JPerl!
By the way, if you didn't notice, I decided to try out this Twitter thingy. I am thinking that I will Twitter from JavaOne in between sessions. Are there any other programmer type Twitterers out there? Anyone else Twittering from JavaOne?
Tuesday, April 22, 2008
.NET is leapfrogging Java and fairies just stole my underpants
I received an email from Manning along with a link to their forum that caused me to pause. It was titled "Has .NET Leapfrogged Java?".
It seems that an anonymous author had three facts that made this true.
1. .NET has LINQ
2. C# had generics/foreach before Java.
3. .NET has Python, Ruby, F# and others.
These "facts" seemed a little light to me, and I said as much.
If you have an opinion on the matter of Java vs. C#, you might want to check this out.
It seems that an anonymous author had three facts that made this true.
1. .NET has LINQ
2. C# had generics/foreach before Java.
3. .NET has Python, Ruby, F# and others.
These "facts" seemed a little light to me, and I said as much.
If you have an opinion on the matter of Java vs. C#, you might want to check this out.
Saturday, March 08, 2008
Review: Project Management with Jira
According to the Jira website, Jira is...
I had already been a casual user of Jira, but due to a business need I needed to look into it a bit further. I am rather impressed with the product, so I wanted to share some of my experiences.
Try Before You Buy
Atlassian, the company that publishes Jira really got this one right. Once you register on the site for free you can then download the application and generate a 30-day key. The key is tied to the "server ID", so you need to install Jira first before you generate the key. Once you plug in the key you now have access to the Enterprise edition, fullt functional in every way.
In my case we wanted to get some real-life experience, so we started using it to manage a real project. The project ran longer than the 30-days, but to Atlassian's credit, I went back to their website and generated another key that allowed us to use Jira for an additional 30-days. I did this several times, so you could potentially evaluate the product for many months if you needed to.
Sales Calls - None!
When you sign up on the Atlassian website and generate a product key, you will receive an automatic email from a Jira sales representative that says something to the affect of "call me if you need help". Besides that automatic email, I received no emails or calls from a sales representative. I really appreciated this.
The Install
This was incredibly easy. You simply download the zip/tar, decompress it, and run the start script. Jira has a stand-alone download option that ships with Tomcat and by default uses Hypersonic DB, which for evaluation purposes is all you need. The longest part of the install process for me was downloading the Jira distribution and copying it up to a server.
The Tools - Security
Jira has quite a few options for security. You can manage users locally, or point Jira to an LDAP server (which I have not tried yet). The users can then belong to groups, and users and groups can be assigned to projects. Each project as "users", "developers", and "admin" roles, and you can define what each of those roles mean on a per-project basis. And if that isn't enough, you can create individual issue level security schemes, and specify the visibility of comments and work logs.
In short, the security capabilities will meet all but the fringe use cases. You can read more about Jira's security setup in the Jira on-line documentation.
The Tools - Mailboxes, Workflows, and More...
There are just way too many good features to explain each in detail, so perhaps a quick summary will suffice.
The easiest way to find out if the tool has the features you will need is to just download it and try it out. For me this provided much more information than just reading the documentation on the Jira website.
Enhancing and Extending Jira
Jira makes it pretty easy to create plug-ins that you can distribute as a jar file and just drop into Jira's lib folder. Each plug-in jar file has an atlassian-plugin.xml file in it that described the nature of the plug-in to Jira. The plug-in may include reports, admin tools, extensions to the project and issue interfaces, or resources like images and CSS files. And not only can you add features, you can also extend existing ones in order to change their behavior.
This is all well and good, but I quickly found some limitations. First, the documentation is typically light on needed information, is buggy, or is a little old. In general I found that I needed to do a bit of searching between the Jira documentation, issue tracking system, and the forums in order to get the information I needed, and it still didn't answer all of my questions. For the rest I took a look at existing plug-ins to see how they had solved the problems.
Beyond the documentation one missing piece I found was that you can't use the Jira persistence engine without tweaking Jira's configuration files. This is problematic for me because I want to stay away from that, as altering Jira's configuration will make upgrades harder as I would need to re-alter the config for each upgrade, and retest the application as well.
In all fairness this is often not needed, as you can often attach arbitrary properties to whatever business object you want. In my case though, the Project object does not support this, so I needed to look at other alternatives for persisting additional project properties.
So mixed results here, but in general Jira plug-in system is pretty good.
If you need to perform customizations that go beyond a simple plug-in, paying customers of Jira can download the Java source code of Jira. This allows you to change the guts of Jira to your hearts content. Of course, this will make upgrading extremely difficult, but it is nice to know the option exists. This is also useful if you want to see what Jira is doing under the covers, making plug-in development a little easier.
The Cost
Software is expensive, especially if it isn't your standard home-user type software. So for what Jira is, it is fairly inexpensive, with options from around $500 to $4000. This also includes a year of upgrades plus email support for a year. This is a deal compared to other business applications.
For open-source teams the deal is even sweeter. You can use Jira for free! This would explain why a lot of open-source projects have adopted Jira.
In Summary
Features great, extensibility not bad, the price good. If you are looking for an issue/bug tracking tool and want a little more than the open-source alternatives, Jira is a great choice.
JIRA is a bug tracking, issue tracking, and project management application developed to make this process easier for your team. JIRA has been designed with a focus on task achievement, is instantly usable and is flexible to work with.
I had already been a casual user of Jira, but due to a business need I needed to look into it a bit further. I am rather impressed with the product, so I wanted to share some of my experiences.
Try Before You Buy
Atlassian, the company that publishes Jira really got this one right. Once you register on the site for free you can then download the application and generate a 30-day key. The key is tied to the "server ID", so you need to install Jira first before you generate the key. Once you plug in the key you now have access to the Enterprise edition, fullt functional in every way.
In my case we wanted to get some real-life experience, so we started using it to manage a real project. The project ran longer than the 30-days, but to Atlassian's credit, I went back to their website and generated another key that allowed us to use Jira for an additional 30-days. I did this several times, so you could potentially evaluate the product for many months if you needed to.
Sales Calls - None!
When you sign up on the Atlassian website and generate a product key, you will receive an automatic email from a Jira sales representative that says something to the affect of "call me if you need help". Besides that automatic email, I received no emails or calls from a sales representative. I really appreciated this.
The Install
This was incredibly easy. You simply download the zip/tar, decompress it, and run the start script. Jira has a stand-alone download option that ships with Tomcat and by default uses Hypersonic DB, which for evaluation purposes is all you need. The longest part of the install process for me was downloading the Jira distribution and copying it up to a server.
The Tools - Security
Jira has quite a few options for security. You can manage users locally, or point Jira to an LDAP server (which I have not tried yet). The users can then belong to groups, and users and groups can be assigned to projects. Each project as "users", "developers", and "admin" roles, and you can define what each of those roles mean on a per-project basis. And if that isn't enough, you can create individual issue level security schemes, and specify the visibility of comments and work logs.
In short, the security capabilities will meet all but the fringe use cases. You can read more about Jira's security setup in the Jira on-line documentation.
The Tools - Mailboxes, Workflows, and More...
There are just way too many good features to explain each in detail, so perhaps a quick summary will suffice.
- Jira can listen to a POP3/IMAP mailbox and auto-create new issues for you.
- Define workflows, the steps from project creation to completion.
- Create custom issue fields, with a dozen or so field types built-in.
- Backup everything to, and restore from XML files.
- Turn on logging/profiling on the fly to debug Jira performance issues.
- Built in SOAP and XML-RPC interfaces (although creating a SOAP client to connect to Jira was non-trivial in my case)
- Run Jelly scripts to perform maintenance tasks.
- Customize resolutions, priorities, and statuses.
- CVS (and SVN with a plug-in) integration.
- Integration with FishEye and Crowd, other Atlassian products.
- And more...
The easiest way to find out if the tool has the features you will need is to just download it and try it out. For me this provided much more information than just reading the documentation on the Jira website.
Enhancing and Extending Jira
Jira makes it pretty easy to create plug-ins that you can distribute as a jar file and just drop into Jira's lib folder. Each plug-in jar file has an atlassian-plugin.xml file in it that described the nature of the plug-in to Jira. The plug-in may include reports, admin tools, extensions to the project and issue interfaces, or resources like images and CSS files. And not only can you add features, you can also extend existing ones in order to change their behavior.
This is all well and good, but I quickly found some limitations. First, the documentation is typically light on needed information, is buggy, or is a little old. In general I found that I needed to do a bit of searching between the Jira documentation, issue tracking system, and the forums in order to get the information I needed, and it still didn't answer all of my questions. For the rest I took a look at existing plug-ins to see how they had solved the problems.
Beyond the documentation one missing piece I found was that you can't use the Jira persistence engine without tweaking Jira's configuration files. This is problematic for me because I want to stay away from that, as altering Jira's configuration will make upgrades harder as I would need to re-alter the config for each upgrade, and retest the application as well.
In all fairness this is often not needed, as you can often attach arbitrary properties to whatever business object you want. In my case though, the Project object does not support this, so I needed to look at other alternatives for persisting additional project properties.
So mixed results here, but in general Jira plug-in system is pretty good.
If you need to perform customizations that go beyond a simple plug-in, paying customers of Jira can download the Java source code of Jira. This allows you to change the guts of Jira to your hearts content. Of course, this will make upgrading extremely difficult, but it is nice to know the option exists. This is also useful if you want to see what Jira is doing under the covers, making plug-in development a little easier.
The Cost
Software is expensive, especially if it isn't your standard home-user type software. So for what Jira is, it is fairly inexpensive, with options from around $500 to $4000. This also includes a year of upgrades plus email support for a year. This is a deal compared to other business applications.
For open-source teams the deal is even sweeter. You can use Jira for free! This would explain why a lot of open-source projects have adopted Jira.
In Summary
Features great, extensibility not bad, the price good. If you are looking for an issue/bug tracking tool and want a little more than the open-source alternatives, Jira is a great choice.
Thursday, December 20, 2007
Testing Servlets with JUnit
In my day-to-day job I write a lot of unit tests, and I try to test everything I can. Recently though I had written a service that I was unable to test completely. One of the methods in the service had the job of sending an HTTP request to a remote server and responding with the results. For my project I used commons-httpclient to send the request.
If you have used commons-httpclient, this is about as simple as it gets, but I still wanted to have a unit test for it. So the problem became, "how can I test this method when it requires that I hit a web site". After some searching I found that the Jetty servlet-container has a ServletTester class just for this purpose.
In my Maven 2 I included the ServletTester using the following repository and dependency information. I always use Maven 2 as allows other developers to quickly set up their IDE and download all required JARs. If you don't use Maven, you will need to manually download all of the dependencies.
With this in place the next step was writing the JUnit test cases. For me I wanted to initialize the servlet-container once, then run a set of tests against it. In JUnit 4 you can use the @BeforeClass and @AfterClass annotations to mark methods that should be executed before and after all of the tests.
The code highlighted in blue is where we start an instance of the server. I created a new instance of the ServletTester, setting the context path and adding a servlet mapping. This alone does not bind the server to a port, for that you need to call createSocketConnector(true), which binds the server to a local port and returns the URL. The port used will be a high unused port. I save an instance of the ServletTester so that I can stop the service in the @AfterClass block, and I save the baseUrl so that I can target it in my tests.
The servlet I added to the container I called EchoServlet. This servlet simply echos the parameters passed to it.
The servlet takes the parameter map and creates a TreeMap out of it. The TreeMap is needed so that the parameter names are returned in sorted order. Being able to predict the order of the returned keys is required in order to test against the output of the servlet.
To finish things up, I just needed to write a unit test.
After that, run the test and watch it work.
@SuppressWarnings("unchecked")
public String sendHttpPost (String url, String queryString)
throws Exception
{
String result = null;
try {
HttpClient client = new HttpClient();
PostMethod post = new PostMethod(url);
post.setQueryString(queryString);
client.executeMethod(post);
result = post.getResponseBodyAsString();
post.releaseConnection();
}
catch (Exception e) {
throw new Exception("post failed", e);
}
return result;
}If you have used commons-httpclient, this is about as simple as it gets, but I still wanted to have a unit test for it. So the problem became, "how can I test this method when it requires that I hit a web site". After some searching I found that the Jetty servlet-container has a ServletTester class just for this purpose.
In my Maven 2 I included the ServletTester using the following repository and dependency information. I always use Maven 2 as allows other developers to quickly set up their IDE and download all required JARs. If you don't use Maven, you will need to manually download all of the dependencies.
<repositories>
<repository>
<id>codehaus-release-repo</id>
<name>Codehaus Release Repo</name>
<url>http://repository.codehaus.org</url>
</repository>
</repositories>
...
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-servlet-tester</artifactId>
<version>6.1.6</version>
<scope>test</scope>
</dependency>
With this in place the next step was writing the JUnit test cases. For me I wanted to initialize the servlet-container once, then run a set of tests against it. In JUnit 4 you can use the @BeforeClass and @AfterClass annotations to mark methods that should be executed before and after all of the tests.
public class HttpPostServiceTest
{
private static ServletTester tester;
private static String baseUrl;
/**
* This kicks off an instance of the Jetty
* servlet container so that we can hit it.
* We register an echo service that simply
* returns the parameters passed to it.
*/
@BeforeClass
public static void initServletContainer () throws Exception
{
tester = new ServletTester();
tester.setContextPath("/");
tester.addServlet(EchoServlet.class, "/echo");
baseUrl = tester.createSocketConnector(true);
tester.start();
}
/**
* Stops the Jetty container.
*/
@AfterClass
public static void cleanupServletContainer () throws Exception
{
tester.stop();
}
}
The code highlighted in blue is where we start an instance of the server. I created a new instance of the ServletTester, setting the context path and adding a servlet mapping. This alone does not bind the server to a port, for that you need to call createSocketConnector(true), which binds the server to a local port and returns the URL. The port used will be a high unused port. I save an instance of the ServletTester so that I can stop the service in the @AfterClass block, and I save the baseUrl so that I can target it in my tests.
The servlet I added to the container I called EchoServlet. This servlet simply echos the parameters passed to it.
public class EchoServlet extends GenericServlet
{
@SuppressWarnings("unchecked")
@Override
public void service (ServletRequest request, ServletResponse response)
throws ServletException, IOException
{
PrintWriter out = response.getWriter();
Map<String, String[]> params =
new TreeMap<String, String[]>(request.getParameterMap());
out.println("SIZE=" + params.size());
for (Entry<String, String[]> entry : params.entrySet()) {
out.println(entry.getKey() + ":::"
+ StringUtils.join(entry.getValue(), ","));
}
}
}
The servlet takes the parameter map and creates a TreeMap out of it. The TreeMap is needed so that the parameter names are returned in sorted order. Being able to predict the order of the returned keys is required in order to test against the output of the servlet.
To finish things up, I just needed to write a unit test.
@Test
public void testPost () throws Exception
{
PostService svc = new PostService();
String res = svc.sendHttpPost(baseUrl + "/echo",
"foo=bar&baz=%25%26%3D%2F");
String[] resList = StringUtils.split(res, "\n");
assertEquals(3, resList.length);
assertEquals("SIZE=2", resList[0].trim());
assertEquals("foo:::bar", resList[1].trim());
assertEquals("baz:::%&=/", resList[2].trim());
}
After that, run the test and watch it work.
Saturday, September 22, 2007
Grokking Domain Specific Languages
A Domain Specific Language (DSL) is a programming language that is designed for a specific task. Although this definition is overly simplistic, it is good place to start.
As a Java developer I use domain specific languages in every non-trivial project that I am involved in. I use Hibernate, which uses an XML configuration file to map Java properties to data in a database. This XML file is a DSL. This also goes for DBUnit dataset files, Spring bean definitions, and even HTML. If might not be natural to think of a "configuration file" as a DSL, but it is.
For a really good discussion on DSL's and related concepts you should watch Language-oriented Programming and Language Workbenches, a presentation given by Neal Ford and Martin Fowler.
In watching the presentation I learned that DSL's don't always mean that you need to create a new language with a new syntax. Let me show you am example so that you can see what I mean.
The code below is a testing "language" of my own concoction that you would use in a JUnit test. It allows you to easily spin up a Spring config, prepopulate the database with DBUnit, then test the output of a Spring controller.
I'll bet that without providing any documentation for the API that you can easily tell what the test is doing. In the presentation I noted, martin Fowler states that readability is one of the cornerstones of a DSL, and I think this one passes the test.
My DSL is written in Java, I didn't create my own programming language for my DSL. This is beneficial because it makes it easy for any Java developer to understand my DSL, and IDE's like Eclipse will be able to provide code completion, making it even easier to use.
DSL's make a lot of sense for simplifying tasks, in turn reducing the cost of a project.
Some references:
As a Java developer I use domain specific languages in every non-trivial project that I am involved in. I use Hibernate, which uses an XML configuration file to map Java properties to data in a database. This XML file is a DSL. This also goes for DBUnit dataset files, Spring bean definitions, and even HTML. If might not be natural to think of a "configuration file" as a DSL, but it is.
For a really good discussion on DSL's and related concepts you should watch Language-oriented Programming and Language Workbenches, a presentation given by Neal Ford and Martin Fowler.
In watching the presentation I learned that DSL's don't always mean that you need to create a new language with a new syntax. Let me show you am example so that you can see what I mean.
The code below is a testing "language" of my own concoction that you would use in a JUnit test. It allows you to easily spin up a Spring config, prepopulate the database with DBUnit, then test the output of a Spring controller.
WebRequest req = createRequest()
.forURL("/app/login")
.usingPostMethod()
.withParams("user", "rhanson". "pass", "r@ckin")
.withSessionAttributes("user", null);
executeSpringController(
"loginController",
usingSpringConfig("applicationContext.xml"),
usingDBUnitData("dataset.xml"),
usingRequest(req)
)
.assertThatViewEquals("loginSuccess");
.assertThatSessionAttribute("user", isNotNull());
.assertThatSessionAttribute("user.name", equalTo("rhanson"));
I'll bet that without providing any documentation for the API that you can easily tell what the test is doing. In the presentation I noted, martin Fowler states that readability is one of the cornerstones of a DSL, and I think this one passes the test.
My DSL is written in Java, I didn't create my own programming language for my DSL. This is beneficial because it makes it easy for any Java developer to understand my DSL, and IDE's like Eclipse will be able to provide code completion, making it even easier to use.
DSL's make a lot of sense for simplifying tasks, in turn reducing the cost of a project.
Some references:
- Presentation by Martin Fowler and Neal Ford - Language-oriented Programming and Language Workbenches
- Martin Fowler - Fluent Interfaces
- InfoQ - Language-Oriented Programming
Subscribe to:
Posts (Atom)