Chris' Tech

Using the HDFS APIs without the hadoop main

2011-02-28T09:43:00.002-05:00

I recently spent way too much time trying to run a simple Java application that uses the HDFS APIs to copy files into HDFS.

While using these APIs works great from within an application launched via the 'hadoop' command line, building one that is called from a Java main() was more challenging.

Why would I want to do this? Mainly because the process that looks for files, figures out where they should live in HDFS, selects from and updates an external database. Using Hibernate and Spring. But the Cascading application that uses the data in HDFS doesn't need Hibernate or Spring. So building a single Jar that supports both seemed overkill (and had its own issues with dependencies and final size of the jar)

First challenge was figuring out what parts of the hadoop shell file to duplicate, since not everything was needed or welcome. Turns out the information is in two places /bin/hadoop-config.sh and /conf/hadoop-env.sh

First try was to set $HADOOP_HOME and try '. $HADOOP_HOME/bin/hadoop-config.sh' in the script that calls my java main().

Except this stepped on HADOOP_HOME for some reason. Turns out the hadoop-config.sh file is playing some games with the path used to call the script to figure out what to set HADOOP_HOME to.

So calling '/opt/hadoop/bin/hadoop-config.sh' (note no '.' or shell variable!) set the path correctly.

then you can use '. $HADOOP_HOME/conf/hadoop-env.sh'

Finally, in your java classpath you need to set '$HADOOP_HOME/conf' before your classes.

For example: java -classpath .:$HADOOP_HOME/conf:/mycode ...

For the record, here is the error I was receiving that helped me find this:

java.io.IOException: Mkdirs failed to create /my_dir/my_file

Tired of hibernate lazy load errors on collections?

2011-02-10T14:04:00.002-05:00

I have a love/hate relationship with Hibernate (Spring w/Hibernate to be fair). It makes some things very, very easy and others so difficult or convoluted.

For example, what should be trivial turns out to be a major pain:

Have an Entity object that contains two child collections. For example

@Entity
public class Client implements Serializable {
...

@OneToMany(mappedBy = "client")
private List users;

@OneToMany(mappedBy = "client")
private List sites;
...

}

Where User and Sites are also simple @Entity classes.

By default the access to these collections are Lazy Loaded. But when you want to access them both you run into problems. Setting both to 'eager' gets an error about too many buckets. Loading one Eager, then passing the object to a JSP for example, with get you the infamous:

org.hibernate.LazyInitializationException: failed to lazily initialize a collection of role: ...Client.organizations, no session or session was closed

(look on Google or StackOverflow for this error, there are no good solutions.)

So the hack/workaround? Create a custom Find method on the DAO and EXPLICITLY tell Hibernate to load the collection.

Note that using the collection directly in 'normal' code doesn't work since Spring/Hibernate/the compiler sees you aren't using the loaded collection and doesn't do the load.

However this works:

Hibernate.initialize(rtn.getOrganizations());

after your default 'find' method returns.

Details of how/why this works here.

Even more bizarre, when debugging this, setting a breakpoint in the 'find' method everything works correctly. No breakpoint and you get the exception.

(Thanks to Scott Mitchell for his help on pointing me to this solution.)

Logging in Groovy shouldn't be this hard

2010-08-24T10:50:00.003-04:00

Spent about 30 minutes this morning doing what should have been easy: logging from within my application.

Groovy includes/wraps Log4j so I thought it would be easy. All the documentation I found suggested it would be easy.

However all the examples left off one key thing: Defining the 'root' logger.

So, in your Config.groovy, find the log4j section and add/uncomment:

appenders {
console name:'stdout', layout:pattern(conversionPattern: '%c{2} %m%n')
}

Then add below the standard error and warn items:

root {
info 'console'
}

Now in your code you can add 'log.info 'blah blah' and it will appear on the console. The 'appenders' section is where you can add your rolling file loggers for production.

Here is what mine looks like:

// log4j configuration
log4j = {
// Example of changing the log pattern for the default console
// appender:
//
appenders {
console name:'stdout', layout:pattern(conversionPattern: '%c{2} %m%n')
}

error 'org.codehaus.groovy.grails.web.servlet', // controllers
'org.codehaus.groovy.grails.web.pages', // GSP
'org.codehaus.groovy.grails.web.sitemesh', // layouts
'org.codehaus.groovy.grails.web.mapping.filter', // URL mapping
'org.codehaus.groovy.grails.web.mapping', // URL mapping
'org.codehaus.groovy.grails.commons', // core / classloading
'org.codehaus.groovy.grails.plugins', // plugins
'org.codehaus.groovy.grails.orm.hibernate', // hibernate integration
'org.springframework',
'org.hibernate',
'net.sf.ehcache.hibernate'

warn 'org.mortbay.log'

root {
info 'console'
}

Grails keeps rebuilding classes

2010-06-14T15:32:00.002-04:00

I found an interesting problem today after replacing my laptop. My Grails 1.2.2 application that ran fine on Friday wasn't running today on 1.3.1 and the new machine.

I thought it was a 1.3.1 issue, but it wasn't. Turns out that if you have a file under src/groovy NOT grails-app AND the filename isn't the name of a class within the file, Grails will keep rebuilding the source file and clearing the Tomcat cache.

basically, I had a file 'GroupByCreator.groovy' which didn't have a class named "GroupByCreator" in it. For some reason IntelliJ doesn't complain and everything compiled correctly.

Instead at runtime I kept getting:

[groovyc] Compiling 1 source file to C:\Development\clouds\reportingui\target\classes

[groovyc] Compiling 2 source files to C:\Development\clouds\reportingui\target\classes

[delete] Deleting directory C:\Documents and Settings\ccurtin\.grails\1.3.1\projects\reportingui\tomcat

Running Grails application..
Server running. Browse to http://localhost:8080/reportingui

[groovyc] Compiling 1 source file to C:\Development\clouds\reportingui\target\classes

[groovyc] Compiling 2 source files to C:\Development\clouds\reportingui\target\classes

[delete] Deleting directory C:\Documents and Settings\ccurtin\.grails\1.3.1\projects\reportingui\tomcat

Running Grails application..
Server running. Browse to http://localhost:8080/reportingui

To figure this out, I went to the \reportingui\target\classes directory and sorted by last modified time. From that it was obvious which classes were being rebuilt, just not why. Finally after looking at the name of the file and the classes (for a few hours Doh!) I figured it out.

Arrgh.

GORM Criteria using child to parent relationship

2010-05-18T10:44:00.001-04:00

I was surprised that I couldn’t find an example of this in any of the Grails and GORM docs or online examples.

I want to find the names and number of occurrences of the CHILD in a 1 to many association based on an attribute in the parent.

The domain classes are:

class Parent {
static hasMany = [children:Child]
String firstname
}

class Child {
static belongsTo = [parent:Parent]
String name

}

To find all the children and # of occurrences across ANY Parent where firstname is 'John':

def fName = 'John'
def children = Parent.withCriteria {
projections {
groupProperty 'name'
count 'id'
}
parent {
eq 'firstname', fName
}

}

March 2010 AWSome Meeting - Cloud Security

2010-03-10T12:54:00.002-05:00

Last night’s AWSome Atlanta meeting was about Virtualization and Cloud Security topics. Taylor Banks presented about 50 slides on the different things to pay attention to, first in a virtualized environment, then in a Cloud. He correctly pointed out that all the issues with virtualization are also present in a Cloud environment, so make sure you get them right the first time.

Taylor was a good presenter, though I do wonder when he managed to get a breath in. He talked a lot, but he wasn’t rambling. He was noticeably excited and involved in the materials he was presenting.

Very early on he presented an new acronym that I think I’ll start using : K.I.S.S.M.Y.A.S.S

Meaning:
Keep It Simple, Stupid Make Your Architecture Simpler to Secure

Yes he mentioned it a few more times during the presentation.

He also said something very profound:

Cloud Security is often more of a process than technology

Yes securing the servers and services is important, but aren’t you already doing that in your self-hosted environments? So Cloud Security is about making decisions about what data you are sharing, how important the data is, what formats you are protecting it in and who can access it.

Taylor also took a few shots at the pundits around Cloud Security and did a pretty funny impression of a DBA. He had a comical routine about how giving two different DBAs identical SQL Server instances in a VM, but only telling one that it was a VM. You can guess what he was making fun of ;-)

The presentation is here.

Taylor’s twitter handle is @TaylorBanks.

AWSome Atlanta February 2010

2010-02-10T14:19:00.002-05:00

Last night’s AWSome Atlanta meeting was one of the best attended that I can remember. The main topic was the Chef configuration management tool for computer infrastructures. The crowd was a mix of the ‘usual suspects’ and quite a few new faces.

During the usual introduction from John Willis we learned there were a number of consultants, end users and a few academics in the crowd. Probably half indicated they were starting to learn about Cloud Computing.

Josh Timberman @jtimberman from Opscode presented about Chef, the concepts behind it and ways to use it in your environment. His presentation was good, but it took a few minutes to get a ‘big picture’ view of what he talked about. However once you understood the goal was to standup a new server (or upgrade an existing server) consistently the materials made a lot of sense.

One of the more interesting things I learned about Chef is they are in beta of a SaaS version where they (Opscode) will host the Chef server which can then be reached via a client within EC2, Rackspace etc. or within your own environment. This is interesting because it removes the need to have a server hosted someplace other than the IaaS provider for the Chef cookbooks and recipes. (Thus no need for IT resources.)

The 30 second overview of Chef: you create recipes and cookbooks for the applications and configurations you need to standup a server. So you can create an “apache 2, Tomcat 6, JDK 6.10 MySoftware 2.00” recipe that knows how to install and configure an exact copy of the environment you want. And do it repeatedly without any intervention or manual steps. (The full explanation took an hour, so there is a lot more to it though.)

Very useful when you are in the cloud and spinning up new instances, but also useful in an internal environment when you need to bring up a new server due to hardware failure (or faster boxes!) or when you want to quickly deploy a new version of software.

Consider building a new Chef recipe for the next release of your software. You create a new server, validate against QA then just change the configuration that defines the production locations/databases etc. No manual check lists, no forgetting about a new service or cron job.

After Josh’s presentation we had about an hour of open discussion. Lots of topics, including EC2, where the cloud is going to impact business, my views on the return of the ISV and a short religious discussion on Ruby ;-) (Sorry Keith)

I hope the new folks got enough out of the session to keep coming back.

Opscode can be found here.

AWSome Atlanta can be found here.

2010-01-28T10:03:00.001-05:00

In my last post I commented that sometimes “drinking the Kool Aid” is a bad thing. Here’s an example that bit me for a couple of hours this week.

Groovy and Grails have done a lot to ‘make things work’ by expanding many of the core Java classes to remove the need for all the ‘work’ needed to get something to work. For example, database connections ‘just work’ and you don’t need tons of exception handling code or thinking about how to release them.

Groovy also makes interacting with existing Java code trivial.

However, the combination of the two leads to some problems. Consider my case. I wanted to upload a comma separated value (CSV) file in Grails, split the file apart and build objects from it. Pretty straight forward right? Done it many times in Java with Struts or GWT.

Searching for examples also showed a number of ways to do it. I took the ‘approved’ version from the Grails site.

Searching for ‘groovy csv’ also showed a lot of examples, including the use of CSVReader, which I’ve used before.

Putting the two examples together was trivial and took a few minutes to test and everything went perfect. Until I tried to upload the same file a second time to test the ‘update’ logic. I received an exception, because the file was locked on the host I couldn’t replace the ‘working’ copy.

Turns out that CSVReader is a Java class with very specific lifecycle steps that hadn’t been ‘groovy-fied’ yet. So while Groovy handled all the exception framework, collection logic etc, it didn’t know to close the CSVReader at the end. I added the explicit close() on CSVReader object and things work now.

I was (am?) trying to learn the Groovy paradigm, not just the language so I’m deliberately not writing ‘java in Groovy’. So I didn’t look at exceptions or lifecycle events since for a lot of other ‘common’ thinks Groovy just works. I’ll pay a little more attention to lifecycle from now on.

2010-01-26T10:18:00.002-05:00

This is the first of hopefully several posts about my experiences and frustrations learning Groovy and Grails. I’ve been programming for over 20 years and have seen a lot of techniques, technologies and tools that claim to make a developer more productive. Many did make you more productive. Some did until you needed to modify the code a year later, or modify someone else’s code. So I’m pretty skeptical about all the claims from fan boys about any technology.

So with the hype around dynamic languages including Ruby and Groovy, I started looking into how useful they could be. For the record, when Bruce Tate talked about ruby at AJUG in August 2005 I was both skeptical and annoyed with what he was presenting. As a ‘fan boy’ he could see nothing wrong with Ruby or the impact it would have on a production system. In particular we got into a mini-debate about real costs of running a production server vs. benefits of Rails for faster development based on the runtime performance back then.

In November 2009, Pratik Patel did a Grails and Groovy presentation to AJUG, which was right after my NOSQL East presentation where I was asked why I don’t use Groovy instead of Java for my applications. Pratik’s presentation got me thinking more about Groovy and researching the differences and benefits from the Java world I’ve been working in for over 9 years now.

So, I’ve started my first Goovy/Grails project. I’ve been doing it part time for a couple of weeks and have some initial impressions.

Pros

GORM for ORM hides a lot of the data tier and is pretty straightforward, especially if you’ve ever fought with understanding Hibernate. Some of the bizarre things are still there, but for the most part it understandable
Auto creation of the UI and the scaffolding in Grails is impressive. After years of screwing up Struts and Tiles configurations and having no idea why, this is cool
Being able to call down to Java as needed. I have A LOT of helper code and business logic that I don’t want to rewrite
‘duck typing’

Cons

GORM. It took me most of a week and a lot of trial and error to figure out how to return certain fields from a multiple table query in a reporting interface without dropping into HSQL. (Yes, I plan on writing a post about this soon)
Dynamic language support in tools. I use IntelliJ which is supposed to have the best Groovy/Grails support and it still sucks. I don’t want to wait until runtime to find out that I spelt a parameter name wrong. Or that a variable name is reserved word in Groovy (Category anyone?)
Real examples. The IBM series is pretty good, but I spent a lot of time banging my head on file uploads and GORM beyond the basics.
Are some of the ‘features’ really a step back? Naming parameters on method calls has been available since PL/1 (at least) and has always been a criticized as too verbose. It just 'feels' wrong to be using them

The biggest negative so far is drinking too much of the Kool Aid ;-) Grails/Groovy makes some things so simple that when you get outside what they’ve “groovy-fied” you need to think again like a Java programmer. I’ll give an example of this in my next post.

So far though I’m impressed with the language and the ease of doing things.

Video of my NoSQL East presentation

2010-01-11T15:45:00.002-05:00

Here is the video from my NoSQL East presentation from last fall.

You can't hear the questions at the end so my answers sound strange ;-)

Advice for a college Freshman

2009-11-19T11:42:00.002-05:00

My post about what you'd teach a 15 year old reminded me of a blog post I have been meaning to write since September. One of the message boards I frequent, SVT Performance had a thread about advice for engineers. Initially it was targeted at women written by an ME, but turned into some good advice for anyone in college. Sorry I can’t deep link to the posting but it is in 'road side'.

I added a few comments, but I have thought more about it so here goes:

Here is my advice to a freshman today in a technical field.

Ask for help. Join study groups. Don’t be afraid of getting to know your peers. You may have been the smartest person in your high school, but there are going to be people better than you in some subject and you better than others in a different one. Get to know your peers so you can help each other. Figure out who are the good people that help others and who are just leaching from the group because they are in over their heads. The guy who kicks ass in Calculus and helps you may struggle in Fluid Dynamics or Data Algorithms were you shine.

Get an internship/co-op as soon as possible. Even as a freshman. Here is one of the last times that you can use your parents and family and not be accused of nepotism. Ask everyone you know if they know anyone in the field you are interested in. Talk to them about summer jobs or co-ops. Of course work through your school’s placement and co-op departments, but sometimes a friend of a friend knows about better positions.

In your internship, make sure the people you work with know who you are. They are going to expect you to ask lots of questions, not know much and even make mistakes. Don’t be the ‘kid’ that no one remembers because you sat in the corner. Here is a chance to introduce yourself to a lot of people in the industry you are interested in. Unlike the rest of the world, these are people you DO have something in common with, so don’t feel awkward about your intelligence or interests. I’ll bet they have felt the same way!

Get a Linked In profile. Yes, I know Facebook is cool (I have one too), but Linked In is where professional people keep track of their networks. It is very rare to hear of someone removing a professional contact in Linked In. Why? Because the network is valuable for years after you’ve stopped working with someone or at a specific job. Being able to search on ‘who works where I am interviewing’ is powerful, but so is asking (or someone asking about you) ‘this would be a great job for Jill, I wonder what she is up to’ AND being able to find them.

Now this doesn’t mean you ask everyone you meet in your internship to be a contact. Instead wait until a couple of weeks BEFORE your internship ends for the semester and ask the people you’ve worked best with for contacts. This isn’t Facebook so you don’t ask everyone.

As you get to know your peers, invite them to Linked In as well. Not the leaches, but the people who genuinely are good at what they do and are people you’d like work with some day. The reason here is the same as above: these are people who are going to be in your industry one day. They are going to be looking for jobs one day or might be able to help you. They may also have the solution to a problem you have (or vice versa). By maintaining the relationship you have at least one way to find them (or them find you) when the time comes.

Again, don’t add everyone in your class. Only add the people you think you’d want to work with. A plus to having a medium sized college network with a small to medium sized professional network is when the hiring manager looks at your profile, they see real professionals AND peers, which tells them a lot about you.

Finally, update your Linked In profile at the beginning and end of each semester at a minimum. As you get closer to graduation the profile and resume should reflect your internships instead of your classes. Update your status regularly about what you are working on. For example, having a status of ‘taking thermo 313 this quarter’ or ‘loved the lecture on np-completeness’ may get people in your network to think of you and reply. This again keeps you in their thoughts.

This isn’t one way. By regularly checking your network you can also see where you could offer help to a peer or a contact. You will be surprised by how many people post about job openings. You may not qualify (or be looking), but the guy that helped you with Calculus might be. By putting them together you help all three of you.

One more thing: don’t cross professional networks and Facebook. While it is tempting to friend someone from your internship, you are in college and you are allowed some stupid things, but don’t let non-friends find out!

What would you teach a 15 year old interested in technology?

2009-11-18T09:28:00.002-05:00

At last night’s AJUG meeting Burr asked the group a couple of questions about technologies needed today to be hired directly onto a team. The question morphed a little into ‘what would you tell a 15 year old to learn’ if they wanted to be in technology. That question got things going in a different direction with a lot of good feedback.

This post is about my thoughts on this, since I have a 15 year old, although one that isn’t interested in being a programmer.

First would be to learn how to think. I know that sounds strange, but learn how to take data and ask questions or given a problem think about how to solve it. Even when I was in elementary school 30+ years ago, a lot of what we learned was how to apply the processes and algorithms we were taught to the examples/questions we were given. Not, given this poorly defined problem, figure out how to solve it with your full knowledge, not just what the teacher gave you yesterday. Think of it as the difference between geometry on calculating the angle on a graph and assembling a Lego house without instructions.

How do you teach someone to think? Unfortunately I don’t really know. With 3 kids I see that each of them learns differently and the processes I use to help each learn something is different. My only advice on learning how to think: has them questions that aren’t what the teacher taught them in the same area, or ask a different set of questions from what the homework asked. For example, Courtney is working on European history during the colonization times. The expected answer about why the Europeans wanted colonies (from the book) didn’t include the greed factor. And the arrogance factor. So I pushed her to think like the King or Queen during that time and see what you’d want. While she was able to give the 'book' answer, she is also learning to think in someone else shoes.

Back to technology. Thinking about technology, I think learning any language that is ‘backend’ focused isn’t a good place to start. Java, C, Groovy, Ruby, C++, Perl and non-UI languages won’t interest the kids. Instead pair them with something that shows immediate usefulness. So use Grails or Ruby on Rails or even GWT. Get them working on making something they can see and show off.

When I learned to program, most people didn’t interact with computers each day, so showing a green-screen output from the Fortran or C I was writing to my peers or parents was novel and unique. Today, everyone knows the web and a browser so showing them a text output isn’t going to get the response the programmer needs as encouragement to continue. Show someone a web page you built and I’ll bet most kids will get ‘wow, I wish I could do that’ as a response.

Once they have a basic language under their belts, look at the Platform as a Service cloud systems like Microsoft’s Azure, Google’s AppEngine or even Force.com. Why? Because I believe these platforms are going to be where a lot of the jobs are will be in the future. Whether it is a startup taking an idea that MAY require greater scale later, but not today, or the IT developer putting together applications for departments to address specific business problems, these platforms are going to make it easy to deploy solutions fast, instead of waiting on hardware purchases. Of course the whole ISV space is going to make a comeback because of these technologies.

Finally, encourage the kids to get out of the house and play. Build things like go-carts and bird houses. Learn to cook, even if using a recipe from a big cook book (Better is a hand written card from Grandma) Encourage them to take things apart to see how they work. The digital world makes this hard, but what about their bike, scooter or old rusted car sitting in the garage?

ControlTier Presentation At AWSome Atlanta

2009-11-02T14:27:00.003-05:00

A co-worker of mine presented about how we are using ControlTier at AWSome Atlanta.

I missed the presentation due to a softball tournament for my daughter, but heard it was great. Watching/Listening to the video I have to agree!

AWSome Atlanta ControlTier Demo

The return of the small business ISV?

2009-08-23T20:46:00.002-04:00

I've been doing a lot of research and thinking about Clouds lately. I know, Clouds is an overused and abused term, but the more I thought about the business uses of a cloud and less about the technology, one thing kept sticking in my mind: the ISV is coming back.

For those around during the rise of the MIS/IT department, business groups used to get what they needed done using computers by hiring a one or few person shop to build software to their exact needed. Ask most 40+ year old developers (and Directors and VPs!) and you'll find that they cut their teeth writing custom code for everything from TRS-80's and HP-1000's to VisiCalc and dBase to sell to small businesses or to larger businesses that didn't have IT yet.

See, before MIS/IT put a clamp down on all non-'approved' applications, it wasn't unusual for a business person who 'knew a computer guy' to hire him (or her, even in those days!) to do exactly what they wanted. So a dBase application to run HR or the dentist office or to handle commissions. In the 80's and 90's I heard a lot of 'so Bob from my Church group did this for us'.

As MIS/IT took control, these small ISVs were starved out by bigger consulting agencies that the IT guys had relationships with (there is a lot more to the decline of the ISV, but I won't get into it today.). Today in most organizations it is impossible to get an application build and installed for a department without IT involvement.

The main reason departments don't do (much) outside work is they don't have any place to run it. With desktops locked down and networks monitored a department couldn't just install an application on a machine bought at Best Buy for their use. I'll talk about all the Access applications that sprung up because of this in a minute.

Now Software as a Service (SaaS) has started to whittle away at this reliance on IT since the apps aren't deployed locally and pretty much all IT sees is the HTTP traffic.

Yes, an department head could get a Rackspace account and run their own hardware, but that is really pushing the box even for someone likes sales (just kidding.) It would also mean they need OS help, which wasn't that easy to come by.

Things are changing though. Take a look at Microsoft's Azure or Google's AppEngine (or the dozen or more similar systems for PHP, Ruby, Java etc.). Here the applications are running inside a container. Containers with well define interfaces with tons of information on the web about how to access them. Containers without any OS access, so you don't need to know Windows, Linux, Tomcat, IIS etc.

Unlike SaaS which typically requires a contract that involves the legal department, Azure and AppEngine can be paid for with a credit card. It wouldn't be a stretch that soon business people are going to 'know a computer guy' who can build apps in these containers. For very little money. Like the kind of money a department manager or director has signing authority over.

Remember those Access databases I talked about a minute ago? Well most of them came out because the business people used Access in school or their interns did and Access is a typical part of Office. So tons of simple (and eventually not-so simple) applications are built with Access, under IT's nose.

What do you think the interns are learning in school now? Not Access so much, but Big Table in AppEngine or SQL Server in Azure. Soon those 'hidden' applications are going to be deployed in the cloud instead of local machines.

So why do I think the small business ISV is coming back? How many business users have problems that could be solved by a small application. One that they'd pay $2000 or $3000 for? How many would be able to hide that amount from IT and the CFO?

Who do you think she'd get for $3000 to build a basic app? Not the big consulting companies, probably not even the local ones IT uses. It will be back to individual developers who will do it at night and over the weekend.

That guy you know from Church.

Hadoop and Cascading presentation for AJUG July 2009

2009-07-23T21:01:00.003-04:00

This week I presented about Hadoop and Cascading for the Atlanta Java Users Group. It was the first time I've presented to a group this big, but I think it went well.

Here is a link to my slides.

Here is a link to the video

Jeff Sutherland at Agile Atlanta

2008-10-28T11:14:00.001-04:00

Last night I attended the Agile Atlanta meeting last night where Jeff Sutherland, co-creator of Scrum, presented his experience with distributed teams and Scrum. He was a really impressive presenter.

The general idea of his presentation was that distributed teams should be working on integrated Scrum teams instead of isolated. He presented a number of cases where the teams had 'local' resources with the Product Owners and 'remote' resources elsewhere. For example one company built a scrum team of 4 engineers, with 2 in Denmark and 2 in India. Initially they all were together in Denmark for 2 sprints, then they broke them up. Jeff claimed that the velocity and quality of the team from the 2 sprints 'together' was continued when they were separated.

His reasoning was that because the team had started together, knew (and to some extent trusted) each other and were using the same tools they continued the same velocity when they changed location. He also said that the team had a shared sense of urgency about what they were trying to accomplish and the remote team knew the product owner and what he was trying to accomplish instead of him being a voice on the phone. He showed their velocity over a dozen or so sprints and pointed out a couple of anomalies, including one major one where the velocity took a nose dive. They found that it was the culture of the Indians, where the junior members (they now had 8 total members) were waiting on the senior engineer to tell them what to do. Once the team understood this, the team made the senior engineer the scrum master and told him to fix his velocity problem.

He also presented a number of cases where non-integrated distributed teams (both outsourcing and different offices) didn't provide anywhere near the same velocity as the co-mingled teams, as he admitted, contradicting all the Agile best practices of putting the teams in the same room. He presented that this was due to the silo building that went on with these teams since they had different priorities and sense of urgency.

He also had a great comment to someone asking about the lone expert who is the only one who knows how to do something: he advises his teams to not commit to anything where there is only one person who can perform the task. if they find a task that only one person knows how to do (he used the example of a deployment configuration manager at one of his clients), then the sprint should stop and the team immediately begin cross training.

The main presentation is here.

A posting about one of the teams he worked with is here.

A paper about the findings is here.

50+ hours and still a full charge

2008-08-24T20:32:00.000-04:00

I really think the battery problems are related to the devices hunting for a signal. I didn't really go out this weekend, so I used the phone only a few times for a few calls. However, the # and time of calls were definitely less than last week when I didn't have a signal most of the time.

Anyone else notice poor battery life when the phone is always hunting for a signal?

Thoughts on cell phone (and iPhone) batteries

2008-08-22T21:49:00.001-04:00

I was at an off site meeting most of this week and one very odd thing happened (well, several odd things happened, but I'll only write about this one ;-)

My AT&T Razor's battery would last less than 24 from the minute we arrived until we left (a little over 3 days). I have never had problems with this phone and batteries. What is odd is when I got home and fully charged it, it is now 11 hours later and I have a full charge.

One piece of information: I didn't have a signal at all in the buildings were we were staying. No edge or 3g. Is it possible that the reason so many people have bad battery life is that the phones are wasting a lot of power trying to find signals, or switching between edge and 3g?

If the phone is still fully charged in the morning that I know it has something to do with the lack of a network, since I get a strong 3g signal in the house.

IASA Atlanta - August 08 Meeting

2008-08-14T09:44:00.002-04:00

Last night's IASA Atlanta meeting was a good one. The main topic was a panel discussion around 'Design Patterns'. Unfortunately, I thought the names of the speakers were on the IASA website, so I didn't write them down (Please comment if you know their names and I'll update this.)

The discussion started with a definition by each of the presenters of what 'Design Patterns' are. The crowd then got involved with their definitions, where they have seen abuses of patterns and suggestions for how to present patterns to their work environments. Having worked almost exclusively with experienced engineers (or junior engineers who ate up anything new or interesting that they didn't know) I was surprised by some of the suggestions and challenges those working in bigger companies had to introduce these concepts. As part of the discussion on how to introduce patterns into an organization we got into a discussion about how people learn and how some (like me) have a 'gut' feeling about what parts of a complex topic can be skipped until later to get the bigger picture. Great discussion on this with the group as well.

About an hour into the session the subject changed to hiring questions, including a favorite of mine: 'what have you read recently?', or how I ask it 'what do you do to keep current with the changing technologies?'. Several good suggestions for Pod casts and RSS/email lists. My suggestion was IBM's developerWorks. I get both the email and the RSS feeds. A number of people joked about the answers they received to these questions. My personal favorite is a candidate who 'reads the internet'. When asked for more details he said he surfs around looking for articles and posts.

One of the group asked about certifications and whether they were useful metrics when hiring or when looking for a job. I personally don't put a lot of weight into them. Some are non-trivial to obtain, but a lot of them have a lot of fraud behind them. I've (unfortunately) interviewed enough engineers (software, network and hardware) who were certified MS-this or Novell-that and Cisco-whatever who couldn't explain the basics of where they used these technologies or answer non-trivial questions. There was also a lot of fraud during the earlier part of this decade where people were paying examiners to pass them and some 'training' companies even advertised they could guarantee you'd pass. One of the IASA officers countered with the rigorous testing that you need to reach the more advanced Java certifications, so I need to look into that some more.

Bringing the discussion back on topic (part of the reason I enjoy these meetings is they tend to be very interactive with the crowd and the speakers/organizers let things get off topic for a little while if the group wants it to.) we talked about Anti-patterns. The general consensus was that 'anti patterns' are mostly about personnel and management, but have a place in development. One comment was the the people interested in patterns hate anti-patterns. I personally enjoy reading about anti-patterns so I can see if I'm doing something others flag as bad, or my organization is, or to see what things can go wrong as we implement a new process or technology so we can avoid them.

The last few minutes were a discussion of patterns again, with Burk asking if this was a topic we wanted to cover again. I chimed in that I thought it was a good idea, but to include more advanced/complex patterns since most people with an architect title or role should know the basics like singleton or command.

There was one mention of scrum, in passing. One mention of offshoring, again in passing. (Both of these usually lead to some detailed discussion, but not tonight.)

It was also one of the best attending meetings in a long time. Lots of the same faces, but plenty of new ones.

At the end I got to check out the Kindle that one of the speakers uses. Pretty impressive device, I may need to add it to my Christmas list.

Norton 360

2008-08-10T21:09:00.001-04:00

I've been a user of Norton Antivirus for over a decade now (I think I bought it when I joined Earthlink back in 1995?). A couple of the releases have sucked (2006 had to be reinstalled every couple of months after a failed dictionary upgrade for example)

My subscription was up this weekend so I went an bought an upgrade to Norton 360 this weekend at Best Buy. They had for $10 less than online. Plus I was there getting a WII Fit and Mario Kart ;-)

First I had trouble uninstalling the old versions on both PCs. Eventually found a tool on the Symantec site to remove ALL products. This tool even found left overs from 2004 versions. With them gone, the install went pretty smooth.

First thing I noticed was the difference in memory footprint. Easily 150 MB smaller with the new version installed than with 2008. I also noticed that the disk thrashing has stopped. To the point that the disk on my main desktop isn't spinning at all, and sometimes goes into powersaving mode. That NEVER happened with the previous version.

This version also has an anti-spam gateway, firewall and anti-malware systems. They've also bundled the disk defragmenter from the standalone tools product.

Odd thing is they added an on-line (Internet) backup tool. I've been using Mozy for a month now (I'll write about it next week after I test restore some files.) I have 20 GB backed up and the Norton tool comes with 2 GB free, but I didn't use it.

I also kicked off a full system scan, then did email and surfed. I didn't notice any delay. In the past the machine was worthless when it was scanning.

Only downside: Norton 360 doesn't like Windows Defender. Not sure why, I'll research that some more, but it told me to uninstall it before it could install.

XP SP3, 512 MB RAM - fun with an OLD PC

2008-08-09T12:57:00.002-04:00

My home PC is nothing to brag about. Actually it was 5 years ago when it was new, but not today. These days I use it for Quicken, surfing the web and email. Anything 'cool' for work I use my laptop, which is a pretty powerful machine.

Last week MS decided to force my XP OS to upgrade to SP3. I looked around quickly for 'gotcha's' and didn't see anything, so I let it install.

30 minutes later, the PC reboots and hangs. I reboot it again, and it starts, but starts thrashing the disk. I leave it for 30 minutes, but it still hasn't fully started. Turn it off again, this time it wouldn't show the login screen. I gave up and went to bed.

Next day I found some interesting issues with SP3: first, you need to disable your anti-virus before installing. (Reason is most have a feature that prevents critical system files from being updated. The same ones SP3 wants to change.)

I got home, used the fallback features to the checkpoint before the install and started over. This time I disabled Norton anti-virus and Windows Defender. Still had to let it sit for 30 minutes while the disk thrashed, but it eventually settled down.

Frustrated with the PC, I wanted a new one. Of course ;-) Researching found some good Dell's for around $500, but since I had mine working now I couldn't justify a new one. Of course Deb's PC is few months older than mine, so I'd need to get 2 if I wanted any peace in the house.

So I went and bought another 512 MB of RAM for my PC and wow, what a difference. Since I was only surfing and doing email I didn't think I needed a lot, but with the new ram things are much faster. Looking at Task Manager I see I'm using 614 MB now, which means I was swapping pretty heavily before.

I still want a new machine, but can't justify it. Maybe we need a good thunderstorm ...

July 2008 AJUG Meeting

2008-07-18T13:51:00.001-04:00

I joke that I get two 'get of out of jail free' cards a month for work-related activities. Twice a month I go to user group meetings, one for AJUG, the other for IASA in the evenings. These really are related to work since we do a lot of Java and I've always been involved in Architecture of software systems.

The July meeting of AJUG was about Terracotta, a tool I had heard about, but didn't have a lot of understanding of what it did and how.

The meeting started as usual with Burr Sutter asking about what's new in the community and if anyone had any job openings. Several people described what they were looking to hire and we started a discussion about why it is hard to find good people in Atlanta and about expected salary range of most of the open positions ($85 to $105k was the range thrown around). I relayed a comment a friend at another company made: 'I know the devils here, why would I risk going someplace where I didn't?'. He was referring to the economy and while tech is supposed to be weathering the storm fine, it is a concern for most people. This led to a couple of other comments about losing vacation, negotiating with HR and the reasons why HR wants things cookie cutter for all employees. Very interesting discussion.

Burr then introduced Kunil Bhasin an engineer from Terracotta. Kunil had a good presentation and was willing to take questions from the audience. At first I wasn't sure what Terracotta was, but as he gave more examples and explained how the system works, the more sense it made.

High level: Terracotta provides a file-system backed, in memory 'database' of the name/value pairs available in a network fashion. Very similar to a cache, but with enough differences to not be just considered a cache. At first I was thinking about a JBoss Cache type of solution, but Kunil showed that what was being cached wasn't an object (and thus all the issues with marshalling, changing signatures etc that caches have) but rather a primitive, like a string or integer. So in Terracotta there is an object identifier, an attribute (or class member) name and the current value. Changes in an application are transmitted to the Terracotta server and available for other servers if necessary.

The service is transparent to the application (once you tell Terracotta which objects to manage). So I don't need to do things like checking the cache using a cache specific API, then loading the object via JDBC and putting the object into the cache. Or worse, worrying that the object in the cache is an older version because I'm doing a rolling deployment and the cache server is shared between two apps, one updated, one not.

For the gory details on how this works, go to their website.

Kunil was a very good speaker and handled the crowd very well. I was frustrated though by the crowd. We had a really good turn out, especially for a summer meeting, but it was one of the most argumentative crowds I could remember since Bruce Tate told everyone Ruby solved every problem we've ever had. (Yes, I was one of the ones arguing with Bruce ;-) It seemed that each time Kunil introduced a new concept someone would ask him about performance or why what Terracotta was doing wouldn't work, usually in an offensive tone. He nicely dodged those questions by telling us he'd talk about performance later in the presentation (unfortunately we ran out of time before he could.) He did address a number of questions about locking and concurrency as the questions were asked. It felt like they thought Kunil was attacking their solutions instead of presenting a tool to consider next time. Very strange.

While some of the questions were argumentative, a lot were good, asking details about how they addressed problems people who use caches today encounter. He answered my question about what happens if a new member is added to a class being monitored (Terracotta knows that the old object doesn't need that member so it doesn't try to set it if the member is requested. Having had problems with object caches in this area, this is a nice solution. )

This was also one of the longer sessions we've had a while. Kunil ran out of time and had to blow through several slides about scale and examples of ways to use Terracotta. I think we could have listened to Kunil for another hour, possibly longer if he got into the details of how to use Terracotta with Tomcat for session management (the first thing I thought of when I understood this) or building caches to offload data from the database. Hopefully he or someone from Terracotta will come back, possible for Dev Days (Burr, you listening?)

Welcome to my Tech Thoughts

2008-07-16T11:25:00.000-04:00

I've been hesitant to put things related to work or technology on my family blog, so I've decided now to create a tech specific blog.

While I won't link specifically to my employer or write about work-specific things here, I will write about technology and business things I find interesting.