Jeff Borst - Blogs at Near Infinity

Hadoop for Managers

Mon, 07 Mar 2011 13:09:22 -0500

So, you have a ton of data and are trying to figure out what to do with it. Big data is the newest industry buzzword and there are a myriad of solutions out there that claim to solve the big data problem. One of the industry leading technologies at the moment is Hadoop. But what is Hadoop and how does it make working with big data manageable? Lets see if we can take some of the essentials of Hadoop and explain them in more business terms. While no analogy is perfect, I think that this one is pretty good.

Coca Cola started in Atlanta Georgia. Coke products were all made in one plant and distributed locally in Georgia. As the popularity of Coke increased over the years, the Coca Cola Company had to change the production and distribution of it's product in order to meet global demand. Now there are bottling plants all over the world that use raw materials and Coke recipes to produce and distribute billions of servings of Coke products a year. As your data grows like the demand for Coke, one plant will not be enough to satisfy your data production needs. Enter Hadoop.

Hadoop is a distributed data crunching architecture very much like Coke's bottling and distribution network. So, why is it impractical to just scale one bottling plant as demand grows? Let's see. Physical infrastructure becomes a problem. As production scales up, you need more raw material. This means a larger loading dock, more trucks, and larger roads to accommodate increased traffic. You also need more machinery, more power, and more people to run the plant. Of course all of this can scale to a point but could you imagine the infrastructure that Coca Cola would need in Atlanta to produce all of the billions of cans of Coke that they sell a year?

You also need a delivery network that delivers product in a timely manner. This is more problematic. Delivering Coke to new markets that are further away from the plant means more trucks and aged product. For all of these reasons Coca Cola made a transformational decision to ship raw material all over the country and eventually world and have regional plants bottle Coke locally. You can do the same thing with data.

Instead of having one large plant that has a finite production capability, Hadoop allows you to have thousands of smaller distributed plants that work together in a network of data production. This solves many big data problems in the same way that Coke solved their production problem. A Hadoop cluster of computers is made up of as many production plants as you need to process your data. Your raw material is your data. Hadoop automatically distributes this raw material to all of your data production plants or nodes. When you run a Hadoop job, instead of pulling data to the program, it pushes the program to the data just like a recipe would be distributed to all of the bottling plants that produce Coke. This approach has many benefits over large singular databases or having one huge bottling plant.

Increase capacity without affecting production

Adding a new plant has no impact on the other plants in the network. When a new plant comes online, the Hadoop system automatically distributes raw material to it and sends it data crunching recipes so that it can immediately increase capacity.

Upgrade individual plants without halting production

If there is a new conveyor belt that can increase the capacity of a plant, while one plant is being upgraded the rest can increase production slightly in order to absorb the temporarily reduced capacity.

Absorb plant failures

With one large system, if something catastrophic happens, all production stops. With a bottling plant network, if a plant in Wisconsin has to halt production because of local flooding, the plants in Illinois and Ohio can ratchet up capacity temporarily to meet demand while flood waters subside.

And then there's scalability

If I have one big plant that that is at capacity, what do I do? Do I build another big plant and double my capacity even though I may initially only need to increase production by 5%? Since the plants are smaller, Hadoop allows you to add what you need when you need it.

There is much more to Hadoop than described here, but I think that this gives you a good idea as to why you should take a look at it if you have big data that you want to exploit in some way. This distribution and production technique did wonders for Coca Cola, just think of what it could do for you.

The Agile Attitude Part 2: Letting Go

Sun, 14 Mar 2010 23:04:03 -0500

For many of us in the IT industry, the popularity of Agile development has been a freeing experience. When an organization understands its role in Agile development and allows the team to self organize and come up with creative solutions, many of the stress and tensions that can be felt by the development team can melt away. This is because Agile processes allow development teams to do what they do best: develop software. The drivel that follows is advice for development teams on how to let go of some habits or preconceptions in order to better benefit from using an Agile development process.

Follow the Agile Process - it will make your life easier

One of my favorite things about Agile development is that it removes the burden of decision making from the development team. The team gets to provide facts to the decision maker (Product Owner) and then execute once a decision is made. There is no more need to hide problems or try to make up schedule time (although I'm sure that no IT professional has ever done such a thing). Developing software is what it is - unpredictable - and Agile development accounts for that through continuous planning. The best thing that a team can do is be as transparent as possible, help decision makers gather all the information that they need in order to make a decision and then sit back and wait for the next Sprint's marching orders. This allows the team to spend more time and energy on what they do really well - building good software.

Remember who owns the system

The majority of software developers out there are building a system for somebody else. Yes, there is a tremendous amount of pride and sweat equity that is built up when building any system, but we always need to keep in mind who it is for. Many times its not our system that we our building, its our customer's. Software developers get paid to develop software and thanks to Scrum, Product Owners now get paid to make decisions that directly shape the software. They get to make big decisions every month, and they should be making little decisions every day. So remember, what makes sense to you should be a starting point for implementation. Many times as developers we understand how our users think and can implement features in ways that are both elegant and intuitive - but not always. Sometimes we completely fail to grasp our customer's environment or needs and if left to our druthers would create a system that would completely miss the mark. This is the way of the past. The future (powered by Agile process) is to let our customer's be intimately involved with the software right from the beginning, and let them shape our actions so that the systems we build will be as useful as possible to their users.

Help Make Decisions

It may sound like I want to purge thought and creativity from the development team, but I really don't. The team is the Product Owner's biggest asset. The team should let the Product Owner understand the possibilities that technology can provide. We are responsible for keeping our Product Owners informed, providing suggestions, and laying out the pros and cons of various options so that they can make good decisions. There is nothing more beautiful in software development than a well oiled development team that has an intimate relationship with its Product Owner. With ideas flying all over the place, creativity and experience can come together to create game changing software that really adds value on top of the regular automation of business processes that we have learned to do so well over the past 10 - 20 years. Just remember, ultimately it is the Product Owner who is responsible for the return on investment for the system, and when it is all said and done, they have final say for implementation.

Don't fight the process

Agile requires a different way of thinking than many IT professionals are used to. Agile development asks us to give up some control over the process for building software. For many, this change can be hard to swallow, especially when we are asked to do things that we don't agree with, be it adding a particular feature or changing the layout of the UI. In these cases, feel free to voice your opposition, but remember your role. System design and implementation is your thing. Deciding what to do and when is up to your Product Owner. You will likely have very good input on these matters, and if you voice your opinion respectfully (not forcefully) in time your Product Owner will come to trust and depend on your sage advice. Allow the iterative nature of the Agile process to work to your advantage. If your Product Owner fails to follow your advice and makes a bad decision, it will be brought to light rather quickly. Its OK to cut them some slack. One thing that as developers we love about Agile is that it tells us that we don't always need to have all of the answers. I think its reasonable that we don't expect our Product Owners to always have all of the right answers either.

Let me sum up

Agile development isn't easy. Bridging the gap between the business side of an organization and the IT side of an organization is extremely hard. As software developers, by giving up a little bit of control and not giving in to our egos always telling us we have the right answer, it will be easier to benefit from all of the wonderful joys that Agile Development has to offer.

The Agile Attitude Part 1: Transparency

Mon, 03 Nov 2008 15:55:38 -0500

Transparency: Implies openness, communication, and accountability

Agile has always espoused the importance of transparency within the development team, but it is also important to be as transparent as possible with your customer. In fact, the use of transparency can build the foundation of your customer relationship.

At Near Infinity most of our projects are consulting efforts where we build software for other people. In these situations, our customers place an enormous amount of trust in us, and it is incumbent upon us to do everything that we can to earn this trust.

Most Agile development methodologies have clear roles that serve to simplify the development process. In Scrum, the development team is responsible for delivering complete, tested, working software each iteration. The product owner is responsible for making decisions about which features should be developed at what time, and for deciding when there is enough new functionality in the software to make a release to production.

For anything not related to feature development the product owner is the decision maker on the project, but many times the software developers have more information about the state of the software than the product owners do. This puts the development team in a position of power. As we all know, with great power comes great responsibility. You have the knowledge that your product owner needs to effectively drive the project to success. You should share as much of this information as your product owner can stomach - whether it is good news or bad news. Especially when it is bad news.

Too many times in our industry, development teams will hide bad news from decision makers, hoping that they can adjust resources or shuffle the order of features that need to be developed in order to make up schedule time. We rationalize this by saying that our customers wouldn't want to know all of the "implementation details" and there is no reason to get the customer involved with something that may not even be an issue. This idea of the development team insulating their customers from development issues is flat out wrong. Nothing good can happen from this. In software development, we almost never make up schedule time and things don't magically get better because we shuffle around resources.

So instead, share the bad news. You may think that it makes the development team look bad, but it won't. Software development is not an exact science. Pretending that it is can be a recipe for disaster. A good product owner will know this and should expect some setbacks during the course of the project. Sharing these setbacks shows your customer that you are their partner in this effort. It gives your product owners the valuable information that they need in order to make the critical decisions that will affect the overall success of your project.

As software developers, we want to prove our expertise. This is our profession; we are supposed to know what we are doing, right? (Right!) So then doesn't admitting our mistakes and pointing them out to our customer make us look incompetent? Absolutely not. The truth is that by owning our mistakes and exposing them, we look more confident in our abilities. It shows your customer that making mistakes is part of the development process and is normal. Confidently owning mistakes, learning from them, and addressing them with better solutions shows your customer that you are experienced and you know how to deal with adversity. It shows that you can be trusted to make decisions that will benefit the project and the customer, and that you are not just looking out for yourself.

It is easy to fall into the trap of always wanting to please our customers. This is a very shortsighted view of software development. If you aren't concerned with repeat business, then by all means, always tell the customer what they want to hear. The truth of the matter is that sharing information that upsets your customer is almost always the right thing to do. It may mean having an awkward meeting or a tense few days, but it allows your customers to do their jobs more effectively, to plan appropriately and it better enables them to meet their expectations in the long run.

Don't mistake honesty for transparency. Customers expect honesty. They expect truthful responses to their questions. Transparency goes one step further. It means sharing information when you don't have to. It means sharing information when things go wrong. It means partnering with your customer in order to deliver the best software possible in order to return value to the business.

Being transparent isn't always easy. Just remember, the most important time to be transparent is when it is the hardest.