1. Revenge of the Tablet: Just as Android created some serious competition for Apple with smartphones, it’s about to do the same with the tablet. 2012 will be the year when we finally see some serious alternatives to the iPad.
2. Gesture-Based TV: The last few years have seen gesture-based technology come to the phone, the tablet and the game console. Next up will be the TV. Why search for the remote when you can just snap your fingers? And why not run apps directly on your TV?
3. Machine to Machine: Next year we’ll see more and more real-world applications of M2M. Our cars already talks to the cloud; very soon anything and everything of value will.
4. Social Goes Enterprise: I predict that 2012 will be the year when most corporations start to embrace social networks and look for ways to harness their value. Think: “Who in my company knows how to optimize Android for video the best?”
5. Return of Zero Install: With the introduction of HTML5 and 4G/LTE, our tablets and smartphones will no longer be dependent on apps and app stores. We’ll be able to access an app’s functionality without the installation.
6. A Planet’s Worth of Data: With sensors showing up in everything, the amount of data we can gather and keep is exploding. What if all this data found its way back to the cloud? Well soon it will, and we’ll need BI tools that can help us understand not just a company’s worth of data, but a planet’s worth.
7. Consumerization of Healthcare: With government incentives pushing doctors from one side and patients pulling from the other, we’ll soon see an explosion of consumer-to-doctor options. We’re already seeing sites like Zocdoc change the way you schedule an appointment. A patient record that you can actually see can’t be far behind.
8. Political Apps: Technology is having an increasing impact on politics, especially elections. With the 2012 U.S. election less than a year away, you can bet this cycle will see an explosion of political apps for tablets and smartphones.
9. Voice Gets Smart: With the Siri feature on iPhone 4S, you can bet voice apps are about to get a major boost. We’ll see voice interfaces getting built into more and more devices.
10. Agile in Business: This rapid development approach, which favors continuous results over analysis paralysis, is making its way into business. Agile makes sense for companies that want to get into new markets quickly and use learning over planning as the way to “get it right.”

Since that date, I have been part of multiple teams with a wide variety of experienced team members (in one project I’ve worked with the punch card developers of the ages of 1960s). One particular project with a very small group of developers was at Liebert,Ohio. My colleagues at Liebert, and many others around the world, treated  integration as a non-event. Any individual developer’s work is only a few hours away from a shared project state and can be integrated back into that state in minutes. Any integration errors are found rapidly and can be fixed rapidly.

This contrast is not the result of an expensive or for that matter any complex tool. The essence of all of this lies in a very simple practice of everyone on the team integrating frequently, usually daily, against a controlled source code repository or any source code configuration management tool like SVN, CVS, VSS, Git, Mercurial, Perforce etc. to name a few.

This practice becomes all the more very important when teams are Agile or practice XP (extreme programming) or any such other method of development like TDD etc. Creating continuous builds for every change that’s committed into the repository becomes a matter of seconds and it also eases the troubleshooting any broken feature. There are multiple tools available in the market. The most commonly used and heard is the CruiseControl.

The story is, however, very different when it comes to mobile projects especially iOS based projects. We had a tough time initially having to come up with something that builds continuously and communicates any issue(s) found as soon as they happen. I am going to describe here in as detailed way as possible for setting up an automated process for building the apps for iOS based projects. But before we do that I think it will be wise to consider the following: Integration is primarily about communication. Integration allows developers to tell other developers about the changes they have made. Frequent communication allows people to know quickly as changes develop. The one prerequisite for a developer committing to the mainline is that they can correctly build their code. This, of course, includes passing the build tests. As with any commit cycle the developer first updates their working copy to match the mainline, resolves any conflicts with the mainline, then builds on their local machine. If the build passes, then they are free to commit to the mainline. By doing this regularly and frequently, developers quickly find out if there’s a conflict between two developers. The key to fixing problems quickly is finding them quickly. With developers committing every few hours a conflict can be detected within a few hours of it occurring, at that point not much has happened and it’s easy to resolve. Conflicts that stay undetected for weeks can be very hard to resolve. So if your team is not accustomed for this practice yet, this process is going to be more of a headache for you rather than an ease. If your developers keep their code changes for weeks for fear of losing them, it won’t work for your team.

The general rule of thumb is that every developer should commit to the repository every day. Practically it’s often more useful if developers commit more frequently than that. The more frequently you commit, the less places you have to look for conflict errors, and the more rapidly you fix conflicts. Besides it also an excellent advantage. And that is that frequent commits encourage developers to break down their work into smaller chunks of a few hours each. This helps track progress and provides a sense of progress for the entire team. Often people initially feel they can’t do something meaningful in just a few hours, but I’ve found that mentoring and practice helps them learn. This way team becomes more Agile and aligned to achievement.

Having said that we have come to the first necessity of setting up the process and that is:

1. Everyone commits to the mainline everyday.

After understanding the requirement, it was learnt that the application doesn’t need to access any of the native feature of mobile device like camera, accelerometer. In other words the application was a data driven application having rich user interface. It triggered the thought!!! Why not have Mobile Web Application for this?

What is Mobile Web Application?
Mobile Web Application is web application, optimized to run in mobile device web browser. Some of the parameters under consideration are:

• Screen size
• Bandwidth
• Device orientation
• Native user interface

Present day mobile web browsers contained by smart phones like IPhone and Android can show almost every website that we can open on desktop browser.
Advantages
One of the biggest advantages of Mobile Web App is, it can be uses over the variety of smart phones. If any customization is required for specific mobile device model, then it can be done on the server side. There is no need to create ports for different mobile devices.
In case of there is any update in the application, then the web server contents only needs to be updated. The mobile client will always get the updated contents.

Conventional Mobile Web App

Conventional mobile web can be created in any web platform like PHP, ASP.NET. It essentially consists of a home page which uses HTML mark-up along with CSS and javasript. Mobile user starts the web browser and hits the home page and starts using the application. For each such page, the server sends and HTML mark-up and images to have a decent user interface. One drawback here is the look and feel. Once the user starts the web application, he/she gets an off-deck user experience. The mobile web application look and feel in most of the cases is very different from the mobile device native user interface.

Leveraging HTML 5 & CSS3

HTML 5 is fast catching up. W3C has already created the draft for HTML 5 and CSS3 standards. Popular desktop browser like Chrome, Firefox 4.0 and Opera 9 supports most of HTML5 and CSS3 standards.
Good news is, IPhone mobile safari and Android device web browser also supports most of the HTML 5 and CSS3 features. With support for CSS3 and HTML 5, the developers get extra arsenals for creating more sophisticated mobile web applications

Native Look & Feel using CSS3

With the use of CSS3 it is possible to have native look & feel for IPhone and Android devices. Some of the CSS3 features are:

• Text effects, Gradients
• 2D transformation
• 3D transformation
• Transitions
• Animations

With above feature we can add the animation and transition effect to mobile web applications similar to what IPhone and Android devices provide. Due to support for gradient and text effects in CSS3, we can minimize use of images. This also helps further to reduce the page pay load

Test Web page without any specific CSS.

Test Web page in Android emulator after applying CSS

Test Web page in IPhone emulator after applying CSS

HTML 5 features

HTML 5 provides many new features as compared to HTML 4. IPhone and Android mobile web browsers now support most of the HTML 5 features.  Some of the important HTML 5 features are:

• Local Web storage. Can be used to store the intermediate data locally and when required the data can be saved to cloud or the web server
• Access to Geolocation. Can be used to location based application
• Audio API. Can be used to control and play audio
• Video API
• Canvas support. A 2D drawing mechanism.
• Web Sockets. Real time application

Full Screen Browser Or Native application with embedded web browser (Hybrid)?

The most common way to start a Mobile Web Application on mobile device is to start the web browser and type the application URL.  We can use JavaScript to set the web browser to full screen mode. For some people still this may be dislike.
For such, we can have a native device application. This application then embeds the Web view. Of course, this device application needs to be created to all supported platform. The user downloads this native application from app store. When launched, it displays the embedded web view in full screen view and hits the application home URL. This way the user has a seamless user experience. With this hybrid one can also provide a mechanism to intercept the events in the web view. For example we can intercept click events for all hyperlinks in the view and then may call some native function using the device specific language.

Accessing phone’s native features

There are many instances, in which we may need to access the native phone features like Geo-location, accelerometer. At present not all native phone features can be accessed using JavaScript API from HTML5. Unfortunately not all native features can be access using the JavaScript API. Currently JavaScript APIs are available to access the Geolocation and Address book.
For other features like camera, accelerometer we need to use some bridge. Most common way is to use PhoneGap. It provides an interface to access the native features of mobile devices. It is available for IPhone, Android and other smart phones

Directions
Start the game by drawing or printing out a large tree with roots, branches, and leaves. Roots represent the foundation of your company, and thick limbs are designated for major areas of functionality within your system. Use the inner leaves to symbolize features in the current release, and the outer leaves show possible new features. Label what each part of the tree represents. Have your customers write their fresh ideas for potential new features on sticky notes or note cards, ideally shaped like leaves, and place them on the outer region of the tree near the branches that categorize them. Once complete, analyze the shape of your tree. How is the growth structured? What can you do to maintain a balanced formation? Does one core-feature branch get more leaves than another? Do the roots extend at least as far as the canopy, providing strong support for your product’s success?

Play Online
The goal of this online version of Prune the Product Tree is to analyze the benefits of a meeting, conference, or event. This is just one example of how you can use the game to benefit customers and yourself.

The layers and regions on the tree help attendees to think visually and provide you with a larger variety of ideas. Clicking this image will lead you to an “instant play” game at innovationgames.com. You will see this image as the “game board” and three icons at the upper left corner of the site:

• Red Apples: Benefits you expected and got.
• Rotten Apples: Benefits you expected but didn’t get.
• Presents: Unexpected benefits that made the conference great.

The multi-layered regions of this tree are designed to capture a variety of information about these benefits. Where did the players receive these benefits (at the conference or at work)? What was the nature of the benefit (personal or professional)? And what about the conference infrastructure – the roots of the tree (before the conference or after the conference)? By exploring these dimensions with players, you can create better conferences in the future.

Players add the benefits to the game board by dragging an icon to the tree and describing it. For this collaborative online game, you and the players are able to view each move in real time. Use the integrated chat facility to inquire about the plays and communicate with your customers.

Bottom Line
Rather than using a complicated timeline that only shows your product’s evolution as a confusing linear road, this exercise allows you to analyze multiple aspects of your product’s future growth. With its visual organization and impactful metaphor, customers can express their desires and show where your product needs to grow or shrink in order to succeed. While you may be flowing extra energy into branches that you think are more valuable to customers, this could lead to an uneven distribution of growth, causing a function integral to the completion of the product to shrivel away. Prune the Product tree provides a fun, interactive way to get valuable customer feedback and to discover what you can do to maintain a balanced product.

The book attempts to teach Innovative Doing, Communicating and Thinking to kids using three key tools for innovation: Approximation, Tenacity and Savvy Smarts. Innovative Doing is teachable using the workbook exercises; Innovative Communicating to a lesser degree; Innovative Thinking is much harder to “teach” – kids grasp it intuitively based on how successfully they understand the first two steps.

If you come up with something novel, it is often useful in the sense that it has value to you: it brings you and your loved ones some pleasure and occasionally, you can monetize it. But in a business context, how do you give rise to product innovations and translate those innovations into business value?

Back in 2002, I wrote a whitepaper, “Ubiquitous Rate Optimized Connectivity for Wireless Data Services”, in which I cast data service product innovation in the context of optimization. To quote,

“For the technology and business to succeed, the following factors have to be simultaneously optimized:

1. Data rate

2. Cost to the end user

3. Productive use for work and entertainment

4. Revenue for the service provider

5. Useful mobile life (blended active and stand-by battery life)

The dynamics of jointly optimizing the above factors will lead to many local optima, thereby providing business entities with many opportunities to experiment with various combinations of factors. Your business enterprise will have to be highly skilled to identify and exploit the dynamic local optima before moving on to capture the next opportunity.”

For the engineering-oriented reader, this is a classic multi-dimensional optimization problem. For complex problems such as Wireless Data Services (or allocation of aircrafts to multiple locations and routes by a major airline), non-linear optimization techniques will have to be employed that give rise to many local “optima.” There may or may not be one grand *best* solution; even if there is, it may not be easily reachable. But there will be many “pretty good” solutions – in this case, money-making wireless data service business options. Finding the optima is doubly challenging in real life (business) problems since the optimization problem is “non-stationary”: they keep changing over time.

In common parlance, optimization solution surface is like a mountain range with many sub-peaks (with pretty good views) and a few high peaks (with fabulous vistas). In addition, the mountain keeps moving around or jiggling - hence the “Jell-O Mountain” metaphor.

For a product strategist, the challenge is to identify and get to the closest sub-peak (a new product version) and do this repeatedly (create the product roadmap). Innovation allows you to jump from one sub-peak to the next and create “breakout” value at each sub-peak.

How do we apply the Jell-O Mountain Approach to a specific product or service development example?

• Identify the key factors involved in a successful product or service; for mobile data service, the five factors were listed earlier.

• Create innovations in *each* of the factors; they will tend to happen in different time scales.

• When a sizable group of innovations have happened in some of the factors, blend them and create a prototype product or “proto-product”.

• Test the proto- product with a lead customer and then expand testing. If the business value becomes demonstrable, roll out the first product version. (This is sub-peak #1).

• In the mean time, innovations in some of the other factors are ongoing. Collect them, create a new proto-product, test and roll out. (This is sub-peak #2).

• Continue innovations on all factors, repeatedly culling features that are ready and good, to create other new proto-products.

Notice that with this approach, the non-stationarity or variability or “Jell-O-ness” of the environment is accommodated (indeed exploited) through the rapid prototyping, releasing and repeating, which is a highly desirable characteristic feature of any adaptive system. It is also likely that some of the innovations are large enough that a new product line could emerge out of the current one.

The major challenges in the Jell-O Mountain Approach are: (1) proper identification of the optimization factors and (2) continuous micro-innovation. #1 is dependent on the quality of domain-expertise that is available to you (i.e., you team and consultants). #2 goes back to the level of team members’ skills in Innovative Doing, Communicating and Thinking. As I mentioned earlier, some can be taught but others have to be abstracted by the team members. Specifically, coaching in the development of the skills of Approximation, Tenacity and Savvy Smarts will help your team navigate the Jell-O Mountain!

Due to the exponential acceleration of digital infrastructure, innovation at a fast pace has taken the place of strategy at technology, media and telecommunications (TMT) firms. Adopting a Jell-O Mountain Approach will serve them well by permitting rapid delivery of compelling new services that TMT customers desire.

So to end on a provocative note,

“Strategy is dead; rapid innovation is the way forward”.

“Jell-O mountain-ness is a good thing; allows you multiple chances to innovate”.

Enjoy the haiku . . . PG

Hanami Festival

“Sakura falling on office-mates
Cool evening
Warm sake”

PG Madhavan, Ph.D. | AVP Technical Advisory | GlobalLogic Inc

Leaders in Software R&D Services

ARGENTINA | CHINA | INDIA | ISRAEL | UKRAINE | UK | USA

10900 NE 4^th St. Suite #2300, Bellevue, WA 98004

Mobile: +1.703.286.9036

pg.madhavan@globallogic.com

]]> http://blogs.globallogic.com/jell-o-mountain-approach-to-product-innovation/feed http://blogs.globallogic.com/framework-to-prioritize-requirements http://blogs.globallogic.com/framework-to-prioritize-requirements#comments Sat, 11 Dec 2010 02:37:06 +0000 Sachin Saxena http://blogs.globallogic.com/?p=1009 This is a simple model to prioritize requirements.

Assumptions

* It assume you can accurately determine the cost and value of each “feature”.
* It assumes the homogeneity of skills to perform these function.
* It assumes no interdependence of the requirements.
* It assumes requirements have been broken down consistently.

The framework provides a way to prioritize what gets build in the next iteration of the product.

The color scheme, indicates suggested order in which the requirements can be discussed
1. Blue
2. Green
3. Orange
4. Red.

The foot notes indicates some comments on the capacity, skills, process or platform related issues that this framework brings out.

*IF YOU ARE CONSISTENTLY PICKING ITEMS FROM THIS BUCKET, THEN YOU MIGHT HAVE TOO MUCH CAPACITY. IT MIGHT BE TIME TO RE-PURPOSE SOME PARTS OF THE TEAM.

** IF YOU ARE CONSISTENTLY UNABLE TO PICK REQUIREMENTS FROM THIS BUCKETS, IT MIGHT BE TIME TO EXTEND THE TEAM.

*** IF YOU ARE CONSISTENTLY UNABLE TO EXECUTE ITEMS FROM THESE BUCKETS, SOMETHING IS WRONG WITH TEAM SIZE, SKILLS PROCESS OR TECHNOLOGY PLATFORM.

Before we start our journey I would request the reader to be patient enough to reach the end of the blog and not to draw a conclusion in the middle of it.

So what did I mean when I said “Agile the revolution”? Let us go back to the days when waterfall was considered as the de facto approach for executing software projects. People use to spend months and months on the analysis phase and more months on the design phase producing lot of documents in the due course and finally some code. The worst part was to seen the project failing after all this time and effort just because of all the wrong assumptions and predictions made during the design phase. This brought together a revolution which lead to the birth of completely new software development process “The Agile”.

Let us try to compare the same to any of the revolution that we have come across in the history of the mankind. The story begins with the existence of a government which exploited the citizen or to the least played blind eyes to the obvious problems that existed in the society. The state of anarchy grew to an extent that people revolted and toppled the so called government. Now what do we have, a completely lawless society. When we walk further down the roads of history we will find that soon some new group of people came together and put together new set of rules or constitution but this time based on the hindsight of the past.

So if we look closely the phase just after the revolution there was a small phase which leads to higher degree of anarchy. Closely examining the phase will make us realize that it was during this phase when the very ideology that inspired the people to revolt was blindly followed. So what is the moral of the story? The very ideology which brought about the revolution cannot be practices in its purest form on a normal course.

Before we go further with our discussion let us all try to summarize on what the software revolution was against:

1.       Counterproductive and irrationally long design phase.

2.       Piles and piles of document which would soon turn into some sought of relic.

Now let us step back and ask these questions:

1.       Don’t we need designing to develop a world class software product?

2.       Don’t we need documentation of the software product we are developing?

Don’t run for cover yet. Let us put the very first reason for the revolution under the microscope and try to derive our conclusion. Revolution was not against the process of software modeling per say but against the counterproductive way it was done. Upfront modeling was the real culprit that made the entire modeling process a victim. I cannot agree any more with Eric Evans (author of the book Domain-Driven Design: Tackling Complexity in the Heart of Software) when he said “The worst model I build was on the day one of the project and the best model I build was on the last day of the project”

What was he trying to convey? Software solutions are manifestation of the actual domain for which the solution is laid down. So how can we model a system to manifest a domain which we are least aware of as in day one of the project? Any attempt to derive a model on day one will fall victim to all wrong assumption and prediction about the future of the domain. But then due to all the frustration the desperation against the counterproductive software modeling phase which brought about the agile revolution there is a slight element of anti-model in it.

I am sure anyone who have spend enough time with software development agree that solving a problem without proper direction and guidelines will only lead to more problems. So what we need is not a complete no-modeling software development software process but rather a more productive and flexible modeling approach. Before I explain any further on the need of a model let us try to see the second evil which triggered the revolution.

Piles and piles of documents which would soon turned into relic. I am sure anyone who has been on those waterfall projects will recollect the nightmare of spending weeks reading those document to understand what they are needed to do and still find themselves as “Alice in Wonderland”. Now a pinch of reality “Can we survive with no documentation?” Most common argument I hear is minimal documentation but then how little is minimal documentation. One other interesting comment I here is “Code should be self documenting”. As soon as a developer is enlightened with this single line of software oracle they start decorating there code in all possible means.

Let us take a simple example of imaginary financial software module which calculates the principle at the end of every month. Basically there is a starting principle from which based on the monthly installment paid some amount will be deducted. But then for every 6 months which the customer has paid the loan without any default they will receive a small bonus which will again be deducted from the principle to finally derive end of the month principle. Now the developer who was assigned the user story came up with following classes:

·         Principle

·         Installment

·         DeltaDeduction

Now delta deduction is the amount that will be deducted based on the bonus earned over the period of 6 months.  Now no matter how well the DeltaDeduction class is commented it will be a source for uncertainty for anyone reviewing the code. First question will be “What should I relate the DeltaDeduction class to in the domain”. Rather if the developer would have named it as something like “NonDefaulterBonus” it would be have been self documenting even when it was not decorated with lot of comments.

Now even to name our classes correctly we need to understand the domain that we are working in and have some model of the same in our mind.

We need to realize that the end goal is not to complete a user story but to deliver a shippable piece of software with a tolerable degree of bugs.

Another common excuse made regarding the modeling is “We know we need some modeling to be done but cannot find time”.

So when did software development become a 100m sprint where all we care about is time. Software development without adequate modeling take us to the very software anomaly which is called as “Big Ball of Mud” (http://laputan.org/mud/ ).

“Co-owning the code” is another very religiously followed principle in agile. I am very sure where this is coming from. If you remember the old days when certain piece of the code is completely owned by one single person. It became completely impossible to make any change in his/her absence. But then does just co-owning the code solve this problem. Let us take a scenario consider a user story which span two modules A and B. The developer who is implementing the user stories works with the team which own the module A. Now to implement the user story he/she need to modify the module B. Risk is that the modification will be done without understanding the actual motivation that went behind the design of module as he/she was not part of the team which co-owned the module B. So the ownership cannot be code -centric but rather should be a domain context centric. Such contexts are called as Bounded Context in Domain Driven Design or DDD.

“Re-factor when your code starts smelling”. Again this is inspired by the fact that many of the waterfall project failed due to ignoring the known issues in the project which acted as time bombs which finally called upon the doomsday for the project.

In reality all large system are not well designed and investing the time and effort to re-factor every bit of the system to adhere it to all know design principle will not yield the expected benefit. The idea is to identify the core modules that the business relies on the most and keeping it clean all time. It is also very beneficial to identify the axis of frequent change within your project and re-factoring it enough to accommodate more changes. There are always going to be certain module of the software which would look more or less like Big Ball of Mud. But as we said before investing time and effort to get these modules re-factored might not yield the desired return. But again left unattended will make the chaos in these modules to creep into rest of the modules.

I am sure most of you are familiar with the Trojan horse story. Modules which are not well designed and re-factored are similar to the Trojan horse. If left unguarded the enemy (bad design) will creep into our castle (code). So how do we deal with such scenario? If the Trojan’s had put the horse in a fenced premise with a strong gate and a gatekeeper the enemy would have failed in their plan. So how do we build wall around these “Big Ball of Mud”. DDD offers a beautify pattern call the Anti-Corruption layer (ACL) that will mask the chaos from infusing into the rest of the software module.

So in short what need is not a completely no-design and no-documentation software development process rather a smatter way to model the domain and implementing the model in our code in such a way that it requires little documentation to be explicit.

I idea of this blog was to set the stage for my next blog “Introduction of Domain Driven Design” in which I will explain how DDD will allow you to achieve high degree of agility and still enjoy the benefit of having domain model in place. Most importantly I will also cover how DDD allow modeling and agile go hand in hand.

“Collector” is probably too generous a word, but I am definitely an aficionado of handcrafted items. While true hand craftsmanship has become rare and generally prohibitively expensive in the West, I’m fortunate that I get to travel to places where hand-made items are still relatively affordable. Over the years I’ve managed to accumulate artwork, metal-craft, furniture pieces, embroidered tablecloths, hand-woven shawls, silks and other items that my family and I really love. Except for the occasional tribal spear or walking stick, I’ve been more successful in finding great household items and women’s clothing than I have in finding real “guy stuff”, but at least that has made me a popular gift-giver in my family!

Having beautiful, handmade things in my life is a source of real satisfaction to me. I like the thought that an actual human being made something I have or use and, I like to think, perhaps they cared about what they made, and took pride that what they made was really good. Maybe they even hoped that someone like me would come along who appreciates what they made. A handcrafted item puts me in touch with a different way of life—that of the craftsperson. And the best of these items have an elegance and, well, “soul” to them that machine-made items just don’t seem to have, at least for me.

When I’m looking at these items, it strikes me that my work is not so different: software is the last handmade thing in common use in the developed world.

For those of us in the software industry, that software is “handcrafted” is no great revelation. To us it’s clear that beyond the technology—which, though sophisticated, is at least largely amenable to human control—the true difficulty in producing a software product are the human factors and the imponderables that human beings and human interactions introduce. Humans misunderstand directions, give unclear requirements, make mistakes and wrong assumptions, have prejudices and divergent career goals, are good at some things and bad at others, can act lazy or be overly aggressive, and generally are very—human. Though the technology is compelling, to me the people aspect of the business is the bigger challenge. How do you take a collection of imperfect human beings, and get them to work together to quickly produce a quality product that does what you and your customers really want it to do? That’s the challenge, and why software product development is so much a human activity.

A few years ago I acquired a small carpet, which I really love. This is a fairly elderly appearing “tribal” pattern carpet, and the seller made extravagant claims for its origins and history. After a series of visits and hard-bargaining sessions I finally bought it, but still had some doubts about whether I had really purchased a hand-made masterpiece or an artificially aged, cheap machine-made knock-off. After living with the carpet a while, I began to notice a lot of small asymmetries in the intricate patterns. A series of “gulls” or “C” shapes make up part of the border, for example. I began to notice that some of the “C” shapes opened to the right, and others to the left, and that while the two sides of the carpet were close to being mirror images of each other, they actually were not. There were a number of other such small asymmetries here and there throughout the carpet. After a few weeks of observing these imperfections, I became completely convinced that this was indeed a laboriously hand-made carpet. It would be prohibitively expensive or even impossible to achieve this degree of asymmetry and imperfection by machine—the only way it could be done was by hand, knot-by-knot. I’ve also read that in more than one culture, such asymmetries are inserted purposely as “intentional flaws”. This is to avoid tempting fate by aspiring to make something perfect. While I think this philosophy is probably a good example of making a virtue out of necessity, I nonetheless appreciate the sentiment. It is, unfortunately, not for us humans to produce perfect work—at least I’ve never seen it or, to be honest, done it. Machines—maybe; but a human endeavor, no.

As another case in point, I once tried to get a business suit made for myself in India, figuring the labor cost would be low—which it was. I had read that one of the hallmarks of a good tailor-made suit were buttonholes made by hand. I had also read that you can tell whether a buttonhole is hand-made or not by looking at the side of the buttonhole on the inside of the suit, facing the wearer. If the buttonhole is rough and imperfect on the inside, it means it has been hand embroidered; if it’s perfect on both sides, it’s machine made. In the West, you will pay a lot more for a suit with “imperfect” buttonholes than for a suit with “perfect” ones, because hand embroidery is a sign of extra, skilled effort.

However, when I asked the Bangalore tailor “Do you put hand-made button holes on your suits?”, he looked embarrassed and responded: “Yes. We’ve been trying to save up for a machine, but so far we can’t afford one.” The tailor’s perspective regarding handwork and my own were quite different in this situation. To me the value of the suit was increased by skilled handwork, even if the result was in some sense imperfect. To the tailor, the imperfections and, certainly, the extra time that came from the required handwork were a negative.

It’s imperfection that’s the hallmark of a hand-made item, not perfection. Both the Bangalore tailor and I agreed on that point. But where I valued the “imperfection” in this particular case, he was embarrassed about it.

As consumers, our standard for software products really is “perfection”. We like our buttonholes perfect on both sides. Like that tailor in India, though, as software developers we are in a situation where there are no tools available to us that will rapidly produce a perfect product. In large part, such tools are not even theoretically possible, because the goals or requirements for a software product are invariably “fuzzy” to a smaller or larger degree when we begin a project.

Years ago, in the early 1990’s, I was on the team that developed Rational Rose “1.0”. I believed—as did a number of my colleagues at the time—that we were helping to create a new way of developing software. We felt that in the future people would generate code directly from a graphical version of their software architecture, and then would be able to use the same tool to reverse-engineer hand-modified code back into an updated graphical representation. Alternatively, you could start with an implementation, semi-automatically extract the architecture from it, and proceed from there. The round-trip nature of this process would overcome what we then saw as one of the major obstacles to good software architecture, which was keeping the architecture diagrams up-to-date with the actual implementation. The reverse-engineering piece, once fully in place, would allow people to effortlessly toggle between a high-level architectural view of their code, and the detailed implementation itself, we thought.

Now, fifteen years down the road, why isn’t a rigorous architectural approach to software development widely used? Why don’t people architect their product and then just mechanically generate the implementation code directly from the detailed architecture? Surely enough time has passed since Rose 1.0 that its descendents and competitors could have made this approach completely practical if that’s what the market wanted. There are probably many reasons why this is not the approach most companies actually tend to take today, but I would argue that a key one is that in reality people generally do not know exactly what their product is supposed to do when they start work on it. I would also argue that in most cases, they can’t know. Even in the case of a relatively mechanical port, the urge to add features, fix problems, act on mid-course learning and/or exploit the possibilities of a new technology generally prove irresistible, and in real life the resulting product will invariably be different from the original concept.

There is no tool yet devised that will mechanically turn a “fuzzy” or currently unknown requirement into one that is clear and unambiguous. There are definitely techniques—Innovation Games, Agile kick-off meetings, requirements as test cases, rapid prototyping and the “fail fast” approach—which can help the product owner refine his or her vision into something less ambiguous. And I can still imagine a world where completely unambiguous requirements are fed into a machine, and something perfect pops out of the other end—like code from a Rose diagram or, to return to our tailoring analogy, a buttonhole from an automatic sewing machine. What I cannot imagine, though, is human beings producing specifications so perfect that what comes out is actually what is desired—perhaps even for a buttonhole.

Until humans are out of the requirements and specification loop—which I can’t imagine if the software is to be used by human beings—I think we need to live with imperfection in this sense. To be sure, our ability to implement requirements mechanically has and will continue to steadily increase. Programming paradigms like convention over configuration (Ruby) and aspect-oriented programming (Spring) are reducing the amount of code required to implement a new feature, reducing development times and eliminating systematic sources of error. Tools, languages and reusable components and frameworks have vastly increased programmer productivity over the last few decades, and I am sure this trend will continue. But to date, people are very much part of the process of creating software as well as specifying it.

Should we hope that human beings are eventually eliminated from the software development process? Right now, a large part of the work of a professional software engineer could arguably be characterized as identifying and eliminating ambiguities in the specifications, to the point where a machine can mechanically carry them out. A software engineer often makes dozens of decisions each day about what the software should do in a given situation not specified or anticipated by the product owner or requirements, often because the situations are considered “edge cases” or “too low level”. A theoretical device that completely eliminated these ambiguities would have to first identify them, and then surface them for resolution by the product owner at requirements-generation time. But would the product-owner be able to cope with a huge volume of decisions about, say, an unhandled exception in a very specific code path many levels deep?

My guess is that while in the future “programming” will be done at a higher level, with better tools, more reusable frameworks and even perhaps “artificially intelligent” assistance, it will remain a human activity at core for years to come. As long as humans are better at understanding the intent of other humans than machines are, I think this must be the case. At some point, machines will probably become so smart, and the collection of re-usable frameworks so deep, that artificially intelligent systems can assemble better software from vague requirements than people can. Until then, however, I think we will have to learn to appreciate our buttonholes with one rough imperfect side, and use approaches like Agile that acknowledge that software development is a human activity.

The hand off happens typically via an architecture spec written by the architect, composed of a set of UML diagrams. The programmers read these diagram and create code from there.  Although most CASE tools provide code generation capability, it is not commonly used because the level of abstract in the design and code implementation are quite different. After the developer digest the meaning of the UML diagram, he/she go ahead to write code.

What is the problem ?

However, the progression is not as smooth as we expect. There is a gap between the architecture diagrams and the code which requires a transformation. During this transformation step, there is a possibility of mis-interpretation, wrong assumption. Quite often, the system ends up being wrongly implemented due to mis-communication and misinterpretation which can happen because the architect hasn’t described his design clear enough in the document, or because the developer is not experienced enough to fill in some left-out detail that the architect consider obvious.

One way to mitigate this problem is to have more design review meetings, or code review session to make sure what is implemented is correctly reflecting the design intention. Unfortunately, I found such review sessions are usually not happening either because the architect is too busy in other tasks, or he/she is too “hands-off” and doesn’t want to work at the code level.  It ends up the implementation doesn’t match the design. Quite often, this discrepancy is discovered at a very late stage and left no time to fix. While developers continuously patching the current implementation for bug fixing or adding new features, the architect start to lose control on the architecture evolution.

Is there a way for the architect to enforce her design at the early stage given the following common constraints ?

1. The architect cannot afford frequent progress/checkpoint review meetings
2. While making sure the implementation compliant with the design at a higher level, the architect doesn’t want to dictate the low level implementation details

Test As A Specification

The solution is to have the architect writing the Unit Tests (e.g. JUnit Test Classes in Java), which acts as the “Spec” of her design.

Note the “component” is an important artifact. The architect will provide a responsibility description of each component.  Each component expose 2 sets of interfaces

1. Facades - defines the interface that this component “provides”
2. Colloborators - defines the interfaces that this component “uses”

But more importantly, the architect also write unit tests against these interfaces.

By writing concrete Unit Tests against the “Facades”, the architect fully specify the external behavior of the system from the perspective of the component’s client.

By expressing “Collaborator” in terms of Mock Obects, the architect fully specify the expectation of its underlying supporting objects.

In other words, the unit test defines how this component interact with external parties, including its client (how this system will be used), as well as the collaborators (how this component uses other components).  It becomes a very concrete design spec of the component and leave no room for mis-interpretation.  The unit tests serves a number of important purposes

1. It is the conformance test of the implementation (even in future evolution)
2. It is a usage example document
3. It is a design specification (now developers knows exactly when the architect is expecting)

Note that the architect is not writing ALL the test cases. Architecture-level Unit Test are just a small subset of the overall test cases specifically focus in the architecture level abstraction. This unit test are specifically written to ignore the implementation detail so that it gives enough freedom for the developers to choose how the implementation should be done. On the other hand, the architecture itself becomes more stable as it will not be affected by future changes of implementation logic.

Other than the architectural unit tests coded by the architect, the developers provide a different set of implementation level unit tests.  Usually, this set of “Impl Level TestCase” which change when the developers change the internal implementations, but this won’t affect the “architectural test cases”. By separating these 2 sets of test cases under different categories, both sets of test cases can evolve independently when different aspects of the system changes along its life cycle, and resulting in a more maintainable system as it evolves.

To illustrate, lets go through an example using a User Authentication system. There maybe 40 classes that implements this whole UserAuthSubsystem. But the architecture-level test cases only focused in the Facade classes and specify only the “external behavior” of what this subsystem should provide. It doesn’t touch any of the underlying implementation classes because those are the implementor’s choices which the architect doesn’t want to constrain.

User Authentication Subsystem Spec

Responsibility:

1. Register User — register a new user
2. Remove User — delete a registered user
3. Process User Login — authenticate a user login and activate a user session
4. Process User Logout — inactivate an existing user session

Collaborators

1. Credit Card Verifier — Tell if the user name match the the card holder
2. User Database - Store user’s login name, password and personal information

public class UserAuthSystemTest {
  UserDB mockedUserDB;
  CreditCardVerifier mockedCreditCardVerifier;
  UserAuthSystem uas;

  @Before
  public void setUp() {
    // Setup the Mock collaborators
    mockedUserDB = createMock(UserDB.class);
    mockedCardVerifier =
        createMock(CreditCardVerifier.class);
    uas =
        new UserAuthSubsystem(mockedUserDB,
                              mockedCardVerifier);
  }

  @Test
  public void testUserLogin_withIncorrectPassword() {
    String uName = "ricky";
    String pwd = "test1234";

    // Define the interactions with Collaborators
    expect(mockUserDB.checkPassword(uName, pwd)))
                     .andReturn("false");
    replay();

    // Check the external behavior is correct
    assertFalse(uas.login(userName, password));
    assertNull(uas.getLoginSession(userName));

    // Check the collaboration with collaborators
    verify();
  }

  @Test
  public void testRegistration_withGoodCreditCard() {
    String userName = "Ricky TAM";
    String password = "testp";
    String creditCard = "123456781234";
    expect(mockCardVerifier.checkCard(userName,creditCard)))
                           .andReturn("true");
    expect(mockUserDB.addUser(userName, password)));
    replay();
    uas.registerUser(userName, creditCard, password));
    verify();
  }

  @Test
  public void testUserLogin_withCorrectPassword() { .... }

  @Test
  public void testRegistration_withBadCreditCard() { .... }

  @Test
  public void testUserLogout() { .... }

  @Test
  public void testUnregisterUser() { .... }
}

Summary

This approach (”Test” as a “Spec”) has a number of advantages …

1. There is no ambiguation about the system’s external behavior and hence no room for mis-communication since the intended behavior of the system is communicated clearly in code. Such code sitting at the boundary also tends to be more readable.
2. The architect can write the TestCase at the level of abstractions she choose. She has full control in what she wants to constraint and what she wants to give freedom.
3. By elevating architect-level test cases as the spec of the system’s external behavior. They become more stable and independent of implementation level. This makes the architecture more stable.
4. This approach force the architect to think repeatedly from an external perspective, what is the “interface” of the subsystem and also what are the collaborators. So the system design is forced to have a clean boundary.
5. The test cases can also serve as the usage examples