Cricket World Cup 2011 is approaching and I'm interested in analyzing one day international cricket data to predict some results and share interesting information about cricket.
For the analysis, I need cricket data and tried several things to get it...
Personal research: Explored the web but couldn't find aggregated cricket data anywhere. There are many cricket-statistics oriented websites but none of them were useful (except Cricinfo, my favorite cricket website)
Reach out to my network: I requested my friends for advise last month and received many emails with information and offer to help compile the data
Reached out to Sports data companies: I contacted Opta Sports to buy who this data. Although they have the data, it was too expensive for my personal experiment.
So I decided to collect this data myself by web scraping cricket scorecards. I first tried to use R libraries to web scrape but found it lacking. So I switched to Ruby, which has a great library for web scraping - Hpricot (thanks Satty for getting me started and Amit/Thomas for solving my newbee issues).
I'm happy to report that now I have a robust Ruby script that can download all One Day International Cricket data (3000+ matches) in 3 handy files:
This Ruby script takes about 40 minutes on a fast internet connection to collect the data. It took me ~ 40 hours to write and fine tune the script. Most of the time was spent in dealing with typical data issues associated with web scraping and making the script generic to handle Test cricket and T20 cricket scorecards as well.
Comparison of population mix and number of crimes per year in Indian states
More populous states seem more dangerous - law of crouds
Comparison of population mix and number of crimes per person in Indian states
Is there a correlation between the religious mix in a state and riots in that state?
Which are the best and worst states to live in?
Using GDP/person as a proxy to growth and opportunities in a state, the following plot shows the states with high/low growth/opportunities and high/low crimes...
Examples of the implications of growth and opportunities and state of crime in Indian territories...
What could Indian states do to move to the "magic quadrant"?
Here's an attempt to plot crime in Indian states. The data was obtained from Systemic Peace Organization. Next I plan to overlay crime data on the state population.
As population differs significantly in Indian states, here are the crimes/person to determine the safest and the most dangerous states.
Daman & Diu has the highest crime rate per person. This territory has liquor license whereas the state close to it, Gujarat doesn't. Could that be a contributing factor?
Bihar, Assam, Jharkhand and Rajasthan don't surprise as the most dangerous states but its interesting to see Kerela with high crime rate per person given its 100% literacy rate
The safe places to live include Nagaland, Delhi, Punjab, Haryana, Chandigarh.
Here's a quick R plot of population in Indian states and the breakdown by the religion people practice. The data was obtained from this Indian government site. I'm looking for other state-level data to overlay on this to find interesting patterns e.g. crime, literacy etc.
As next steps, I would like to: 1) "zoom" into the areas of chart that are too small to be interpreted properly, and 2) plot this data on the map of India.
Test driven development isn’t actually testing, but designing.
The code I might write in my unit test describes the design I’d like to see reflected in the code I’ll write. The running unit test doesn’t confirm my code is bug free, just that it meets the design I described in my unit test. Test Driven Development is a thinking technique; a designing technique.
Couple of weeks ago, I was looking for interesting data sets to play with using R. I came across this post and got inspired to use World Bank data to extract interesting patterns/facts about developing countries. World Bank has 300+ World Development indicators on 200+ countries from 1960 to 2008.
I plotted BRIC (Brazil, Russia, India, China) nations against USA. Here are some plots on a few topics...
I'm interested in analyzing and plotting Cricket data to discover interesting facts and relationships...
I'm looking for "raw" Cricket scorecard data for One-day internationals and Test matches only. Google didn't find anything useful this morning. So, I'm looking for advice and I'm happy to pay for it too. Any suggestions?
I want to share my experience in generating the data for social network analysis using R and analyzing it using Gephi...
WHICH DATA STRUCTURE TO USE FOR LARGE GRAPHS? I quickly realized that using edge lists and adjacency matrix gets difficult as the graph size increases. So I needed an alternative graph format that was efficient (for storage) and flexible to capture details like edge weight. I chose Gephi's gexf file format as it can handle large graphs, and it supports dynamic and hierarchical structure. Checkout gexf comparison with other formats for details.
HOW TO HANDLE LARGE DATA SETS IN R? As I tried to process millions of rows of email log to derive the edgelist, I realized a couple of things...
1) R cannot handle data larger than my computer's RAM. So I had to look for a way to use R for large data sets. R packages like RMySQL and SQLDF came in handy for this. SQLDF uses SQLlite, an in-memory database. If your data cannot fit into RAM then you can instruct SQLLITE to use persistent store for handling large data sets. Note: There are many other ways to handle large data in R effectively, e.g. R multicore package for parallel processing, R on MapReduce/Hadoop, etc. Check out the presentation on high performance computing in R for other techniques like ff and bigmemory. Please shout if there are other ways that you used...
2) Some operations are better suited for database/RDBMS: I offloaded RDBMS-suited tasks to SQLlite, the default database used by SQLDF.
3) Learn memory management in R: - By default R allocates ~1.5GB memory for its use. I allocated more memory for R to handle larger objects using the command "memory.limit(size=3000)" - Remove unwanted objects from the R session e.g. rm(raw_emails, emails, to_nodes,from_nodes,all_nodes, unique_nodes) gc() # call garbage collection explicitly
LOADING THE GRAPH IN GEPHI Gephi wasn't able to handle very large graph files (e.g. for files > 500MB size, Gephi was either too slow or stopped responding). So I had to do a couple of things...
1) Increase the amount of memory Gephi allocates for the JVM at startup: By default Gephi allocates 512MB memory for JVM. This wasn't enough to load the large graph file, so I increased the max. memory Gephi allocated for JVM to 1.4GB.
Edit C:\Program Files\Gephi-0.7\etc\gephidesktop.conf file and changing the line default_options="--branding gephidesktop -J-Xms64m -J-Xmx512m" to default_options="--branding gephidesktop -J-Xms64m -J-Xmx1400m"
2) Decrease the file size by reducing the text in the graph file e.g. use shorter node_ids, edge_ids etc.
Also, Gephi complained about incorrect file format (it expects UTF-8 encoded XML files). I fixed this simply by opening the graph file generated by R in Textpad and saving it in UTF-8 format before feeding it to Gephi.
LESSONS LEARNED 1) R is more than a statistical tool. I was able to manipulate and clean large data sets (500+ million rows) easily. I will continue learning it. Its fun and rewarding.
2) There are other sophisticated tools for visual social network analysis like Network Workbench I will explore it for heavy analysis, but Gephi is very easy to use and continues to be my favorite.
3) Use a machine with a lot of RAM, as both Gephi and R are memory hungry
MY CODE FOR GENERATING THE GRAPH By the way, here's the R code I used for preparing the graph from email logs for social network analysis using R and Gephi. I'm sure there are better ways to accomplish this. Please shout if you notice any.
I want to share my experience in generating the data for social network analysis using R and analyzing it using Gephi...
WHICH DATA STRUCTURE TO USE FOR LARGE GRAPHS? I quickly realized that using edge lists and adjacency matrix gets difficult as the graph size increases. So I needed an alternative graph format that was efficient (for storage) and flexible to capture details like edge weight. I chose Gephi's gexf file format as it can handle large graphs, and it supports dynamic and hierarchical structure. Checkout gexf comparison with other formats for details.
HOW TO HANDLE LARGE DATA SETS IN R? As I tried to process millions of rows of email log to derive the edgelist, I realized a couple of things...
1) R cannot handle data larger than my computer's RAM. So I had to look for a way to use R for large data sets. R packages like RMySQL and SQLDF came in handy for this. SQLDF uses SQLlite, an in-memory database. If your data cannot fit into RAM then you can instruct SQLLITE to use persistent store for handling large data sets. Note: There are many other ways to handle large data in R effectively, e.g. R multicore package for parallel processing, R on MapReduce/Hadoop, etc. Check out the presentation on high performance computing in R for other techniques like ff and bigmemory. Please shout if there are other ways that you used...
2) Some operations are better suited for database/RDBMS: I offloaded RDBMS-suited tasks to SQLlite, the default database used by SQLDF.
3) Learn memory management in R: - By default R allocates ~1.5GB memory for its use. I allocated more memory for R to handle larger objects using the command "memory.limit(size=3000)" - Remove unwanted objects from the R session e.g. rm(raw_emails, emails, to_nodes,from_nodes,all_nodes, unique_nodes) gc() # call garbage collection explicitly
LOADING THE GRAPH IN GEPHI Gephi wasn't able to handle very large graph files (e.g. for files > 500MB size, Gephi was either too slow or stopped responding). So I had to do a couple of things...
1) Increase the amount of memory Gephi allocates for the JVM at startup: By default Gephi allocates 512MB memory for JVM. This wasn't enough to load the large graph file, so I increased the max. memory Gephi allocated for JVM to 1.4GB.
Edit C:\Program Files\Gephi-0.7\etc\gephidesktop.conf file and changing the line default_options="--branding gephidesktop -J-Xms64m -J-Xmx512m" to default_options="--branding gephidesktop -J-Xms64m -J-Xmx1400m"
2) Decrease the file size by reducing the text in the graph file e.g. use shorter node_ids, edge_ids etc.
Also, Gephi complained about incorrect file format (it expects UTF-8 encoded XML files). I fixed this simply by opening the graph file generated by R in Textpad and saving it in UTF-8 format before feeding it to Gephi.
LESSONS LEARNED 1) R is more than a statistical tool. I was able to manipulate and clean large data sets (500+ million rows) easily. I will continue learning it. Its fun and rewarding.
2) There are other sophisticated tools for visual social network analysis like Network Workbench I will explore it for heavy analysis, but Gephi is very easy to use and continues to be my favorite.
3) Use a machine with a lot of RAM, as both Gephi and R are memory hungry
MY CODE FOR GENERATING THE GRAPH By the way, here's the R code I used for preparing the graph from email logs for social network analysis using R and Gephi. I'm sure there are better ways to accomplish this. Please shout if you notice any.
- Learning both has been fun - "dude, i'm playing with R and love it." The excitement doesn't end!
I'm totally addicted.
- I keep discovering new and nifty things. Every time I hit an interesting problem, there's an elegant way to deal with it. Very Ruby like - "If it is taking more then 5 lines of code, you are probably doing it wrong."
- Both, R and Ruby have Plugin architecture that gives you a lot of power and flexibility. For example, just discovered that R has a RDBMS package to offload database-specific/suited tasks to a database, while using R for what its good at...
- Both have an active, passionate community behind it.
- Learning both has been fun - "dude, i'm playing with R and love it." The excitement doesn't end!
I'm totally addicted.
- I keep discovering new and nifty things. Every time I hit an interesting problem, there's an elegant way to deal with it. Very Ruby like - "If it is taking more then 5 lines of code, you are probably doing it wrong."
- Both, R and Ruby have Plugin architecture that gives you a lot of power and flexibility. For example, just discovered that R has a RDBMS package to offload database-specific/suited tasks to a database, while using R for what its good at...
- Both have an active, passionate community behind it.
After learning the basics of R, I decided to learn something harder last week. I picked Social Network Analysis (SNA) to learn the concepts of SNA and R. My primary interest in SNA is visual exploration of networks, so I needed to find a tool first.
Which tool to use for visual SNA? Features needed:
1) graphical representation of network
2) visually navigate the graph (zoom in/out, drag) to explore large graphs
3) manipulate the graph (filter nodes, edit/delete/group nodes and same for edges)
4) free, preferably open source. I'm not currently interested in commercial tools (can't justify their steep price tag for my experiments).
I found out that R has good libraries like SNA (checkout Drew Conway's tutorial) and iGraph (see this tutorial) for social network analysis. However, they lack features to deal with large graphs (nodes > 200, edges > 500 seem to make the process slow and the plots unusable), navigate and manipulate the graph visually. All I could do was plot simple networks (shown below).
So I continued my hunt for a good tool for visual SNA and discovered Gephis, an open source app for visual exploration. Think "photoshop for graphs".
WARNING: SNA with Gephis is addictive. I've had my share of sleepless nights and dreams of nodes and edges
I played with Gephi for several hours to learn it (its kewl) and to impress my daughter (her dad is no fireman, who saves people, but he can do nifty things with a computer ;) I was able to discover interesting facts from the data, including:
- Avg. degree: the number of edges/connections attached to a node - Network diameter: The longest path between the nodes in the graph - Average path length: In how many steps (on avg) can one can reach any node from any other node in the graph - Degree power law: The higher this number, the more unequal is the distribution of connections within the network, which means that some nodes are very well-connected and some are not at all - Average clustering coefficient: Shows how well the nodes are embedded in their neighborhood i.e. is there a "small world" effect within the network - Modularity: The higher this parameter, the more defined are the communities within the network. A result of 0.4 or more is usually considered meaningful - Betweenness centrality was calculated for each node, which shows how often the node appears on the shortest path between any two random nodes in the network. The higher this parameter, the more influential the node is. The nodes which have high betweenness centrality are not necessarily the ones that have the most connections and don't have to be the most "popular" ones Here's a video of Gephi features (older version)
I was also able to discover interesting patterns in the data, like the communities that emerge, popular people and the connectors. See the graph below to see nodes with different colors (communities), size (popularity) and how most connections between the two communities flow through a few nodes (connections)
Web rendition of these graphs is also possible. Checkout this visualization of Perl authors (you can drag the graph with left click and dragging your mouse; you can zoom-in/out on the graph with your mouse wheel)
Lessons learned so far: - I quickly realized that you need a good machine for using Gephi effectively (a good video card, enough memory and fast CPU) - There is a great, active community behind Gephi, so expect frequent releases to resolve critical issues and new features. I'm waiting for this month's release that fixes some issues I've faced :-) If you know of other tools to visually explore graphs, please leave a comment.
Which network data I used? There's a wealth of network data available today, including social networking sites, phone logs, work history, chat logs, email logs etc. I decided to create test data for email traffic to test the hypothesis of "who we send emails to or receive emails from" is a good indicator of our social network. My test data has 50,000+ nodes and 150,000+ edges.
I used R to create the create data for graph. R is good at handling millions of rows of data and is powerful for data manipulation (cleaning, creating edge lists, adjacency matrix etc.). I used it to format raw email traffic test data into graph formats (edgelist, adjacency matrix etc.) It took me a couple of hours to write code for creating the data set to feed into Gephi. Using R to solve a real need has been a good learning experience so far.
Leave a comment, if you're interested in seeing the code.
After learning the basics of R, I decided to learn something harder last week. I picked Social Network Analysis (SNA) to learn the concepts of SNA and R. My primary interest in SNA is visual exploration of networks, so I needed to find a tool first.
Which tool to use for visual SNA? Features needed:
1) graphical representation of network
2) visually navigate the graph (zoom in/out, drag) to explore large graphs
3) manipulate the graph (filter nodes, edit/delete/group nodes and same for edges)
4) free, preferably open source. I'm not currently interested in commercial tools (can't justify their steep price tag for my experiments).
I found out that R has good libraries like SNA (checkout Drew Conway's tutorial) and iGraph (see this tutorial) for social network analysis. However, they lack features to deal with large graphs (nodes > 200, edges > 500 seem to make the process slow and the plots unusable), navigate and manipulate the graph visually. All I could do was plot simple networks (shown below).
So I continued my hunt for a good tool for visual SNA and discovered Gephis, an open source app for visual exploration. Think "photoshop for graphs".
WARNING: SNA with Gephis is addictive. I've had my share of sleepless nights and dreams of nodes and edges
I played with Gephi for several hours to learn it (its kewl) and to impress my daughter (her dad is no fireman, who saves people, but he can do nifty things with a computer ;) I was able to discover interesting facts from the data, including:
- Avg. degree: the number of edges/connections attached to a node - Network diameter: The longest path between the nodes in the graph - Average path length: In how many steps (on avg) can one can reach any node from any other node in the graph - Degree power law: The higher this number, the more unequal is the distribution of connections within the network, which means that some nodes are very well-connected and some are not at all - Average clustering coefficient: Shows how well the nodes are embedded in their neighborhood i.e. is there a "small world" effect within the network - Modularity: The higher this parameter, the more defined are the communities within the network. A result of 0.4 or more is usually considered meaningful - Betweenness centrality was calculated for each node, which shows how often the node appears on the shortest path between any two random nodes in the network. The higher this parameter, the more influential the node is. The nodes which have high betweenness centrality are not necessarily the ones that have the most connections and don't have to be the most "popular" ones Here's a video of Gephi features (older version)
I was also able to discover interesting patterns in the data, like the communities that emerge, popular people and the connectors. See the graph below to see nodes with different colors (communities), size (popularity) and how most connections between the two communities flow through a few nodes (connections)
Web rendition of these graphs is also possible. Checkout this visualization of Perl authors (you can drag the graph with left click and dragging your mouse; you can zoom-in/out on the graph with your mouse wheel)
Lessons learned so far: - I quickly realized that you need a good machine for using Gephi effectively (a good video card, enough memory and fast CPU) - There is a great, active community behind Gephi, so expect frequent releases to resolve critical issues and new features. I'm waiting for this month's release that fixes some issues I've faced :-) If you know of other tools to visually explore graphs, please leave a comment.
Which network data I used? There's a wealth of network data available today, including social networking sites, phone logs, work history, chat logs, email logs etc. I decided to create test data for email traffic to test the hypothesis of "who we send emails to or receive emails from" is a good indicator of our social network. My test data has 50,000+ nodes and 150,000+ edges.
I used R to create the create data for graph. R is good at handling millions of rows of data and is powerful for data manipulation (cleaning, creating edge lists, adjacency matrix etc.). I used it to format raw email traffic test data into graph formats (edgelist, adjacency matrix etc.) It took me a couple of hours to write code for creating the data set to feed into Gephi. Using R to solve a real need has been a good learning experience so far.
Leave a comment, if you're interested in seeing the code.
After experimenting with MapReduce (Hadoop, Pig) last year, we recently ran some tests to check if its worth pursuing further for large data analytics.
Test details: - Environment: We ran these tests on Amazon's cloud (quick, cheap, no hassles :-) - Test data: 500 Million and 1 Billion rows of simple observations (2 column data - Customer_ID and Amount) - Computation: Simple (group_by and summation)
Here's what we found: 1) Hadoop scales well: Even when we doubled the data volume, processing time did not increase proportionately (notice the vertical distance between the curves) 2) MapReduce gains flatten outafter a certain point: Beyond a certain # nodes, there are no more savings in computation time (notice the flattening of the curves). Scaling up infinitely won't make drop computation time to seconds :-| 3) Pay more to save more time: Processing time is a factor of # nodes. We can easily decide how much we're willing to pay based on the time savings (RoI) example, a high priced expert might chose to pay more to save more time, compared to what an analyst chooses.
Now that Hadoop has passed the "sniff" test, we plan to run a real-life computation on it. I'm also looking for experts to drive this forward now. If there's anyone interested, please leave a comment.
We need to keep Amdahl's law in mind to estimate the max. savings expected from parallelization.
Note: I'm also interested in running R on Mapreduce sometime in the future
After experimenting with MapReduce (Hadoop, Pig) last year, we recently ran some tests to check if its worth pursuing further for large data analytics.
Test details: - Environment: We ran these tests on Amazon's cloud (quick, cheap, no hassles :-) - Test data: 500 Million and 1 Billion rows of simple observations (2 column data - Customer_ID and Amount) - Computation: Simple (group_by and summation)
Here's what we found: 1) Hadoop scales well: Even when we doubled the data volume, processing time did not increase proportionately (notice the vertical distance between the curves) 2) MapReduce gains flatten outafter a certain point: Beyond a certain # nodes, there are no more savings in computation time (notice the flattening of the curves). Scaling up infinitely won't make drop computation time to seconds :-| 3) Pay more to save more time: Processing time is a factor of # nodes. We can easily decide how much we're willing to pay based on the time savings (RoI) example, a high priced expert might chose to pay more to save more time, compared to what an analyst chooses.
Now that Hadoop has passed the "sniff" test, we plan to run a real-life computation on it. I'm also looking for experts to drive this forward now. If there's anyone interested, please leave a comment.
We need to keep Amdahl's law in mind to estimate the max. savings expected from parallelization.
Note: I'm also interested in running R on Mapreduce sometime in the future
I'm deeply saddened by the demise of Dr. C. K. Prahalad, an eminent thought leader and renowned management thinker. I've read many of his books and had the privilege to listen to him as well. He served as a distinguished professor at the Ross School of Business, University of Michigan.
I'm deeply saddened by the demise of Dr. C. K. Prahalad, an eminent thought leader and renowned management thinker. I've read many of his books and had the privilege to listen to him as well. He served as a distinguished professor at the Ross School of Business, University of Michigan.
Arthur Benjamin says, "Statistics is the future of Math". I believe it and I'm learning Statistics. Statistics is a foundational skill I need to build Prediction Markets (my goal for 2010). I'm learning R (a statistical programming language). We've formed a study group and are following Google/Stanford Stats 202 lectures.
Its only been a couple of weeks but I'm excited about R. It seems powerful, fast (<1M rows dataset) and a lot of fun! I've been spending time at work and home to learn the constructs of the language and plot charts.
I'll share code, links, plots and my experiences of learning R in the following posts.
Arthur Benjamin says, "Statistics is the future of Math". I believe it and I'm learning Statistics. Statistics is a foundational skill I need to build Prediction Markets (my goal for 2010). I'm learning R (a statistical programming language). We've formed a study group and are following Google/Stanford Stats 202 lectures.
Its only been a couple of weeks but I'm excited about R. It seems powerful, fast (<1M rows dataset) and a lot of fun! I've been spending time at work and home to learn the constructs of the language and plot charts.
I'll share code, links, plots and my experiences of learning R in the following posts.
As I reflect on the book, I feel inspired. There are many tips that we practice as an organization but many that we need to adopt
My favorite part is how the authors conclude... "Inspiration is perishable. If you want to do something, you've got to do it now... Inspiration is a magical thing, a productivity multiplier, a motivator. But it won't wait for you"
It was a quick read... took me ~ 3 hours in all (between all the interruptions during the day :) It was easy to read and hard to put down once I started (read a couple of chapters while walking the streets of midtown Manhattan... I don't recommended it, unless you want New Yorkers to give you a stare when you bump into them ;)
Here's what I was doing while reading this book...
I was laughing... yes, laughing... not something you see while reading a book on such a topic but it made me laugh... Jason and DHH have used a casual yet striking style to send across their message
I was reading it in 15 minute bursts. Great idea to break the book into many small chapters... the authors took their own advise and broke down a 200+ page book into many 1 - 2 page chapters. Finishing each chapter gave me a sense of accomplishment :-) and was also a great way to leave it at a meaningful point to pick it up again... Small chapters with large actionable titles also made it easy to refer back...
I started highlighting some sections in the book and kept a highlighter with me throughout. I ended up marking more than I've ever done in other business books (Yes, all my science books look like my daughter's color book). I marked sections as must-read for my friends, my family and myself
I was using it in my conversation (already!). The book is filled with great, actionable tips and "soul-searching" questions. Do you expect anything short of that from the guys at 37Signals? In fact, the insights are so actionable that I requested a couple of team members to read a chapter or two while discussing improvement opportunities
I stopped and reflected. "Are you working on your best idea?" and other soul searching questions like this made me search deeper... Inspired me to keep looking...
I felt good.... there are some tips that I breezed through smiling as we practice it already in our team (planning is guessing, ignore the details early on, start at the epicenter, meetings are toxic, less mass, embrace constraints, throw less at the problem, launch now, interruption is the enemy of productivity, good enough is fine, quick wins, hire when it hurts, test drive employees, own your bad news etc.)
Parts that will help me re-focus, re-energize, and improve overall...
Ignore the real world: "The real world isn't a place, its an excuse. It's a justification for not trying. It has nothing to do with you."
Learning from mistakes is overrated: "What do you really learn from mistakes? ... what not to do again, but how valuable is that? You still don't know what you should do next." "Success is the experience that actually counts"
Make a dent in the universe: "What you do is your legacy."
No time is no excuse: "...the perfect time never arrives"
"Let your latest grand ideas cool off for a while"
Build an audience: "share information that's valuable and you'll slowly but surely build a loyal audience"
Everything is marketing: "... everyone in your company is doing 24/7/365.... phone... email..."
"Instead of explaining what something sounds like, hum it. Do everything you can to remove layers of abstraction."
Hire the better writer: "Clear writing is a sign of clear thinking." It reminded me how badly I need to improve my writing skills. As a start, I need to finish the book Jeanine gave me (On Writing Well by William Zinsser)
Here are some favorite parts...
Workaholism: "Workaholics aren't heroes. They don't save the day, they just use it up"
"When you don't know what you believe.... everything is debatable. But when you stand for something, decisions are obvious."
"Scaring away new customers is worse than losing old customers"
Out-teach your competition: What a great tip! I became a fan instantly (although I don't use any software products from 37Signals). It actually inspired me to pen down how I felt about the book (something I have'nt done before... so bear with me while I struggle thru my first book review). It reinforced the idea that we can build trust with a large online audience too (one-on-one interactions aren't the only way and I can't use it as an excuse).
Emulate drug dealers: "Make your product so good, so addictive, so "can't miss" that giving customers a small free taste makes them come back with cash in hand"
Do it yourself first: "Never hire anyone to do a job until you've tried to do it yourself first." We've practiced this successfully in past and second it completely. For example, before hiring our first Ruby engineer (Matt), Amit and I took time to understand it, try it out on a project for 3 - 4 months. It worked really well (no, not the code we produced but the process of identifying what exactly we needed).
How to say you're sorry
"Don't create a policy because one person did something wrong once. Policies are only meant for situations that come up over and over again." This reminds me of a public calendar incidence - someone shared (unintentionally of course) official meeting information on a public calendar and boom! - we shut off access to that calendar for everyone. Should we reconsider such decisions?
This book has passed my test for a great book. My test is pretty simple - Is this a book I would want my daughter to read when she grows up? This one has made it to that list (a list of mostly biographies of world leaders). Yes, the four letter word did not stop me from recommending it to her. Keep up the good work. I'm a fan.
As I reflect on the book, I feel inspired. There are many tips that we practice as an organization but many that we need to adopt
My favorite part is how the authors conclude... "Inspiration is perishable. If you want to do something, you've got to do it now... Inspiration is a magical thing, a productivity multiplier, a motivator. But it won't wait for you"
It was a quick read... took me ~ 3 hours in all (between all the interruptions during the day :) It was easy to read and hard to put down once I started (read a couple of chapters while walking the streets of midtown Manhattan... I don't recommended it, unless you want New Yorkers to give you a stare when you bump into them ;)
Here's what I was doing while reading this book...
I was laughing... yes, laughing... not something you see while reading a book on such a topic but it made me laugh... Jason and DHH have used a casual yet striking style to send across their message
I was reading it in 15 minute bursts. Great idea to break the book into many small chapters... the authors took their own advise and broke down a 200+ page book into many 1 - 2 page chapters. Finishing each chapter gave me a sense of accomplishment :-) and was also a great way to leave it at a meaningful point to pick it up again... Small chapters with large actionable titles also made it easy to refer back...
I started highlighting some sections in the book and kept a highlighter with me throughout. I ended up marking more than I've ever done in other business books (Yes, all my science books look like my daughter's color book). I marked sections as must-read for my friends, my family and myself
I was using it in my conversation (already!). The book is filled with great, actionable tips and "soul-searching" questions. Do you expect anything short of that from the guys at 37Signals? In fact, the insights are so actionable that I requested a couple of team members to read a chapter or two while discussing improvement opportunities
I stopped and reflected. "Are you working on your best idea?" and other soul searching questions like this made me search deeper... Inspired me to keep looking...
I felt good.... there are some tips that I breezed through smiling as we practice it already in our team (planning is guessing, ignore the details early on, start at the epicenter, meetings are toxic, less mass, embrace constraints, throw less at the problem, launch now, interruption is the enemy of productivity, good enough is fine, quick wins, hire when it hurts, test drive employees, own your bad news etc.)
Parts that will help me re-focus, re-energize, and improve overall...
Ignore the real world: "The real world isn't a place, its an excuse. It's a justification for not trying. It has nothing to do with you."
Learning from mistakes is overrated: "What do you really learn from mistakes? ... what not to do again, but how valuable is that? You still don't know what you should do next." "Success is the experience that actually counts"
Make a dent in the universe: "What you do is your legacy."
No time is no excuse: "...the perfect time never arrives"
"Let your latest grand ideas cool off for a while"
Build an audience: "share information that's valuable and you'll slowly but surely build a loyal audience"
Everything is marketing: "... everyone in your company is doing 24/7/365.... phone... email..."
"Instead of explaining what something sounds like, hum it. Do everything you can to remove layers of abstraction."
Hire the better writer: "Clear writing is a sign of clear thinking." It reminded me how badly I need to improve my writing skills. As a start, I need to finish the book Jeanine gave me (On Writing Well by William Zinsser)
Here are some favorite parts...
Workaholism: "Workaholics aren't heroes. They don't save the day, they just use it up"
"When you don't know what you believe.... everything is debatable. But when you stand for something, decisions are obvious."
"Scaring away new customers is worse than losing old customers"
Out-teach your competition: What a great tip! I became a fan instantly (although I don't use any software products from 37Signals). It actually inspired me to pen down how I felt about the book (something I have'nt done before... so bear with me while I struggle thru my first book review). It reinforced the idea that we can build trust with a large online audience too (one-on-one interactions aren't the only way and I can't use it as an excuse).
Emulate drug dealers: "Make your product so good, so addictive, so "can't miss" that giving customers a small free taste makes them come back with cash in hand"
Do it yourself first: "Never hire anyone to do a job until you've tried to do it yourself first." We've practiced this successfully in past and second it completely. For example, before hiring our first Ruby engineer (Matt), Amit and I took time to understand it, try it out on a project for 3 - 4 months. It worked really well (no, not the code we produced but the process of identifying what exactly we needed).
How to say you're sorry
"Don't create a policy because one person did something wrong once. Policies are only meant for situations that come up over and over again." This reminds me of a public calendar incidence - someone shared (unintentionally of course) official meeting information on a public calendar and boom! - we shut off access to that calendar for everyone. Should we reconsider such decisions?
This book has passed my test for a great book. My test is pretty simple - Is this a book I would want my daughter to read when she grows up? This one has made it to that list (a list of mostly biographies of world leaders). Yes, the four letter word did not stop me from recommending it to her. Keep up the good work. I'm a fan.
It's been 3 years since we introduced Ruby in our firm for application development. In the posts ahead I will share where we are, our experience and the key shifts in development approach.
It's been 3 years since we introduced Ruby in our firm for application development. In the posts ahead I will share where we are, our experience and the key shifts in development approach.
My daughter is hooked to Sid The Science Kid and Super Why!
She has got me hooked to these shows as well and I love Sid The Science Kid. Its the greatest kid show ever! What a way to learn and get kids interested in science. May be its the geek in me but who doesn't like kid scientists!
My daughter is hooked to Sid The Science Kid and Super Why!
She has got me hooked to these shows as well and I love Sid The Science Kid. Its the greatest kid show ever! What a way to learn and get kids interested in science. May be its the geek in me but who doesn't like kid scientists!
I'm a member of firm's IT Governance that's responsible for ensuring successful IT deployments across the firm. We've worked together for 12+ months now and its time to reflect on how we're working as a team i.e. time to inspect and adapt => Retrospective time!
I requested Jeanine to facilitate IT Governance retrospective.We're following the typical retrospective process I discussed earlier... Here it is in nutshell:
Collect data (pluses + opportunities) from the team - Jeanine is interviewing team members before retrospective
Synthesize findings - Jeanine will assimilate her findings and share with the group before retrospective
Prioritize opportunities - During retrospective
Analyze root cause - Scheduled for retrospective discussion, if time permits
Formulate action plan - Post retrospective
I'm looking fwd to the session and will you posted...
I'm a member of firm's IT Governance that's responsible for ensuring successful IT deployments across the firm. We've worked together for 12+ months now and its time to reflect on how we're working as a team i.e. time to inspect and adapt => Retrospective time!
I requested Jeanine to facilitate IT Governance retrospective.We're following the typical retrospective process I discussed earlier... Here it is in nutshell:
Collect data (pluses + opportunities) from the team - Jeanine is interviewing team members before retrospective
Synthesize findings - Jeanine will assimilate her findings and share with the group before retrospective
Prioritize opportunities - During retrospective
Analyze root cause - Scheduled for retrospective discussion, if time permits
Formulate action plan - Post retrospective
I'm looking fwd to the session and will you posted...
Intense cricket from both sides. Congratulations to the Indian team for coming back strongly after a loss in Nagpur. Hats off to Hashim Amla (South African batsman) for scoring almost 500 runs in 2 test games.
What a thriller... A 5 day game decided in the last 15 mins! Who says test cricket is dead?
Intense cricket from both sides. Congratulations to the Indian team for coming back strongly after a loss in Nagpur. Hats off to Hashim Amla (South African batsman) for scoring almost 500 runs in 2 test games.
What a thriller... A 5 day game decided in the last 15 mins! Who says test cricket is dead?
... in the 2nd test match at Eden Gardens, Kolkata, at 3am EST. India has to win this match to retain the top spot in test cricket. 3 more wickets... that's all we need in the next 25 overs or so... go India!!!
... in the 2nd test match at Eden Gardens, Kolkata, at 3am EST. India has to win this match to retain the top spot in test cricket. 3 more wickets... that's all we need in the next 25 overs or so... go India!!!
Our team provides software development services to our firm. Here's our mission:
Excellence in Software Development - Delight our users with simple, fast and valued solutions - Consistently exceed sponsor expectations - Inspire your colleagues - Understand best practice. Embrace next practices - Demonstrate software craftsmanship - Create the best place to work, rich with fun and passion
Our team provides software development services to our firm. Here's our mission:
Excellence in Software Development - Delight our users with simple, fast and valued solutions - Consistently exceed sponsor expectations - Inspire your colleagues - Understand best practice. Embrace next practices - Demonstrate software craftsmanship - Create the best place to work, rich with fun and passion
Many organizations ask us, why we chose Agile. This post is an attempt to compile my thoughts on the answer.
For us, it was the need of the hour... Agile came as a perfect solution for the problems we were facing in software development (discussed below). These problems prompted introspection and Agile had common sense approach to a lot of those issues...
Problems we were facing in software development
1) knowing what to build - fat documents, UML had all failed "we wished we could just get the dev teams to talk to the sponsors/end users"
2) knowing when we'll be done - upfront planning and estimations were bogus... always off... "we wished we could update our estimates during the project lifecycle"
3) knowing exactly how we'll do it (at the finest level of detail) - we spent countless hours in predicting our system design (Big Upfront Design) and always had surprises during development that forced us to change our design "we wished we could design as we go" Emergent design
4) development time was the shortest period in the project - sponsor discussions, requirements elaboration, visual design, detailed design, syndication etc. took long time and that put pressure on development team to rollout ASAP "we wished we started development from the first day IT was engaged with sponsors"
5) most of the work happened towards the end of the project - 80-20 rule implemented in the wrong way :) 80% of the work was done in 20% of the time (towards the end of the project) "we wished we could produce software in a rhythm throughout the project"
6) focus on process instead of product - we had countless reviews to ensure proper artifacts were created and signed-off, prescribed process (waterfall and Unified Process) was followed "we wished we could do just enough to start and do more as needed" ceremony vs. essence
7) change management - we used to finalize requirements before starting development and every small change started a long change management process that was wasteful and created negative energy between the development and sponsor teams "sponsors wished they had infinite flexibility to change their minds at any time during the project"
8) our product launches were delayed and costed more - no one trusted our detailed plan and always added 50 - 100% more time/$$ for delivery date/cost "we wished we could be more accurate in our planning" Predictability
9) our users/sponsors thought they knew exactly what they wanted - by the time they saw what was developed, they always changed their minds "we wished we could show the product early to them and co-design it with them"
10)
Agile gave us a "natural" way to build software that addressed these issues, a framework to build better products with simple common sense approach on process (e.g. helps us to know what to build, how to build, when will it be built).
Agile brought back the basics of good teamwork and collaboration. It enabled: - good communication. After all "good software needs good communication" - short feedback loops - which helps in reducing the risk of mistakes and misunderstandings - rhythm in development - predictability in estimations on cost/delivery - focus in building the most valued features first - flexibility to change our minds on both functional and technical design to eventually build a better product - focus on building the product (and not change management procedures, sign-off process, requirement documentation standards, etc.)
Is it time for you to adopt Agile?
Well,I think if you hear any of these words in your software development organization, its at least time for some introspection...
Change management UML Month long QA 25+ page Word documents for requirements or design First product demo towards the end of the project Detailed plan in Microsoft Project ...
Many organizations ask us, why we chose Agile. This post is an attempt to compile my thoughts on the answer.
For us, it was the need of the hour... Agile came as a perfect solution for the problems we were facing in software development (discussed below). These problems prompted introspection and Agile had common sense approach to a lot of those issues...
Problems we were facing in software development
1) knowing what to build - fat documents, UML had all failed "we wished we could just get the dev teams to talk to the sponsors/end users"
2) knowing when we'll be done - upfront planning and estimations were bogus... always off... "we wished we could update our estimates during the project lifecycle"
3) knowing exactly how we'll do it (at the finest level of detail) - we spent countless hours in predicting our system design (Big Upfront Design) and always had surprises during development that forced us to change our design "we wished we could design as we go" Emergent design
4) development time was the shortest period in the project - sponsor discussions, requirements elaboration, visual design, detailed design, syndication etc. took long time and that put pressure on development team to rollout ASAP "we wished we started development from the first day IT was engaged with sponsors"
5) most of the work happened towards the end of the project - 80-20 rule implemented in the wrong way :) 80% of the work was done in 20% of the time (towards the end of the project) "we wished we could produce software in a rhythm throughout the project"
6) focus on process instead of product - we had countless reviews to ensure proper artifacts were created and signed-off, prescribed process (waterfall and Unified Process) was followed "we wished we could do just enough to start and do more as needed" ceremony vs. essence
7) change management - we used to finalize requirements before starting development and every small change started a long change management process that was wasteful and created negative energy between the development and sponsor teams "sponsors wished they had infinite flexibility to change their minds at any time during the project"
8) our product launches were delayed and costed more - no one trusted our detailed plan and always added 50 - 100% more time/$$ for delivery date/cost "we wished we could be more accurate in our planning" Predictability
9) our users/sponsors thought they knew exactly what they wanted - by the time they saw what was developed, they always changed their minds "we wished we could show the product early to them and co-design it with them"
10)
Agile gave us a "natural" way to build software that addressed these issues, a framework to build better products with simple common sense approach on process (e.g. helps us to know what to build, how to build, when will it be built).
Agile brought back the basics of good teamwork and collaboration. It enabled: - good communication. After all "good software needs good communication" - short feedback loops - which helps in reducing the risk of mistakes and misunderstandings - rhythm in development - predictability in estimations on cost/delivery - focus in building the most valued features first - flexibility to change our minds on both functional and technical design to eventually build a better product - focus on building the product (and not change management procedures, sign-off process, requirement documentation standards, etc.)
Is it time for you to adopt Agile?
Well,I think if you hear any of these words in your software development organization, its at least time for some introspection...
Change management UML Month long QA 25+ page Word documents for requirements or design First product demo towards the end of the project Detailed plan in Microsoft Project ...
1. What is our purpose for existing? 2. Who is our target customer? 3. Why does anyone need what we're selling? 4. If there is a need, is it enough to support a profitable business? 5. What were our competitors up to? 6. Can you reduce expenses--without harming the product? 7. Do we have the right leadership? 8. Do we have the right employees? 9. How will we continue to drive revenue? 10. How are our employees holding up?
I wonder what are the equivalent questions for a service organization like an enterprise IT shop.
1. What is our purpose for existing? 2. Who is our target customer? 3. Why does anyone need what we're selling offering? 4. If there is a need, is it enough to support a profitable ??? business? 5. What were our competitors up to? 6. Can we reduce expenses--without harming the product? 7. Do we have the right leadership? 8. Do we have the right employees? 9. How will we continue to drive revenue ??? ? 10. How are our employees holding up?
1. What is our purpose for existing? 2. Who is our target customer? 3. Why does anyone need what we're selling? 4. If there is a need, is it enough to support a profitable business? 5. What were our competitors up to? 6. Can you reduce expenses--without harming the product? 7. Do we have the right leadership? 8. Do we have the right employees? 9. How will we continue to drive revenue? 10. How are our employees holding up?
I wonder what are the equivalent questions for a service organization like an enterprise IT shop.
1. What is our purpose for existing? 2. Who is our target customer? 3. Why does anyone need what we're selling offering? 4. If there is a need, is it enough to support a profitable ??? business? 5. What were our competitors up to? 6. Can we reduce expenses--without harming the product? 7. Do we have the right leadership? 8. Do we have the right employees? 9. How will we continue to drive revenue ??? ? 10. How are our employees holding up?
I'm a huge cricket fan. It makes me happy, keeps me going and makes me sad, mad as well.
Yesterday I witnessed an exciting match between the Aussies and the Indians. In that match, Sachin Tendulkar, the best batsman the world has produced, reached another major milestone - 17,000 runs in One Day International.
He carries the load of the expectations of 1 billion people of India and has been a master in every form of the game and in every sense of gamesmanship.
He has played international cricket for 20 years now. I can only wish that we get to watch him play for another 5 years! Here's his one day cricket record against the top international teams so far...
I'm a huge cricket fan. It makes me happy, keeps me going and makes me sad, mad as well.
Yesterday I witnessed an exciting match between the Aussies and the Indians. In that match, Sachin Tendulkar, the best batsman the world has produced, reached another major milestone - 17,000 runs in One Day International.
He carries the load of the expectations of 1 billion people of India and has been a master in every form of the game and in every sense of gamesmanship.
He has played international cricket for 20 years now. I can only wish that we get to watch him play for another 5 years! Here's his one day cricket record against the top international teams so far...
When people have options on how to accomplish a given task, they typically follow the path of least resistance. So if we want to influence the choice they make, we should make our preferred choice really easy.
For businesses, it translates to making our preferred option really cheap and quick. Traditional approaches (policies, checkpoints) cause a lot of friction and people find easier alternatives, for example:
individuals don't follow mail archive or working paper policy (instead copy mail/docs on DVDs)
business units don't use standard web content management platform (instead use other preferred solutions).
When people have options on how to accomplish a given task, they typically follow the path of least resistance. So if we want to influence the choice they make, we should make our preferred choice really easy.
For businesses, it translates to making our preferred option really cheap and quick. Traditional approaches (policies, checkpoints) cause a lot of friction and people find easier alternatives, for example:
individuals don't follow mail archive or working paper policy (instead copy mail/docs on DVDs)
business units don't use standard web content management platform (instead use other preferred solutions).
CSS Sprites is a well known technique for webpage optimization.
It typically takes about ~ 2 to 3 days of work by UI designers and web developers to implement it. Steve Souders (UI magician) has shrunk it to zero! Try it out -> http://spriteme.org/demo.php
It automates all "spriting" activities: - finding all images on a page - grouping them into sprites - you can even generate the sprite! - it computes the complex CSS positions as well - and yeah get ready for the magic, it also injects the sprite in your current page
Could we ask for more? We can't make any excuse to not use sprites now :-)
http://spriteme.org
We need more such ideas to decrease the barrier of doing the right thing. Lets make doing the right thing, really easy!
CSS Sprites is a well known technique for webpage optimization.
It typically takes about ~ 2 to 3 days of work by UI designers and web developers to implement it. Steve Souders (UI magician) has shrunk it to zero! Try it out -> http://spriteme.org/demo.php
It automates all "spriting" activities: - finding all images on a page - grouping them into sprites - you can even generate the sprite! - it computes the complex CSS positions as well - and yeah get ready for the magic, it also injects the sprite in your current page
Could we ask for more? We can't make any excuse to not use sprites now :-)
http://spriteme.org
We need more such ideas to decrease the barrier of doing the right thing. Lets make doing the right thing, really easy!
First of all, I have to acknowledge that conducting effective retrospectives is hard... Its not easy, so hang in there... don't give up. Lets just try to get better at it, one step at a time...
Lets own our retrospectives and make the best use of our time. If it didn't work well, lets make sure the next one works better. Lets raise the bar every time and demand better retrospectives (for that matter, any discussion).
Here are some things I learned and plan to apply in the next retrospective...
My key takeaway is that retrospectives are really problem solving sessions. We're better off structuring retrospectives for effective problem solving and leverage online/offline and asynchronous/synchronous means to achieve the desired result. For example, problem solving is ineffective in a distributed team setting, so decrease the barrier as much as possible using video conferencing etc.
1) Allow more time for problem solving We put a lot of pressure on ourselves when we expect to walk out of a 1-2 hour discussion with most important opportunities, their root causes and action plans identified.
Effective problem solving (root cause identification, action planning) can take longer (sometimes days, weeks and its okay!). So I plan to let the team mull over the opportunities longer via online, asynchronous means. This can be effective as more ideas, discussions can happen without any time pressure. E.g. online discussions, brainstorming, debates.
2) Use team time efficiently - Don't force the team to come together for activities that can be done as/more effectively without all of them present at the same time e.g. data gathering, data sharing, voting.
- Perform all activities (possible) before the retrospective to focus the team on problem solving. Leverage easy and effective online tools like wiki, microblogs/discussions, survey.
3) Don't follow the routineblindly Be thoughtful of what you do and when.
- Don't follow all steps suggested for retrospectives blindly. For example, once you've identified the opportunities for your team, you may not need to collect data in the next few weeks to identify the opportunities again. The team might be better off focusing on actions for those opportunities. When a new opportunity comes up, discuss it and if it has higher priority, add it to your list.
- Forget the rhythm of doing it every 1or 2 weeks just because you have to reflect in each sprint... Have a specific purpose in mind and conduct a retrospective only when needed... Make sure you know exactly why and when is the right time to do it. People get bored with ineffective discussions.
4) Go slow Don't take big bites. Too many teams try to improve too many things, too quickly, which often results in inaction. We're better off identifying the biggest opportunity and working on it, letting it become a part of our team's DNA before moving ahead to the next opportunity.
Bottom line: Effective retrospectives require everything that effective meetings need. Think about it. Don't follow the process blindly. Demand improvements.
First of all, I have to acknowledge that conducting effective retrospectives is hard... Its not easy, so hang in there... don't give up. Lets just try to get better at it, one step at a time...
Lets own our retrospectives and make the best use of our time. If it didn't work well, lets make sure the next one works better. Lets raise the bar every time and demand better retrospectives (for that matter, any discussion).
Here are some things I learned and plan to apply in the next retrospective...
My key takeaway is that retrospectives are really problem solving sessions. We're better off structuring retrospectives for effective problem solving and leverage online/offline and asynchronous/synchronous means to achieve the desired result. For example, problem solving is ineffective in a distributed team setting, so decrease the barrier as much as possible using video conferencing etc.
1) Allow more time for problem solving We put a lot of pressure on ourselves when we expect to walk out of a 1-2 hour discussion with most important opportunities, their root causes and action plans identified.
Effective problem solving (root cause identification, action planning) can take longer (sometimes days, weeks and its okay!). So I plan to let the team mull over the opportunities longer via online, asynchronous means. This can be effective as more ideas, discussions can happen without any time pressure. E.g. online discussions, brainstorming, debates.
2) Use team time efficiently - Don't force the team to come together for activities that can be done as/more effectively without all of them present at the same time e.g. data gathering, data sharing, voting.
- Perform all activities (possible) before the retrospective to focus the team on problem solving. Leverage easy and effective online tools like wiki, microblogs/discussions, survey.
3) Don't follow the routineblindly Be thoughtful of what you do and when.
- Don't follow all steps suggested for retrospectives blindly. For example, once you've identified the opportunities for your team, you may not need to collect data in the next few weeks to identify the opportunities again. The team might be better off focusing on actions for those opportunities. When a new opportunity comes up, discuss it and if it has higher priority, add it to your list.
- Forget the rhythm of doing it every 1or 2 weeks just because you have to reflect in each sprint... Have a specific purpose in mind and conduct a retrospective only when needed... Make sure you know exactly why and when is the right time to do it. People get bored with ineffective discussions.
4) Go slow Don't take big bites. Too many teams try to improve too many things, too quickly, which often results in inaction. We're better off identifying the biggest opportunity and working on it, letting it become a part of our team's DNA before moving ahead to the next opportunity.
Bottom line: Effective retrospectives require everything that effective meetings need. Think about it. Don't follow the process blindly. Demand improvements.
We didn't want to get hung up on improvement opportunities only and ignore what this team of timezone warriors had achieved. So we started with celebrations - looked back at what was accomplished, shared thank you notes and acknowledged that we were proud of what we produced.
4) Highlight strengths and opportunities from data gathered (10 mins)
We reflected on team's strengths and opportunities collected from the survey and discussed: - if there were any surprises - if there was anything missing to add - if there was anything incorrect to remove
Team agreed with the strengths and opportunities identified from the survey.
I also highlighted the areas where Business and IT had a disconnect (voted differently).
5) Ask the team to select top 3 opportunities to discuss today (30 mins) Team members voted (hidden*) to identify top 3 opportunities that would have the biggest impact on the future releases.
* to ensure team members were not influenced by each other
We wasted a lot of time here. Wish I had done more effectively. Perhaps before the discussion? Perhaps another survey for this completely distributed team?
6) Facilitate problem solving session to identify root causes for these opportunities (60 mins)
The team discussed various issues freely to try to identify the root causes. I wish we had more time for this.
7) Brainstorm action plan to avail the opportunities identified
We left this for the new team that was staffed on the project. I've emailed the survey responses and discussion deck to the team to carry the root cause analysis and action planning forward. Looking forward to that discussion.
At the end, I requested team's input to make future retrospectives better thru an anonymous 1-minute survey and indicate if in this session - - Received value < Time invested - Received value = Time invested - Received value > Time invested
They also had a place to leave general comments to improve our retrospectives.
-I got a copy of Agile Retrospectives - Making good teams great and skimmed thru it quickly (1 hour) to identify relevant techniques for this discussion (FYI, I read it earlier this year, so skimming was possible).
- I attended other retrospectives - one in person (1 hour) and another on the phone (1 hour)
- I discussed it with a colleague who's good at facilitating retrospectives
B) Get data (2 hours)
- I interviewed team members - spoke to business (sponsor, program manager) and IT (BA, tech lead, portfolio lead) teams to identify relevant topics, as well as understand why past retrospectives weren't perceived to be useful
- I learned that data gathering took a lot of time in previous retrospectives (teleconference with 20 people in 6-7 geographic locations), and was one of the reasons why the team could not engage in effective root cause identification and action planning for continuous improvement.
I wanted to eliminate this friction by making data gathering about the product and the process super easy. So I created a survey to identify team's strengths and opportunities. I added team barometer questions to get the pulse of the team and launched an anonymous 15-minute survey (30 questions* with satisfaction scale) to minimize team's time investment and to capture their actual feeling/emotions.
C) Analyze data (1 hour)
The survey response was fantastic (100% participation by team members). I extracted the data into a spreadsheet to find patterns (team's interpretation of their strengths/opportunities, disconnect between IT and Business' satisfaction).
D) Share data with the team before the retrospective (5 mins)
I emailed survey results to the team before the retrospective so that they could understand and reflect on it and share insights on the day.
I got ~ 2.5 days advance notice to facilitate the retrospective. Given other prior commitments, I had ~ 6 hours to prepare.
The retrospective was scheduled for 2 hours (teleconference) and I was told that the team (sponsors and development) didn't find retrospectives particularly useful i.e. Received benefit < Time invested
So my challenge was to make this retrospective useful or at least tilt the scale towards it being useful.
Scrum recommends periodic retrospectives for continuous improvement (inspect and adapt).
I recently facilitated a release retrospective for a large project team*. In the next few posts, I will share my experience and learnings.
* This was no doubt a large project... ~ 20 team members distributed across New York, San Francisco, Atlanta, Bangalore, Mumbai, Dusseldorf, Paris that have worked together for ~ 2 years with multiple 3-month releases.
I had to create a quick online survey last week for a retrospective. I used Zoomerang and loved the ease of use such online survey tools offer (I dislike typical enterprise offerings in this space. They are bulky, complicated, slow and self-service is next to impossible for business-users).
I had one issue using a free account though. Survey creation and launch only took 10 minutes, whereas data export took 45 mins! (export wasn't available for a free account, so I had to manually export data into a spreadsheet).
In the future, I plan to use Google forms instead.
Often our peers and leaders ask us "so what?"... by this they're implying that they don't clearly understand why we're saying what we're saying... Here's a good way to structure our verbal and written communication...
One of the few positive outcomes of economic recession is that business executives in IT are getting interested in Open Source Software (OSS). The economic pressure has put it on their radar.
Us developers have always loved it for the innovation, "socialism" and benevolence that open source is all about. Now with the budget crunch, there's growing interest in sponsors even with feature compromises.
Here's a quote from a recent IT governance discussion:
"An open source/package scan should be conducted to determine if there is a suitable replacement that will solve the above problems"
Currently most of our teams have a dedicated person to play the role of Scrum Master for all 3 types of projects we do. Is that the right setup? Here's how I see it...
Type 1
Small teams working on simple project demands e.g. Client Connect, SugarCRM >> My take: A project team member (product owner or someone from the dev team) can play the Scrum Master role. They can rotate it among them as well. More thoughts below...
Type 2
Teams working on projects with complex sponsor, vendor, technology, coordination risks and demands e.g. Digital Strategy, MyTime >> My take: We need a dedicated person to play the role of a SM for large projects
Type 3
Everything between 1 and 2 >> My take: Same as type 1
We should consider the following facts during staffing project types 1 and 3: - everyone in a team may not be able to play the role of a Scrum Master (only a few can in our 150 person team) - heads-up (forest) vs. heads-down (trees) perspective on projects need to be ensured - balance between "what" (product owner focus) and "how" (Architect’s focus ) something is built - Scrum discipline: teams might compromise e.g. its “okay to live with these impediments” - Anything else to consider?
Most front-end developers suffer a lot of pain due to hacks needed for IE 6 in their stylesheet. This article sums up nicely how IE6 peculiarities can be dealt with in stylesheets...
I particularly like the suggestion to keep IE6 hacks separate from the main stylesheet to keep main stylesheet clean...
So my focus on pixel perfect in the last several weeks has led me to a bigger, "nobler" goal of improving the overall web experience.
Our firm members have a different web experience in the intranet (slow and unintuitive) compared to the internet (fast and simple). The consumer internet has set a new bar for effective web solutions.
We have the opportunity to exceed our user’s expectations by delivering fast and simple intranet solutions.
We can have the highest impact by focusing on speed (sub-second web interactions) and consistency (firm branding and usability).
We've staffed a passionate team* to figure out how to take our development team to this state.
* Tim M (PM), James (front-end engineer and BA) and Thomas N (architect)
as we get closer to the release, lets make sure that we've dotted all the i's and crossed all the t's...
here's my mental checklist (anything else to consider?) for the site... can we pls discuss how we're addressing these areas soon? pls advise. - prasoon
code quality
test coverage
valid markup
security testing
cross-browser testing
performance under load
low bandwidth optimization
character encoding (proper character entry/display/treatment)
error msgs
consistency in site visual and behavioral conventions
I) Goal: Improve front-end quality of our applications
II) Measure: The number of applications that conform to Firm’s new brand guidelines
III) Areas to tackle
Learn ? Technical skills: HTML/CSS ? Web design skills: Best practices for web application design (e.g. when to use a list) ? Firm brand guidelines
Develop ? Use templates ? Export-to-web the design comp (xhtmlized.com, Allan)
Test ? Automated check for inclusion of common CSS* in application (during continuous integration build-test-deploy cycles) ? Manual check with design comps (or suggested design changes) with Pixel Perfect
IV) Execution strategy Work with project teams to achieve the goal
* Common CSS refers is the stylesheet (part of template) to start building applications with to ensure compliance to Firm brand guideline
Which developer tools do you use to develop high quality front-end?
Here are some of my favorites:
Web Developer (Update: June 3, 2009 >>> removed MeasureIt and Resize Window from the list. Web Developer has a ruler (under Miscellaneous tab) and window resize capability too)
ur new brand guidelines are available now. And the best part isn't just the slick/open look-and-feel, but also that it applies to both internal and external digital properties.
Creating one brand guideline for all (internal and external) digital properties is a huge step for us. Other firms that do this already might consider it common sense but in our firm, where a thousand flowers are allowed/encouraged to bloom, such governance is challenging.
I'm excited about it as it'll help our: 1) end users experience these applications in a consistent way 2) development teams us with: - reuse of templates, icons, styles, techniques, experts, tools - familiarity of applying one brand to app apps they develop
1) Set a standard for front-end development - create a template that development teams can use as a starting point - it will include page layout and CSS
Next steps: James is creating the template
2) Validate compliance to the standard If all our apps include the standard CSS then we can ensure that: - every HTML element has a style specified in the markup (reset.css) - the style specified in the markup is present in the CSS (default HTML elements + common styles. Overrides and app-specific styles will be in app's stylesheet) - the CSS class is as per the standard (e.g. errors should be size: 8, color: red, font: arial) Next steps: We're investigating ways to test for the standard CSS include in our continuous integration cycle
1) Raise the bar of front-end development skills in our engineering pool - Set expectations for engineers: All engineers are expected to have a certain level of proficiency in front-end development - Conduct hands-on training: Allan, who has front-end development experience, will conduct the class.
2) Develop front-end engineers - Develop 2 - 4 front-end specialists who can champion front-end development standards, testing, coaching
Lets discuss why aren't we pixel perfect? I see two main reasons for this...
1) We lack front-end development skills - Our engineers understand and practice Java/Ruby/SQL quite well but most don't know (and ever want to touch) the HTML/CSS code
2) We're not disciplined in creating user interfaces - Our struggle in being pixel perfect includes page (HTML element) styling... We don't follow a standard to ensure consistent apps. It results in inconsistent rendition of pages in our apps - style (font, color, size) and behavior (link, forms, errors).
Yes, we cannot be lazy about the user interface. Ignoring it affects overall quality. Our applications need to be polished! We must be proud of what we've produced.
Lets discuss "How should we make it pixel perfect?" and "Who should make it pixel perfect?" in this post.
Here's the flow I think of...
The designer on the project creates the UI spec.
The UI developer takes the UI spec and creates HTML rendition of the UI spec.
- The HTML produced is the result of negotiations and agreements between the designer and the UI developer. Everything may not be possible and sometimes UI developer will have to modify the designs to be able to actually make it work in a web page. But the designer should always be aware of the changes the UI developer plans to make, and they should discuss to see what are the possibilities.
- He/she can use free tools like Firebug (Firefox plugins) to edit/debug CSS/HTML live in the web page and compare their work to the UI spec using Pixel Perfect compare their to validate it
The development team then takes the HTML code and builds the application using it. They don't have to worry about being pixel perfect as long as they don't change any "style" elements/code
So, it is the UI developer's responsibility to make the page components pixel perfect. With this process being pixel perfect can become much simpler.
Lets discuss question #1 in this post... What should we make pixel perfect?
What is the unit of measure? How should we measure progress?
We should use page components as the unit of measure (not the entire page). The example of such page components include page header, footer, navigation etc.
A fully developed page with "real" content will often look different from the UI spec, which has mock-up content.
If a page component is reused across pages and isn't pixel perfect, then its better to track progress with "one component is not pixel perfect yet" rather than saying "two pages aren't pixel perfect yet" as work left is only on making one component pixel perfect
So lets start making page components pixel perfect. We can test page components to measure progress (# page components that are pixel perfect) and validate sites for pixel perfection [addressing "What should we test to be pixel perfect?"]
As developers we code for functionality/business rules/performance, and measure/test them to claim being "done" on a story.
We haven't yet put "conform to UI spec" in our done list (because it is hard to measure/test), which causes many issues like -
not being able to get a true sense of completeness/progress
designers complain that most front-end developers don’t seem to respect the designs when they build the templates. They just align stuff approximately, don’t notice all the details
This causes a big quality issue in our software that we (developers) don't notice/acknowledge.
I challenge us to add "conform to UI spec" in our done list to take our application quality to the next level.
I will post my thoughts and findings on how to achieve this in future posts...
I'm not against the use of Javascript in web apps. However, we need to reinforce the mantra of "not starting with Javascript/AJAX-based designs" while designing a new web app.
It can help ensure graceful degradation for which basic HTML forms are needed anyway and starting with them ensures simple, straight forward app is built first to highlight any issues etc.
It also keeps things relatively simple...
Does the following order make sense to write a new web app?
HTML (content) -> CSS (style) -> Javascript (behavior/interactivity)
In the last post on enabling cross-team learning we discussed the reasons for limited cross-team learning. In this post, lets discuss ideas to solve for (3) no one is aware of all discussions* happening in our group.
We have an opportunity to significantly reduce the barrier for cross-team learning by solving for (3) in the short-term, by compiling the schedule of all team discussions in one central place.
We could do this simply by using a team/group calendar (we use Notes team rooms). All the teams would need to do is cc: this group calendar in their discussion invitations. And the end result will be a compilation of all schedules in the group calendar, which all of us can see to chose the relevant discussions to join...
There is limited cross-team learning in our teams today. In a recent townhall, we discussed the reasons for this and the common themes were -
(1) Time: busy with our own projects (2) Interest: limited value perceived in cross-team learning (3) Transparency: no one is aware of all discussions* happening in our group
* CoE/initiatives/interest groups, townhalls, sprint/release planning/reviews/retrospectives etc.
We can solve this problem by working on the themes identified above in the reverse order 3 -> 2 -> 1, i.e. (3) help everyone get visibility into all team discussions, and then, (2) showcase some learning/value from cross-team learning and (1) team members will create time for cross-team learning.
How do you communicate project risks to your stakeholders? Is it effective? Does it help stakeholders - - focus on the right risks? - understand what you need from them or how can they help?
I've mostly used and seen a high/medium/low categorization and recently saw a simple and effective model (yes, I'm a sucker for models) that helped me zoom into the important issues right away (Thanks Erika and Daniel!)
How do you communicate risks? Does this model help? Can you improve it?
Professional development 1.0 Professional development is a focus for many organizations and over the years, we've witnessed this responsibility move from the hands of human resources to our managers. This change made it much more relevant and actionable because of frequent (from every year to every month) and contextual (from generic to patterns) interactions.
Professional development 2.0 The next leap forward in professional development could be by opening it up, by making it a social process (peer-led professional development). Our colleagues who we work with day and night (if you party together) can give us most relevant and actionable coaching and feedback. This is different from seeking their input once/twice a year as in Professional development 1.0. In 2.0, its a continuous process to make interactions more frequent (from monthly to daily) and contextual (from patterns to concrete).
Why restrict professional development coaching and feedback to a single person (manager)? Why not tap into the entire pool of "coaches" we have in our teams? This is a big shift for a group and will need experimentation to test and fine tune. I plan to conduct a mini-experiment internally (democratize coaching and feedback for myself) and will keep you posted...
Some considerations Some might argue that such a process will become a finger-pointing, criticism match. And I agree. The prerequisite for social professional development is the right culture, an environment where people care about each other, want to see each other succeed. A group will need to invest in creating such a culture before trying this. The group will to be trained as well, similar to the manager training in organizations practicing professional development 1.0.
If what you work on isn't your dream job, then read on...
Yes, most of us aren't in our dream jobs yet, whether it is running a school in India, opening a restaurant in backwaters of Kerala, becoming a professional soccer player/photographer/dog groomer/painter).
My dream job is to run a school in India and it'll take me few more years to get there. The question I ask myself everyday is what should I do till then? How can I make the most of the next 5 - 10 years "working" towards my dream job?
Here's how I look at it... 1. Find work I want to do Find my organization's priorities and - get involved in areas that I'm passionate about (maximize my time here) - develop in areas that I'm not as good at or as interested in (its work after all!)
2. Make work I want to do Make my interests public and "create" opportunities at work by influencing/exploring unchartered territories - "create my own workplace here"
3. Ignore everything else Why should I spend any minute on something that's not important to me and the organization
I want to run a school in India one day. What excites me is the opportunity to help shape the future of bright kids.
In my school, there will be a good mix of formal education and sports (more on using sports as a tool to teach key behaviors like team work, leading, coaching, helping others, in my future posts).
This morning, Thomas and I were discussing how somethings can take longer than usual in our organization, like design and security reviews.
We could continue as-is but what's the fun in that...
So, I challenged him to finish his project's security review in one session. Doing this in only one session will earn him EE (exceeds expectations) on this effort.
What should Thomas think about? What's your advice for him?
This can revolutionalize our personal (shopping, socializing) and professional (network, problem solving) life by meshing the information that was accessible only through a PC/handheld so far, in our daily interactions.
While writing someone's year-end evaluation in 2008, I highlighted how their style was exemplary and resulted in delighted stakeholders, including sponsors, development partner and other IT teams (Security, Infrastructure, Operations).
We have talented individuals with many hard skills but we often ignore the softer aspect of service delivery, understanding the wins for sponsors and other stakeholders.
Do you have "soft hands" to deal with all the undertakings in your project?
Litmus test to check if you have "soft hands" - Do you have a win-win mindset and practice it in your interactions/negotiations? - Do you ask yourself "was I helpful?" after every interaction - Are you okay with agreeing to disagree? - Do you have a calming effect on volatile discussions? - Do your colleagues reach out to you for help/advice in dealing with tough situations? - Do you lead tough negotiations or take the back seat? -
Litmus test to check if you don't have "soft hands" - Do you often come out of security discussions deflated? - Do you shut down when its a business or product adoption problem? - Do you find stakeholders reaching out to you through someone else in your team/portfolio? -
Tips for success: - Listen, be in the moment - 50% of what we do has nothing to do with technology - Learn from your colleagues who do it well - Be passionate but never raise your voice - What else has worked for you?
Its that time of the year that most of us don't like... No, not the coldest month... I'm talking about finalizing our yearly goals (PDP... professional development plan).
Although I cannot offer a perfect solution, I'm experimenting with a framework that some have found useful to determine their goals for the year. Let me explain...
Personal: What do you plan to learn, practice, teach, innovate this year? How will you position yourself in 2009 to inch closer to your target profile (separate post coming soon)?
Community: Center of Excellence (CoE), Portfolios, Interest groups, Application Development group, IT
External: Outside Firm (IT forums, IT industry, consumer internet, open source, etc.). How will you engage externally to improve and contribute back?
The framework is meant to present a holistic view of your impact. An honest self-examination can shed light on patterns like "oops, I didn't give back anything to the community. My contributions are limited to projects only" or "whoa, I don't have any impact on my project. I'm a chicken".
With this framework you can paint both, the as-is and the target picture (impact planned for the year). It can help set your goals and direct your efforts.
Disclaimers - You don't need to have a plan for every cell in the framework. Don't stretch yourself too thin - The framework is merely to assist in goal setting. It is not a tool for planning and time frames - -
1) Bootstrapping testing of legacy applications - All legacy Java applications will be enabled for one-click build, test and deployment - Development teams working on these applications will increase app test coverage
2) Simplify People Systems applications - Significantly decrease support costs and cost of change
3) Learning - We will publish a training schedule with topics that our internal experts will discuss. Anyone within IT is welcome to attend - This in turn gives us the opportunity to learn, teach, give back to the community and is a creative way to train ourselves given the economic situation
Neal shared his SWOT analysis of Architecture CoE with us last month.
Neal saw no major holes in architecture skill set. He rated architects as Proficient and Expert (top 2 using the Dreyfus scale).
STRENGTHS - Introspection: Willingness to get external input. Constantly looking at what we are doing and trying to get better. Focus on application architecture (presence of CoE) - Participation: We have a very engaged Arch CoE population. Most initiatives (in other companies) start out strong but die quickly. Ours has high participation and continues to do so (rotating responsibilities and ownership helps with sustained participation and enthusiasm) - Will To Change: Not stuck in ways of the past, looking to learn and grow - Awareness: of legacy code issues - Design choices: Buy, build, open source
WEAKNESSES - Lots of sub par legacy products / solutions in the environment. Affecting our support costs and slowing the evolution (end result is high support costs and high cost of change) - Very high inertia in applications. We have the willingness to change but does not see it happening as quickly as it could - Large monolithic applications, should move towards smaller composable solutions - Lack of overall support cost transparency: e.g. infrastructure costs might decrease with a certain design choice but associated support costs might be higher - Less focus on (1) enterprise architecture (we have application architects (~ building planners) and solution architects (how apps talk to each otehr) but not many enterprise level architects (~ city planners))
OPPORTUNITIES - Chance to re-architect some applications at enterprise level - Using budget constraint as a mechanism to drive creativity/innovation and improve efficient of development - Upgrade contractor staff: Talented engineering pool can be tapped into
THREATS - Economy / Budget Constraints - Predatory vendors - watch out for large vendor products that promise to do everything
SUGGESTIONS FOR 2009 GOALS - Shore up weaknesses - Build consistencies across projects (similar to rails). For example identify a consistent directory structure / package layout / etc for Java projects. - Take advantage of automation using these consistencies - Prioritize any items that give us a strategic advantage
Software development and integration methods continue to evolve...
Last week Matt discussed how he's thinking about Digital Strategy integration and we discussed ROA (Resource Oriented Architecture) and WOA (Web Oriented Architecture). They are specialized types of SOA (Service Oriented Architecture). Neal called them " SOA that actually works".
If you haven't heard about it or used it yet, go understand it, learn it, use it. Gerardo, Matt and other engineers can help...
- We grew to become a high performing team. Everyone has improved and, in their own ways, led the CoE... from Jaspreet creating design patterns to Ketan introducing peer architect concept.
- We developed hands-on skills. No more chickens... we're proud to be pigs on projects.
- We leveraged each other. Never hesitated in asking for help, always ready to help and have unquestionable trust in each other.
- We learned new skills and domains. Ruby, PHP, Finance, Learn.
- We improved sponsor confidence. Senior IT leadership and Firm functions recognize and seek CoE member's advice.
- We increased developer reach. ODC isn't 7000 miles aways anymore. Our engineers are talking to us now.
- We enabled Scrum at our organization. We really did. How else can design decisions be finalized in short sprints of 40 concurrent projects?
- We had fun doing it. Meeting daily never became a drag. Central park outing, Amazing race skits, birthday celebrations are all part of it.
I'm crazy for cricket. When Indian cricket is on, I cannot do anything else. I have to watch the game. And tonight is one of those nights. India and England will play a test match in Chennai (if the rain gods don't go there). Up late tonight to watch it. I predict that India will win by more than 4 wickets or more than 150 runs. Go Sachin! Go Zaheer!
Yes, cricket is my religion. I grew up playing the game and can play/watch all forms of cricket - test, one-day, 20-20 and even street cricket! Some of you have seen a cricket bat in my office. Always up for a quick knock in the lobby...
I'm a decent medium pace bowler and a dependable middle order batsman. My love for pace shaped how I bowl and bat... I don't get to play much cricket in the US but I'll get to play again daily when I go back to India (yes, some day).
...this is what Neal Ford said in a discussion yesterday and it stuck with me. It sounds paradoxical but is totally true for software. We do not observe this anywhere else in the physical world.
That's why we need good engineers to design and write simple software.
Neal Ford puts it much more eloquently and I've requested him to blog about it... I'll post a link when it comes out...
Here are some of my thoughts on Architecture CoE's performance this year. I'll discuss what went well and what didn't go so well in separate posts.
Things that didn't go well
- We were not fearless. E.g. we failed to take bold steps to achieve great things e.g. we tolerated the pain from our heavyweight survey platform and failed to reject it
- We were not thorough. We released 3-5 applications that needed hotfixes because we broke existing functionality. Yes, its a small % (total 120+ releases) but nothing short of perfection is expected from us in such maintenance releases. We cannot shortchange regression testing (case for automated testing but manual if/while tests aren't fully automated)
- We were not focused. We ended up spreading our attention on many things (Presentation Service, Common Service, Reporting Service, Real-life testing, Improve our design, Automated build/test/deploy/CI of Java apps, Ruby/LAMP). We could not make much progress in half of these areas
I've requested external perspective on the CoE from our external coach - Neal Ford. He will share his SWOT (Strengths, Weaknesses, Opportunities, Threats) analysis with us in January 2009.
Often we're asked to consult/advise on technology solutions built within the Firm but not at IT. This isn't rare because the Firm encourages thousand flowers to bloom and also IT cannot meet all software demands in the Firm.
These situations are challenging as the overall cost of building and supporting a solution isn't considered in the effort to build a cheap and effective solution. The existing IT ecosystem does matter. If IT doesn't support a certain product/platform, then using that product/platform has downstream support implications (cost $$$) that are often ignored when pitching a product/platform for a cheap and effective solution.
We need to consider total costs while making these technology decisions, for IT needs to create transparency on support/maintenance costs.
In our organization, we've decreased software launch duration from 9+ months to 100-days and have now done it consistently for several months now. Its done. We're now programmed to do this now!
Isn't it time to push it down further? Come on! Lets break the 100-day barrier. I'm thinking of 30-days launch and want to try it out if the following are true:
- Its a new web app (existing apps to be considered later) - Visual design of app (wireframes, etc.) is ready - App has well defined and limited integration points - Product Owner is available throughout development - Well defined testing approach for end users - Anything else to consider?
If you haven't seen Wanted you should (if you like action flicks). It's awesome. Sujit, Praful and I watched it this weekend and loved it. Its pace didn't give you a second to think or relax...
My favorite parts are the scenes where they're curving the bullets and where Wesley says "What the f*** have you done lately?" Inspiring ha?
Let me draw a parallel from project timelines to explain.
Till 2007 we used to have many projects that took 9 - 12 months to release working software. Not anymore. Now, we release working software in production/staging every 100-days. No, we haven't magically shrunk a Document Management or ERP system implementation to 100-days from several months. All it means is that we've broken down the 9 or 12 month project into 3 or 4 100-day releases now. This has simplified a lot of things and resulted in successful delivery of large projects.
Can't we apply the same concept to a "complex application" we need to build and break it up into multiple simple applications/components. Many engineers in our team practice it really well and I'm learning this skill from them. Let me know if you want to join the class!
Traditionally web application development has required web developer(s) and database developer(s).
Those web developers are dying or already dead now.
They're being replaced by a new breed of "web application developers" who can do much more than their predecessors. This new specie can develop a web application from A to Z... from top to bottom. They possess not only the typical web development skills but also enough database skills to crank out a web application of simple to medium complexity all by themselves.
This trend is here to stay... When I look back, till about 2 - 3 years ago all development teams needed a database developer; now some teams build an application without a database specialist.
There are many reasons for this shift including maturity in technologists (polyskilled) and technology (abstraction of database in frameworks like Ruby on Rails) etc.
Mark wrote about variety in work and I subscribe to that philosophy as well. I believe we all undergo a constant cycle of Learn -> Practice -> Teach in our lives and keep creating new cycles periodically...
In fact, I'm encouraging people I work with to be explicit about their Learn -> Practice -> Teach cycles and share it (their 2009 "development plan") broadly. Yes, why not get everyone's help in having a successful year, why not declare - what we're working on, what we're getting better at and what we've experienced already.
Thank you for your comments Rajesh. The styles of directing, supporting and coaching are well known and I believe in different strokes for different folks too (see Rajesh's post)...
An interesting question that comes to mind is -At what frequency are these styles practiced and why?
I've noticed the frequency as: directing > supporting > coaching
Some leaders might reason that it is because of the level and distribution of talent and maturity in their team. It is true in various cases, but cannot be applied universally (more and more we're surrounded by talented professionals). Could the following be some reasons for it? Thoughts?
(a) the level of difficulty of these styles directing < supporting < coaching
(b) leader's comfort factor Most leaders started and grew in the organization as individual contributors and default to it (or achieving the goal itself) when faced with another question or goal
(c) how leaders are measured (what expectations are set for them) You get what you measure... If you measure leader's output as tasks done, that's what they'll focus on and perhaps use directing more often to achieve those goals. On the other hand, if your measure is how they develop talent, they might flex their coaching muscle...
I'm trying to learn and practice coaching skills as well given the good talent pool in our organization. Please give me feedback on what has (has not) worked and feel free to suggest techniques that have worked well for you...
Different leaders approach getting their teams to a solution differently. Here are some patterns I've observed and their effectiveness...
1) "Let me show you the way"- Leader's focus is to get a task done - "I know, follow me" style
2) "We don't know, lets figure it out" - Leader's focus is to let the team find a solution. Leader shepherds the team and is a coach, a facilitator in this case
3) "Let me help the team figure it out themselves" - Leader's focus is to grow the team - Leader knows a solution but doesn't give it away to the team. He/she uses techniques like questioning to help focus the team directionally - In this process, the team solution could be different from what the leader imagined but he/she is fine with it (a different answer/outcome isn't bad if it is directionally correct)
Each visit to the offshore development center highlights the vast communication gap that exists between onshore and offshore. These teams don't fully understand each other's goals, issues etc. and don't have shared views on organizational priorities.
On a recent visit, I noticed two issues with this communication gap:
1) disconnect: lack of knowledge and/or understanding of each other's priorities. E.g. there are no common posters at onshore and offshore; technical priorities weren't known to the technologists offshore; onshore members doesn't realize/understand/appreciate the state of offshore - many advancements it has made in several areas...
2) lag: shifting priorities, goals were not well communicated and understood resulting in the offshore teams focusing on "old" priorities.
Geographical and time separation, project focus (narrow), not a priority for anyone are typical reasons why this has continued for several years.
This gap is counter-productive and slows down the speed at which the we can improve as a development organization.
To leap from the current state (ODC is at an inflection point), we've identified sustaining communication with offshore teams as a priority and want to make significant traction on it from now on. The guiding principles for the solution are:
1) establish something that we can sustain (not high activity for a few months that dies down later)
2) create a "need" to communicate (optional doesn't stick/sustain)
Here are some ideas we've discussed so far (more to come). We need to evaluate these against the guiding principles (Can we sustain it? Will people do it?) to select the right ones to focus on...
Let everyone know its a priority
Build trust and understanding - e.g. periodically conduct offshore week, where onshore teams we come in early so that offshore can leave on time (not at 9pm their time)
Make it someone's responsibility - Assign the responsibility of information radiators (or making sure its happening) as someone's "day job"
Distribute the work of information radiators - onshore members (technologists and some BA and PM) mentor/coach/connect with offshore members
Connect the champions at each location - Each initiative onshore and offshore should have champions/radiators in the other location
Our IT leader discussed high performing teams with us today . Here's what I gathered...
What is a high performing team? - A team whose output is much greater than what team members could achieve individually (whole is greater than the sum of its parts)
High performing teams are rare...
Their foundational qualities are: - Unquestionable trust - Respect and understanding of team member strengths (and weaknesses) - Unquestioned commitment to task
Once they have the foundational qualities, high performing teams usually exhibit following behaviors... - Cover for each other - Personal achievements are secondary to team achievements - Push project boundaries
Here are a few things we can do to help create high performing teams: - Set expectations for commitment to task - Send message that you trust team members - Give team members chance for new beginnings (eliminate/minimize preconceptions). Consider each day as a fresh start for yourself and others too... - ?
If you think about it, the foundational and behavioral requirements for high performing teams seem similar to Maslow's hierarchy of need for individuals. May be we can conceptualize high performing team requirements with "Maslow's hierarchy of need for teams".
After my recent post on vendor product usage scenarios, one of my colleagues asked me to elaborate what I meant by product customizations... Here's how I think about it. Please join this discussion to help define it...
Customization: You've customized a software product, when you've made a change in the product that the vendor doesn't provide a published and supported* mechanism for...
* The published and supported mechanism for change in a software's behavior are through configuration and API.
Tests to detect customizations: - Did you use published and supported mechanisms (configuration, API) in the product to change product behavior?
- Will your changes be compatible with new product versions? I.e. will your changes work seamlessly after you've upgraded the software product to a newer version? (This test is necessary but not sufficient, as any changes to published and supported configuration and/or API can also make this test true)
- Does the vendor support your changes through its "regular" support or does it charge you extra?
- ?
Open question: How does this differ between Open Source Software and proprietary vendor software?
Earlier this month I visited our offshore development center (ODC) in India. Many of you have asked me about the current state of ODC. This post captures some of my thoughts...
This year, I noticed distinct improvements compared to last year. Although there are improvements each year but this year was special! Changes were not just incremental but a large step forward...
Given the passion, energy and excitement in the group there (~170 ) I felt like being in the middle of a college classroom. A place where people are learning, growing and having fun while doing it. The energy onsite is not even half of what I experienced there.
Some words/phrases, I've used to describe how I felt are - - high energy - passionate - learning agility - college classroom feel - share successes - share failures - high performing team - desire to do more - desire to make an impact - change agility
Our ODC is at an inflection point...
We need to create an environment, build a platform to help the ODC leap forward to become a higher performing team. If we fail to do this we will loose an opportunity and the ODC will continue on its regular growth path.
Over the next few weeks I'll think about the necessary conditions to create this environment,, meanwhile here are some factors that motivates them... Lets do more of it... - direct interaction with onshore (sponsors/POs/CoEs) - hear their ideas - create transparency - exciting work - variety in work to learn new technologies/domains - timely feedback (praise, development areas) - play multiple roles (support, QA, development, BA, PM) - celebrate success (project accomplishments) - opportunity to become specialist (full coverage in an area like testing, quick starts) - ?
And here are some things that kill their motivation... Lets do less of it... - delays - insufficient resource commitments - no idea of roadmap, - old technologies - ?
Often a vendor software package we buy doesn't exactly meet our needs, so we want to alter it's behavior. This can be done in following ways with these advantages/disadvantages and preference...
OUT-OF-BOX FEATURES (functionality provided in vendor's base product as-is or with configuration changes)
Effort required: None Vendor support:Yes Maintenance costs:None Preference: High
Comments - Simple product upgrades - Applications bolted onto the vendor product pose lower migration costs than customizations to the vendor product itself BUILD CUSTOM FUNCTIONALITY USING VENDOR PRODUCT API (extend product functionality using vendor published and supported API)
Effort required:Low - Medium Vendor support: Yes (API) Maintenance costs: Medium Preference:Medium Comments - Simple product upgrades (upgrades don't impact enhancements as vendor APIs don't change, hence custom code written using vendor API works with product upgrade. So no need to change our code as functionality won't break) - Customizations outside of the system source code pose limited migration costs because minimal development is required with every upgrade CUSTOMIZE VENDOR PRODUCT (outside API) i.e. change in vendor's base product
Effort required:High Vendor support: No Maintenance costs: High
Preference: None, as vendor product customizations lead to - (1) against our core design principles (2) long-term issues with support and maintenance (3) incompatibility with vendor upgrades (4) potential delays in releasing the product due to unforeseen issues with custom code and standard code integration
Comments - Risky/time consuming product upgrades because customizations can break - Requires significant analysis and testing and potentially recoding to upgrade - Support issues due to non-standard nature of the product - Customizations don't work properly with upgrades -Once product code is changed (even slightly) the product no longer can be upgraded without losing those customizations. So we might face a no-win choice: If we want to upgrade, we have to forgo all the customizations that made the product fit our specific needs. Thus customizations leave our systems badly compromised
Often our sponsors ask us to bite the bullet and customize a vendor product to meet their specific requirement(s) rather than changing their business process around the vendor product.
Here's a template we've used for our discussions with sponsors if they insist on product customizations...
Does the business benefit of *** doing blah *** offset the cost and impact of customizing *** this vendor product ***?
E.g. Does the business benefit of pushing staffing data in Time System offset the cost and impact of customizing the vendor time management product?
Why vendor product customizations are bad?
UPFRONT AND MAINTENANCE COSTS
upfront costs/effort/time
development (customization)
testing
deployment
maintenance costs/effort/time with product upgrade (typically every 12 - 18 months)
analysis (impact of upgrade on customizations)
development (redeploy customization)
testing
deployment
OPPORTUNITY COST
Cannot use new features or new product releases as upgrading to new product version becomes harder (i.e., more time consuming, more risky) with customizations
SUPPORT ISSUES
If vendor agrees to support customizations, the vendor would need to ensure:
redundancy in resources knowledgeable of the customizations
the customizations are thoroughly documented
knowledge transfer of customizations as vendor resources knowledgeable of the customizations can move out of your IT account at the vendor
The above items pose a support risk because our IT department cannot be certain that these things happen effectively, thus putting service levels at risk
If vendor does not agree to support customizations then our IT department needs to bear the cost and risk of documenting and supporting customizations on an ongoing basis
RESOURCING
In cases where we own the customization, we may not be able to access the engineers who customized the product leading to resource constraints. This is often because we don't have enough work to keep that engineer busy/allocated full-time. This often results in a big lead time to source the proper expertise
For a particular IT requirement how do you decide when to buy and when to build?
Here are some considerations... How do you think about it?
If your objectives are strategic, it often makes sense to build to your needs, so that you can further enhance the product to meet your changing needs.
If your objectives fall into a commodotized IT domain, then it makes sense to buy a product if it meets your current and anticipated needs (you really don't want to customize a vendor product. This is where open source products become attractive...). In such cases, requirements are fairly well understood and even if you've found a product that meets your requirements, following need to be true before you decide to procure it:
- good integration capabilities/API for data access, messaging etc. - ?
There is a distinct gap in the technical proficiency we need to build a quality software product vs. what is offered by our offshore software development partner today.
Over the years, we've asked them to bridge this gap (raise technical competency level) by getting experienced/talented engineers in the pool... but it hasn't happened yet...
So we're addressing it as follows. What are you doing?
Bridge the gap from our end - i.e. build and maintain a pool of talented engineers (FTE, Consultants) onshore to make key technical decisions. For this we've hired top talent in Java/Ruby/SQL and trained our engineers to be hands-on.
Encourage offshore partner to hire/build/retain talented technologists - with techniques like offering higher rate for such engineers offshore
Just got back from Vegas, where I heard C.K. Prahalad (renowned management thinker at University of Michigan) discuss his new book - "the new age of innovation". Enjoyed it!
One of my key take aways was the way he defined and contrasted Best Practices with Next Practices.
We often use the phrase "Best Practices" in IT. I discourage its all purpose use, primarily because of how its interpreted by most... Often it is understood as "here's the gospel, use it, don't question it, just follow it". It doesn't encourage thinking, questioning conventional wisdom. It might be fine in some cases, but not always.
Moreover, there are factors to consider while using Best Practices like context: What worked in a certain situation may not work everywhere; time: What you're facing today is very different from what you/someone else faced a week ago; etc.
Here's what I understand and like to practice...
To be distinctive we need to innovate - question the norm, think, adjust continuously. C.K. Prahalad calls this Next Practices. These practices are live i.e. designed to change. If you want to be "better than the rest" in an area, you need Next Practices, typically in core areas (e.g. software development is core to development teams, whereas infrastructure isn't)
In non-core areas, we expect only to be as good as others. In this case we can learn/follow what experts have defined - Best Practices. This can make us "only as good as the rest"
We introduced Ruby/LAMP in 2008 to develop applications. In our experience, Ruby provides significant developer productivity gains. However, several questions have been raised in IT as we've been a Java/J2EE shop for several years.
Ruby/LAMP isn't production ready in the enterprise:- Enterprises are used to vendor software that provide facilities like GUI-driven application setup/deployment, a single vendor to call to yell at for production issues etc. This offers them great comfort that they don't want to loose for open source software. It will take time to take Ruby/LAMP to the same place (engineering, development, support) as Java/J2EE in IT. - Managing/supporting Mongrel is painful: Passenger mod_rails is an attractive solution as it simplifies deployment of Ruby applications. We're moving ahead with it
Why use MySQL if you have Oracle? Most enterprises run and support Oracle, so incremental Oracle database cost is zero. Why should one use MySQL then? Does it matter? If so, where/what and how much? - MySQL is primary database for most open source software (including Ruby) and hence some software components (plugins etc.) only work on MySQL (at least in early phases) - MySQL enables agile development practices - Enterprise/core/reused datasets vs. "Application" datasets
How will we choose between Java and Ruby? - We plan to continue maintaining our Java apps using existing technology (Java/J2EE) - For new apps, I suggest Ruby as the starting point considering our internal needs...Here's why: Java offers high efficiency of execution with byte code compilation (JVM), whereas Ruby offers simplicity, maintainability and productivity gains. Most internal apps can be developed using Ruby unless complex integration with Java systems is needed.
(a) Most internal enterprise apps are simple and don’t require high efficiency of execution (e.g. many thousands of users and high-volume, real-time data processing)
(b) Performance gains from Ruby to Java aren’t significant from end-user perspective (server-side performance is negligible from end-user perspective)
This discussion in IT raises some important questions?
Should we leverage existing infrastructure (Oracle, Java/J2EE (JRuby)) for deploying Ruby applications?
Is/should MySQL be at par with Oracle in the enterprise?
Is/should LAMP be at par with J2EE in the enterprise?
?
How are you dealing with this in your IT department?
An offshore team might have the following configuration: 8 offshore engineers in development team + 2.5 FTEs (offshore teams need more onshore FTEs for coordination/ collaboration. Our experience shows ~ 2.5 FTEs for 6 - 8 person offshore team.)
Now compare this with an onshore team. In this case, you have access to experienced engineers, so you need fewer (say 4) and they can also complete the project faster (say 400 hours). Since they're onshore you'll need fewer FTEs (say 1.5) considering increased quality and quantity of collaboration ...
In essence the total costs are the same and if companies started measuring total costs and consider intangible costs like quality and collaboration, they may make different decisions...
Caution: This also doesn't imply that you fire your offshore development partner.
In fact, many offshore vendors have realized this from their experiences in last several years and are offering near-shore or onshore development options to their clients. E.g. Tata opened up a development center in Cincinnati in 2008)
This does not apply to all software development. There are several factors to consider while making on/offshore decision. E.g. Technology matters! Team size matters! Project duration matters! Offshore development seems more cost effective for commoditized technologies; large teams and/or long project timelines.
May be we need a decision tree to decide where to execute a project?
Many engineers I talk to hate the enterprise. They crave for start-ups... their environment, their fun, their energy, their perks. They find enterprises boring... and why not... that's exactly what most of them are!
How can enterprises address this?
To begin with... recognize the issue! Most enterprises are boring. What do you expect with all legacy technologies, overcomplicated systems, old school thinking, bureaucracy, corporate structures/controls etc.?
Don't throw money at it. Most talented IT engineers don't work just for the money. They see software as a community activity, a benevolent act (heard of open source yet?)
Understand that, to hire and retain top talent in the industry, you need to offer -
competitive environment: Will the new talent find your engineers at their intellectual level? What can you offer to grow them as professionals - paid conferences, ?
flexibility: go the Google way. Let your developers dream on you money. Give them a day to follow their dreams
insulate them from your mess: handle it yourself. What are FTEs for anyway? :)
break artificial corporate boundaries: let them contribute back to the open source world, let them take stuff outside the corporate boundaries (blog etc.) - that Drupal fix doesn't give you competitive edge, after all. In essence, enable cross-pollination with external world, don't stop it!
keep pace with technology: if you run mainframes, fuggetabout attracting top IT talent
I'll be in Bangalore in November and I'm looking forward to attending Ignite Bangalore on Novemeber 13th. It'll be great to meet some of the technology leaders in India's Silicon Valley.
If you have a lightening talk, submit your presentation topic.
I've exercised and noticed different practices in design and writing code... Sometimes we've spent weeks creating blueprints (so 2001) before starting actual development. At other times we've jumped into code directly (prototypes typically) and designed while writing code. There's also the hybrid approach of writing code and designing at the same time - emerging design.
Depending on the case, all practices were legitimate and useful. How should one decide what to do when?
What do you practice?
What are the factors that influence it?
What comes more natural to you and why?
Does it depend on the technology?
Does it depend on experience?
?
These practices don't seem black and white... there's really a continuum... In this spectrum of various practices, where do you fall? What do you practice? And why?
1------------------- 2 ----------------------3
Upfront design (speculative design): design, then code
Emerging design: design and code together
No design: Fuggetaboutit! start writing code and design will follow
More and more, we're practicing emerging design in our new initiatives. Less speculative, more practical...
We've struggled to find a team file sharing application that supports:
Offline access
Seamless access from desktop tools (Windows Explorer, Microsoft Office applications)
Web based file sharing doesn't always work well in the business world, where road warriors need offline access. Enterprise products like Groove (Microsoft), eRoom (EMC-Documentum) are good attempts but they don't meet the 2nd requirement- seamless access from desktop tools - as they create a separate space.
Businesses have suffered a lot already. They've waited long enough for IBM, Microsoft, others to solve it for them but no luck yet...
I believe it is time for some businesses to come together and jointly solve this problem as an open source effort.
Has anyone thought of building a custom solution to address this problem?
Windows Explorer integration with version control systems like Git/SVN?
Some Scrum teams have suggested that their Scrum Master doesn't add much value in the project... that their primary function is to communicate upwards/outwards by polling team members for updates... that they are chickens on the projects, not pigs*.
Such team dynamics sounds dangerous and should be avoided... Will the following help?
Let a team member take up the Scrum Master role - whether its the BA, Tech Lead, Developer or Tester in the team. It can even be rotated among them. Scrum Masters are pigs then and are close to the project to know/coach/communicate well... This would help keep the Scrum teams small and effective!
For this to happen, the following needs to be true though:
Team is matured to practice Scrum
Project doesn't demand a separate Scrum Master due to risk, size etc.
?
* To clarify, this isn't the case everywhere but just in a few of our teams
Software development is a social process and needs synergistic interaction within the team and with the business decision maker as well. I've noticed the following factors affecting our development team's effectiveness -
Team location: spacial and time-zone separation causes collaboration gaps
Space gap: Co-located vs. distributed teams (Geographical separation)
Time gap: Time-zone separation affects communication and collaboration. It results in less interaction between US-based business decision makers (sponsors/product owners) and offshore teams
Team agility: Agile mindset, engineering practices, soft-skills (influencing, negotiation, pushing back, saying no etc.)
Team experience: Senior vs. Junior team members. Domain knowledge and technical competency.
Can the following ideas address these issues?
Team location: Co-located teams (prominently at one place)
Leverage development partner’s global capabilities to execute projects locally
Access to global talent pool
Enable personalized, direct interactions
Team agility: Strengthen development partner relationships and mix
Establish new development partner relationships that meet skills gap
Tap into strengths of development partners
Shift from scale to skills
Team experience: Hire and retain technical talent: FTEs can deliver projects too... don't rely solely on external talent for all your needs
Build and sustain a pool of highly talented software engineers
My ideas for succeeding with offshoring might be different from what you hear/read elsewhere, so grab a seat...
eliminate penalties - share risks. You only need a master agreement, not many contracts/CRs. Trust-based relationship is key afterall
fixed-price contracts won't produce high-quality software- if your focus is quality, why ask your development partner to optimize on cost
talent matters - all engineers aren't created equal. If you can, hand-pick the engineers in your offshore team that you trust with your software application and then give them all the freedom (and caffeine)
involve them (make 'em think) - don't make every decision for them, instead share what's important for you and have them make those calls. This is the ONLY way to sustain offshoring. Otherwise you'll need daily sync-up with your offshore team members and you're not gonna like it
have a few partners and stick with them - having a new organization understand your company, your priorities, your culture takes time. So think hard before adding another development partner in the mix and forget about the 6-month return policy
By the way, inspite of all this the fact is that overall:
