How to do continuous integration for hardware design -- a story of release Wednesday.
Background
Recently I come across this issue of performing continuous integration (CI) in hardware development flow (mostly RTL front end design). There are discussions of the tools and framework for this tasks. Most of the available tools are for software development. Thus it is not easy to chose one.
I am more concern of adapting the approach (or the common practices) of software CI flow to the hardware dev environment. The company I worked with has this software CI infrastructure in place for the RTL projects. Despite the large efforts put in it, the system did not generate many significant results (at lot less then what I expected). Without the incentives to maintain the system, it die a slow death.
We can spend a long day to discuss the differences between hardware and software development and be very academic about it. But what I want to present is another story happened in the same company.
Wednesday
It was in the early years and the concept of CI is not that popular (at least not in that hardware company). Projects still have the same need to regularly check the health of the code base. Without any fancy tools, we come up the the idea of release Wednesday.
Every Wednesday, the project technical lead (this is a role, not a title) will check with the engineers to see what can be (and should be, following the project schedule) ready for verification and integration. Then the technical lead will actively pull the modules in the main branch and allow the integration to produce a coherent and consist copy of the project code base. Once the release is process, nobody to mess with the revision control system until released.
It was a stress day. For the module developers, it is the time they have to present their work to the team. All the sloppy work, every stupid bugs and any schedule slip will be transparent to the whole team. For the technical lead, it is the most busy day. All conflicts between modules has to be resolved to prevent further diverse between modules. A lot of decisions (e.g. which feature to be integrated first, which conflicts can be grouped while resolving, when to notify the verification team for the quick tests) must be made to deliver the release. For the verification team, the pressure is on the quick turn around time. The release candidate (RC) is usually available mid-day. Then a battery of tests is run to assure the quality of the RC. There may be several iterations and thus several RC to be checked.
Work as a team is very much the reality of this day as everyone must focus on the release and standby for unexpected work (or rework).
Achievements
After the release, everyone can start working on a common code base which is known to work.
The verification team will check if the previously reported bugs/issues are resolved; if the planned features are integrated and functioning; if the there are any new bugs/issues introduced.
The front end developers can start working on new features. If their work depend on the features from others, this is a good time to check if it is presented and then plan the development work accordingly.
The technical lead has a very good understanding about the status of the code and the progress of the project. It is always a refresh of the functional and implementation specifications while integrating and resolving the integration conflicts.
Base on the reports from the technical lead, the project manager can then review the project schedule and plan the resources better for next week.
The end result of this Wednesday release is not different from the result of an automatic CI tool. The real benefit is the impact on the team members (i.e. the human). The difference is between waiting an machine generated integration/regress reports and involving actively in the integration progress. The exposure to the full project scale integration and focus of resolving issues with other teammates are the most valuable part. In my experience, everyone in the team get to know the project better each time and the release Wednesday is pretty smooth after the team gets used to the integration process.
There are very little noise generated in this approach when comparing to the automatic CI process. All involved persons are happier in this way during the project.
Notes
While adapting this approach, there are a few notes should be taken.
- The involved team size should not be too large. My experience is with a team across UK and India with less than 10 people. The actual number of engineers contributed to the code may be larger but they can select a representative person for this task (a single voice for a sub-team).
- The technical lead must have some degree of authority to make decisions impacting the project schedule. He/She also need to have very good knowledge in both specification requirements and the practical work of front-end development.
- The members of the team need to be self-motivated and understand the reasons behind this approach. They have to be in standby mode for the day and actively help resolving the issues even the issues may not be caused by their work.
- It must be a regular recurrent task. In the initial stage of the project, the integration work may be rough due the the large delta changes. In the later stage, there may be very minor updates within a week. All this should not be an excuse to skip the release Wednesday.
