Release Wednesday

How to do continuous integration for hardware design -- a story of release Wednesday.

Background

Recently I come across this issue of performing continuous integration (CI) in hardware development flow (mostly RTL front end design). There are discussions of the tools and framework for this tasks. Most of the available tools are for software development. Thus it is not easy to chose one.

I am more concern of adapting the approach (or the common practices) of software CI flow to the hardware dev environment. The company I worked with has this software CI infrastructure in place for the RTL projects. Despite the large efforts put in it, the system did not generate many significant results (at lot less then what I expected). Without the incentives to maintain the system, it die a slow death.

We can spend a long day to discuss the differences between hardware and software development and be very academic about it. But what I want to present is another story happened in the same company.

Wednesday

It was in the early years and the concept of CI is not that popular (at least not in that hardware company). Projects still have the same need to regularly check the health of the code base. Without any fancy tools, we come up the the idea of release Wednesday.

Every Wednesday, the project technical lead (this is a role, not a title) will check with the engineers to see what can be (and should be, following the project schedule) ready for verification and integration. Then the technical lead will actively pull the modules in the main branch and allow the integration to produce a coherent and consist copy of the project code base. Once the release is process, nobody to mess with the revision control system until released.

It was a stress day. For the module developers, it is the time they have to present their work to the team. All the sloppy work, every stupid bugs and any schedule slip will be transparent to the whole team. For the technical lead, it is the most busy day. All conflicts between modules has to be resolved to prevent further diverse between modules. A lot of decisions (e.g. which feature to be integrated first, which conflicts can be grouped while resolving, when to notify the verification team for the quick tests) must be made to deliver the release. For the verification team, the pressure is on the quick turn around time. The release candidate (RC) is usually available mid-day. Then a battery of tests is run to assure the quality of the RC. There may be several iterations and thus several RC to be checked.

Work as a team is very much the reality of this day as everyone must focus on the release and standby for unexpected work (or rework).

Achievements

After the release, everyone can start working on a common code base which is known to work.

The verification team will check if the previously reported bugs/issues are resolved; if the planned features are integrated and functioning; if the there are any new bugs/issues introduced.

The front end developers can start working on new features. If their work depend on the features from others, this is a good time to check if it is presented and then plan the development work accordingly.

The technical lead has a very good understanding about the status of the code and the progress of the project. It is always a refresh of the functional and implementation specifications while integrating and resolving the integration conflicts.

Base on the reports from the technical lead, the project manager can then review the project schedule and plan the resources better for next week.

The end result of this Wednesday release is not different from the result of an automatic CI tool. The real benefit is the impact on the team members (i.e. the human). The difference is between waiting an machine generated integration/regress reports and involving actively in the integration progress. The exposure to the full project scale integration and focus of resolving issues with other teammates are the most valuable part. In my experience, everyone in the team get to know the project better each time and the release Wednesday is pretty smooth after the team gets used to the integration process.

There are very little noise generated in this approach when comparing to the automatic CI process. All involved persons are happier in this way during the project.

Notes

While adapting this approach, there are a few notes should be taken.

1. The involved team size should not be too large. My experience is with a team across UK and India with less than 10 people. The actual number of engineers contributed to the code may be larger but they can select a representative person for this task (a single voice for a sub-team).
2. The technical lead must have some degree of authority to make decisions impacting the project schedule. He/She also need to have very good knowledge in both specification requirements and the practical work of front-end development.
3. The members of the team need to be self-motivated and understand the reasons behind this approach. They have to be in standby mode for the day and actively help resolving the issues even the issues may not be caused by their work.
4. It must be a regular recurrent task. In the initial stage of the project, the integration work may be rough due the the large delta changes. In the later stage, there may be very minor updates within a week. All this should not be an excuse to skip the release Wednesday.

Verilog TestBench Design

I have be working on my side project of digital circuit design in Verilog for a while. One thing that bugs me and pops up regularly is the way to write a test. I have a few options here:

1. I can go for the old school style to embedded the test as blocks of Verilog code in the testbench. This give me the largest flexibility in terms of precise control of the testing environment. But this is also the most troublesome style. It requires the test author to code in Verilog. It exposes the complexity and diversity of the DUT to the test author. It is difficult to build and hard to debug. It also require re-compilation for even the simplest change in the test.
2. Most comercial Verilog simulator supports a build scripting engine which allows an interactive user experience through the simulator consoles. These include VCS, Incisive and ModelSim. TCL is the most commonly supported scripting language in these EDA tools. So one can build the complete test environment around this technology. The largest advantage is the easy of coding in a familiar syntax while maintaining fine control over the DUT through the TCL commands provided by the EDA tool. No recompilation is required. I have seen this approach in practical environment for IP design. It works very well, as long as you have paid for the simulator. Unfortunately, I cannot find any free Verilog simulator supporting this, not to said that I have to run it on my Macbook Air.
3. UVM+SV is the trendy thingy in SoC/IP design. I sure you that there are tons of information you can get from Google telling its advantages. But again, I cannot find a free (either as beer or for speech) Verilog simulator supporting this.
4. The approach I used is adapted from the company I worked for. The implementation is completely different by the underlying idea is the same. To write a standard test vector generator which will read in a text based test file. It then alters the signals connected to the DUT based on parsing the text file line by line. A simple programming language with a handful of instructions and a text editor are all you need to create and modify a test. This works fine until you try more complicate tests with branch and arithmetic operations

Now I am working on another approach which:

• doest not require recompilation of the Verilog code (this is very important as we cannot assume the person who creates and runs the test will have access to the same Verilog compiler, also, you can ship binary instead of source code to the testing environment)
• has a simple programming interface. It may not be as flexible and mature as TCL, but it should support simple looping and arithmetic operations.
• is independent from the Verilog compiler and simulator. This will ensure the portability of the tests and allow future extension.