System iNetwork

Here's a surprisingly obvious question that IT managers don't often ask. Why not?

**Okay, I don't see a way to format this post. I give up. It's ugly and squirmed together. Many apologies. Blame the guys behind this site. I should be able to do basic HTML. Or maybe blame those idiots who keep spamming this site.

Probably because it's a very delicate, nasty question :-)

If you're a developer, how many times has anyone ever asked you this question?

Never. In my 5 years of professional experience, no one ever asked that question. We strove for absolute perfection in the code we produced (which is, of course, absolute nonsense. If the user can C-M-a M-It seems odd (at least in my experience) that some variation of this question isn't posed more frequently.

Honestly, after reading this article, I have to say you have an excellent point.

sometimes quality can be lowered to lower production costs and thus the price.

MS has proven this. A product can profitably ship with hundreds of known bugs.

Esp. if those bugs are little nits that most users will never encounter, because all the really big stuff has already been tested.

Of course, MS has also proved that, as long as you spend enough money on good advertising, you can also let the testing dept slip...

And there are even published specs on various properties, such as noise levels in sound reporduction (which, of course, don't tell the whole story).

Which raises an interesting question. Is sound reproduction as complicated as software engineering? [I don't know the first thing about "real" sound production. I assume it's at least as complicated].

So, random train of thought, what about a bit of software that describes "This is the *right thing* to do here." But, you can run 10x faster using a routine that does *almost right thing,* and 100x faster by running *roughly right thing.* And the compiler should be smart enough to recognize these potential optimizations and let the person doing the compiling (or the end user...go ahead and compile all 3 options...hard drives are cheap now) decide which way to go. Not that I see that happening.

I guess the reason questions about software quality are rare is that there may seem to be few, if any, simple and inexpensive ways to measure quality.

Or maybe grad students are blowing off this complicated question and playing with toys like implementing a sort-of-posix open-source operating system? :-)

It looks almost to me as if you're describing a bug-tracking -reporting -evaluating system. Which implies serious testers in place to actually use the system.

I suppose it might happen automatically, in a few academic languages like scheme, but I don't see that level of AI becoming mainstream.

a. Number of detected defects per 1,000 executable source statements (or lines)

Absolutely vital. Actually, this metric could be important for any number of reasons.

b. Same as a), but using weighting for seriousness of the defect (minor, severe, etc.) and/or time intervals (defects per month since release).

Sure, why not both? Or either? Trivial to store this, once you have a.

c. Same as a) and b), but measuring deficiencies, rather than defects

I should probably google this, but it's past time for bed. Do you mean (deficiencies = (things users expect to find) vs (things users do find, but shouldn't))?

d. End-user satisfaction ratings (ease-of-use, reliability, functionality, overall satisfaction).

Now you're getting really subjective, but you're right.

e. Code quality measures, such as using one of the "complexity" formulas, or review by a "master-level" programmer.

The problem, of course, defining a "master-level" programmer. "That idiot gave me a low rating because he's stupid."

It's a worthwhile objective, of course. But I think CS is far from it. It's easy enough to pick apart a junior-level programmer. But a master-level programmer? Hmm.

f. Performance test results; performance measurements from production use

Of course

g. "Maintainability" ratings by programmers who code revisions to the software.

This one seems really problematical. Lots of back-stabbing possibilities here. If you're such an ass that everyone hates you, you lose, even if your code is the best-designed ever. Then again, if you're a generally nice person no one wants to offend, yet you write crap code, this falls off the other edge of the map.

Good idea, horribly difficult [decent] implementation.

Quality measurements are mostly relative, rather than absolute,

Are they? (Examples snipped)

How do you think iSeries IT shops should approach the question?

Here's the crux, isn't it?

Heh. You should allow a developer to have his cake and eat it too.

I want to develop code in some 4GL. I want the underlying compiler to create the most efficient machine code possible.

-- Paul

After the enlightning and coherent insight given by James, this topic has been fully covered, sound reporduction included. Next please!

qwrqrqrqrqr

After reading this, I had to conclude this is a rather silly and pompous question. In most companies, users are not shy about letting IT and others know about how well the IT infrastructure is working. The remaining companies where this question is never raised would react negatively to the question, thinking suspiciously this is a turf grab of some type.

If company management allows buggy software to leave IT as a mattter of regularity, then company management is getting what it asked for. Stupid in, stupid out.

The QC measurements outlined above are also ineffective and somewhat rediculous in an IT context. If a program works as designed, but it solves the wrong problem, or solves the correct problem inadequately, this is not an IT issue. It is a design team issue or possibly an industry issue.

Defects per lines of code is small time thinking. Why not pay coders on the basis of lines compiled? It makes about as much sense. Validation and test scripts go along way towards dealing with superficial errors.

Asking coders, particularly AS/400 coders, to individually assess quality by asking them comment on the work of their co-workers is a great way to enforce mediocrity. Excellence should be imposed by leadership, not committee. My recollection of most AS/400 coders is of narrow focused, snappy, do-it-my-way, exceptionally average thinkers. Putting a gaggle such as this in charge of a code review is a good way to chase away anyone who has real talent. Besides, that person would probably be fired for not fitting in.

Think I am wrong? Go into your department and work over the next project by treating it as a business problem as opposed to a need to for programs. Assemble the project team. Is IT in the majority or just one department represented? Does the design team focus on defining the problem or go straight to code after a brief discussion? Is there a considerable amount of time spent discussion solutions before thinking about how IT will be involved in the solutions? Or is IT and custom code considered the solution from the get-go?

The AS/400 IT coder gagle will probably react with negativity or hostility if asked to think in business terms first and be a part of an actual team. Or they will sit through it politely and later scheme out how to get the tail to wag the dog again.

Manufacturers depend on knowing the quality of what they produce, as do major software companies (Intuit is an interesting company that's taken a very proactive approach to software quality with their Quicken product).

Sorry if the way I posed this question here didn't strike a favorable chord, but the basic question about software quality is hardly "silly".

The possibilities for measuring quality were listed to kick off a constructive discussion, with pros and cons of these and other approaches to quality measures. As the introduction noted, there "seem to be few, if any, simple and inexpensive ways to measure quality." It would be interesting to learn how some iSeries IT groups tackle the problem.

Your perception of "AS/400 coders" is quite different than mine. I've found many iSeries developers' greatest strength lies in how they absorb knowledge about the business.

Addressing the quality of software that gets produced is a way to improve the other side of iSeries development -- actual implementation design and coding.

-- Paul

Paul,

Of course quality matters. However, my experience during my past life as an AS/400 consultant (about 15 years) is rather negative. Quality is usually not even an issue and draws blank stares if even mentioned. Coders confuse quality with RPG fads such as coding methods and are quite brittle whenever alternatives of any type are mentioned.

In the best places I visited, IT was a part of a design team and how to implement IT in the solution was a part of the discussion. This is where quality started, not in acceptance testing or SPC measurements. These places were rare and refreshing.

Quality adds value only when it is a part of the entire process, not just the IT department. Defects such as coding errors (example 2 + 2 = 5) can be uncovered using simple scripted tests. This is the low end of QC as it relates to IT, but still necessary.

If the solution does not fit the objective, this is another type of problem and requires more work to uncover why. Was an unqualified but confident coder / analyst pair put in charge of the show or was IT a part of a team and expected to handle only the IT part of a project plan?

Does IT go passive-aggressive whenver users complain the solution driven by the IT team does not work as well an anticipated? If so, any lack of complaints does not indicate acceptable work. It is a normal reaction to not wanting to waste time arguing with IT professionals who don't understand the problem and have learned over time that they don't have to if they stonewall well enough.

No, not all are this bad, but I have seen incredible incompetence during my former life, and this incompetence was institutionalized among the RPG coders in these companies. These brittle people were best at maintaining the status quo and effective at isolating anyone in IT who had a broader view of the business. Also, innefective management supported this culture. I suspect this is one of the many reasons why RPG and the AS/400 is fading from view. IBM did not push evolutionary innovation and, thus, nobody else did either.

On another thread here, I wrote a reply concerning how to make a project fail. This was the topic of the thread and it brought back a lot of memories. Nothing I wrote was fiction. In fact, I left out some of the more bizzare situations I ran across. All were indicitive of low quailty environments, in which IT was only a part.

If you want high quality, then remove as much individuality (variation) from the process as possible. Bring it in only when it adds value or innovation and is required because this is the only way to achieve the necessary results. Consider all quality inititives to be a part of the entire process, from conception to post-mortum analysis and not a static measurement that can only be loosely tied to system success. Management that drives the entire company forward as a matter of routine is also a prerequisite.

signed

A former AS/400 application developer

Some questions for the former AS/400 developer:

What do you develop in now?

How do you see that development environment enabling you and presumably fellow application developers you respect to produce better business solutions, and for what reasons do you see them being better?

Are you in the same vertical industries as throughout portions of your 15 year RPG career? Same kinds of applications? Any of the same companies?

Can you give a specific example of the flexibilty you see your current colleagues demonstrate that you can contrast with examples of brittleness you observed with your RPG colleagues?

Can you give specific examples of how your current applications serve your clients better than the RPG programs you wrote for 15 years?

rd

You would be supirsed how many narrow minded Developer Managers there are out there that never get user input before building an application. I can give examples.

-David

That"svery kind of you

Software is hardly ever perfect, but it should always get better. Every test should be more extensive than the last and should be built on what was tested before. Otherwise you might as well chuck out the source code and start again afresh - we don't, we build on what was coded before so the testing should do exactly the same.

The trouble is not many people work this way, some do and have installed best of breed testing tools designed for iSeries like TestBench so they don't have to worry about testing the existing code, only what has been changed.

Attitudes have changed, users and business people expect us to get it right, just like anything else they buy. I met a bunch of developers last year who claimed they were highly responsive to the business needs. They might put live a new version of a progam 5 times a day! Of course, it turned out each was to fix bugs, and each implementation introduced more. A month later they learned their whole deptartment had been outsourced off-shore, they were all out of work.
If you can't deliver quality, don't expect to keep your job - so you might be a little wary if management asks "how good is our software?

Ralph Daugherty,

Your reply reminds me of days gone by.

The quality inititives I described are common in all other departments in all industries that respect TQC. Basically, you center the process within allowable limits, then take steps to reduce variation.

An IT department removes variation by removing individuality. You don't find accounting clerks or nurses or firemen doing their jobs however they want. All follow procedures so each is as interchangable as possible. IT and programmers have historically felt immune from these rules and frequently create crap as a result.

Putting a strawboss or brittle coder in charge as a master programmer just makes the entire department no better than whoever snagged that job. In order to count an error, you all must first agree on the definition of an error. Politics and office bullying is a poor way to come to an 'agreement'.

A former AS/400 application developer

For the record, here was my reply. Whcih of these questions I asked that you didn't answer reminds you of days gone by?

Some questions for the former AS/400 developer:

What do you develop in now?

How do you see that development environment enabling you and presumably fellow application developers you respect to produce better business solutions, and for what reasons do you see them being better?

Are you in the same vertical industries as throughout portions of your 15 year RPG career? Same kinds of applications? Any of the same companies?

Can you give a specific example of the flexibilty you see your current colleagues demonstrate that you can contrast with examples of brittleness you observed with your RPG colleagues?

Can you give specific examples of how your current applications serve your clients better than the RPG programs you wrote for 15 years?

rd

Ralph,

The tone of your reply is reminds me of how IT considers itself different from everybody else. Quality has little if anything to do with such activities as forbiding GOTO statements or firing employees who speak of politically unpopular programming techniques. In fact, an argument could be made that this level of micromanagement is evidence of a low quality environment. (Yes, I know you did not write of this. In my opinion, these concepts are just a stone's throw away.)

With respect to brittleness, I know you have read countless newsgroup threads where one programmer or another has hissy fits when they see someone programming in a way they just can't stand. Maybe they use GOTO. Maybe they used called programs. Maybe they use procedures. Maybe they use PDM option 14 and refuse to change. We have both read from managers proudly state that they would fire a programmer who uses one style of coding over another. These micromanagers or brittle coders almost never write with passion about problem solving or competiveness or anything else outside their excessively narrow view. In fact, usually they just shun anyone who seems 'different'.

You imply that certain applications or industries permit an approach that places the programmer apart from and above others. By asking about my personal successes, you infer that I am somehow a person to take down ... and by doing so my TQC perspective is invalidated.

Why not pick up a book on basic Quality Control from Amazon or elsewhere. Used texts can be had for very little. You will not read that IT is exempt from the same approachs used by other departments. In fact, you will read that much of what I have written is considered foundational.

Rush Limbaugh uses a rather devious technique to create the appearance of winning arguments. He berates others for not providing 'evidence' for their claims. If someone supports their statement, he demands evidence for the support statements. Then evidence in support of the support is demanded. This cycle continues until Rush's adversary gives up. Then Rush proclaims victory. For his evidence, Rush will use one flaky opinion or another and claims eloquently that his source is irrefutable.

Agree with everything you said and unfortunately there are those that would rather look for perceived weaknesses in fast written prose rather than support their own justifications.

Keep in mind that being in both worlds(well there's more than one, grin) that this is everywhere and is not by any means indicative of the iSeries, Windows, Unix, etc.

IT somehow managed to become an independent entity in most business like acctg. I can remember the wars and battles undergone with controllers and CFO's who thought they should run the IT dept.

That said, corporate IT accountability is on par with the industry, that is, there is relatively none and there are many who would rather keep it that way. IT has changed in a very narrow perspective but generally it's the same as any capitalistic competitive environment. Keep inventing a new product and sales surely follow.

I think the question should really be "How good does our software have to be?"
Good examples of this are CMM and RUP, which focus on trying to bring structured process to the perceived mess. Problem is they are extremely costly to implement and have never really proven to provide the elasticity necc. to build robust software. OTOH if you're building embedded software for automotive braking systems you'd better damn well have exacting specifications and testing processes(structure). What process did NASA use to orbit the moon back in the 60's?

Most all software written or designed is within all constraints we all live by, so if the environment and processes for the IT dept are poor it's probably symptomatic of the whole company mgmt style. OTOH, I just left a project 6 mos ago where CMM and RUP basically killed any chance to renew the contract. It's was too costly, although we got our CMM Lvl 3 out of it but lost the renewal.

This may come across as simplistic, but I generally define quality as "conformance to requirements", and measure quality by the "cost of non-conformance".

What are examples of the cost of non-conformance? Redesign. Revisions. Rework. Downtime.

You might also consider measuring the cost of an application, created by a 3rd party, that conforms to requirements better than yours, and ends up replacing yours.

This approach won't work if you try to define quality by comparing the attributes of one application to another in a way that one might compare a high-end Mercedes Benz to a low-end Ford, because the differences may be due to the target customer base.

If you define quality as conformance to requirements, then both the Ford and the Mercedes Benz may have equivalent quality, though their attributes are different.

The attributes of applications may be different because their runtime contexts may be different, or the intended user base may be different.

Quality may come down to how well a company understands requirements, is able to document them, and validates that they are met, and measures the cost of non-conformance.

Some organizations might define the cost of quality as the cost of upgrading individual knowledge and skills and implementing better procedures. But measuring the cost of non-conformance is a better approach, in my opinion. Seeing the cost of rework/replacement, hits the mark, and motivates change.

One thing attracting me to software development as a profession was having witnessed a number of failed projects and thinking I could do better.

If you fail at an attempt to replace something existing, it might be a clue that something about your idea of quality, or your definition of requirements is lacking.

Nathan, I'm thinking along the same lines you did. For me, quality of a software product is determined by measuring who much the product deviates from the design specifications (the functional design), how much it deviates from the general accepted functionality of the sector or industry or niche where this products operates in and lastly, how much it deviates from standard accepted IT functionality.

The lesser it deviates, the higher the quality.
Of course that leaves to quantify the gravity and the accompanying cost of a deviation (an error) to be discussed, which naturally is also influenced by the developemt stage the product is in. The earlier an error is detected, the least cost, is common knowledge.

I wouldn't go as far to determine the quality of the functional design of a software product. It may depend on the size of the development budget or on the kind of targeted public who is going to use the product, like you would have a different public who buys either a Ford or a Mercedes.

Nathan, I'm thinking along the same lines you did. For me, quality of a software product is determined by measuring how much the product deviates from the design specifications (the functional design), how much it deviates from the general accepted functionality of the sector or industry or niche where this products operates in and lastly, how much it deviates from standard accepted IT functionality.

The lesser it deviates, the higher the quality.
Of course that leaves to quantify the gravity and the accompanying cost of a deviation (an error) to be discussed, which naturally is also influenced by the developemt stage the product is in. The earlier an error is detected, the least cost, is common knowledge.

I wouldn't go as far to determine the quality of the functional design of a software product. It may depend on the size of the development budget or on the kind of targeted public who is going to use the product, like you would have a different public who buys either a Ford or a Mercedes.

One poster first condemned RPG developers saying that they are incapable of solving the problem themselves but, the alternatives he offered, he condemned as too expensive. Large companies will pay for QuickTest Pro and the test teams to use it. Smaller companies with much less code don't have the problem large companies face. Soon, a new generation of developers who learned about unit testing in college will solve the problem cheaply and efficiently.

test comment