Visibility as literal as it may sound gives us a path to source of information, which in turn gives way to align ourselves in accordance to that information so that accurate and timely decisions making can become a reality.

Information visibility becomes a major issue when considered in the context of a corporate environment. Problem becomes worst when information visibility is required beyond the boundaries of single organization. Let’s consider a traditional supply chain, where a manufacture procures raw material from its suppliers, delivers finished good to a distributor. Distributor later resells those good throw wholesalers and retailers and finally good are delivered to end user /consumer.

Suppliers Manufacturers Distributors Wholesaler Retailer Consumer

Every member of this supply chain forecasts the future demand and prepares themselves to meet that demand by maintaining a certain level of inventory, this stocked inventory is known as safety stock. Customer demand is rarely stable and as a consequence participates of supply chain from retailer to raw material supplier make forecasting errors. Even with a stable demand downstream member (retailer and wholesalers) increase there orders in anticipation of increased demand and so upstream members (manufactures and distributors) are forced to produce and maintain high level of safety stock.

This process of amplification of demand (which in turn increases the level of inventory) based on forecast errors as one move upstream in the supply chain is known as Bullwhip Effect. This effect was first demonstrated by the MIT Sloan School of Management in the form of a Beer Distribution Game.

The key to resolve these complex problems is collaborative planning of all the important functions and processes of various stake holders (within the organization and beyond the boundaries of organizations) based on the near-real time sharing of information, in other words Integration Business Planning is the need of the hour.

There are some prerequisites to Integration Business Planning.

Willingness to share the information.

Ability to share information.

Often companies are extremely conservative when it comes to sharing their crucial information (customer centric information, supplier’s raw material details etc), but there is a need to realize that their reluctance to share information can lead to far serious consequences such as ineffective production, excessive inventory, increased cost , financial losses and overall inaccurate decision making. Beside this, information visibility may also expose certain intentional or unintentional departmental blunders in front of management, which can lead to unnecessary explanation and at the extreme level can result in firing of the relevant managers.

Once different stake holders have agreed to share information, focus has to shift toward there ability to share information. When it comes to there ability of sharing information the major challenge lies in the intra and inter-organizational Data Integration.

Data integration is the process of combining data residing at different sources and providing the user with a unified view of these data. There are two stages of data integration

Internal Data Integration

External Data Integration

Organizations find it difficult to share information because there the internal processes and systems are not properly aligned and information/data resides in departmental files, legacy systems or other transaction processing systems.

Internal Data Integration

As depicted in the diagram, enterprise-wide data warehouse is the path toward enterprise-wide information visibility, but life is never simple. When effort are made to initiate such huge project (enterprise-wide data warehouse), usually such initiates are abandon because these initiatives pose enormous cost, time and resource constraints before they yield some results.

A better approach to resolve such issues as such suggested by Ralph Kimball, (often referred to as GOD of data warehousing) is to start with one or two major departments, that is extract information from their information repository, then transform the extracted data in accordance with the agreed standards and finally load that information into the data warehouse. In this manner some fruitful conclusions can be derived, in a relatively short time span and considerably small cost, further it will give other functions motivation to join hands in this project.

Let focus our attention toward external data integration, coming back to supply chain, as portrayed in the figure, there is a continuous requirement of data sharing between supply chain member, for instance raw material supplier needs access to manufacturer’s production plans, distributors have a need to examine end-customer’s demand available only with the retailer and so on.

External Data Integration

Again data integration provides a guiding light, as depicted there is a need to build a central information repository, data from all the supply chain participants have to be extracted and is to be loaded into a central data warehouse after applying various business rules and validations in the form of transformations. This central data repository will act as a data warehouse, where information would be uploaded as per business requirements in a periodic (daily or weekly) manner. All supply chain participants can have access each other’s information in the form of reports.

Internal or external data integration is usually performed by the dominant stake holder, who uses its influence to enforce the necessary alterations in the internal processes of other stake holder. Better way to initiate such a project is to consult a separate entity, whose core business revolves around data warehousing. This approach will not only reduce dominant member hypocrisy, future it will ensure that business intelligence/data warehousing professional will analyze and resolve all the data integration complexity and produce a best of breed solution.

It was a bit late in the evening and I was staring at a large diagram that has a mishmash of rectangular shapes, many of them joined by lines criss-crossing across the whole diagram. What started out as a simple data model has become a smallish monster of a diagram!

I started to have second thoughts about my data modeling: Am I over-designing everything? Why are they so many tables? I started muttering the names of the tables: Book, BookFormatType, BookPresentationType, BookCategory, BookPrice, BookPriceType, BookRatingType, BookPrice…

How can I get rid of so many tables that are classifying and qualifying the main entities? Having attributes instead of separate tables would introduce the classic problems of anomalies in data that normalization generally solves. This could be eliminated by putting constraints on the domain of possible values against these attributes but that would mean that any alterations on these value would require a redefinition of the constaint.

I have spent a significant amount of time in painstakingly describing all the tables, their descriptions, and their relationships to parent tables and as a consequence have extended the scope of the work by introducing new screens to maintain those tables.

My though process was interrupted by Zafar who almost hollered at me: “What are you doing?” I said “Nothing. Usual stuff”. I was a bit reluctant to discuss this with him because of my professional pride as I was supposed to be the software designer and The Data Modeling Person while he was a Consultant cum Project Manager. However, on his insistence, I discussed my beef with the model I was working on.

With his eyes lit, he said: “My friend, you need a type and type class here”. Not understanding what he meant, I just stared at his face. He explained that he has noticed that all these reference/type tables are using a code and a description as attributes. “What you need is instead of all these tables, what you need are two tables or entities, let’s call them Type and Type Class. Type Class will keep a list of all these reference table names that you have and the other entity ‘Type’ will have the actual instances or records against each Type Class row.” He said.

I took a couple of minutes to digest his idea and fired my questions: “How would you ensure referential integrity”?

Zafar smiled: “Well. The table where these type records are referred exactly knows what type class they belong to. Either you can put database level constraint on the attribute so that type records belonging to the intended type class are the available values or you do this in the application code”.

“Hmm.” This is clever, but I had my next question ready: “There are some maintainable tables and some ready-only tables in my original design. How do you tackle that in the maintenance screens”.

“That’s simple. In the Type Class table, you keep a flag of what data class is maintainable. Tell you what. I have requirements where certain preset records were immutable whereas the rest can be amended or deleted. You can handle them with another flag in the ‘Type’ table”.

“This is awesome. Is this your idea?” I had to ask him this question. “Well, this is something that I have done with Tooraj and Tara”*. Zafar replied.

[* The writer would like to acknowledge that he was introduced to this pattern during one of the development projects in 1996 where Zafar Omar, Tooraj Dervish and Tara Collinson were the principle business analysts and data modelers. Whether this was borrowed from somebody else’s idea is not known and is for the purposes of this acknowledgement, beside the point]

Design Pattern

TBA

Formal Definition

Alternate Terms/Synonyms (Enumeration and Enumeration Values; PeopleSoft calls it Translate Values)

Eligibility Criteria

Sample

Restrictions & Possible Cons

Consider the following design for exchange rate storage:

The Currency table holds the list of currencies whereas the exchange rate holds the rate of exchange between two currencies. The ‘Effective From’ date is part of the primary key so that history of rates (as well as future rates) can be kept in the same table.

This is just a hypothetical design and probably not a very suitable example: where do we ever have future rates? Why keep exchange rates between US$ and UK£ and between US$ and € separately when you can derive exchange rates, etc. However let’s disregard these issues for now. The key point is the use of effective date in the primary key.

Now, let’s try and write SQL to get the current (today’s) exchange rate between US$ and UK£.

Select Rate

From ExchangeRate

Where FromCurrency = ‘USD’ and ToCurrency = ‘GBP’

And EffectiveFrom = (today’s date)

The above assumes that exchange rates are maintained on a daily basis. If not, the query becomes:

Select Rate

From ExchangeRate

Where FromCurrency = ‘USD’ and ToCurrency = ‘GBP’

And EffectiveFrom = (

Select Max(EffectiveFrom)

From ExchangeRate

Where FromCurrency = ‘USD’

And ToCurrency=’GBP’

And EffectiveFrom <= (today’s date)

)

Similar is the query for getting exchange rate for a particular date. This is tedious to write and could have serious performance implications. How do you solve this problem?

Solution: Use Current

In situations where the current row is of primary concern, consider using the following design:

In the above design, a new Yes/No (Boolean) column called ‘Current’ is introduced. Only one row against each from-to currency combination is set to ‘Yes’, signifying the current rate. With this the query to get current rate becomes:

Select Rate

From ExchangeRate

Where FromCurrency = ‘USD’ and ToCurrency = ‘GBP’

And Current = 1 (Yes)

The above is only useful when bulk of the enquiries are for current rate as getting rates other than current rates is still tedious. Also, for systems that hold future rates as well, a batch job is required to point to the correct current row.

Solution: Introduce EffectiveTo

Another solution is to use an ‘Effective To’ date, as illustrated below:

The query becomes:

Select Rate

From ExchangeRate

Where FromCurrency = ‘USD’ And ToCurrency = ‘GBP’

And EffectiveFrom >= (current date)

And EffectiveTo <= (current date)

*************** End of post *****************

Usually we tend to design using the following approach; we create a promotion entity and tie it with book categories by creating the following entities to handle inclusion of categories and exclusion within “included” categories:

TABLE: PROMOTION {Primary key: PromotionID}

Sample Data

TABLE: BOOK {Primary key: BookCode}

Sample Data

TABLE: PROMOTION-CATEGORY {Primary key: PromotionID + CategoryID}

Sample Data

PROMOTION-CATEGORY-EXCEPTION {Primary key: PROMOTIONID+CategoryID+BookCode}

This approach though straight forward is difficult to program. When we try to launch a PROMOTION with a particular discount rate we need to be able to process at BookCode level. When a customer purchases a book you want the system to pick the correct PROMOTIONID (if any is applicable) and then apply the discount rate on the book based on the above setup.

In certain cases an easier way to handle this is to add PROMOTIONID to the Book entity. The program will then be a very simple and efficient one. All it has to do is check whether a PROMOTIONID is applied to the particular book.

Sample Data

This creates a problem for the end user who may find it very difficult to maintain PROMOTIONID at book level. To help out the end user, you can design a screen which gives an easy interface to the end user. So the end user has the flexibility to add book categories to Promotions and just define the exclusions. The system will need to additionally file the (un-normalize) field of PromotionID to BookCodes at the time of “Save”. This is more work for the UI but less work for the Sales Order form (because now it has to check at the BookCode level whether a promotion is applicable or not).

End of the day, you can argue that the complexity has not been circumvented. Instead it has just been shifted from the processing to the screen. It will be the decision of the software designer/architect to choose which one works for the particular scenario.

A point which would seem obvious to an accountant but is not so obvious to technical people is that, we cannot let any transaction be “edited” in a double entry system. If for any reason it must be edited then the original transaction should be fully reversed and a new transaction should be created. This is necessitated by the fact that each financial period should be reconcilable with the detailed transactions.

Example

Let’s say that in the month of January we generated the revenue of Rs. 1,000,000. We received cash payments of Rs. 8,00,000 against the revenue and the rest is receivable

The accounting transactions would be as follows

Cash & Bank (Debit) 8,00,000

Receivable (Debit) 2,00,000

Revenue (Credit) 1,000,000

Now suppose on the 3rd of February there is a need to edit a transaction of 31st January and the GL Journal/batch is already uploaded to GL. What would happen if we go and change a revenue transaction of Rs. 1000 to Rs. 2000? The GL Journal will not reconcile because the Journal is frozen at month end.

One way to handle this is by leaving the original transaction untouched. Instead you can create two transactions on 3rd February, one for Rs. -1000 and the other for Rs. 2000. There are ways to keep track of the original transaction if it has been later reversed. It is up to the software designer to design the UI in such a way that the process is easy for the system user. For example, you may design the screen in such a way that the user just goes and pulls an old transaction and edits it. The system does the necessary prompting but once the go ahead is given (by the user) then it generates the reversal transaction and new transaction without requiring any user intervention.

Conclusion

The above approach will ensure a reconcilable system which serves as the subsidiary ledger to the GL. The examples quoted here are quite simplistic for the purpose of illustration and real life is never that simple. Sometimes the transaction load of a double entry system could be quite taxing on the hardware and certain architecture/design decisions need to be taken up early in the project.

Most financial systems like billing, accounts payables, accounts receivables, etc. write Journals / Batches to the General Ledger application on a daily or monthly basis. These journals are expected to balanced, meaning that the total figure of debit and credit should match

Example

A typical accounts payable system which write daily journals to the general ledger, will write at the time of voucher creation for an expenditure of Rs. 100/-

Stationary Expense (Debit) = 100

Accounts Payables (Credit) = 100

At the time of payment the transaction would be

Accounts Payables (Debit) = 100 {this entry is a contra entry which will reduce the account payable figure in the accounting books/ledger}

Cash & Bank (Credit) = 100

If you look at the above account types the stationary expense is an expense and is a debit line. Similarly the accounts payable is a liability and is a credit line. The Cash & Bank is an asset and is a debit line.

When looking at the above transactions one can work out that if the above transaction was the only transaction in the whole day, then the daily journal will be balanced.

Challenge

How do you ensure that the system will always generate balanced transactions?

One way to approach it is that at the time of transaction you build rules in the system that the system will only allow selection of accounts (for example stationary expense in the above example) which are debit lines at the time of voucher creation. This makes sense because at the time of voucher creation the system will automatically generate the balancing transaction of accounts payable (credit line).

Another Example

In a guest billing system for a hospitality concern could have transactions such as

Receivable (Debit line) = 100

Revenue (Credit line) = 100

At the time of payment the transaction would be

Cash & Bank (Debit line) = 100

Receivable (Credit line) = 100 {contra entry which will reduce the receivable)

In the above case we need to build controls in the system that all services given to the guest have to have a Credit line (revenue) behind them (in the dictionary setup). In this case the receivable is automatically generated (which is a debit line).

Conclusion

In the above cases we have managed balanced transaction in systems like accounts payables and billing which are traditionally double entry systems. Meaning that they need to send balanced accounting transactions to the general ledger.

Sometimes in an integrated system, there is a need to segregate the system from various modules or third-party systems through the use of queues. It is possible that in this case you have a background program which reads from the queue and posts the transactions to the integrated system. Sometimes there are programs within the system which also want to post these type of transactions and hence a locking mechanism needs to be incorporate. This complicates the overall system. In certain cases it is possible to take a slightly different approach which is given as follows:

Transferring Data from other modules

1) These modules should write to the centralized queue using a common executable library (for example, DLL in Windows applications)

2) They should not write directly to the system to avoid locking issues

Generating Data from local module/system

3) Any programs local to the integrated application should also write to the centralized queue instead of directly posting the transaction from within the system

4) This will allow for only one background program to read from the queue and serially post transactions into the system, hence by-passing the locking issues

The Issue of Latency

5) It is possible that the requirement of the system may be that the transactions generated from within the system should be given higher priority for the purpose of speeding up a particular customer order, etc.

6) In that case we will have to create a mechanism in the queue to give priority to such transactions at the time of queue reading and posting

When we perform such field level checks, we forget some basic common sense checking such as re-checking for the existence of parent/master rows at the time of the actual saving. (I have intentionally taken this simple example to illustrate the point). The point being that these business rules that we implement on field level should also be re- implemented on the SAVE button.

It seems like extra work and extra load on the machine but depending on the application it could mean lost revenue. Consider the example of a computer store inventory. During check-out a certain computer was available for purchase as it is indeed in the inventory. However, if it is purchased by another concurrent user at another check-out, it is no longer available when we finalize the sale.

This is especially important in web based applications where you may not trust whether your java script will always run at the field level or you have a stable connection.

Another reason why we recommend keeping all business rules behind the SAVE button is that it simplifies the code and maintains a single code base (read: maintenance). How many times have we wished that all the checks were placed in a single location after a programmer’s exit so we would not have to look behind each field and open up each event?

No matter which approach you choose, remember that consistent user experience is the holy grail of application development. Do not subject your user to field level checking on one screen and then SAVE button checking on another screen.

Some design decisions need to be taken up early whether you are developing a Product or a customized solution.  The decisions are easier in the customized solution case because there is less market research involved.  You basically need to assess the current strength the application servers and the database servers and then decide how much load can be placed on each cluster.

Example

For example, you are developing a heavy report which the client insists should respond within 2 minutes of invocation.  You have looked at the business scenario and it seems that the 2 minute requirement is a fair one.

Approach

For the example above you may start by considering the following

1. Write a stored procedure which would give a very good turn around but it will load the DB server beyond acceptable parameters

2. Develop a memory record set based approach where the whole output is first prepared in memory and then passed to the reporting engine (like Crystal Reports, Jasper etc.) to print.  This will also perform quite well (perhaps not as well as the first approach) but the down side with this is that it will load the Application server beyond acceptable parameters

3. Develop a temporary physical table where the whole output is first prepared and then passed to the reporting engine to print.  This is most feasible because it does not load the Application or the DB servers.  The only problem is that it may not be able to meet the 2 minute turn-around time requirement of the client

4. Develop using some hybrid of the above and also doing some processing on crystal (app server) but usually report engines are  very slow

5. Develop statistical tables

1. In certain cases it can be beneficial to populate statistical tables at the time of transaction posting.  This slows down the post process but greatly improves the report turn-around time
2. Another way to do it is to write a background job which runs at midnight; that way there is no additional load at the time of transaction posting.  The downside it that the customer will not get up-to-the-minute results, instead the results will reflect the position as of last night

6. Add data redundancy into your tables to avoid too many query joins

Challenge

How do you achieve this without a Production level UAT hardware?

How best to write diagnostic utilities which will ensure the integrity of statistical tables or redundant field values?

Conclusion

For the purpose of simplicity we have not brought the web server in the discussion above.  The thinking pattern would be similar and this chapter would have hopefully provided some options to consider.

1) Tab ordering is correct
2) Capitalization standards are followed (Title case vs. Sentence case in different controls)
3) The use of controls should be standardized, if such a standard exists. For instance, if the project demands or requires the use of a particular Grid Control (True DB Grid or Infragistics Ultra Grid), it should be used throughout.
4) The names & position of controls should be consistent between screens.
         a. Standard Names and icons for buttons, for e.g., Add, Edit/Update, Delete, Save, Cancel/Close/Exit
        b. If search is provided at the top of the screen, it should be at the top everywhere.
5) Mandatory fields are handled.
6) Correct field lengths are used and set.
7) Hour glass or a progress dialog (with an ability to cancel) is provided for long operations.
8. Enable and Disable controls properly so that actions that are not applicable or valid are not available to the user. This reduces a lot of validation code.

User Interface Design

1) Consider creating a common control (control class) for visual controls that are common across multiple screens.
2) If the development platform allows inheritance of forms, use it.
3) Common validation routines should be put in a common class and should be re-used rather than each developer making their own function to validate a date etc.

Application Code

1) Compliance with coding standards including variable prefixes, class, file and function names, commenting standards, use of constants (vs. magic numbers)
2) Appropriate handling of null values, especially when null values are returned from the database
3) Ensure that exceptions are caught by the application code and handled as per the case may be. This may be in the form of:
   a. Corrective action, in order to continue program execution
   b. the user is provided with an appropriate user-friendly message
   c. the application or the module is stopped with a graceful exit
4) Ensure that user exceptions are thrown as per the agreed standards and guidelines.
5) Ensure that appropriate error logging library is used and appropriate error priority is set for different error conditions or as a result of different exceptions.
6) Handle concurrent access cases as per the design.
7) Handle deadlock errors.
8. Use transactions where required.
9) Ensure that connections are closed after use. Use disconnected record sets unless a connected record set is absolutely required.

Class Design

1) Ensure that appropriate classes and interfaces are design in compliance with the application architecture (n-tier, etc.).

Database Queries

1) Avoid using Select * (use specific column names instead)

Other Checks

1) Ensure that code is checked in to the central source control system (almost daily).
2) Ensure that each version or release can be traced back to its source code.
3) Ensure that all required post-analyzers are run on the code:
    a. No warnings or hints are outstanding during compilation
    b. FxCop is run and there are no outstanding warnings or errors
4) If using NDoc or any other tool to generate documentation, ensure that it is run and produces the correct output.
5) If using NUnit/JUnit or any other unit testing framework, ensure that it is run and all tests are successful.
6) Ensure that all documentation, including use case description, data dictionary, and program specifications are updated.