Keeping the e-Learning/e-Portal service in good shape: Performance Audit and Tuning, Capacity Planning for the Blackboard System

The Blackboard (Bb) Academic Suite has been deployed in the university since July 2005. The Sun Solaris/Oracle DBMS based production system is supported by four application servers via a load-balancer and an additional application server is used for virtual classroom. These application servers work with the database servers and file servers which are both in clustered configurations and connected to the SAN storage and backup devices.

The Learning Management System and the e-Portal System including the Content Collect (i.e. Content System) are fully deployed. CityU is achieving 100% adoption of e-Learning; in 2006/2007 academic year, Bb was host to 30,000 students and 4,000 staff, and over 3,000 course sites.

In the new academic year it is expected that experienced users/courses will utilize more Bb features in their teaching and learning, already evidenced are the demands for wikis and blogs in English writing enhancement, peer learning and evaluation; use of e-Portfolio by students to document learning outcomes; on-line quizzes for continuous assessment; external teaching and learning collaborations.

The university is mindful of the user expectation on system stability and reliability and the need to monitor usage growth in planning for the new 4-year degree programme. A Consultant from Blackboard Inc. in the US was commissioned to carry out a system capacity planning and performance audit project to assess our hardware capacity, to identify thresholds and bottlenecks in the application, database and file services; and to perform overall system tuning and optimization.

The project was done in two phases – mid March in 2006/07 Semester B and mid July in 2007 Summer Term. The first phase focused on the Learning Management System and e-Portal, and the second phase was primarily on the Content System. This approach helped identify performance issues involving major components as well as their inter-relating factors.

The Consultant came to CityU for a week to understand our system and network infrastructure. He also had interviews with different stakeholders to get a better idea of our development plan like new initiatives for teaching and learning. The system auditing and tuning exercise involved a thorough analysis of our user access pattern and usages which resulted in a performance baseline. The data was then programmed in a simulation tool to emulate various controlled load tests - double load, triple load and quadruple load, on the system to identify and optimize the weak points. To minimize the impact of the load tests on our users, they were run between midnight and 4a.m. by the Consultant remotely from the US.

The findings from the various load tests indicated that the current system hardware architecture was generally well optimized. It revealed that the increased usage had no significant effect on the application servers and file server tier, however, it had an alarming impact on the database CPU resource. Similar observation was recorded in the Content System audit exercise. This suggested that the bottleneck, as illustrated in Figure 1, would be the database should usage increase. The findings were shared with the Bb product development team for advice and probable development in the future product release.

Figure 1 - Database Server CPU Utilization Comparison

In the interim, some database processes and re-configurations are implemented to help minimize resources locking. The tuning effects are shown in Figure 2 and 3 indicating a more sustainable setting with a 17% increase in throughput. This also calls for careful monitoring of the database resource utilization and the planning for an increase in the database hardware.

Figure 2 – Before Tuning	Figure 3 – After Tuning
Going through the audit exercises gave us a better understanding of Bb for application and hardware resources deployment. Moreover, we now know how our students use and utilize e-learning in their total learning experience which is invaluable information for the planning and promotion of new teaching and learning initiatives.