{'fraction_non_alphanumeric': 0.026270604395604396, 'fraction_numerical': 0.011976304945054946, 'mean_word_length': 5.209221748400853, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
INTEGRATION OF BALANCED SCORECARD AND SIX SIGMA IN MEASURING OPEN UNIVERSITY ACADEMIC SERVICES PERFORMANCE

Performance appraisal is one of the activities carried out by the organization to assess performance quantitatively. One of the performance assessment methods used is Six Sigma and the Balanced Scorecard. Six Sigma is an organizational approach to improve operational excellence, while the Balanced Scorecard provides a framework for transforming organizational strategies into work matrices that help organizations compete. Strategic business plans can be implemented using the Balanced Scorecard (BSC) performance management system approach, while various action programs can be applied using the Six Sigma approach. Both methods can be integrated to synergize in achieving the company’s strategic goals. This study aims to measure the performance of Open University academic services with the integration of BSC and Six Sigma in the field of academic services. Through this integration, performance measurement was focused on quality control by exploring UT’s academic service system as a whole and combined with four perspectives in the Balanced Scorecard. The results showed the level of student satisfaction with UT’s academic services of 90.20% which means that students are very satisfied with UT’s academic services which include programs of distance education and programs, models, teaching materials (modules and non-print teaching materials), face-to-face tutorials and online tutorials, learning assistance counselling services, and learning evaluation. Meanwhile, measurement using Six Sigma shows that UT academic services are at level 3, which means there are still some UT services that are not perfect, especially in tutorial and teaching materials services.


INTRODUCTION
Performance is the work of an organization to realize its strategic objectives, customer satisfaction and contribution to the strategic environment. So that the determination of performance indicators based on the formulation of the strategic plan, vision, mission and organisational goals (Akdon, 2011). To know with certainty whether an organisation performance has been able to achieve the strategic objectives set, a series of performance measurement and evaluation is needed. The primary purpose of implementing a performance measurement system is to improve organizational performance so that it can better serve customers, employees, owners, and stakeholders. So that performance in the organization can be managed properly, it requires performance management. Armstrong (2006) states that: performance management defined as a systematic process for improving organizational performance by developing the performance of individuals and teams.
There are various methods of performance measurement that are applied in organizations, for example, Six Sigma and Balanced Scorecard. Six Sigma is an organizational approach to operational excellence that has existed since it launched at Motorola in the 1980s (Hidayat, 2007). Meanwhile the Balanced Scorecard provides a framework for transforming organizational strategy into a work matrix forward-looking that helps organizations compete. Strategic business plans can be implemented using to management system approach Balanced Scorecard, while various action programs can be implemented using the approach Six Sigma.
The study discusses the integration of Six Sigma and the Balanced Scorecard in measuring the performance of the organization has not so much, especially in educational studies. . Research that discusses the integration of Six Sigma and the Balanced Scorecard was proposed by Heavy and Murphy (2011) which explains the added value of the integration of the Balanced Scorecard with Six Sigma. Other research that discusses the integration of these two methods in the field of education is explained by Holmes, et.al., (2014). Both of these studies provide references and guidelines on how to integrate the Balanced Scorecard and Six Sigma for performance and specific measurements in the world of education.
Performance measurement with the integration of BSC and Six Sigma is very appropriate to be carried out at the Open University besides the performance measurement that has applied so far. This is done so that the Open University can measure UT performance through different perspectives, to obtain additional data on UT performance, especially those relating to student academic services. The implementation of academic services is carried out at the Central UT and 39 Distance Learning Service Units which are spread across 34 provinces and one UT UTB Foreign Service UPBJJ. The services provided are in accordance with standards that have been standardized in the UT Quality Assurance System (SIMINTAS UT) which has accredited, both national and international accreditation. The purpose of this quality assurance is the creation of continuous improvement in academic and non-academic services and will ultimately affect student satisfaction.
Due to the broad reach of the UT working area and the variety of academic services provided, the integration of BSC and Six Sigma needed as one of the measurements of Open University performance in the field of academic services. Through this integration, performance measurement is focused on quality control by exploring UT's overall academic service system and combined with the four perspectives in the Balanced Scorecard. This service is related to the vision and mission of UT and the focus of Academic Quality Development which includes policies regarding educational programs and curriculum, teaching materials, learning assistance services, evaluation of learning outcomes, as well as research and community service.
This research is based on the results of research by Heavy and Murphy (2011) and Holmes, et.al. (2014), but using a different approach. In previous studies, they have discussed how the results of the integration of BSC and Six Sigma, while in this study will begin with the development of Six Sigma and BSC. While the evaluation and measurement of performance with the integration of these two models will be carried out the following year. Russel and Taylor (2006) define Six Sigma as "a process for developing and delivering near products and services". Six Sigma defined as a process for developing near-perfect products and services in order to obtain results that are close to "zero defect". Six Sigma's focus is to prioritize customers by using data to get facts and data to get better solutions. The target of Six Sigma achievement is in three main areas, namely: (1) Improving customer satisfaction; (2) Reducing cycle times; and (3) Reducing defects. Six Sigma quality improvement programs can be implemented with a variety of methodologies. One method commonly used is to use the DMAIC (Define, Measure, Analyze, Improve, and Control) model approach. But there are also other methods that can be used, namely the IDOV method (Identify, Design, Optimize, and Verify) (Gaspersz, 2007). For the Six Sigma method to be implemented with an intensive Six Sigma quality improvement program, it must involve the top-level management intensively.
The Balanced Scorecard (BSC) is a measure of company performance so that companies can measure longterm performance by using indicators and specified benchmarks. The balanced scorecard provides answers to four fundamental questions on four perspectives, namely: (1) Customer and stakeholder perspectives; (2) Financial Perspective; (3) Employee and organization capacity Perspective; and Internal Business Process Perspective. Many organizations use the BSC as a foundation for a strategic management system. The BSC reflects a balance between short-term and long-term goals, financial and non-financial measures, lagging and leading indicators and perspectives on external and internal performance (Hepworth, 1998).
The added value of the BSC is a combination of all major business areas and identifying and clarifying the interrelationships of each perspective to produce success (Hepworth, 1998). This added value allows the company to know the financial condition simultaneously by monitoring progress in building capabilities and obtaining the intangible assets needed for growth (Kaplan and Norton, 2007). Scorecards place strategy and vision at the core of organizational goals, not control (Kaplan and Norton, 1992).

INTEGRATION OF THE BALANCED SCORECARD WITH SIX SIGMA
Based on an understanding of the objectives and workings of Six Sigma and the BSC it can be concluded that Six Sigma provides a structured tool for defining business problems through customer perception, measuring performance baselines, and prioritizing the root causes of implementing solutions and controls. Increasing Six Sigma metrics to a higher level indicates a smaller production error rate. This increase can collectively influence the achievement of the level of performance indicators to a higher level. The combination of Six Sigma with BSC can overcome the weaknesses of BSC in providing solutions and provide problem-solving capabilities for high-level performance metrics. Six Sigma is driven by a deep understanding of customer needs, rigorous use of facts, data and statistical analysis, and diligent attention to managing, improving and re-creating business processes (Pande et.al., 2000). The strength of the BSC is its ability to translate strategies into relevant organizational metrics and performance measures while Six Sigma provides tools and methods to improve performance metrics and achieve the organization's vision and mission. Six Sigma also helps provide problem-solving in achieving performance in a systematic and structured manner.
Performance measurement indicators at BSC are poured into KPI (Key Performance Indicator). But if KPI does not reach the target after performance measurement, then Six Sigma programs consisting of Define, Measure, Analyze, Improve, and Control (DMAIC) stages can use. DMAIC can overcome various problems such as reducing defective products, reducing production costs due to waste, improving product quality, etc. The expected quality target in implementing the Six Sigma methodology is to improve process capability by achieving 3.4 DPMO (defects per one in the process production) 3.4 DPMO means 3.4 defects in 1 million opportunities DPMO is one of the process capability assessments to measure how good a production process is, Alastair (2003) explains that the combination of the balanced scorecard with Six Sigma is a breakthrough in business performance named "Business Improvement System." Ad five components in combining a balanced scorecard with Six Sigma, including (1) Voice of the customer (VOC); (2) Optimal process; (3) Reform management; (4) Project or initiative selection; and (5) Project implementation or initiative. The method for integrating the two management systems is to see the suitability of the two, which begins by translating the DMAIC or IDOV models on Six Sigma and then linking them to the activities on the Balanced Scorecard. The focus of attention is primarily on how the objective components and measures of each Balanced Scorecard perspective are linked to the continuous quality/performance improvement program of Six Sigma, with the DMAIC methodology approach.

METHOD
Development of integration of balanced scorecard and six sigma in measuring the performance of open university academic services developed in this study uses a combination model of the Major Steps in the R&D Cycle (Borg and Gall, 1983) with the Steps of Systems Approach Model of Educational Research and Development (Gall, Joyce & Borg, 2007) in Suparman (2016). At steps 1, steps are carried out (a) research and information collecting in the form of literature reviews, surveys and interviews; (b) planning; and (c) develop preliminary products. Data obtained from the results of the questionnaire for BSC analysis were processed with the Customer Satisfaction Index (CSI) and Importance Performance Analysis (IPA). While processing six sigma uses Pareto analysis, fishbone and process capability report. Key Performance Indicators (KPI) were developed to measure the performance of distance education services (Alstete and Beutell, 2004), Griggs and Smith (2012), Mary and Santovec (2004).

Respondent
Retrieval of development integration of balanced scorecard and six sigma research data in measuring the performance of open university academic services in step 1 conducted for UT students with a survey method using a questionnaire. The focus of the study compiled with the Key Performance Index (KPI) Parmenter(2010), Alstete & Beutell (2004), Powar, KB, Panda, Santosh., Bhalla (2000) on six service groups consisting of: (1) study programs and distance learning ; (2) registration; (3) teaching material; (4) face to face tutorial and online tutorial; (5) counselling services and learning assistance; and (6) evaluation learning.
The research sample was UT students from 13 UT Regional Offices in Indonesia, with a total sample of 370 students. Samples were taken by purposive sampling method by considering the representation of the regions of Western Indonesia, Central Indonesia and Eastern Indonesia. Most of the respondents are S1 Basic Education (PGSD and PAUD students (Figure 1).

Analysis Method
Analysis of student satisfaction with Open University academic services quantitatively using Importance-Performance Analysis (IPA) (Lewis, 2004) and Customer Satisfaction Index (CSI) (Fandy Tjiptono and Gregorius Chandra, 2011) analysis tools. IPA is used to determine the gap between performance and expectations of service products and the CSI which is used to analyze the overall level of customer satisfaction (Handi Irawan, 2004;Nigel Hill, Self Bill, 2002;Ilieska, 2013). The results of the analysis with IPA and CSI explained in Table 1and Figure 2.   Table 1 shows that the level of student satisfaction with UT academic services in the six service groups is very high (90.16%). This table in line with the performance analysis index (Figure 2) which shows that 54% of UT academic service performance is in Quadrant B, which means that most of the academic services provided are in line with student expectations. 6.3% performance of UT academic services is in quadrant A, which means this service is considered very important for students, but its performance is still low. 29.2% of the performance services of UT academics assessed by students as being in Quadrant C, which means that the service is not too important for students and has low performance. 10.4% of service performance is considered not necessary by students and has reasonably high performance (quadrant D).

Balanced Scorecard Analysis
The performance analysis index shows that 46% of the performance of the services provided is not in line with student expectations, so the service must be improved. Meanwhile, services that considered not important should be reviewed. UT should focus on improving the performance of academic services that are important by students.
Measurement with BSC shows that KPI determined by UT was not reached 100% so that further analysis needs to be done using Six Sigma. The Six Sigma program consists of Define, Measure, Analyze, Improve, and Control (DMAIC) stages. DMAIC application can overcome various problems in the production process such as reducing defective products, reducing production costs due to waste, improving product quality and so on. Meanwhile in academic services, it is hoped that the DMAIC application can be used to analyze service lags, enhance the quality of service, reduce service costs, and improve academic service quality. So the combination of BSC with six sigma can provide a breakthrough in UT academic service performance (Alastair, 2003).

Six Sigma Analysis
Six sigma compiles the UT academic service mapping process and compares the level of importance with performance. Each academic service group is assessed through the DMAIC stages. Next, an analysis of six academic service groups with fishbone was conducted.  The analysis shows that the results of UT's academic services are "the quality of distance education". Whereas the six academic service groups are factors that will influence these results. Fishbone analysis is used to help identify, sort, and display various causes of UT academic services that are not in line with the performance expected by students and cannot reach the overall KPI.
Fishbone analysis continued with Pareto chart analysis. The purpose of the Pareto chart is to clarify the most important factors of several influencing factors. In quality control, this often represents the source of defects that are most frequently encountered, the types of defects that occur most often, or the reasons that most commonly occur.  When there are complaints from consumers, based on the results of the Pareto chart in the study program, it appears that nearly 80% of academic service problems occur in students of Basic Education (PGSD and PAUD). This is because 55.62% of UT students are students of basic education study programs https:// www.ut.ac.id/ut-in-angka. So, if UT wants to improve academic service performance and achieve the KPI that has been set to 80%, focus on improving academic services for students in basic education (Figure 4). Improvement of services for Basic Education students will have a very significant impact on improving services for Ut students as a whole.
Meanwhile, an increase in the performance of academic services that are considered important by students is also almost 60% also experienced by PGSD students and PAUD students and management study program students ( Figure 5).  i P e m b a n g u n a n A k u n t a n s i M a n a je Improvement of UT services in services considered important by students is focused on three study programs consisting of PGSD, PGPAUD and Management. The effect of improving service quality for three students in the study program will greatly affect the percentage of UT academic improvement by 80%. The effect of this policy is very significant because of the large number of students served in PGSD, PGPAUD and Management study programs.   Figure 6 shows that 74.9% of UT's academic service problems are dominated by problems in (a) study program and distance learning; (b) face to face and online tutorials; (c) teaching material; and (d) counselling services and learning assistance. The results of the analysis with the Pareto chart in the learning service group showed a significant problem that occurred in the study group and distance learning service groups. Although in the BSC analysis the level of student satisfaction reached 91.05%, only 45% of the types of services were considered to have good performance by students. 18% of service types are considered not so important by students and have low performance. While 27% of the types of services, despite having high performance but are not considered important by students. 9% of the types of services in this group are considered important and require increased performance. Process Capability Report for Rata-rata Figure 7. Process capability report for rata-rata The subsequent analysis of the Six Sigma method is the analysis of process capability. Process capability is the ability of a process to produce a product/service in accordance with the needs/requirements of consumers or the specifications expected. Capability analysis is also used to find out a process running capable and produce products/services according to its specifications. The expected quality target is to increase process capability by reaching 3.4 DPMO in the production process. Figure 7 shows that the value of z.bench is 0.68, which means that the z.bench is still below 1. The value is below sigma capability 2.5σ or 1.7 sigma. By looking at the comparison in Table 2, it can be seen that the performance of UT academic services is still not satisfactory. With a sigma level below 1, there are 70% of UT academic services that still need to be improved, because these services are not in accordance with the specified KPIs. Figure 7 also shows Cpk <1, so the process is said to be not capable and there needs to be an improvement in the process. The high level of errors in the delivery of academic services causes low student satisfaction.

DISCUSSION AND CONCLUSION
This research is preliminary research to build an integration between BSC and Six Sigma in improving academic service performance in distance education. The results of the analysis of UT academic services with BSC and Six Sigma show that the performance of UT academic services still needs to be improved. CSI analysis shows the level of student satisfaction is very high (90.16%), but the science analysis revealed 46% of the services provided were not in line with student expectations. This fact means that there are still many UT services that although they have high performance, they are not considered important services for students. The BSC has translated the organization's strategy into relevant organizational metrics in the form of Key Performance Indicators (KPI) and made measurements on the perspective of the BSC (Kaplan, 2001;Parmenter, 2010).
Analysis with Six Sigma states that UT's academic service level is at the level of sigma ability 2.5 σ or sigma level below 1 (one). This analysis means that 70% of UT's academic services must be improved by the specified KPIs. Analysis of six sigma by four stages Define, Measure, Analyze, Improve and Control (DMAIC) shows that services academic need attention, especially in (a) courses and distance learning; (B) face to face and online tutorials; (c) teaching materials; and (d) counselling and learning assistance services (Alastair, 2003). The Pareto analysis shows that service improvement must first be provided primarily for academic services in FKIP Basic Education students.
Measurement of organizational performance based on BSC analysis shows that some academic services still need improvement. Open Universities can choose service improvements that allow the most efficient use of time and resources and provide significant results for universities (Holmes, et.al., 2014). The six sigma approach provides detailed steps about what the organization must do to achieve the established performance (Pande et al., 2000). An increase in Six Sigma metrics to a higher level indicates a smaller level of a production error and an increase in production quality. If related to BSC, this increase can collectively influence the achievement of level indicators to a higher level.
Similar to the results of research conducted by Heavy and Murphy (2011) and Holmes (2014) the merging of BSC and six sigma performance measurement is very effective to describe the performance of Open University academic services. The BSC is an effective tool for translating strategies into high-level performance while six sigma provides the ability for organizations to control performance improvements systematically and measurably. The combination of BSC with Six Sigma will articulate and execute business strategies and will act as a platform for business excellence and continuous improvement (Heavy and Murphy (2011).