The Performance Focus
September - October | July - August | May - June
Welcome to “The Performance Focus”, the regular online round-up of news, technical tips and events of interest to the community at large from Enteros.
In this issue we have included a short article by Hanan Hit our Director of Customer Support. The complete article appears in the NoCOUG Journal August Issue Please contact me and let me know what additional features you would like to see as part of the Performance Focus. Paul.firstname.lastname@example.org
Human factors effect successful change
The productivity of knowledge work– still abysmally low – will become the economic challenge of the knowledge society. This is dependent on the competitive position of every single country, every single industry, and every single institution within society.
- - Peter Drucker
Lately I started to study the concepts and policies recommended by the Information Technology Infrastructure Library (ITIL), and I was amazed that many organizations are still inefficient and ineffective. Rather than analyzing risks organizations tend to avoid them altogether based on technical assumptions which have not been validated, preferring intuitions over methodical research. Planning is often inadequate due to lack of knowledge and the outcome is IT projects which cost more and deliver less than expected. Successful projects result from application of methodical problem solving skills based on deep understanding of how the systems being deployed function. Such problem solving skills include, among others, the ability to identify and eliminate database bottlenecks, and the ability to develop efficient SQL statements. Adequately preparing for an Oracle project greatly increases the chance of success and reflects positively on the organization as well as our career. Preparing and planning will help. Successful project boost the contributor's career both directly and by contributing to the organization's revenue, and improving its competitive advantage.
Director of Customer Support Enteros
Enteros announces Load 2Test for EC2 is available immediately and its unique technology can dramatically reduce the cost of load testing for organizations of any size, ensuring optimal performance for enterprise systems. This latest innovation from Enteros removes the need to maintain array of load test servers and the hardware, power and infrastructure currently required, providing fast ROI and adoption of the EC2.
Load2Test for EC2 greatly reduces the cost of creating and executing tests with semi-automated load tests based on a TCP protocol with a base price of four dollars (U.S.) per instance per hour and ability to support thousands of concurrent users on each instance. Load2Test represents the most advanced cloud computing test platform currently available, enabling load testing for both Web and Ajax user interfaces and applications, including service oriented architectures. In addition Load2Test provides support for backend infrastructure components, such as databases, messages buses and application servers in a VPN environment, integrating cloud and internal operational components.
Cloud computing platforms are moving beyond simple Web hosting applications, becoming viable replacements for internally hosted business applications and operations infrastructures,” said Ron Warshawsky, CTO, Enteros. “Our Load2Test for EC2 reduces the need to perform certain types of maintenance that are strategically important for business-critical systems and provides seamless integration between the cloud and existing internal systems, ensuring that the systems function according to internal and external service level agreements.”
With the today's dynamic nature of Web and enterprise computing, organizations find it difficult to build realistic load tests and support the required server farms and infrastructure components. Applications are often improperly tested with load simulations that are not representative of real production activity. This on demand cloud computing solution from Enteros, as well as future software-as-a-service (SaaS) offerings, allows customers to quickly create user friendly infrastructure that manages load testing group activity, analyzes load tests bottlenecks, generates load tests reports and establishes labs to operate load testing software. Enteros Load2Test Framework now available for Microsoft® ADO.NET users
Enteros announces the extension of support for its Load2Test 2.0 Framework to ADO.NET. This latest addition to the Enteros load testing offering comes in both 32- and 64-bit capacity, enabling organizations additional supporting technology to improve server performance and meet the stringent SLAs for critical business line applications.
With the new functionality in Load2Test, connections to the database directly are no longer needed as it can communicate within the middle tier of ADO.NET calls from applications to data sources. When bottlenecks and performance degradation are found, Load2Test can easily and efficiently find the location of the ADO.NET calls and identify them for immediate remediation, ensuring little to no down time. Customers using the .NET Framework for its application development platform can now have the scalability for testing in both the staging and production portions of the development lifecycle. Unlike other load testing solutions, Load2Test uses real production data that provides a complete picture of overall system performance.
ADO.NET is a set of computer software components that can be used by programmers to access data and data services from sources such as Microsoft® SQL Server as well as those exposed through OLE DB and XML. It is a part of the base class library that is included with the Microsoft .NET Framework and is commonly used by programmers to access and modify data stored in relational database systems as well as non-relational sources. Users of ADO.NET can not only connect to a database, but can also execute commands and retrieve, manipulate and update data.
“Load testing continues to be a key indicator for future performance of critical business systems,” said Ron Warshawsky, CTO, Enteros. “The Microsoft .NET Framework is one of the leading development platforms for high performance production data and applications. Providing ADO.Net support ensures that investments made in this modern architecture support business lines and meet the rigorous demands placed on today's systems.”
Northern California Oracle User Group (NoCOUG), was held on August 20, Chevron, San Ramon, CA.
We featured our Enterprise Production Performance Management Solutions for Oracle. Many of the attendees visited with our team and learned how our solutions helped to make them more proficient and agile while leveraging their knowledge for developing more capabilities for the following : fine-grain, multi-application performance problem identification, real-time performance problem remediation of databases, advanced performance analysis for root cause, change impact and scalability of databases and advanced performance analysis for root cause, change impact and scalability of multi-tiered applications. Every visitor was given a pin number to download a free copy of our performance offerings. To receive your pin number please contact Enteros at 1 (408) 824-1292 and ask for sales
Increase Agility and Boost Production Performance! Gain Awareness. Perform Analysis. Remove Risk.
Join prior eBay production SWAT team principal Ron Warshawsky, Founder and CTO of Enteros, and Hanan Hit, Director of Customer Support at Enteros, and President of Northern California Oracle User Group, to learn how to solve threatening production performance issues. You will get insight into the strategies that top organizations use today to deal with critical performance matters quickly and before they impact business. Professionals in organizations that depend on the peak performance of business systems should not miss this event.
You will learn:
• Methods and best practices for a complete approach to enterprise performance management
• What types of analysis can be used to uncover production issues and what the results mean
• Where to look for signs of accumulating performance problems and what preventive measures work
To view webcast go to www.enteros.com
The Tip - How to
use Linux magic SysRq Key in order to simulate kernel panic directly from
Enteros Inc. Load2Test suite.
The magic SysRq key is a key combination in the Linux kernel which allows the user to perform various low level commands regardless of the system's state using the SysRq key. It is often used to recover from freezes, or to reboot a computer without corrupting the filesystem. To be able to use this functionality the CONFIG_MAGIC_SYSRQ option has to be enabled at kernel compile time. Much like Sun Microsystems's Open Firmware (OpenBoot), this key combination provides access to powerful tools for software development and disaster recovery. In this sense, it can be considered a form of escape sequence. Principal among the offered commands are means to forcibly unmount file systems, kill processes, recover keyboard state, and write unwritten data to disk. With respect to these tasks, this feature serves as a tool of last resort. The reader will be presented with detailed instructions on the required setup (if needed), and the way to simulate a kernel panic directly from Load2Test suite.
To enable the “magical” sysrq, you need to have it compiled in the kernel. Luckily all major Linux distribution today has sysrq compiled in by default. To see the status if sysrq, issue:
By default this value is "1" on Debian/Ubuntu and "0" on RHEL. "0" disables sysrq and "1" enables all functions of sysrq.
You might also use "sysctl to check and enable sysrq:
# sysctl kernel/sysrq
kernel.sysrq = 0
# sysctl –w kernel/sysrq=1
kernel.sysrq = 0
As we would like to use the sysrq from Load2Test we obviously are not capable to use the console. In order to crash the system we just need to redirect the command-key to /proc/sysrq-trigger. So to crash the running server we should do:
# echo “c” > /proc/sysrq-trigger
Load testing will never be executed with “root” credentials and the above command has to be executed with the “root” credentials so we have to use the good and old “sudo” capabilities. So what we had done is to add the following under /etc/sudoers
Below are the major two sections that will cause the Linux server to crash:
1. Setup of the credentials while using dummy command as sudo
2. echo “c” > /proc/sysrq-trigger | sudo /bin/sh (Pipe the command to sudo usage).
*** Remark – Load2Test is using XML parser and as such we had to escape the “ with regular escape characters.
The double-quote character (") can be symbolised with this character entity reference when you need to embed a double-quote inside a string which is already double-quoted.
As can be seen in figure 3 we had chosen to crash the server after 10 sec by setting the “time_start_exact” to 10,000 ms.
Once the load test execution had reached the 10sec timeframe one will notice the below valid errors on the Load2Test TestNode which symbols that the connection was lost to the Linux server.
Enteros Inc. customers can make usage of simple Linux sysadmin API's and Enteros Inc. Load2Test suite, in order to simulate various production failures that will ensure that the cluster depending on where the failure has occurred will continue to function.
During our testing we had noticed a bizarre behavior from the VMware that seems to be a bug.
Once crashing the system it was noticed that the system UI (if used) is still intact and the “SCROLL LOCK” is alternating between “OFF” and “ON'
Load2Test iteration 5 significantly revamps Load2Test User Interface - from the look of the scenario text editor and to the way the real-time load test data is displayed to the user. Iteration 5 establishes Load2Test as a more powerful flow recording tool, with added support for many more web protocols such as pop3 and jdbc. Additionally, flow playback has been made more versatile through added support for .xml agent configuration file, which allows the user to pass additional parameters to the databases, as well as configure output and rt logging independently. Furthermore, iteration 5 continues to build upon the report generation functionality. Formerly introduced multipane_chart and multipane_chart_overview tags have been expanded to support data smoothing, and typecasting. Plus, graphing of transaction data on sequence and flow levels has been added, and support for regular expressions has been implemented for XY_chart tag. Minor changes have been applied to the pfc data charting mechanism, which allows for data with the same pfcname, or pfcfield to be displayed on the same grid.
The new user interface gives Load2Test a cleaner look, through the introduction of dual views – Design and Execute. The UI redesign allows for a much larger viewing area of real-time charts. Also, the ability to customize the look of the scenario text editor by setting custom background color, font, and text size has been added as part of preference configuration. The introduction of the additional protocols virtually doubles the number of protocols supported in previous iteration. Accompanying the Load2Test iteration 5 distributions is a collection of examples which document and demo the usage of the added protocols.
In iteration 5, agent configuration has been transitioned to XML file, versus a formally used text file. While the text file configuration is still supported for backward capability purposes, XML configuration offers a lot for versatility to the user. User of XML configuration file is able to specify several databases and identify additional parameters besides username and password.
The introduction of multipane_chart and multipane_chart_overview tags are expanded to include the window attribute. Configuring the window attribute allows the user to specify the number of data points to be used in calculating a moving average for the charted line. This makes the line appear smoother and helps filter out the “noise”.
Source typecasting has been introduced to help manage data when the output contains tens or hundreds of sources. Source type is specified in the scenario file before a load test is launched. This triggers the controlled to assign a user specified type to all data which relates to the source. After the load test completes, a user is able to refer to a collection of sources by the type, instead of having to list them one by one: source=" type::Type
In addition to being able to chart Performance Capturer statistics and system performance data, the reports generating functionality now has the ability to add sequence, flow, and thread info to the charts. The granularity of displayed data is controlled by the level attribute which has been added to multipane_chart_overview, multipane_chart, and XY_chart tags. The level attribute can be set to “SYSTEM”, “SEQUENCE”, “FLOW”, and “THREADINFO”, and the default if the level attribute is omitted is system level.