EMUSIM: Integrated Emulation and Simulation for Evaluation of Cloud Computing Applications
Introduction
Different evaluation methodologies can be used to evaluate concepts, algorithms, hypothesis, and other elements in the context of Computer Science. The most popular methods include:
- Analytical Modelling. In this method, the object of evaluation is modelled following a specific formal method. Popular formalisms are Markov Chains, Queueing Networks Petri Nets. We include Model Checking in this category as well. The goal here is proving that the object of evaluation is correct given some formal definitions or that it has certain properties. Objects being evaluated in this method are very abstract, so it is more suitable for evaluation of algorithms and protocols rather than implemented programs.
- In situ. In this method, real artefacts (like a running program) runs in a real infrastructure subject to real software management. This method is appropriate when all the elements to be evaluated are available. However, technical issues (such as bugs, hardware failures, etc) can make experiments using this method longer than experiments using other methods.
- Emulation. In this method, real software artefacts are evaluated in models of hardware artefacts. It makes the experimental platform more controllable and experiments more reproducible than in situ methods.
- Benchmarking. In this method, a model of software artefact (such as an application model or a software developed for evaluation purposes) is submitted to the hardware artefacts. This method is typically used for hardware evaluation.
- Trace-based Simulation. In this method, software and hardware are models. Input of the software is based on traces obtained from real systems under similar conditions than the simulated one.
- Model-based Simulation. This method is similar to the above one, with the difference that the input is based on statistical models extracted from one or more traces.
- Interview/Assumptions/Manual Input. These methods are based on human interaction and are used for evaluation of qualitative aspects of systems, as commonly found in Information Systems research.
A good further reading in this subject is: Jens Gustedt, Emmanuel Jeannot, Martin Quinson. Experimental Methodologies for Large-Scale Systems: a Survey. Parallel Processing Letters 19(3):399-418. World Scientific, Sep. 2009. DOI:10.1142/S0129626409000304.
EMUSIM
EMUSIM combines emulation (AEF) and simulation (CloudSim) to enable more accurate models of software artefacts (obtained via profiling during emulation) to be used during simulations. This is especially useful when the tester has no idea on the performance of the software under different levels of concurrency and parallelism, which impedes utilization of simulation. These can replace in situ experiments when such experiments would require a scale that is either unavailable for the tester or too expensive to run in a public Cloud.
System Requirements
In order to be able to use EMUSIM, you will need to have knowledge on the following software technologies:
- Linux
- Java
- Xen
- CloudSim
You will also need access to a cluster whose nodes run Xen's dom0. The cluster frontend can be either a Linux host or a VM running Linux.
Installation and Configuration
To install EMUSIM, simply extract the file below to your directory of choice in the cluster frontend. For configuration, follow the below steps:
- Edit the file cluster.xml to include description of your cluster. Following the syntax described in the page 4 of the "AEF tutorial" included in the package. Basically, you have to describe the cluster's netowrk connections, and each host specification (CPU power, disk, memory, IP and MAC addresses).
- Create a XML, with the same syntax above, to describe your emulated environment (i.e., number and characteristics of the VMs), as described in page 5 of the tutorial.
- Edit the files aef.properties and emusim.properties to include information about the specific emulation to be performed.
- Make sure you have a VM image that is pre-configured for the application to be executed, and include the qAppDeployer (in the lib directory of the EMUSIM package). The path to the image has to be included in the emusim.properties file.
- Also make sure that the VMs and the cluster nodes are configured to be accessible via ssh from the frontend without password (instructions on how to enable it are found in the tutorial).
- Define your CloudSim simulation parameters in the file simulation.properties.
- Execute ./runemusim.sh to start the emulation and emulation process. At the end of the emulation stage, a file system.properties will be generated. It contains the profiling information from the application and can be reused in later simulation experiments.
- If you are familiar with CloudSim, you can modify the simulation code to include any behaviour you would like to evaluate, change workload pattern, etc. you don't have to run the (time-demanding) emulation again for this application, as all the information that EMUSIM requires for the simulation is stored in the system.properties file.
