summaryrefslogtreecommitdiffstats
path: root/documentation/writing_your_first_test.html
blob: 9b1253046231ada63bda10370c68f4c9cd54dc04 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<Title>Phoronix Test Suite - A Quick Guide To Writing A Test Profile</Title>
<link href="includes/pts-documentation.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="pts_doc_header"><div class="pts_doc_fixed"><a href="http://www.phoronix-test-suite.com/"><div id="pts_doc_logo"></div></a></div></div>
<div class="pts_doc_fixed">
<div class="pts_doc_notice"><div style="float: left"><a href="index.html">&lt;&lt; Documentation Home</a></div><div style="float: right;">Writing A Test Profile</div></div>
<div class="pts_doc_main">
<!-- PTS AREA -->
<h1>Writing A Test Profile</h1>
<p>Writing a test profile for the Phoronix Test Suite is a relatively quick and easy 
process for anyone familiar with common Linux commands and the basics of XML. 
To help you understand the design of the Phoronix Test Suite, this guide covers 
the steps needed to write a testing profile for a very simple application.</p>
<p>The first step in the profile writing process is to, well, have a piece of 
software you'd like to use with the Phoronix Test Suite. This software can be 
closed-source or open-source and be virtually anything as long as it is compatible 
with the Linux operating system.</p>
<p>For this guide, the piece of software being used for demonstration is just 
a simple C++ program that calculates Pi to 8,765,4321 digits using the Leibniz 
formula. Below is this sample piece of software intended just for demonstration 
purposes.</p>
<blockquote>#include &lt;iostream&gt;<br>
#include &lt;math.h&gt;</p>
<p>int main()<br>
{<br>
double pi = 0;</p>
<p> for(long int i = 1; i &lt;= 87654321; i++)<br>
pi += (double) pow(-1, i + 1) / (2 * i - 1);</p>
<p> pi *= 4;<br>
std::cout &lt;&lt; &quot;Done Calculating Pi...&quot; &lt;&lt; endl;<br>
return 0;<br>
}</blockquote>
<p>The first step in the actual profile writing process is to name it. If you're 
looking to ultimately push this profile to be included in the Phoronix Test Suite, 
its name must be all lower case and consist of just alpha-numeric characters, 
but can contain dashes (-). For this sample profile, we're calling it <em>sample-program</em> 
and the file-name would be <em>sample-program.xml</em>. Our (very basic) profile is showcased 
below.</p>
<blockquote>&lt;PhoronixTestSuite&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;TestProfile&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;Version&gt;1.0&lt;/Version&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;TestType&gt;Processor&lt;/TestType&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;SoftwareType&gt;Utility&lt;/SoftwareType&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;License&gt;FREE&lt;/License&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;Status&gt;PRIVATE&lt;/Status&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;Maintainer&gt;Phoronix Media&lt;/Maintainer&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;/TestProfile&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;TestInformation&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;Title&gt;Sample Pi Program&lt;/Title&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;TimesToRun&gt;3&lt;/TimesToRun&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;ResultScale&gt;Seconds&lt;/ResultScale&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;Proportion&gt;LIB&lt;/Proportion&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;Description&gt;A simple C++ program that calculates Pi to 8,765,4321 digits 
using the Leibniz formula.&lt;/Description&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;ExternalDependencies&gt;build-utilities&lt;/ExternalDependencies&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;/TestInformation&gt;<br>
&lt;/PhoronixTestSuite&gt;</blockquote>
<p>This XML profile is what interfaces with the Phoronix Test Suite and provides 
all the needed information about the test as well as other attributes. For a complete 
listing of all the supported profile options, look at the specification files 
in the documentation folder. In the case of <em>sample-program</em>, it lets the Phoronix 
Test Suite know that it's composed of free software, is designed to test the processor, 
is intended for private use only, and this profile is maintained by Phoronix Media. 
In addition, it tells the Phoronix Test Suite to execute this program three times 
and as no result quantifier is set, the average of the three runs will be taken. 
This profile also tells the Phoronix Test Suite that the generic <em>build-utilities</em> 
package is needed, which will attempt to ensure that GCC and the standard development 
utilities/libraries are installed on your Linux distribution. This is needed as 
the C++ source-code will need to be built from source.</p>
<p>The next step is to write the <em>install.sh</em> file, which once called by the Phoronix 
Test Suite is intended to install the test locally for benchmarking purposes. 
The <em>install.sh</em> file is technically optional, but is generally used by all tests. 
Alternatively, the file may be named <em>install.php</em> if you wish to write the install 
script in PHP instead of a shell script. Note: The first argument supplied to 
the install script is the directory that the test needs to be installed to. The 
<em>install.sh</em> file (in our instance) is to be placed inside <em>pts/test-resources/sample-program</em>. 
Below is the <em>install.sh</em> for the <em>sample-program</em>.</p>
<blockquote>#!/bin/sh<br /><br />
tar -xjf sample-pi-program-1.tar.bz2<br />
g++ sample-pi-program.cpp -o sample-pi-program<br />
echo &quot;#!/bin/sh<br />
time -f \&quot;Pi Calculation Time: %e Seconds\&quot; ./sample-pi-program 2&gt;&amp;1 
| grep Seconds&quot; &gt; sample-program<br>
chmod +x sample-program</blockquote>
<p>This install file builds the code with GCC, and then creates a small script 
that is run by the Phoronix Test Suite, which times how long it takes to run the 
software. Where does the source-code come into play? Well, it needs to be downloaded 
now from a web server. The Phoronix Test Suite has built-in support for managing 
downloads from multiple servers in a random over, fall-back support if one mirror 
is done, and verification of MD5 check-sums. Below is the <em>downloads.xml</em> file for 
<em>sample-program</em> that covers all of this.</p>
<blockquote>&lt;PhoronixTestSuite&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;Downloads&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;Package&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;URL&gt;http://www.phoronix-test-suite.com/benchmark-files/sample-pi-program.cpp&lt;/URL&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;MD5&gt;e90fb790df8d1544696a1439c9b5bd8d&lt;/MD5&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;/Package&gt;<br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;/Downloads&gt;<br>
&lt;/PhoronixTestSuite&gt;</blockquote>
<p>The final step in the profile writing process is to write a parser to strip 
all information but the reported result from the standard output. The standard 
output is submitted to <em>parse-results.sh</em> or <em>parse-results.php</em> as the first argument 
in quotes. In the case of <em>sample-program</em>, all that's done inside parse-results.php 
is:</p>
<blockquote>&lt;?php<br>
$BENCHMARK_RESULTS = substr($argv[1], strrpos($argv[1], &quot;Pi Calculation Time:&quot;) 
+ 20);<br>
echo trim(substr($BENCHMARK_RESULTS, 0, strpos($BENCHMARK_RESULTS, &quot;Seconds&quot;)));<br>
?&gt;</blockquote>
<p>After that, with all the files in their correct locations, just run: <em>phoronix-test-suite 
benchmark sample-program</em>. The Phoronix Test Suite should now handle the rest by 
installing the test, running the test, and recording the results (if you so choose). 
There is no additional work that needs to be done for the results to be recorded 
in the PTS Results Viewer or even reporting the results to Phoronix Global.</p>
<p><strong>For more information, visit <a href="http://www.phoronix-test-suite.com/">Phoronix-Test-Suite.com</a> and read the <a href="index.html">included documentation</a>. It should also be of great help to just look at the existing test profiles, which can be found inside the <em>phoronix-test-suite/pts/</em> folder with the <em>test-resources</em> and <em>test-profiles</em> sub-directories.</strong></p>
<!-- END OF PTS AREA -->
</div></div><div class="pts_doc_fixed"><div class="pts_doc_bottom"><div style="float: left;"><a href="http://www.phoronix-test-suite.com/">Phoronix-Test-Suite.com</a></div><div style="float: right;">Copyright &copy; 2008 by <a href="http://www.phoronixmedia.com/">Phoronix Media</a>.</div></div></div>
</body>
</html>