<!DOCTYPE html>
<html lang='en'>
<head>
<title>kernel-crypto.git - User-space crypto API development for Linux</title>
<meta name='generator' content='cgit '/>
<meta name='robots' content='index, nofollow'/>
<link rel='stylesheet' type='text/css' href='/cgit-data/cgit.css'/>
<link rel='shortcut icon' href='/favicon.ico'/>
<link rel='alternate' title='Atom feed' href='https://fedorapeople.org/cgit/mitr/public_git/kernel-crypto.git/atom/include/linux/slob_def.h?h=ncr-integration' type='application/atom+xml'/>
<link rel='vcs-git' href='git://fedorapeople.org/home/fedora/mitr/public_git/kernel-crypto.git' title='kernel-crypto.git Git repository'/>
<link rel='vcs-git' href='ssh://fedorapeople.org/home/fedora/mitr/public_git/kernel-crypto.git' title='kernel-crypto.git Git repository'/>
<link rel='vcs-git' href='https://fedorapeople.org/cgit/mitr/public_git/kernel-crypto.git' title='kernel-crypto.git Git repository'/>
</head>
<body>
<div id='cgit'><table id='header'>
<tr>
<td class='logo' rowspan='2'><a href='/cgit/'><img src='/cgit-data/cgit.png' alt='cgit logo'/></a></td>
<td class='main'><a href='/cgit/'>index</a> : <a title='kernel-crypto.git' href='/cgit/mitr/public_git/kernel-crypto.git/'>kernel-crypto.git</a></td><td class='form'><form method='get'>
<input type='hidden' name='id' value='9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'/><select name='h' onchange='this.form.submit();'>
<option value='audit'>audit</option>
<option value='bugfixes'>bugfixes</option>
<option value='herbert'>herbert</option>
<option value='integration'>integration</option>
<option value='master'>master</option>
<option value='ncr-audit'>ncr-audit</option>
<option value='ncr-integration' selected='selected'>ncr-integration</option>
<option value='ncr-rhel5-patches'>ncr-rhel5-patches</option>
<option value='ncr-standalone-master'>ncr-standalone-master</option>
<option value='ncr-standalone-rename'>ncr-standalone-rename</option>
<option value='ncr-submission-2'>ncr-submission-2</option>
<option value='ncr-userspace-crypto'>ncr-userspace-crypto</option>
</select> <input type='submit' value='switch'/></form></td></tr>
<tr><td class='sub'>User-space crypto API development for Linux</td><td class='sub right'>Miloslav Trmac</td></tr></table>
<table class='tabs'><tr><td>
<a href='/cgit/mitr/public_git/kernel-crypto.git/'>summary</a><a href='/cgit/mitr/public_git/kernel-crypto.git/refs/?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>refs</a><a href='/cgit/mitr/public_git/kernel-crypto.git/log/include/linux/slob_def.h'>log</a><a href='/cgit/mitr/public_git/kernel-crypto.git/tree/include/linux/slob_def.h?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>tree</a><a class='active' href='/cgit/mitr/public_git/kernel-crypto.git/commit/include/linux/slob_def.h?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>commit</a><a href='/cgit/mitr/public_git/kernel-crypto.git/diff/include/linux/slob_def.h?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>diff</a><a href='/cgit/mitr/public_git/kernel-crypto.git/stats/include/linux/slob_def.h'>stats</a></td><td class='form'><form class='right' method='get' action='/cgit/mitr/public_git/kernel-crypto.git/log/include/linux/slob_def.h'>
<input type='hidden' name='id' value='9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'/><select name='qt'>
<option value='grep'>log msg</option>
<option value='author'>author</option>
<option value='committer'>committer</option>
<option value='range'>range</option>
</select>
<input class='txt' type='search' size='10' name='q' value=''/>
<input type='submit' value='search'/>
</form>
</td></tr></table>
<div class='path'>path: <a href='/cgit/mitr/public_git/kernel-crypto.git/commit/?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>root</a>/<a href='/cgit/mitr/public_git/kernel-crypto.git/commit/include?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>include</a>/<a href='/cgit/mitr/public_git/kernel-crypto.git/commit/include/linux?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>linux</a>/<a href='/cgit/mitr/public_git/kernel-crypto.git/commit/include/linux/slob_def.h?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>slob_def.h</a></div><div class='content'><div class='cgit-panel'><b>diff options</b><form method='get'><input type='hidden' name='id' value='9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'/><table><tr><td colspan='2'/></tr><tr><td class='label'>context:</td><td class='ctrl'><select name='context' onchange='this.form.submit();'><option value='1'>1</option><option value='2'>2</option><option value='3' selected='selected'>3</option><option value='4'>4</option><option value='5'>5</option><option value='6'>6</option><option value='7'>7</option><option value='8'>8</option><option value='9'>9</option><option value='10'>10</option><option value='15'>15</option><option value='20'>20</option><option value='25'>25</option><option value='30'>30</option><option value='35'>35</option><option value='40'>40</option></select></td></tr><tr><td class='label'>space:</td><td class='ctrl'><select name='ignorews' onchange='this.form.submit();'><option value='0' selected='selected'>include</option><option value='1'>ignore</option></select></td></tr><tr><td class='label'>mode:</td><td class='ctrl'><select name='dt' onchange='this.form.submit();'><option value='0' selected='selected'>unified</option><option value='1'>ssdiff</option><option value='2'>stat only</option></select></td></tr><tr><td/><td class='ctrl'><noscript><input type='submit' value='reload'/></noscript></td></tr></table></form></div><table summary='commit info' class='commit-info'>
<tr><th>author</th><td>David S. Miller &lt;davem@davemloft.net&gt;</td><td class='right'>2009-06-15 03:02:23 -0700</td></tr>
<tr><th>committer</th><td>David S. Miller &lt;davem@davemloft.net&gt;</td><td class='right'>2009-06-15 03:02:23 -0700</td></tr>
<tr><th>commit</th><td colspan='2' class='sha1'><a href='/cgit/mitr/public_git/kernel-crypto.git/commit/include/linux/slob_def.h?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb</a> (<a href='/cgit/mitr/public_git/kernel-crypto.git/patch/include/linux/slob_def.h?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>patch</a>)</td></tr>
<tr><th>tree</th><td colspan='2' class='sha1'><a href='/cgit/mitr/public_git/kernel-crypto.git/tree/?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>8d104ec2a459346b99413b0b77421ca7b9936c1a</a> /<a href='/cgit/mitr/public_git/kernel-crypto.git/tree/include/linux/slob_def.h?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>include/linux/slob_def.h</a></td></tr>
<tr><th>parent</th><td colspan='2' class='sha1'><a href='/cgit/mitr/public_git/kernel-crypto.git/commit/include/linux/slob_def.h?id=ca44d6e60f9de26281fda203f58b570e1748c015'>ca44d6e60f9de26281fda203f58b570e1748c015</a> (<a href='/cgit/mitr/public_git/kernel-crypto.git/diff/include/linux/slob_def.h?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb&amp;id2=ca44d6e60f9de26281fda203f58b570e1748c015'>diff</a>)</td></tr><tr><th>parent</th><td colspan='2' class='sha1'><a href='/cgit/mitr/public_git/kernel-crypto.git/commit/include/linux/slob_def.h?id=45e3e1935e2857c54783291107d33323b3ef33c8'>45e3e1935e2857c54783291107d33323b3ef33c8</a> (<a href='/cgit/mitr/public_git/kernel-crypto.git/diff/include/linux/slob_def.h?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb&amp;id2=45e3e1935e2857c54783291107d33323b3ef33c8'>diff</a>)</td></tr><tr><th>download</th><td colspan='2' class='sha1'><a href='/cgit/mitr/public_git/kernel-crypto.git/snapshot/kernel-crypto-9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb.tar.gz'>kernel-crypto-9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb.tar.gz</a><br/><a href='/cgit/mitr/public_git/kernel-crypto.git/snapshot/kernel-crypto-9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb.tar.xz'>kernel-crypto-9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb.tar.xz</a><br/><a href='/cgit/mitr/public_git/kernel-crypto.git/snapshot/kernel-crypto-9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb.zip'>kernel-crypto-9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb.zip</a><br/></td></tr></table>
<div class='commit-subject'>Merge branch 'master' of master.kernel.org:/pub/scm/linux/kernel/git/torvalds/linux-2.6</div><div class='commit-msg'>Conflicts:
	Documentation/feature-removal-schedule.txt
	drivers/scsi/fcoe/fcoe.c
	net/core/drop_monitor.c
	net/core/net-traces.c
</div><div class='diffstat-header'><a href='/cgit/mitr/public_git/kernel-crypto.git/diff/?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>Diffstat</a> (limited to 'include/linux/slob_def.h')</div><table summary='diffstat' class='diffstat'><tr><td class='mode'>-rw-r--r--</td><td class='upd'><a href='/cgit/mitr/public_git/kernel-crypto.git/diff/include/linux/slob_def.h?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>include/linux/slob_def.h</a></td><td class='right'>5</td><td class='graph'><table summary='file diffstat' width='5%'><tr><td class='add' style='width: 100.0%;'/><td class='rem' style='width: 0.0%;'/><td class='none' style='width: 0.0%;'/></tr></table></td></tr>
</table><div class='diffstat-summary'>1 files changed, 5 insertions, 0 deletions</div><table summary='diff' class='diff'><tr><td><div class='head'>diff --git a/include/linux/slob_def.h b/include/linux/slob_def.h<br/>index 0ec00b39d00..bb5368df4be 100644<br/>--- a/<a href='/cgit/mitr/public_git/kernel-crypto.git/tree/include/linux/slob_def.h?id=ca44d6e60f9de26281fda203f58b570e1748c015'>include/linux/slob_def.h</a><br/>+++ b/<a href='/cgit/mitr/public_git/kernel-crypto.git/tree/include/linux/slob_def.h?id=9cbc1cb8cd46ce1f7645b9de249b2ce8460129bb'>include/linux/slob_def.h</a></div><div class='hunk'>@@ -34,4 +34,9 @@ static __always_inline void *__kmalloc(size_t size, gfp_t flags)</div><div class='ctx'> 	return kmalloc(size, flags);</div><div class='ctx'> }</div><div class='ctx'> </div><div class='add'>+static inline void kmem_cache_init_late(void)</div><div class='add'>+{</div><div class='add'>+	/* Nothing to do */</div><div class='add'>+}</div><div class='add'>+</div><div class='ctx'> #endif /* __LINUX_SLOB_DEF_H */</div></td></tr></table></div> <!-- class=content -->
<div class='footer'>generated by <a href='https://git.zx2c4.com/cgit/about/'>cgit </a> (<a href='https://git-scm.com/'>git 2.34.1</a>) at 2025-08-13 11:29:04 +0000</div>
</div> <!-- id=cgit -->
</body>
</html>
a id='n194' href='#n194'>194</a>
<a id='n195' href='#n195'>195</a>
<a id='n196' href='#n196'>196</a>
<a id='n197' href='#n197'>197</a>
<a id='n198' href='#n198'>198</a>
<a id='n199' href='#n199'>199</a>
<a id='n200' href='#n200'>200</a>
<a id='n201' href='#n201'>201</a>
<a id='n202' href='#n202'>202</a>
<a id='n203' href='#n203'>203</a>
<a id='n204' href='#n204'>204</a>
<a id='n205' href='#n205'>205</a>
<a id='n206' href='#n206'>206</a>
<a id='n207' href='#n207'>207</a>
<a id='n208' href='#n208'>208</a>
<a id='n209' href='#n209'>209</a>
<a id='n210' href='#n210'>210</a>
<a id='n211' href='#n211'>211</a>
<a id='n212' href='#n212'>212</a>
<a id='n213' href='#n213'>213</a>
<a id='n214' href='#n214'>214</a>
<a id='n215' href='#n215'>215</a>
<a id='n216' href='#n216'>216</a>
<a id='n217' href='#n217'>217</a>
<a id='n218' href='#n218'>218</a>
<a id='n219' href='#n219'>219</a>
<a id='n220' href='#n220'>220</a>
<a id='n221' href='#n221'>221</a>
<a id='n222' href='#n222'>222</a>
<a id='n223' href='#n223'>223</a>
<a id='n224' href='#n224'>224</a>
<a id='n225' href='#n225'>225</a>
<a id='n226' href='#n226'>226</a>
<a id='n227' href='#n227'>227</a>
<a id='n228' href='#n228'>228</a>
<a id='n229' href='#n229'>229</a>
<a id='n230' href='#n230'>230</a>
<a id='n231' href='#n231'>231</a>
<a id='n232' href='#n232'>232</a>
<a id='n233' href='#n233'>233</a>
<a id='n234' href='#n234'>234</a>
<a id='n235' href='#n235'>235</a>
<a id='n236' href='#n236'>236</a>
<a id='n237' href='#n237'>237</a>
<a id='n238' href='#n238'>238</a>
<a id='n239' href='#n239'>239</a>
<a id='n240' href='#n240'>240</a>
<a id='n241' href='#n241'>241</a>
<a id='n242' href='#n242'>242</a>
<a id='n243' href='#n243'>243</a>
<a id='n244' href='#n244'>244</a>
<a id='n245' href='#n245'>245</a>
<a id='n246' href='#n246'>246</a>
<a id='n247' href='#n247'>247</a>
<a id='n248' href='#n248'>248</a>
<a id='n249' href='#n249'>249</a>
<a id='n250' href='#n250'>250</a>
<a id='n251' href='#n251'>251</a>
<a id='n252' href='#n252'>252</a>
<a id='n253' href='#n253'>253</a>
<a id='n254' href='#n254'>254</a>
<a id='n255' href='#n255'>255</a>
<a id='n256' href='#n256'>256</a>
<a id='n257' href='#n257'>257</a>
<a id='n258' href='#n258'>258</a>
<a id='n259' href='#n259'>259</a>
<a id='n260' href='#n260'>260</a>
<a id='n261' href='#n261'>261</a>
<a id='n262' href='#n262'>262</a>
<a id='n263' href='#n263'>263</a>
<a id='n264' href='#n264'>264</a>
<a id='n265' href='#n265'>265</a>
<a id='n266' href='#n266'>266</a>
<a id='n267' href='#n267'>267</a>
<a id='n268' href='#n268'>268</a>
<a id='n269' href='#n269'>269</a>
<a id='n270' href='#n270'>270</a>
<a id='n271' href='#n271'>271</a>
<a id='n272' href='#n272'>272</a>
<a id='n273' href='#n273'>273</a>
<a id='n274' href='#n274'>274</a>
<a id='n275' href='#n275'>275</a>
<a id='n276' href='#n276'>276</a>
<a id='n277' href='#n277'>277</a>
<a id='n278' href='#n278'>278</a>
<a id='n279' href='#n279'>279</a>
<a id='n280' href='#n280'>280</a>
<a id='n281' href='#n281'>281</a>
<a id='n282' href='#n282'>282</a>
<a id='n283' href='#n283'>283</a>
<a id='n284' href='#n284'>284</a>
<a id='n285' href='#n285'>285</a>
<a id='n286' href='#n286'>286</a>
<a id='n287' href='#n287'>287</a>
<a id='n288' href='#n288'>288</a>
<a id='n289' href='#n289'>289</a>
<a id='n290' href='#n290'>290</a>
<a id='n291' href='#n291'>291</a>
<a id='n292' href='#n292'>292</a>
<a id='n293' href='#n293'>293</a>
<a id='n294' href='#n294'>294</a>
<a id='n295' href='#n295'>295</a>
<a id='n296' href='#n296'>296</a>
<a id='n297' href='#n297'>297</a>
<a id='n298' href='#n298'>298</a>
<a id='n299' href='#n299'>299</a>
<a id='n300' href='#n300'>300</a>
<a id='n301' href='#n301'>301</a>
<a id='n302' href='#n302'>302</a>
<a id='n303' href='#n303'>303</a>
<a id='n304' href='#n304'>304</a>
<a id='n305' href='#n305'>305</a>
<a id='n306' href='#n306'>306</a>
<a id='n307' href='#n307'>307</a>
<a id='n308' href='#n308'>308</a>
<a id='n309' href='#n309'>309</a>
<a id='n310' href='#n310'>310</a>
<a id='n311' href='#n311'>311</a>
<a id='n312' href='#n312'>312</a>
<a id='n313' href='#n313'>313</a>
<a id='n314' href='#n314'>314</a>
<a id='n315' href='#n315'>315</a>
<a id='n316' href='#n316'>316</a>
<a id='n317' href='#n317'>317</a>
<a id='n318' href='#n318'>318</a>
<a id='n319' href='#n319'>319</a>
<a id='n320' href='#n320'>320</a>
<a id='n321' href='#n321'>321</a>
<a id='n322' href='#n322'>322</a>
<a id='n323' href='#n323'>323</a>
<a id='n324' href='#n324'>324</a>
<a id='n325' href='#n325'>325</a>
<a id='n326' href='#n326'>326</a>
<a id='n327' href='#n327'>327</a>
<a id='n328' href='#n328'>328</a>
<a id='n329' href='#n329'>329</a>
<a id='n330' href='#n330'>330</a>
<a id='n331' href='#n331'>331</a>
<a id='n332' href='#n332'>332</a>
<a id='n333' href='#n333'>333</a>
<a id='n334' href='#n334'>334</a>
<a id='n335' href='#n335'>335</a>
<a id='n336' href='#n336'>336</a>
<a id='n337' href='#n337'>337</a>
<a id='n338' href='#n338'>338</a>
<a id='n339' href='#n339'>339</a>
<a id='n340' href='#n340'>340</a>
<a id='n341' href='#n341'>341</a>
<a id='n342' href='#n342'>342</a>
<a id='n343' href='#n343'>343</a>
<a id='n344' href='#n344'>344</a>
<a id='n345' href='#n345'>345</a>
<a id='n346' href='#n346'>346</a>
<a id='n347' href='#n347'>347</a>
<a id='n348' href='#n348'>348</a>
<a id='n349' href='#n349'>349</a>
<a id='n350' href='#n350'>350</a>
<a id='n351' href='#n351'>351</a>
<a id='n352' href='#n352'>352</a>
<a id='n353' href='#n353'>353</a>
<a id='n354' href='#n354'>354</a>
<a id='n355' href='#n355'>355</a>
<a id='n356' href='#n356'>356</a>
<a id='n357' href='#n357'>357</a>
<a id='n358' href='#n358'>358</a>
<a id='n359' href='#n359'>359</a>
<a id='n360' href='#n360'>360</a>
<a id='n361' href='#n361'>361</a>
<a id='n362' href='#n362'>362</a>
<a id='n363' href='#n363'>363</a>
<a id='n364' href='#n364'>364</a>
<a id='n365' href='#n365'>365</a>
<a id='n366' href='#n366'>366</a>
<a id='n367' href='#n367'>367</a>
<a id='n368' href='#n368'>368</a>
<a id='n369' href='#n369'>369</a>
<a id='n370' href='#n370'>370</a>
<a id='n371' href='#n371'>371</a>
<a id='n372' href='#n372'>372</a>
<a id='n373' href='#n373'>373</a>
<a id='n374' href='#n374'>374</a>
<a id='n375' href='#n375'>375</a>
<a id='n376' href='#n376'>376</a>
<a id='n377' href='#n377'>377</a>
<a id='n378' href='#n378'>378</a>
<a id='n379' href='#n379'>379</a>
<a id='n380' href='#n380'>380</a>
<a id='n381' href='#n381'>381</a>
<a id='n382' href='#n382'>382</a>
<a id='n383' href='#n383'>383</a>
<a id='n384' href='#n384'>384</a>
<a id='n385' href='#n385'>385</a>
<a id='n386' href='#n386'>386</a>
<a id='n387' href='#n387'>387</a>
<a id='n388' href='#n388'>388</a>
<a id='n389' href='#n389'>389</a>
<a id='n390' href='#n390'>390</a>
<a id='n391' href='#n391'>391</a>
<a id='n392' href='#n392'>392</a>
<a id='n393' href='#n393'>393</a>
<a id='n394' href='#n394'>394</a>
<a id='n395' href='#n395'>395</a>
<a id='n396' href='#n396'>396</a>
<a id='n397' href='#n397'>397</a>
<a id='n398' href='#n398'>398</a>
<a id='n399' href='#n399'>399</a>
<a id='n400' href='#n400'>400</a>
<a id='n401' href='#n401'>401</a>
<a id='n402' href='#n402'>402</a>
<a id='n403' href='#n403'>403</a>
<a id='n404' href='#n404'>404</a>
<a id='n405' href='#n405'>405</a>
<a id='n406' href='#n406'>406</a>
<a id='n407' href='#n407'>407</a>
<a id='n408' href='#n408'>408</a>
<a id='n409' href='#n409'>409</a>
<a id='n410' href='#n410'>410</a>
<a id='n411' href='#n411'>411</a>
<a id='n412' href='#n412'>412</a>
<a id='n413' href='#n413'>413</a>
<a id='n414' href='#n414'>414</a>
<a id='n415' href='#n415'>415</a>
<a id='n416' href='#n416'>416</a>
<a id='n417' href='#n417'>417</a>
<a id='n418' href='#n418'>418</a>
<a id='n419' href='#n419'>419</a>
<a id='n420' href='#n420'>420</a>
<a id='n421' href='#n421'>421</a>
<a id='n422' href='#n422'>422</a>
<a id='n423' href='#n423'>423</a>
<a id='n424' href='#n424'>424</a>
<a id='n425' href='#n425'>425</a>
<a id='n426' href='#n426'>426</a>
<a id='n427' href='#n427'>427</a>
<a id='n428' href='#n428'>428</a>
<a id='n429' href='#n429'>429</a>
<a id='n430' href='#n430'>430</a>
<a id='n431' href='#n431'>431</a>
<a id='n432' href='#n432'>432</a>
<a id='n433' href='#n433'>433</a>
<a id='n434' href='#n434'>434</a>
<a id='n435' href='#n435'>435</a>
<a id='n436' href='#n436'>436</a>
<a id='n437' href='#n437'>437</a>
<a id='n438' href='#n438'>438</a>
<a id='n439' href='#n439'>439</a>
<a id='n440' href='#n440'>440</a>
<a id='n441' href='#n441'>441</a>
<a id='n442' href='#n442'>442</a>
<a id='n443' href='#n443'>443</a>
<a id='n444' href='#n444'>444</a>
<a id='n445' href='#n445'>445</a>
<a id='n446' href='#n446'>446</a>
<a id='n447' href='#n447'>447</a>
<a id='n448' href='#n448'>448</a>
<a id='n449' href='#n449'>449</a>
<a id='n450' href='#n450'>450</a>
<a id='n451' href='#n451'>451</a>
<a id='n452' href='#n452'>452</a>
<a id='n453' href='#n453'>453</a>
<a id='n454' href='#n454'>454</a>
<a id='n455' href='#n455'>455</a>
<a id='n456' href='#n456'>456</a>
<a id='n457' href='#n457'>457</a>
<a id='n458' href='#n458'>458</a>
<a id='n459' href='#n459'>459</a>
<a id='n460' href='#n460'>460</a>
<a id='n461' href='#n461'>461</a>
<a id='n462' href='#n462'>462</a>
<a id='n463' href='#n463'>463</a>
<a id='n464' href='#n464'>464</a>
<a id='n465' href='#n465'>465</a>
<a id='n466' href='#n466'>466</a>
<a id='n467' href='#n467'>467</a>
<a id='n468' href='#n468'>468</a>
<a id='n469' href='#n469'>469</a>
<a id='n470' href='#n470'>470</a>
<a id='n471' href='#n471'>471</a>
<a id='n472' href='#n472'>472</a>
<a id='n473' href='#n473'>473</a>
<a id='n474' href='#n474'>474</a>
<a id='n475' href='#n475'>475</a>
<a id='n476' href='#n476'>476</a>
<a id='n477' href='#n477'>477</a>
<a id='n478' href='#n478'>478</a>
<a id='n479' href='#n479'>479</a>
<a id='n480' href='#n480'>480</a>
<a id='n481' href='#n481'>481</a>
<a id='n482' href='#n482'>482</a>
<a id='n483' href='#n483'>483</a>
<a id='n484' href='#n484'>484</a>
<a id='n485' href='#n485'>485</a>
<a id='n486' href='#n486'>486</a>
<a id='n487' href='#n487'>487</a>
<a id='n488' href='#n488'>488</a>
<a id='n489' href='#n489'>489</a>
<a id='n490' href='#n490'>490</a>
<a id='n491' href='#n491'>491</a>
<a id='n492' href='#n492'>492</a>
<a id='n493' href='#n493'>493</a>
<a id='n494' href='#n494'>494</a>
<a id='n495' href='#n495'>495</a>
<a id='n496' href='#n496'>496</a>
<a id='n497' href='#n497'>497</a>
<a id='n498' href='#n498'>498</a>
<a id='n499' href='#n499'>499</a>
<a id='n500' href='#n500'>500</a>
<a id='n501' href='#n501'>501</a>
<a id='n502' href='#n502'>502</a>
<a id='n503' href='#n503'>503</a>
<a id='n504' href='#n504'>504</a>
<a id='n505' href='#n505'>505</a>
<a id='n506' href='#n506'>506</a>
<a id='n507' href='#n507'>507</a>
<a id='n508' href='#n508'>508</a>
<a id='n509' href='#n509'>509</a>
<a id='n510' href='#n510'>510</a>
<a id='n511' href='#n511'>511</a>
<a id='n512' href='#n512'>512</a>
<a id='n513' href='#n513'>513</a>
<a id='n514' href='#n514'>514</a>
<a id='n515' href='#n515'>515</a>
<a id='n516' href='#n516'>516</a>
<a id='n517' href='#n517'>517</a>
<a id='n518' href='#n518'>518</a>
<a id='n519' href='#n519'>519</a>
<a id='n520' href='#n520'>520</a>
<a id='n521' href='#n521'>521</a>
<a id='n522' href='#n522'>522</a>
<a id='n523' href='#n523'>523</a>
<a id='n524' href='#n524'>524</a>
<a id='n525' href='#n525'>525</a>
<a id='n526' href='#n526'>526</a>
<a id='n527' href='#n527'>527</a>
<a id='n528' href='#n528'>528</a>
<a id='n529' href='#n529'>529</a>
<a id='n530' href='#n530'>530</a>
<a id='n531' href='#n531'>531</a>
<a id='n532' href='#n532'>532</a>
<a id='n533' href='#n533'>533</a>
<a id='n534' href='#n534'>534</a>
<a id='n535' href='#n535'>535</a>
<a id='n536' href='#n536'>536</a>
<a id='n537' href='#n537'>537</a>
<a id='n538' href='#n538'>538</a>
<a id='n539' href='#n539'>539</a>
<a id='n540' href='#n540'>540</a>
<a id='n541' href='#n541'>541</a>
<a id='n542' href='#n542'>542</a>
<a id='n543' href='#n543'>543</a>
<a id='n544' href='#n544'>544</a>
<a id='n545' href='#n545'>545</a>
<a id='n546' href='#n546'>546</a>
<a id='n547' href='#n547'>547</a>
<a id='n548' href='#n548'>548</a>
<a id='n549' href='#n549'>549</a>
<a id='n550' href='#n550'>550</a>
<a id='n551' href='#n551'>551</a>
<a id='n552' href='#n552'>552</a>
<a id='n553' href='#n553'>553</a>
<a id='n554' href='#n554'>554</a>
<a id='n555' href='#n555'>555</a>
<a id='n556' href='#n556'>556</a>
<a id='n557' href='#n557'>557</a>
<a id='n558' href='#n558'>558</a>
<a id='n559' href='#n559'>559</a>
<a id='n560' href='#n560'>560</a>
<a id='n561' href='#n561'>561</a>
<a id='n562' href='#n562'>562</a>
<a id='n563' href='#n563'>563</a>
<a id='n564' href='#n564'>564</a>
<a id='n565' href='#n565'>565</a>
<a id='n566' href='#n566'>566</a>
<a id='n567' href='#n567'>567</a>
<a id='n568' href='#n568'>568</a>
<a id='n569' href='#n569'>569</a>
<a id='n570' href='#n570'>570</a>
<a id='n571' href='#n571'>571</a>
<a id='n572' href='#n572'>572</a>
<a id='n573' href='#n573'>573</a>
<a id='n574' href='#n574'>574</a>
<a id='n575' href='#n575'>575</a>
<a id='n576' href='#n576'>576</a>
<a id='n577' href='#n577'>577</a>
<a id='n578' href='#n578'>578</a>
<a id='n579' href='#n579'>579</a>
<a id='n580' href='#n580'>580</a>
<a id='n581' href='#n581'>581</a>
<a id='n582' href='#n582'>582</a>
<a id='n583' href='#n583'>583</a>
<a id='n584' href='#n584'>584</a>
<a id='n585' href='#n585'>585</a>
<a id='n586' href='#n586'>586</a>
<a id='n587' href='#n587'>587</a>
<a id='n588' href='#n588'>588</a>
<a id='n589' href='#n589'>589</a>
<a id='n590' href='#n590'>590</a>
<a id='n591' href='#n591'>591</a>
<a id='n592' href='#n592'>592</a>
<a id='n593' href='#n593'>593</a>
<a id='n594' href='#n594'>594</a>
<a id='n595' href='#n595'>595</a>
<a id='n596' href='#n596'>596</a>
<a id='n597' href='#n597'>597</a>
<a id='n598' href='#n598'>598</a>
<a id='n599' href='#n599'>599</a>
<a id='n600' href='#n600'>600</a>
<a id='n601' href='#n601'>601</a>
<a id='n602' href='#n602'>602</a>
<a id='n603' href='#n603'>603</a>
<a id='n604' href='#n604'>604</a>
<a id='n605' href='#n605'>605</a>
<a id='n606' href='#n606'>606</a>
<a id='n607' href='#n607'>607</a>
<a id='n608' href='#n608'>608</a>
<a id='n609' href='#n609'>609</a>
<a id='n610' href='#n610'>610</a>
<a id='n611' href='#n611'>611</a>
<a id='n612' href='#n612'>612</a>
<a id='n613' href='#n613'>613</a>
<a id='n614' href='#n614'>614</a>
<a id='n615' href='#n615'>615</a>
<a id='n616' href='#n616'>616</a>
<a id='n617' href='#n617'>617</a>
<a id='n618' href='#n618'>618</a>
<a id='n619' href='#n619'>619</a>
<a id='n620' href='#n620'>620</a>
<a id='n621' href='#n621'>621</a>
<a id='n622' href='#n622'>622</a>
<a id='n623' href='#n623'>623</a>
<a id='n624' href='#n624'>624</a>
<a id='n625' href='#n625'>625</a>
<a id='n626' href='#n626'>626</a>
<a id='n627' href='#n627'>627</a>
<a id='n628' href='#n628'>628</a>
<a id='n629' href='#n629'>629</a>
<a id='n630' href='#n630'>630</a>
<a id='n631' href='#n631'>631</a>
<a id='n632' href='#n632'>632</a>
<a id='n633' href='#n633'>633</a>
<a id='n634' href='#n634'>634</a>
<a id='n635' href='#n635'>635</a>
<a id='n636' href='#n636'>636</a>
<a id='n637' href='#n637'>637</a>
<a id='n638' href='#n638'>638</a>
<a id='n639' href='#n639'>639</a>
<a id='n640' href='#n640'>640</a>
<a id='n641' href='#n641'>641</a>
<a id='n642' href='#n642'>642</a>
<a id='n643' href='#n643'>643</a>
<a id='n644' href='#n644'>644</a>
<a id='n645' href='#n645'>645</a>
<a id='n646' href='#n646'>646</a>
<a id='n647' href='#n647'>647</a>
<a id='n648' href='#n648'>648</a>
<a id='n649' href='#n649'>649</a>
<a id='n650' href='#n650'>650</a>
<a id='n651' href='#n651'>651</a>
<a id='n652' href='#n652'>652</a>
<a id='n653' href='#n653'>653</a>
<a id='n654' href='#n654'>654</a>
<a id='n655' href='#n655'>655</a>
<a id='n656' href='#n656'>656</a>
<a id='n657' href='#n657'>657</a>
<a id='n658' href='#n658'>658</a>
<a id='n659' href='#n659'>659</a>
<a id='n660' href='#n660'>660</a>
<a id='n661' href='#n661'>661</a>
<a id='n662' href='#n662'>662</a>
<a id='n663' href='#n663'>663</a>
<a id='n664' href='#n664'>664</a>
<a id='n665' href='#n665'>665</a>
<a id='n666' href='#n666'>666</a>
<a id='n667' href='#n667'>667</a>
<a id='n668' href='#n668'>668</a>
<a id='n669' href='#n669'>669</a>
<a id='n670' href='#n670'>670</a>
<a id='n671' href='#n671'>671</a>
<a id='n672' href='#n672'>672</a>
<a id='n673' href='#n673'>673</a>
<a id='n674' href='#n674'>674</a>
<a id='n675' href='#n675'>675</a>
<a id='n676' href='#n676'>676</a>
<a id='n677' href='#n677'>677</a>
<a id='n678' href='#n678'>678</a>
<a id='n679' href='#n679'>679</a>
<a id='n680' href='#n680'>680</a>
<a id='n681' href='#n681'>681</a>
<a id='n682' href='#n682'>682</a>
<a id='n683' href='#n683'>683</a>
<a id='n684' href='#n684'>684</a>
<a id='n685' href='#n685'>685</a>
<a id='n686' href='#n686'>686</a>
<a id='n687' href='#n687'>687</a>
<a id='n688' href='#n688'>688</a>
<a id='n689' href='#n689'>689</a>
<a id='n690' href='#n690'>690</a>
<a id='n691' href='#n691'>691</a>
<a id='n692' href='#n692'>692</a>
<a id='n693' href='#n693'>693</a>
<a id='n694' href='#n694'>694</a>
<a id='n695' href='#n695'>695</a>
<a id='n696' href='#n696'>696</a>
<a id='n697' href='#n697'>697</a>
<a id='n698' href='#n698'>698</a>
<a id='n699' href='#n699'>699</a>
<a id='n700' href='#n700'>700</a>
<a id='n701' href='#n701'>701</a>
<a id='n702' href='#n702'>702</a>
<a id='n703' href='#n703'>703</a>
<a id='n704' href='#n704'>704</a>
<a id='n705' href='#n705'>705</a>
<a id='n706' href='#n706'>706</a>
<a id='n707' href='#n707'>707</a>
<a id='n708' href='#n708'>708</a>
<a id='n709' href='#n709'>709</a>
<a id='n710' href='#n710'>710</a>
<a id='n711' href='#n711'>711</a>
<a id='n712' href='#n712'>712</a>
<a id='n713' href='#n713'>713</a>
<a id='n714' href='#n714'>714</a>
<a id='n715' href='#n715'>715</a>
<a id='n716' href='#n716'>716</a>
<a id='n717' href='#n717'>717</a>
<a id='n718' href='#n718'>718</a>
<a id='n719' href='#n719'>719</a>
<a id='n720' href='#n720'>720</a>
<a id='n721' href='#n721'>721</a>
<a id='n722' href='#n722'>722</a>
<a id='n723' href='#n723'>723</a>
<a id='n724' href='#n724'>724</a>
<a id='n725' href='#n725'>725</a>
<a id='n726' href='#n726'>726</a>
<a id='n727' href='#n727'>727</a>
<a id='n728' href='#n728'>728</a>
<a id='n729' href='#n729'>729</a>
<a id='n730' href='#n730'>730</a>
<a id='n731' href='#n731'>731</a>
<a id='n732' href='#n732'>732</a>
<a id='n733' href='#n733'>733</a>
<a id='n734' href='#n734'>734</a>
<a id='n735' href='#n735'>735</a>
<a id='n736' href='#n736'>736</a>
<a id='n737' href='#n737'>737</a>
<a id='n738' href='#n738'>738</a>
<a id='n739' href='#n739'>739</a>
<a id='n740' href='#n740'>740</a>
<a id='n741' href='#n741'>741</a>
<a id='n742' href='#n742'>742</a>
<a id='n743' href='#n743'>743</a>
<a id='n744' href='#n744'>744</a>
<a id='n745' href='#n745'>745</a>
<a id='n746' href='#n746'>746</a>
<a id='n747' href='#n747'>747</a>
<a id='n748' href='#n748'>748</a>
<a id='n749' href='#n749'>749</a>
<a id='n750' href='#n750'>750</a>
<a id='n751' href='#n751'>751</a>
<a id='n752' href='#n752'>752</a>
<a id='n753' href='#n753'>753</a>
<a id='n754' href='#n754'>754</a>
<a id='n755' href='#n755'>755</a>
<a id='n756' href='#n756'>756</a>
<a id='n757' href='#n757'>757</a>
<a id='n758' href='#n758'>758</a>
<a id='n759' href='#n759'>759</a>
<a id='n760' href='#n760'>760</a>
<a id='n761' href='#n761'>761</a>
<a id='n762' href='#n762'>762</a>
<a id='n763' href='#n763'>763</a>
<a id='n764' href='#n764'>764</a>
<a id='n765' href='#n765'>765</a>
<a id='n766' href='#n766'>766</a>
<a id='n767' href='#n767'>767</a>
<a id='n768' href='#n768'>768</a>
<a id='n769' href='#n769'>769</a>
<a id='n770' href='#n770'>770</a>
<a id='n771' href='#n771'>771</a>
<a id='n772' href='#n772'>772</a>
<a id='n773' href='#n773'>773</a>
<a id='n774' href='#n774'>774</a>
<a id='n775' href='#n775'>775</a>
<a id='n776' href='#n776'>776</a>
<a id='n777' href='#n777'>777</a>
<a id='n778' href='#n778'>778</a>
<a id='n779' href='#n779'>779</a>
<a id='n780' href='#n780'>780</a>
<a id='n781' href='#n781'>781</a>
<a id='n782' href='#n782'>782</a>
<a id='n783' href='#n783'>783</a>
<a id='n784' href='#n784'>784</a>
<a id='n785' href='#n785'>785</a>
<a id='n786' href='#n786'>786</a>
<a id='n787' href='#n787'>787</a>
<a id='n788' href='#n788'>788</a>
<a id='n789' href='#n789'>789</a>
<a id='n790' href='#n790'>790</a>
<a id='n791' href='#n791'>791</a>
<a id='n792' href='#n792'>792</a>
<a id='n793' href='#n793'>793</a>
<a id='n794' href='#n794'>794</a>
<a id='n795' href='#n795'>795</a>
<a id='n796' href='#n796'>796</a>
<a id='n797' href='#n797'>797</a>
<a id='n798' href='#n798'>798</a>
<a id='n799' href='#n799'>799</a>
<a id='n800' href='#n800'>800</a>
<a id='n801' href='#n801'>801</a>
<a id='n802' href='#n802'>802</a>
<a id='n803' href='#n803'>803</a>
<a id='n804' href='#n804'>804</a>
<a id='n805' href='#n805'>805</a>
<a id='n806' href='#n806'>806</a>
<a id='n807' href='#n807'>807</a>
<a id='n808' href='#n808'>808</a>
<a id='n809' href='#n809'>809</a>
<a id='n810' href='#n810'>810</a>
<a id='n811' href='#n811'>811</a>
<a id='n812' href='#n812'>812</a>
<a id='n813' href='#n813'>813</a>
<a id='n814' href='#n814'>814</a>
<a id='n815' href='#n815'>815</a>
<a id='n816' href='#n816'>816</a>
<a id='n817' href='#n817'>817</a>
<a id='n818' href='#n818'>818</a>
<a id='n819' href='#n819'>819</a>
<a id='n820' href='#n820'>820</a>
<a id='n821' href='#n821'>821</a>
<a id='n822' href='#n822'>822</a>
<a id='n823' href='#n823'>823</a>
<a id='n824' href='#n824'>824</a>
<a id='n825' href='#n825'>825</a>
<a id='n826' href='#n826'>826</a>
<a id='n827' href='#n827'>827</a>
<a id='n828' href='#n828'>828</a>
<a id='n829' href='#n829'>829</a>
<a id='n830' href='#n830'>830</a>
<a id='n831' href='#n831'>831</a>
<a id='n832' href='#n832'>832</a>
<a id='n833' href='#n833'>833</a>
<a id='n834' href='#n834'>834</a>
<a id='n835' href='#n835'>835</a>
<a id='n836' href='#n836'>836</a>
<a id='n837' href='#n837'>837</a>
<a id='n838' href='#n838'>838</a>
<a id='n839' href='#n839'>839</a>
<a id='n840' href='#n840'>840</a>
<a id='n841' href='#n841'>841</a>
<a id='n842' href='#n842'>842</a>
<a id='n843' href='#n843'>843</a>
<a id='n844' href='#n844'>844</a>
<a id='n845' href='#n845'>845</a>
<a id='n846' href='#n846'>846</a>
<a id='n847' href='#n847'>847</a>
<a id='n848' href='#n848'>848</a>
<a id='n849' href='#n849'>849</a>
<a id='n850' href='#n850'>850</a>
<a id='n851' href='#n851'>851</a>
<a id='n852' href='#n852'>852</a>
<a id='n853' href='#n853'>853</a>
<a id='n854' href='#n854'>854</a>
<a id='n855' href='#n855'>855</a>
<a id='n856' href='#n856'>856</a>
<a id='n857' href='#n857'>857</a>
<a id='n858' href='#n858'>858</a>
<a id='n859' href='#n859'>859</a>
<a id='n860' href='#n860'>860</a>
<a id='n861' href='#n861'>861</a>
<a id='n862' href='#n862'>862</a>
<a id='n863' href='#n863'>863</a>
<a id='n864' href='#n864'>864</a>
<a id='n865' href='#n865'>865</a>
<a id='n866' href='#n866'>866</a>
<a id='n867' href='#n867'>867</a>
<a id='n868' href='#n868'>868</a>
<a id='n869' href='#n869'>869</a>
<a id='n870' href='#n870'>870</a>
<a id='n871' href='#n871'>871</a>
<a id='n872' href='#n872'>872</a>
<a id='n873' href='#n873'>873</a>
<a id='n874' href='#n874'>874</a>
<a id='n875' href='#n875'>875</a>
<a id='n876' href='#n876'>876</a>
<a id='n877' href='#n877'>877</a>
<a id='n878' href='#n878'>878</a>
<a id='n879' href='#n879'>879</a>
<a id='n880' href='#n880'>880</a>
<a id='n881' href='#n881'>881</a>
<a id='n882' href='#n882'>882</a>
<a id='n883' href='#n883'>883</a>
<a id='n884' href='#n884'>884</a>
<a id='n885' href='#n885'>885</a>
<a id='n886' href='#n886'>886</a>
<a id='n887' href='#n887'>887</a>
<a id='n888' href='#n888'>888</a>
<a id='n889' href='#n889'>889</a>
<a id='n890' href='#n890'>890</a>
<a id='n891' href='#n891'>891</a>
<a id='n892' href='#n892'>892</a>
<a id='n893' href='#n893'>893</a>
<a id='n894' href='#n894'>894</a>
<a id='n895' href='#n895'>895</a>
<a id='n896' href='#n896'>896</a>
<a id='n897' href='#n897'>897</a>
<a id='n898' href='#n898'>898</a>
<a id='n899' href='#n899'>899</a>
<a id='n900' href='#n900'>900</a>
<a id='n901' href='#n901'>901</a>
<a id='n902' href='#n902'>902</a>
<a id='n903' href='#n903'>903</a>
<a id='n904' href='#n904'>904</a>
<a id='n905' href='#n905'>905</a>
<a id='n906' href='#n906'>906</a>
<a id='n907' href='#n907'>907</a>
<a id='n908' href='#n908'>908</a>
<a id='n909' href='#n909'>909</a>
<a id='n910' href='#n910'>910</a>
<a id='n911' href='#n911'>911</a>
<a id='n912' href='#n912'>912</a>
<a id='n913' href='#n913'>913</a>
<a id='n914' href='#n914'>914</a>
<a id='n915' href='#n915'>915</a>
<a id='n916' href='#n916'>916</a>
<a id='n917' href='#n917'>917</a>
<a id='n918' href='#n918'>918</a>
<a id='n919' href='#n919'>919</a>
<a id='n920' href='#n920'>920</a>
<a id='n921' href='#n921'>921</a>
<a id='n922' href='#n922'>922</a>
<a id='n923' href='#n923'>923</a>
<a id='n924' href='#n924'>924</a>
<a id='n925' href='#n925'>925</a>
<a id='n926' href='#n926'>926</a>
<a id='n927' href='#n927'>927</a>
<a id='n928' href='#n928'>928</a>
<a id='n929' href='#n929'>929</a>
<a id='n930' href='#n930'>930</a>
<a id='n931' href='#n931'>931</a>
<a id='n932' href='#n932'>932</a>
<a id='n933' href='#n933'>933</a>
<a id='n934' href='#n934'>934</a>
<a id='n935' href='#n935'>935</a>
<a id='n936' href='#n936'>936</a>
<a id='n937' href='#n937'>937</a>
<a id='n938' href='#n938'>938</a>
<a id='n939' href='#n939'>939</a>
<a id='n940' href='#n940'>940</a>
<a id='n941' href='#n941'>941</a>
<a id='n942' href='#n942'>942</a>
<a id='n943' href='#n943'>943</a>
<a id='n944' href='#n944'>944</a>
<a id='n945' href='#n945'>945</a>
<a id='n946' href='#n946'>946</a>
<a id='n947' href='#n947'>947</a>
<a id='n948' href='#n948'>948</a>
<a id='n949' href='#n949'>949</a>
<a id='n950' href='#n950'>950</a>
<a id='n951' href='#n951'>951</a>
<a id='n952' href='#n952'>952</a>
<a id='n953' href='#n953'>953</a>
<a id='n954' href='#n954'>954</a>
<a id='n955' href='#n955'>955</a>
<a id='n956' href='#n956'>956</a>
<a id='n957' href='#n957'>957</a>
<a id='n958' href='#n958'>958</a>
<a id='n959' href='#n959'>959</a>
<a id='n960' href='#n960'>960</a>
<a id='n961' href='#n961'>961</a>
<a id='n962' href='#n962'>962</a>
<a id='n963' href='#n963'>963</a>
<a id='n964' href='#n964'>964</a>
<a id='n965' href='#n965'>965</a>
<a id='n966' href='#n966'>966</a>
<a id='n967' href='#n967'>967</a>
<a id='n968' href='#n968'>968</a>
<a id='n969' href='#n969'>969</a>
<a id='n970' href='#n970'>970</a>
<a id='n971' href='#n971'>971</a>
<a id='n972' href='#n972'>972</a>
<a id='n973' href='#n973'>973</a>
<a id='n974' href='#n974'>974</a>
<a id='n975' href='#n975'>975</a>
<a id='n976' href='#n976'>976</a>
<a id='n977' href='#n977'>977</a>
<a id='n978' href='#n978'>978</a>
<a id='n979' href='#n979'>979</a>
<a id='n980' href='#n980'>980</a>
<a id='n981' href='#n981'>981</a>
<a id='n982' href='#n982'>982</a>
<a id='n983' href='#n983'>983</a>
<a id='n984' href='#n984'>984</a>
<a id='n985' href='#n985'>985</a>
<a id='n986' href='#n986'>986</a>
<a id='n987' href='#n987'>987</a>
<a id='n988' href='#n988'>988</a>
<a id='n989' href='#n989'>989</a>
<a id='n990' href='#n990'>990</a>
<a id='n991' href='#n991'>991</a>
<a id='n992' href='#n992'>992</a>
<a id='n993' href='#n993'>993</a>
<a id='n994' href='#n994'>994</a>
<a id='n995' href='#n995'>995</a>
<a id='n996' href='#n996'>996</a>
<a id='n997' href='#n997'>997</a>
<a id='n998' href='#n998'>998</a>
<a id='n999' href='#n999'>999</a>
<a id='n1000' href='#n1000'>1000</a>
<a id='n1001' href='#n1001'>1001</a>
<a id='n1002' href='#n1002'>1002</a>
<a id='n1003' href='#n1003'>1003</a>
<a id='n1004' href='#n1004'>1004</a>
<a id='n1005' href='#n1005'>1005</a>
<a id='n1006' href='#n1006'>1006</a>
<a id='n1007' href='#n1007'>1007</a>
<a id='n1008' href='#n1008'>1008</a>
<a id='n1009' href='#n1009'>1009</a>
<a id='n1010' href='#n1010'>1010</a>
<a id='n1011' href='#n1011'>1011</a>
<a id='n1012' href='#n1012'>1012</a>
<a id='n1013' href='#n1013'>1013</a>
<a id='n1014' href='#n1014'>1014</a>
<a id='n1015' href='#n1015'>1015</a>
<a id='n1016' href='#n1016'>1016</a>
<a id='n1017' href='#n1017'>1017</a>
<a id='n1018' href='#n1018'>1018</a>
<a id='n1019' href='#n1019'>1019</a>
<a id='n1020' href='#n1020'>1020</a>
<a id='n1021' href='#n1021'>1021</a>
<a id='n1022' href='#n1022'>1022</a>
<a id='n1023' href='#n1023'>1023</a>
<a id='n1024' href='#n1024'>1024</a>
<a id='n1025' href='#n1025'>1025</a>
<a id='n1026' href='#n1026'>1026</a>
<a id='n1027' href='#n1027'>1027</a>
<a id='n1028' href='#n1028'>1028</a>
<a id='n1029' href='#n1029'>1029</a>
<a id='n1030' href='#n1030'>1030</a>
<a id='n1031' href='#n1031'>1031</a>
<a id='n1032' href='#n1032'>1032</a>
<a id='n1033' href='#n1033'>1033</a>
<a id='n1034' href='#n1034'>1034</a>
<a id='n1035' href='#n1035'>1035</a>
<a id='n1036' href='#n1036'>1036</a>
<a id='n1037' href='#n1037'>1037</a>
<a id='n1038' href='#n1038'>1038</a>
<a id='n1039' href='#n1039'>1039</a>
<a id='n1040' href='#n1040'>1040</a>
<a id='n1041' href='#n1041'>1041</a>
<a id='n1042' href='#n1042'>1042</a>
<a id='n1043' href='#n1043'>1043</a>
<a id='n1044' href='#n1044'>1044</a>
<a id='n1045' href='#n1045'>1045</a>
<a id='n1046' href='#n1046'>1046</a>
<a id='n1047' href='#n1047'>1047</a>
<a id='n1048' href='#n1048'>1048</a>
<a id='n1049' href='#n1049'>1049</a>
<a id='n1050' href='#n1050'>1050</a>
<a id='n1051' href='#n1051'>1051</a>
<a id='n1052' href='#n1052'>1052</a>
<a id='n1053' href='#n1053'>1053</a>
<a id='n1054' href='#n1054'>1054</a>
<a id='n1055' href='#n1055'>1055</a>
<a id='n1056' href='#n1056'>1056</a>
<a id='n1057' href='#n1057'>1057</a>
<a id='n1058' href='#n1058'>1058</a>
<a id='n1059' href='#n1059'>1059</a>
<a id='n1060' href='#n1060'>1060</a>
<a id='n1061' href='#n1061'>1061</a>
<a id='n1062' href='#n1062'>1062</a>
<a id='n1063' href='#n1063'>1063</a>
<a id='n1064' href='#n1064'>1064</a>
<a id='n1065' href='#n1065'>1065</a>
<a id='n1066' href='#n1066'>1066</a>
<a id='n1067' href='#n1067'>1067</a>
<a id='n1068' href='#n1068'>1068</a>
<a id='n1069' href='#n1069'>1069</a>
<a id='n1070' href='#n1070'>1070</a>
<a id='n1071' href='#n1071'>1071</a>
<a id='n1072' href='#n1072'>1072</a>
<a id='n1073' href='#n1073'>1073</a>
<a id='n1074' href='#n1074'>1074</a>
<a id='n1075' href='#n1075'>1075</a>
<a id='n1076' href='#n1076'>1076</a>
<a id='n1077' href='#n1077'>1077</a>
<a id='n1078' href='#n1078'>1078</a>
<a id='n1079' href='#n1079'>1079</a>
<a id='n1080' href='#n1080'>1080</a>
<a id='n1081' href='#n1081'>1081</a>
<a id='n1082' href='#n1082'>1082</a>
<a id='n1083' href='#n1083'>1083</a>
<a id='n1084' href='#n1084'>1084</a>
<a id='n1085' href='#n1085'>1085</a>
<a id='n1086' href='#n1086'>1086</a>
<a id='n1087' href='#n1087'>1087</a>
<a id='n1088' href='#n1088'>1088</a>
<a id='n1089' href='#n1089'>1089</a>
<a id='n1090' href='#n1090'>1090</a>
<a id='n1091' href='#n1091'>1091</a>
<a id='n1092' href='#n1092'>1092</a>
<a id='n1093' href='#n1093'>1093</a>
<a id='n1094' href='#n1094'>1094</a>
<a id='n1095' href='#n1095'>1095</a>
<a id='n1096' href='#n1096'>1096</a>
<a id='n1097' href='#n1097'>1097</a>
<a id='n1098' href='#n1098'>1098</a>
<a id='n1099' href='#n1099'>1099</a>
<a id='n1100' href='#n1100'>1100</a>
<a id='n1101' href='#n1101'>1101</a>
<a id='n1102' href='#n1102'>1102</a>
<a id='n1103' href='#n1103'>1103</a>
<a id='n1104' href='#n1104'>1104</a>
<a id='n1105' href='#n1105'>1105</a>
<a id='n1106' href='#n1106'>1106</a>
<a id='n1107' href='#n1107'>1107</a>
<a id='n1108' href='#n1108'>1108</a>
<a id='n1109' href='#n1109'>1109</a>
<a id='n1110' href='#n1110'>1110</a>
<a id='n1111' href='#n1111'>1111</a>
<a id='n1112' href='#n1112'>1112</a>
<a id='n1113' href='#n1113'>1113</a>
<a id='n1114' href='#n1114'>1114</a>
<a id='n1115' href='#n1115'>1115</a>
<a id='n1116' href='#n1116'>1116</a>
<a id='n1117' href='#n1117'>1117</a>
<a id='n1118' href='#n1118'>1118</a>
<a id='n1119' href='#n1119'>1119</a>
<a id='n1120' href='#n1120'>1120</a>
<a id='n1121' href='#n1121'>1121</a>
<a id='n1122' href='#n1122'>1122</a>
<a id='n1123' href='#n1123'>1123</a>
<a id='n1124' href='#n1124'>1124</a>
<a id='n1125' href='#n1125'>1125</a>
<a id='n1126' href='#n1126'>1126</a>
<a id='n1127' href='#n1127'>1127</a>
<a id='n1128' href='#n1128'>1128</a>
<a id='n1129' href='#n1129'>1129</a>
<a id='n1130' href='#n1130'>1130</a>
<a id='n1131' href='#n1131'>1131</a>
<a id='n1132' href='#n1132'>1132</a>
<a id='n1133' href='#n1133'>1133</a>
<a id='n1134' href='#n1134'>1134</a>
<a id='n1135' href='#n1135'>1135</a>
<a id='n1136' href='#n1136'>1136</a>
<a id='n1137' href='#n1137'>1137</a>
<a id='n1138' href='#n1138'>1138</a>
<a id='n1139' href='#n1139'>1139</a>
<a id='n1140' href='#n1140'>1140</a>
<a id='n1141' href='#n1141'>1141</a>
<a id='n1142' href='#n1142'>1142</a>
<a id='n1143' href='#n1143'>1143</a>
<a id='n1144' href='#n1144'>1144</a>
<a id='n1145' href='#n1145'>1145</a>
<a id='n1146' href='#n1146'>1146</a>
<a id='n1147' href='#n1147'>1147</a>
<a id='n1148' href='#n1148'>1148</a>
<a id='n1149' href='#n1149'>1149</a>
<a id='n1150' href='#n1150'>1150</a>
<a id='n1151' href='#n1151'>1151</a>
<a id='n1152' href='#n1152'>1152</a>
<a id='n1153' href='#n1153'>1153</a>
<a id='n1154' href='#n1154'>1154</a>
<a id='n1155' href='#n1155'>1155</a>
<a id='n1156' href='#n1156'>1156</a>
<a id='n1157' href='#n1157'>1157</a>
<a id='n1158' href='#n1158'>1158</a>
<a id='n1159' href='#n1159'>1159</a>
<a id='n1160' href='#n1160'>1160</a>
<a id='n1161' href='#n1161'>1161</a>
<a id='n1162' href='#n1162'>1162</a>
<a id='n1163' href='#n1163'>1163</a>
<a id='n1164' href='#n1164'>1164</a>
<a id='n1165' href='#n1165'>1165</a>
<a id='n1166' href='#n1166'>1166</a>
<a id='n1167' href='#n1167'>1167</a>
<a id='n1168' href='#n1168'>1168</a>
<a id='n1169' href='#n1169'>1169</a>
<a id='n1170' href='#n1170'>1170</a>
<a id='n1171' href='#n1171'>1171</a>
<a id='n1172' href='#n1172'>1172</a>
<a id='n1173' href='#n1173'>1173</a>
<a id='n1174' href='#n1174'>1174</a>
<a id='n1175' href='#n1175'>1175</a>
<a id='n1176' href='#n1176'>1176</a>
<a id='n1177' href='#n1177'>1177</a>
<a id='n1178' href='#n1178'>1178</a>
<a id='n1179' href='#n1179'>1179</a>
<a id='n1180' href='#n1180'>1180</a>
<a id='n1181' href='#n1181'>1181</a>
<a id='n1182' href='#n1182'>1182</a>
<a id='n1183' href='#n1183'>1183</a>
<a id='n1184' href='#n1184'>1184</a>
<a id='n1185' href='#n1185'>1185</a>
<a id='n1186' href='#n1186'>1186</a>
<a id='n1187' href='#n1187'>1187</a>
<a id='n1188' href='#n1188'>1188</a>
<a id='n1189' href='#n1189'>1189</a>
<a id='n1190' href='#n1190'>1190</a>
<a id='n1191' href='#n1191'>1191</a>
<a id='n1192' href='#n1192'>1192</a>
<a id='n1193' href='#n1193'>1193</a>
<a id='n1194' href='#n1194'>1194</a>
<a id='n1195' href='#n1195'>1195</a>
<a id='n1196' href='#n1196'>1196</a>
<a id='n1197' href='#n1197'>1197</a>
<a id='n1198' href='#n1198'>1198</a>
<a id='n1199' href='#n1199'>1199</a>
<a id='n1200' href='#n1200'>1200</a>
<a id='n1201' href='#n1201'>1201</a>
<a id='n1202' href='#n1202'>1202</a>
<a id='n1203' href='#n1203'>1203</a>
<a id='n1204' href='#n1204'>1204</a>
<a id='n1205' href='#n1205'>1205</a>
<a id='n1206' href='#n1206'>1206</a>
<a id='n1207' href='#n1207'>1207</a>
<a id='n1208' href='#n1208'>1208</a>
<a id='n1209' href='#n1209'>1209</a>
<a id='n1210' href='#n1210'>1210</a>
<a id='n1211' href='#n1211'>1211</a>
</pre></td>
<td class='lines'><pre><code><span class="hl slc">% Copyright (C) 2005-2007 Red Hat Inc.</span>
<span class="hl slc">% This file is part of systemtap, and is free software.  You can</span>
<span class="hl slc">% redistribute it and/or modify it under the terms of the GNU General</span>
<span class="hl slc">% Public License (GPL); either version 2, or (at your option) any</span>
<span class="hl slc">% later version.</span>

<span class="hl kwa">\documentclass</span><span class="hl opt">{</span>article<span class="hl opt">}</span>
<span class="hl kwa">\usepackage</span><span class="hl opt">{</span>html<span class="hl opt">}</span>
<span class="hl kwa">\usepackage</span><span class="hl opt">{</span><span class="hl kwb">graphicx</span><span class="hl opt">}</span>
<span class="hl slc">% \usepackage{moreverb}</span>
<span class="hl kwa">\usepackage</span><span class="hl opt">{</span>fancyvrb<span class="hl opt">}</span>
<span class="hl kwa">\usepackage</span><span class="hl opt">{</span>listings<span class="hl opt">}</span>
<span class="hl kwa">\usepackage</span><span class="hl opt">{</span>fullpage<span class="hl opt">}</span>
<span class="hl kwa">\usepackage</span><span class="hl opt">{</span>fancybox<span class="hl opt">}</span>
<span class="hl kwa">\usepackage</span><span class="hl opt">[</span>compatible<span class="hl opt">]{</span>nomencl<span class="hl opt">}</span>
<span class="hl slc">% \usepackage{geometry}</span>
<span class="hl slc">% \geometry{letterpaper,text={7in,8.5in}}</span>
<span class="hl kwa">\usepackage</span><span class="hl opt">{</span>charter<span class="hl opt">}</span>

<span class="hl kwa">\newenvironment</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}</span><span class="hl slc">%% Boxed minipage</span>
    <span class="hl opt">{</span><span class="hl kwa">\begin</span><span class="hl opt">{</span>makeimage<span class="hl opt">}</span><span class="hl kwa">\begin</span><span class="hl opt">{</span>center<span class="hl opt">}</span><span class="hl kwa">\begin</span><span class="hl opt">{</span>Sbox<span class="hl opt">}</span><span class="hl kwa">\begin</span><span class="hl opt">{</span>minipage<span class="hl opt">}}</span><span class="hl slc">%</span>
    <span class="hl opt">{</span><span class="hl kwa">\end</span><span class="hl opt">{</span>minipage<span class="hl opt">}</span><span class="hl kwa">\end</span><span class="hl opt">{</span>Sbox<span class="hl opt">}</span><span class="hl kwa">\fbox</span><span class="hl opt">{</span><span class="hl kwa">\TheSbox</span><span class="hl opt">}</span><span class="hl kwa">\end</span><span class="hl opt">{</span>center<span class="hl opt">}</span><span class="hl kwa">\end</span><span class="hl opt">{</span>makeimage<span class="hl opt">}}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>htmlonly<span class="hl opt">}</span>
<span class="hl kwa">\renewcommand</span><span class="hl opt">{</span><span class="hl kwa">\nomenclature</span><span class="hl opt">}[</span><span class="hl num">2</span><span class="hl opt">]{}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>htmlonly<span class="hl opt">}</span>

<span class="hl slc">% \usepackage{draftcopy} % ugly</span>
<span class="hl kwa">\bibliographystyle</span><span class="hl opt">{</span>plain<span class="hl opt">}</span>
<span class="hl kwa">\makeglossary</span>
<span class="hl kwa">\parindent0</span>.0cm
<span class="hl kwa">\parskip0</span>.2cm

<span class="hl kwa">\begin</span><span class="hl opt">{</span><span class="hl kwb">document</span><span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>center<span class="hl opt">}</span>
<span class="hl kwa">\LARGE</span> <span class="hl opt">{</span><span class="hl kwa">\bf</span> Systemtap tutorial<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>center<span class="hl opt">}</span>

<span class="hl kwa">\hfill \begin</span><span class="hl opt">{</span>minipage<span class="hl opt">}{</span><span class="hl num">2.5</span>in<span class="hl opt">}</span>
<span class="hl slc">% contributors please add your names to the list</span>
Frank Ch. Eigler <span class="hl opt">{</span><span class="hl kwa">\tt \small</span> &lt;fche&#64;redhat.com&gt;<span class="hl opt">}</span> \\
<span class="hl kwa">\hfill \today</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>minipage<span class="hl opt">}</span>

<span class="hl kwa">\tableofcontents</span>

<span class="hl kwa">\section</span><span class="hl opt">{</span>Introduction<span class="hl opt">}</span>

Systemtap is a tool that allows developers and administrators to write
and reuse simple scripts to deeply examine the activities of a live
Linux system.  Data may be extracted, filtered, and summarized quickly
and safely, to enable diagnoses of complex performance or functional
problems.

<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>script<span class="hl opt">}{</span>A simple programming language understood by systemtap.<span class="hl opt">}</span>

The essential idea behind a systemtap script is to name <span class="hl opt">{</span><span class="hl kwa">\em</span> events<span class="hl opt">}</span>,
and to give them <span class="hl opt">{</span><span class="hl kwa">\em</span> handlers<span class="hl opt">}</span>.  Whenever a specified event occurs,
the Linux kernel runs the handler as if it were a quick subroutine,
then resumes.  There are several kind of events, such as entering or
exiting a function, a timer expiring, or the entire systemtap session
starting or stopping.  A handler is a series of script language
statements that specify the work to be done whenever the event occurs.
This work normally includes extracting data from the event context,
storing them into internal variables, or printing results.

<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>event<span class="hl opt">}{</span>An identifiable instant in the operating system&#39;s
execution state, such as entry to a function, or expiry of a timer.<span class="hl opt">}</span>
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>session<span class="hl opt">}{</span>A complete run of a systemtap script program.<span class="hl opt">}</span>
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>handler<span class="hl opt">}{</span>A series of statements, written in script, which
is to be performed whenever an event occurs.<span class="hl opt">}</span>
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span><span class="hl kwa">\tt</span> .stp<span class="hl opt">}{</span>The standard file name extension for systemtap
scripts.<span class="hl opt">}</span>

Systemtap works by translating the script to C, running the system C
compiler to create a kernel module from that.  When the module is
loaded, it activates all the probed events by hooking into the kernel.
Then, as events occur on any processor, the compiled handlers run.
Eventually, the session stops, the hooks are disconnected, and the
module removed.  This entire process is driven from a single
command-line program, <span class="hl kwa">\verb</span>+stap+.

<span class="hl kwa">\begin</span><span class="hl opt">{</span>figure<span class="hl opt">}[</span>!ht<span class="hl opt">]</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}{</span><span class="hl num">4.5</span>in<span class="hl opt">}</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
# cat hello-world.stp
probe begin 
<span class="hl opt">{</span>
  print (&quot;hello world<span class="hl kwa">\n</span>&quot;)
  exit ()
<span class="hl opt">}</span>

# stap hello-world.stp
hello world
<span class="hl kwa">\end</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}</span>
<span class="hl kwa">\label</span><span class="hl opt">{</span>fig:hello-world<span class="hl opt">}</span>
<span class="hl kwa">\caption</span><span class="hl opt">{</span>A systemtap smoke test.<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>figure<span class="hl opt">}</span>

This paper assumes that you have installed systemtap and its
prerequisite kernel development tools and debugging data, so that you
can run the scripts such as the simple one in
Figure~<span class="hl kwa">\ref</span><span class="hl opt">{</span>fig:hello-world<span class="hl opt">}</span>.  Log on as <span class="hl kwa">\verb</span>+root+, or even better,
as a user authorized to <span class="hl kwa">\verb</span>+sudo+, before running systemtap.

<span class="hl kwa">\begin</span><span class="hl opt">{</span>figure<span class="hl opt">}[</span>ht<span class="hl opt">]</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}{</span><span class="hl num">4.5</span>in<span class="hl opt">}</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
# cat strace-open.stp
probe syscall.open
<span class="hl opt">{</span>
  printf (&quot;<span class="hl slc">%s(%d) open (%s)\n&quot;, execname(), pid(), argstr) </span>
<span class="hl opt">}</span>
probe timer.ms(<span class="hl num">4000</span>) # after <span class="hl num">4</span> seconds
<span class="hl opt">{</span>
  exit ()
<span class="hl opt">}</span>

# stap strace-open.stp
vmware-guestd(<span class="hl num">2206</span>) open (&quot;/etc/redhat-release&quot;, O_RDONLY)
hald(<span class="hl num">2360</span>) open (&quot;/dev/hdc&quot;, O_RDONLY|O_EXCL|O_NONBLOCK)
hald(<span class="hl num">2360</span>) open (&quot;/dev/hdc&quot;, O_RDONLY|O_EXCL|O_NONBLOCK)
hald(<span class="hl num">2360</span>) open (&quot;/dev/hdc&quot;, O_RDONLY|O_EXCL|O_NONBLOCK)
df(<span class="hl num">3433</span>) open (&quot;/etc/ld.so.cache&quot;, O_RDONLY)
df(<span class="hl num">3433</span>) open (&quot;/lib/tls/libc.so<span class="hl num">.6</span>&quot;, O_RDONLY)
df(<span class="hl num">3433</span>) open (&quot;/etc/mtab&quot;, O_RDONLY)
hald(<span class="hl num">2360</span>) open (&quot;/dev/hdc&quot;, O_RDONLY|O_EXCL|O_NONBLOCK)
<span class="hl kwa">\end</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}</span>
<span class="hl kwa">\label</span><span class="hl opt">{</span>fig:strace-open<span class="hl opt">}</span>
<span class="hl kwa">\caption</span><span class="hl opt">{</span>A taste of systemtap: a system-wide <span class="hl opt">{</span><span class="hl kwa">\tt</span> strace<span class="hl opt">}</span>, just for
the <span class="hl opt">{</span><span class="hl kwa">\tt</span> open<span class="hl opt">}</span> system call.<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>figure<span class="hl opt">}</span>
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>strace<span class="hl opt">}{</span>A standard ptrace-based command line tool to trace system call activity of a process.<span class="hl opt">}</span>

<span class="hl kwa">\section</span><span class="hl opt">{</span>Tracing<span class="hl opt">}</span>

The simplest kind of probe is simply to <span class="hl opt">{</span><span class="hl kwa">\em</span> trace<span class="hl opt">}</span> an event.
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>trace<span class="hl opt">}{</span>A compact textual record of an event occurrence.<span class="hl opt">}</span>
This is the effect of inserting strategically located <span class="hl kwa">\verb</span>+print+
statements into a program.  This is often the first step of problem
solving: explore by seeing a history of what has happened.

This style of instrumentation is the simplest.  It just asks systemtap
to print something at each event.  To express this in the script
language, you need to say where to probe and what to print there.

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Where to probe<span class="hl opt">}</span>

Systemtap supports a number of built-in events.  The library of
scripts that comes with systemtap, each called a ``tapset&#39;&#39;, may
define additional ones defined in terms of the built-in family.  See
the <span class="hl kwa">\verb</span>+stapprobes+ man page for details.  <span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>tapset<span class="hl opt">}{</span>A
reusable script forming part of the automatically searched tapset
library.<span class="hl opt">}</span>  All these events are named using a unified syntax that
looks like dot-separated parameterized identifiers:

<span class="hl kwa">\begin</span><span class="hl opt">{</span>tabular<span class="hl opt">}{</span>rl<span class="hl opt">}</span>
<span class="hl kwa">\verb</span>+begin+ &amp; The startup of the systemtap session. \\
<span class="hl kwa">\verb</span>+end+ &amp; The end of the systemtap session. \\
<span class="hl kwa">\verb</span>+kernel.function(&quot;sys_open&quot;)+ &amp; The entry to the function named
<span class="hl kwa">\verb</span>+sys_open+ in the kernel. \\
<span class="hl kwa">\verb</span>+syscall.close.return+ &amp; The return from the <span class="hl kwa">\verb</span>+close+ system
call. \\
<span class="hl kwa">\verb</span>+module(&quot;ext3&quot;).statement(<span class="hl num">0xdeadbeef</span>)+ &amp; The addressed instruction
in the <span class="hl kwa">\verb</span>+ext3+ filesystem driver. \\
<span class="hl kwa">\verb</span>+timer.ms(<span class="hl num">200</span>)+ &amp; A timer that fires every <span class="hl num">200</span> milliseconds. \\
<span class="hl kwa">\end</span><span class="hl opt">{</span>tabular<span class="hl opt">}</span>

Let&#39;s say that you would like to trace all function entries and exits
in a source file, say <span class="hl kwa">\verb</span>+net/socket.c+ in the kernel.  The
<span class="hl kwa">\verb</span>+kernel.function+ probe point lets you express that easily, since
systemtap examines the kernel&#39;s debugging information to relate object
code to source code.  It works like a debugger: if you can name or
place it, you can probe it.  Use
<span class="hl kwa">\verb</span>+kernel.function(&quot;*&#64;net/socket.c&quot;).call+ for the function
entries<span class="hl kwa">\footnote</span><span class="hl opt">{</span>Without the <span class="hl opt">{</span><span class="hl kwa">\tt</span> .call<span class="hl opt">}</span> qualifier, inlined function
instances are also probed, but they have no corresponding <span class="hl opt">{</span><span class="hl kwa">\tt</span> .return<span class="hl opt">}</span>.<span class="hl opt">}</span>,
and <span class="hl kwa">\verb</span>+kernel.function(&quot;*&#64;net/socket.c&quot;).return+ for matching exits.  Note
the use of wildcards in the function name part, and the subsequent
<span class="hl kwa">\verb</span>+&#64;FILENAME+ part.  You can also put wildcards into the file name,
and even add a colon (<span class="hl kwa">\verb</span>+:+) and a line number, if you want to
restrict the search that precisely.  Since systemtap will put a
separate probe in every place that matches a probe point, a few
wildcards can expand to hundreds or thousands of probes, so be careful
what you ask for.  <span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>debug information<span class="hl opt">}{</span>Data created by the
compiler when the kernel or application was built, sometimes packaged into
<span class="hl opt">{</span><span class="hl kwa">\tt</span> debuginfo<span class="hl opt">}</span> files, for use by a symbolic debugger.<span class="hl opt">}</span>
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>wildcard<span class="hl opt">}{</span>Presence of <span class="hl kwa">\verb</span>+*+ globbing patterns in probe points.<span class="hl opt">}</span>

Once you identify the probe points, the skeleton of the systemtap
script appears.  The <span class="hl kwa">\verb</span>+probe+ keyword introduces a probe point, or
a comma-separated list of them.  The following <span class="hl kwa">\verb</span>+<span class="hl opt">{</span>+ and <span class="hl kwa">\verb</span>+<span class="hl opt">}</span>+
braces enclose the handler for all listed probe points.
<span class="hl kwa">\begin</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
probe kernel.function(&quot;*&#64;net/socket.c&quot;) <span class="hl opt">{ }</span>
probe kernel.function(&quot;*&#64;net/socket.c&quot;).return <span class="hl opt">{ }</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
You can run this script as is, though with empty handlers there will
be no output.  Put the two lines into a new file.  Run
<span class="hl kwa">\verb</span>+stap -v FILE+.  Terminate it any time with <span class="hl kwa">\verb</span>+^C+.  (The
<span class="hl kwa">\verb</span>+-v+ option tells systemtap to print more verbose messages during
its processing.  Try the <span class="hl kwa">\verb</span>+-h+ option to see more options.)

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>What to print<span class="hl opt">}</span>

Since you are interested in each function that was entered and exited,
a line should be printed for each, containing the function name.  In
order to make that list easy to read, systemtap should indent the
lines so that functions called by other traced functions are nested
deeper.  To tell each single process apart from any others that may be
running concurrently, systemtap should also print the process ID in
the line.

Systemtap provides a variety of such contextual data, ready for
formatting.  They usually appear as function calls within the handler,
like you already saw in Figure~<span class="hl kwa">\ref</span><span class="hl opt">{</span>fig:strace-open<span class="hl opt">}</span>.  See the
<span class="hl kwa">\verb</span>+stapfuncs+ man page for those functions and more defined in the
tapset library, but here&#39;s a sampling:

<span class="hl kwa">\begin</span><span class="hl opt">{</span>tabular<span class="hl opt">}{</span>rl<span class="hl opt">}</span>
<span class="hl kwa">\verb</span>+tid()+ &amp; The id of the current thread. \\
<span class="hl kwa">\verb</span>+pid()+ &amp; The process (task group) id of the current thread. \\
<span class="hl kwa">\verb</span>+uid()+ &amp; The id of the current user. \\
<span class="hl kwa">\verb</span>+execname()+ &amp; The name of the current process. \\
<span class="hl kwa">\verb</span>+cpu()+ &amp; The current cpu number. \\
<span class="hl kwa">\verb</span>+gettimeofday_s()+ &amp; Number of seconds since epoch. \\
<span class="hl kwa">\verb</span>+get_cycles()+ &amp; Snapshot of hardware cycle counter. \\
<span class="hl kwa">\verb</span>+pp()+ &amp; A string describing the probe point being currently handled. \\
<span class="hl kwa">\verb</span>+probefunc()+ &amp; If known, the name of the function in which
                     this probe was placed. \\
<span class="hl kwa">\end</span><span class="hl opt">{</span>tabular<span class="hl opt">}</span>

The values returned may be strings or numbers.  The <span class="hl kwa">\verb</span>+print()+
built-in function accepts either as its sole argument.  Or, you can
use the C-style <span class="hl kwa">\verb</span>+printf()+ built-in, whose formatting argument
may include <span class="hl kwa">\verb</span>+<span class="hl slc">%s+ for a string, \verb+%d+ for a number.</span>
<span class="hl kwa">\verb</span>+printf+ and other functions take comma-separated arguments.
Don&#39;t forget a <span class="hl kwa">\verb</span>+&quot;<span class="hl kwa">\n</span>&quot;+ at the end.

A particularly handy function in the tapset library is
<span class="hl kwa">\verb</span>+thread_indent+.  Given an indentation delta parameter, it stores
internally an indentation counter for each thread (<span class="hl kwa">\verb</span>+tid()+), and
returns a string with some generic trace data plus an appropriate
number of indentation spaces.  That generic data includes a timestamp
(number of microseconds since the most recent initial indentation), a
process name and the thread id itself.  It therefore gives an idea not
only about what functions were called, but who called them, and how
long they took.  Figure~<span class="hl kwa">\ref</span><span class="hl opt">{</span>fig:socket-trace<span class="hl opt">}</span> shows the finished
script.  It lacks a call to the <span class="hl kwa">\verb</span>+exit()+ function, so you need to
interrupt it with <span class="hl kwa">\verb</span>+^C+ when you want the tracing to stop.

<span class="hl kwa">\begin</span><span class="hl opt">{</span>figure<span class="hl opt">}[</span>!ht<span class="hl opt">]</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}{</span><span class="hl num">4.5</span>in<span class="hl opt">}</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
# cat socket-trace.stp
probe kernel.function(&quot;*&#64;net/socket.c&quot;) <span class="hl opt">{</span>
  printf (&quot;<span class="hl slc">%s -&gt; %s\n&quot;, thread_indent(1), probefunc())</span>
<span class="hl opt">}</span>
probe kernel.function(&quot;*&#64;net/socket.c&quot;).return <span class="hl opt">{</span>
  printf (&quot;<span class="hl slc">%s &lt;- %s\n&quot;, thread_indent(-1), probefunc())</span>
<span class="hl opt">}</span>

# stap socket-trace.stp
     <span class="hl num">0</span> hald(<span class="hl num">2632</span>): -&gt; sock_poll
    <span class="hl num">28</span> hald(<span class="hl num">2632</span>): &lt;- sock_poll
<span class="hl opt">[</span>...<span class="hl opt">]</span>
     <span class="hl num">0</span> ftp(<span class="hl num">7223</span>): -&gt; sys_socketcall
  <span class="hl num">1159</span> ftp(<span class="hl num">7223</span>):  -&gt; sys_socket
  <span class="hl num">2173</span> ftp(<span class="hl num">7223</span>):   -&gt; __sock_create
  <span class="hl num">2286</span> ftp(<span class="hl num">7223</span>):    -&gt; sock_alloc_inode
  <span class="hl num">2737</span> ftp(<span class="hl num">7223</span>):    &lt;- sock_alloc_inode
  <span class="hl num">3349</span> ftp(<span class="hl num">7223</span>):    -&gt; sock_alloc
  <span class="hl num">3389</span> ftp(<span class="hl num">7223</span>):    &lt;- sock_alloc
  <span class="hl num">3417</span> ftp(<span class="hl num">7223</span>):   &lt;- __sock_create
  <span class="hl num">4117</span> ftp(<span class="hl num">7223</span>):   -&gt; sock_create
  <span class="hl num">4160</span> ftp(<span class="hl num">7223</span>):   &lt;- sock_create
  <span class="hl num">4301</span> ftp(<span class="hl num">7223</span>):   -&gt; sock_map_fd
  <span class="hl num">4644</span> ftp(<span class="hl num">7223</span>):    -&gt; sock_map_file
  <span class="hl num">4699</span> ftp(<span class="hl num">7223</span>):    &lt;- sock_map_file
  <span class="hl num">4715</span> ftp(<span class="hl num">7223</span>):   &lt;- sock_map_fd
  <span class="hl num">4732</span> ftp(<span class="hl num">7223</span>):  &lt;- sys_socket
  <span class="hl num">4775</span> ftp(<span class="hl num">7223</span>): &lt;- sys_socketcall
<span class="hl opt">[</span>...<span class="hl opt">]</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}</span>
<span class="hl kwa">\caption</span><span class="hl opt">{</span>Tracing and timing functions in <span class="hl opt">{</span><span class="hl kwa">\tt</span> net/sockets.c<span class="hl opt">}</span>.<span class="hl opt">}</span>
<span class="hl kwa">\label</span><span class="hl opt">{</span>fig:socket-trace<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>figure<span class="hl opt">}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Exercises<span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>enumerate<span class="hl opt">}</span>
<span class="hl kwa">\item</span> Use the <span class="hl kwa">\verb</span>+-p2+ option to systemtap to list all the kernel
functions named with the word ``nit&#39;&#39; in them.  The probe handlers
might as well be empty.

<span class="hl kwa">\item</span> Trace some system calls (use <span class="hl kwa">\verb</span>+syscall.NAME+ and <span class="hl kwa">\verb</span>+.return+
probe points), with the same <span class="hl kwa">\verb</span>+thread_indent+ probe handler as in
Figure~<span class="hl kwa">\ref</span><span class="hl opt">{</span>fig:socket-trace<span class="hl opt">}</span>.  Interpret the results.

<span class="hl kwa">\end</span><span class="hl opt">{</span>enumerate<span class="hl opt">}</span>

<span class="hl kwa">\section</span><span class="hl opt">{</span>Analysis<span class="hl opt">}</span>

Pages of generic tracing text may give you enough information for
exploring a system.  With systemtap, it is possible to analyze that
data, to filter, aggregate, transform, and summarize it.  Different
probes can work together to share data.  Probe handlers can use a rich
set of control constructs to describe algorithms, with a syntax taken
roughly from <span class="hl kwa">\verb</span>+awk+.  With these tools, systemtap scripts can
focus on a specific question and provide a compact response: no
<span class="hl kwa">\verb</span>+grep+ needed.
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>awk<span class="hl opt">}{</span>A classic UNIX stream processing language.<span class="hl opt">}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Basic constructs<span class="hl opt">}</span>

Most systemtap scripts include conditionals, to limit tracing or other
logic to those processes or users or <span class="hl opt">{</span><span class="hl kwa">\em</span> whatever<span class="hl opt">}</span> of interest.  The
syntax is simple:

<span class="hl kwa">\begin</span><span class="hl opt">{</span>tabular<span class="hl opt">}{</span>rl<span class="hl opt">}</span>
<span class="hl kwa">\verb</span>+if (+<span class="hl opt">{</span><span class="hl kwa">\em</span> EXPR<span class="hl opt">}</span><span class="hl kwa">\verb</span>+)+ <span class="hl opt">{</span><span class="hl kwa">\em</span> STATEMENT<span class="hl opt">} [</span><span class="hl kwa">\verb</span>+else+ <span class="hl opt">{</span><span class="hl kwa">\em</span> STATEMENT<span class="hl opt">}</span><span class="hl kwa">\verb</span>+<span class="hl opt">]</span>+ &amp; if/else statement \\
<span class="hl kwa">\verb</span>+while (+<span class="hl opt">{</span><span class="hl kwa">\em</span> EXPR<span class="hl opt">}</span><span class="hl kwa">\verb</span>+)+ <span class="hl opt">{</span><span class="hl kwa">\em</span> STATEMENT<span class="hl opt">}</span> &amp; while loop \\
<span class="hl kwa">\verb</span>+for (+<span class="hl opt">{</span><span class="hl kwa">\em</span> A<span class="hl opt">}</span><span class="hl kwa">\verb</span>+;+ <span class="hl opt">{</span><span class="hl kwa">\em</span> B<span class="hl opt">}</span><span class="hl kwa">\verb</span>+;+ <span class="hl opt">{</span><span class="hl kwa">\em</span> C<span class="hl opt">}</span><span class="hl kwa">\verb</span>+)+ <span class="hl opt">{</span><span class="hl kwa">\em</span> STATEMENT<span class="hl opt">}</span> &amp; for loop \\
<span class="hl kwa">\end</span><span class="hl opt">{</span>tabular<span class="hl opt">}</span>

Scripts may use <span class="hl kwa">\verb</span>+break+/<span class="hl kwa">\verb</span>+continue+ as in C.
Probe handlers can return early using <span class="hl kwa">\verb</span>+next+ as in <span class="hl kwa">\verb</span>+awk+.
Blocks of statements are enclosed in <span class="hl kwa">\verb</span>+<span class="hl opt">{</span>+ and <span class="hl kwa">\verb</span>+<span class="hl opt">}</span>+.  In
systemtap, the semicolon (<span class="hl kwa">\verb</span>+;+) is accepted as a null statement
rather than as a statement terminator, so is only rarely<span class="hl kwa">\footnote</span><span class="hl opt">{</span>Use
them between consecutive expressions that place unary <span class="hl opt">{</span><span class="hl kwa">\tt</span> +<span class="hl opt">}</span>,<span class="hl opt">{</span><span class="hl kwa">\tt</span> -<span class="hl opt">}</span>
or mixed pre/post <span class="hl opt">{</span><span class="hl kwa">\tt</span> ++<span class="hl opt">}</span>,<span class="hl opt">{</span><span class="hl kwa">\tt</span> --<span class="hl opt">}</span> in an ambiguous manner.<span class="hl opt">}</span>
necessary.  Shell-style (<span class="hl kwa">\verb</span>+#+), C-style (<span class="hl kwa">\verb</span>+/* */+), and
C++-style (<span class="hl kwa">\verb</span>+//+) comments are all accepted.

Expressions look like C or <span class="hl kwa">\verb</span>+awk+, and support the usual
operators, precedences, and numeric literals.  Strings are treated as
atomic values rather than arrays of characters.  String concatenation
is done with the dot (<span class="hl kwa">\verb</span>+&quot;a&quot; . &quot;b&quot;+).  Some examples:

<span class="hl kwa">\begin</span><span class="hl opt">{</span>tabular<span class="hl opt">}{</span>rl<span class="hl opt">}</span>
<span class="hl kwa">\verb</span>+(uid() &gt; <span class="hl num">100</span>)+ &amp; probably an ordinary user \\
<span class="hl kwa">\verb</span>+(execname() == &quot;sed&quot;)+ &amp; current process is sed \\
<span class="hl kwa">\verb</span>+(cpu() == <span class="hl num">0</span> &amp;&amp; gettimeofday_s() &gt; <span class="hl num">1140498000</span>)+ &amp; after Feb. <span class="hl num">21</span>, <span class="hl num">2006</span>, on CPU <span class="hl num">0</span> \\
<span class="hl kwa">\verb</span>+&quot;hello&quot; . &quot; &quot; . &quot;world&quot;+ &amp; a string in three easy pieces \\
<span class="hl kwa">\end</span><span class="hl opt">{</span>tabular<span class="hl opt">}</span>

Variables may be used as well.  Just pick a name, assign to it, and
use it in expressions.  They are automatically initialized and
declared.  The type of each identifier -- string vs. number -- is
automatically inferred by systemtap from the kinds of operators and
literals used on it.  Any inconsistencies will be reported as errors.
Conversion between string and number types is done through explicit
function calls.

<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>type<span class="hl opt">}{</span>A designation of each identifier such as a
variable, or function, or array value or index, as containing a string
or number.<span class="hl opt">}</span>  <span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>string<span class="hl opt">}{</span>A <span class="hl kwa">\verb</span>+<span class="hl kwa">\0</span>+-terminated character
string of up to a fixed limit in length.<span class="hl opt">}</span>  <span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>number<span class="hl opt">}{</span>A
<span class="hl num">64</span>-bit signed integer.<span class="hl opt">}</span>  <span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>type inference<span class="hl opt">}{</span>The automatic
determination of the type of each variable, function parameter, array
value and index, based on their use.<span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>tabular<span class="hl opt">}{</span>rl<span class="hl opt">}</span>
<span class="hl kwa">\verb</span>+foo = gettimeofday_s()+ &amp; foo is a number \\
<span class="hl kwa">\verb</span>+bar = &quot;/usr/bin/&quot; . execname()+ &amp; bar is a string \\
<span class="hl kwa">\verb</span>|c++| &amp; c is a number \\
<span class="hl kwa">\verb</span>+s = sprint(<span class="hl num">2345</span>)+ &amp; s becomes the string &quot;<span class="hl num">2345</span>&quot; \\
<span class="hl kwa">\end</span><span class="hl opt">{</span>tabular<span class="hl opt">}</span>

By default, variables are local to the probe they are used in.  That
is, they are initialized, used, and disposed of at each probe handler
invocation.  To share variables between probes, declare them global
anywhere in the script.  Because of possible concurrency (multiple
probe handlers running on different CPUs), each global variable used
by a probe is automatically read- or write-locked while the handler is
running.  <span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>global variable<span class="hl opt">}{</span>A scalar, array, or aggregate that was
named in a <span class="hl kwa">\verb</span>+global+ declaration, sharing that object amongst all
probe handlers and functions executed during a systemtap session.<span class="hl opt">}</span>
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>locking<span class="hl opt">}{</span>An automated facility used by systemtap to
protect global variables against concurrent modification and/or
access.<span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>figure<span class="hl opt">}[</span>!ht<span class="hl opt">]</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}{</span><span class="hl num">4.5</span>in<span class="hl opt">}</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
# cat timer-jiffies.stp
global count_jiffies, count_ms
probe timer.jiffies(<span class="hl num">100</span>) <span class="hl opt">{</span> count_jiffies ++ <span class="hl opt">}</span>
probe timer.ms(<span class="hl num">100</span>) <span class="hl opt">{</span> count_ms ++ <span class="hl opt">}</span>
probe timer.ms(<span class="hl num">12345</span>) 
<span class="hl opt">{</span>
  hz=(<span class="hl num">1000</span>*count_jiffies) / count_ms
  printf (&quot;jiffies:ms ratio <span class="hl slc">%d:%d =&gt; CONFIG_HZ=%d\n&quot;,</span>
    count_jiffies, count_ms, hz)
  exit ()
<span class="hl opt">}</span>

# stap timer-jiffies.stp
jiffies:ms ratio <span class="hl num">30</span>:<span class="hl num">123</span> =&gt; CONFIG_HZ=<span class="hl num">243</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}</span>
<span class="hl kwa">\caption</span><span class="hl opt">{</span>Experimentally measuring <span class="hl opt">{</span><span class="hl kwa">\tt</span> CONFIG<span class="hl kwa">\_HZ</span><span class="hl opt">}</span>.<span class="hl opt">}</span>
<span class="hl kwa">\label</span><span class="hl opt">{</span>fig:timer-jiffies<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>figure<span class="hl opt">}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Target variables<span class="hl opt">}</span>

A class of special ``target variables&#39;&#39; allow access to the probe
point context.  <span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>target variable<span class="hl opt">}{</span>A value that may be
extracted from the kernel context of the probe point, such as a
parameter or local variable within a probed function.<span class="hl opt">}</span>  In a symbolic
debugger, when you&#39;re stopped at a breakpoint, you can print values
from the program&#39;s context.  In systemtap scripts, for those probe
points that match with specific executable point (rather than an
asynchronous event like a timer), you can do the same.  To know which
variables are likely to be available, you will need to be familiar
with the kernel source you are probing.  In addition, you will need to
check that the compiler has not optimized those values into
unreachable nonexistence.

Let&#39;s say that you are trying to trace filesystem reads/writes to a
particular device/inode.  From your knowledge of the kernel, you know
that two functions of interest could be <span class="hl kwa">\verb</span>+vfs_read+ and
<span class="hl kwa">\verb</span>+vfs_write+.  Each takes a <span class="hl kwa">\verb</span>+struct file *+ argument, inside
which there is a <span class="hl kwa">\verb</span>+struct dentry *+, a <span class="hl kwa">\verb</span>+struct inode *+, and
so on.  Systemtap allows limited dereferencing of such pointer chains.
Two functions, <span class="hl kwa">\verb</span>+user_string+ and <span class="hl kwa">\verb</span>+kernel_string+, can copy
<span class="hl kwa">\verb</span>+char *+ target variables into systemtap strings.
Figure~<span class="hl kwa">\ref</span><span class="hl opt">{</span>fig:inode-watch<span class="hl opt">}</span> demonstrates one way to monitor a
particular file (identifed by device number and inode number).  This
example also demonstrates pasting numeric command-line arguments
(<span class="hl kwa">\verb</span>+$<span class="hl num">1</span>+ etc.) into scripts.
<span class="hl slc">%$</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>figure<span class="hl opt">}[</span>!ht<span class="hl opt">]</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}{</span><span class="hl num">4.5</span>in<span class="hl opt">}</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
# cat inode-watch.stp
probe kernel.function (&quot;vfs_write&quot;),
      kernel.function (&quot;vfs_read&quot;)
<span class="hl opt">{</span>
  dev_nr = $file-&gt;f_dentry-&gt;d_inode-&gt;i_sb-&gt;s_dev
  inode_nr = $file-&gt;f_dentry-&gt;d_inode-&gt;i_ino

  if (dev_nr == ($<span class="hl num">1</span> &lt;&lt; <span class="hl num">20</span> | $<span class="hl num">2</span>) # major/minor device
      &amp;&amp; inode_nr == $<span class="hl num">3</span>)
    printf (&quot;<span class="hl slc">%s(%d) %s 0x%x/%u\n&quot;,</span>
      execname(), pid(), probefunc(), dev_nr, inode_nr)
<span class="hl opt">}</span>
# stat -c &#39;<span class="hl slc">%D %i&#39; /etc/crontab</span>
<span class="hl num">803 988136</span>
# stap inode-watch.stp <span class="hl num">8 3 988136</span>
crond(<span class="hl num">2419</span>) vfs_read <span class="hl num">0x800003</span>/<span class="hl num">988136</span>
crond(<span class="hl num">2419</span>) vfs_read <span class="hl num">0x800003</span>/<span class="hl num">988136</span>
crond(<span class="hl num">2419</span>) vfs_read <span class="hl num">0x800003</span>/<span class="hl num">988136</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
<span class="hl slc">% $</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}</span>
<span class="hl kwa">\caption</span><span class="hl opt">{</span>Watching for reads/writes to a particular file.<span class="hl opt">}</span>
<span class="hl kwa">\label</span><span class="hl opt">{</span>fig:inode-watch<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>figure<span class="hl opt">}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Functions<span class="hl opt">}</span>

Functions are conveniently packaged reusable software: it would be a
shame to have to duplicate a complex condition expression or logging
directive in every placed it&#39;s used.  So, systemtap lets you define
functions of your own.  Like global variables, systemtap functions may
be defined anywhere in the script.  They may take any number of string
or numeric arguments (by value), and may return a single string or
number.  The parameter types are inferred as for ordinary variables,
and must be consistent throughout the program.  Local and global
script variables are available, but target variables are <span class="hl opt">{</span><span class="hl kwa">\em</span> not<span class="hl opt">}</span>.
That&#39;s because there is no specific debugging-level context associated
with a function.
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>function<span class="hl opt">}{</span>A clump of parametrized script statements that
may be repeatedly and recursively called from probe handlers and other
functions.<span class="hl opt">}</span>

A function is defined with the keyword <span class="hl kwa">\verb</span>+function+ followed by a
name.  Then comes a comma-separated formal argument list (just a list
of variable names).  The <span class="hl kwa">\verb</span>+<span class="hl opt">{ }</span>+-enclosed body consists of any list
of statements, including expressions that call functions.  Recursion
is possible, up to a nesting depth limit.  Figure~<span class="hl kwa">\ref</span><span class="hl opt">{</span>fig:functions<span class="hl opt">}</span>
displays function syntax.


<span class="hl kwa">\begin</span><span class="hl opt">{</span>figure<span class="hl opt">}[</span>!ht<span class="hl opt">]</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}{</span><span class="hl num">4.5</span>in<span class="hl opt">}</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
# Red Hat convention
function system_uid_p (u) <span class="hl opt">{</span> return u &lt; <span class="hl num">500</span> <span class="hl opt">}</span>

# kernel device number assembly macro
function makedev (major,minor) <span class="hl opt">{</span> return major &lt;&lt; <span class="hl num">20</span> | minor <span class="hl opt">}</span>

function trace_common ()
<span class="hl opt">{</span>
  printf(&quot;<span class="hl slc">%d %s(%d)&quot;, gettimeofday_s(), execname(), pid())</span>
  # no return value necessary
<span class="hl opt">}</span> 

function fibonacci (i)
<span class="hl opt">{</span>
  if (i &lt; <span class="hl num">1</span>) return <span class="hl num">0</span>
  else if (i &lt; <span class="hl num">2</span>) return <span class="hl num">1</span>
  else return fibonacci(i-<span class="hl num">1</span>) + fibonacci(i-<span class="hl num">2</span>)
<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}</span>
<span class="hl kwa">\caption</span><span class="hl opt">{</span>Some functions of dubious utility.<span class="hl opt">}</span>
<span class="hl kwa">\label</span><span class="hl opt">{</span>fig:functions<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>figure<span class="hl opt">}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Arrays<span class="hl opt">}</span>

Often, probes will want to share data that cannot be represented as a
simple scalar value.  Much data is naturally tabular in nature,
indexed by some tuple of thread numbers, processor ids, names, time,
and so on.  Systemtap offers associative arrays for this purpose.
These arrays are implemented as hash tables with a maximum size that
is fixed at startup.  Because they are too large to be created
dynamically for inidividual probes handler runs, they must be declared
as global.  <span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>array<span class="hl opt">}{</span>A global
<span class="hl kwa">\verb</span>+<span class="hl opt">[</span>+$k_1,k_2,<span class="hl kwa">\ldots</span>,k_n<span class="hl kwa">\verb</span>+<span class="hl opt">]</span>+<span class="hl kwa">\rightarrow</span> value$
associative lookup table, with a string,
number for each index; the value may be a string, number, or an aggregate.<span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>tabular<span class="hl opt">}{</span>rl<span class="hl opt">}</span>
<span class="hl kwa">\verb</span>|global a| &amp; declare global scalar or array variable \\
<span class="hl kwa">\verb</span>|global b<span class="hl opt">[</span><span class="hl num">400</span><span class="hl opt">]</span>| &amp; declare array, reserving space for up to <span class="hl num">400</span> tuples \\
<span class="hl kwa">\end</span><span class="hl opt">{</span>tabular<span class="hl opt">}</span>

The basic operations for arrays are setting and looking up elements.
These are expressed in <span class="hl kwa">\verb</span>+awk+ syntax: the array name followed by
an opening <span class="hl kwa">\verb</span>+<span class="hl opt">[</span>+ bracket, a comma-separated list of index
expressions, and a closing <span class="hl kwa">\verb</span>+<span class="hl opt">]</span>+ bracket.  Each index expression
may be string or numeric, as long as it is consistently typed
throughout the script.
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>arity<span class="hl opt">}{</span>Number of indexes to an array, or number of parameters
to a function.<span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>tabular<span class="hl opt">}{</span>rl<span class="hl opt">}</span>
<span class="hl kwa">\verb</span>|foo <span class="hl opt">[</span><span class="hl num">4</span>,&quot;hello&quot;<span class="hl opt">]</span> ++ | &amp; increment the named array slot \\
<span class="hl kwa">\verb</span>|processusage <span class="hl opt">[</span>uid(),execname()<span class="hl opt">]</span> ++| &amp; update a statistic \\
<span class="hl kwa">\verb</span>|times <span class="hl opt">[</span>tid()<span class="hl opt">]</span> = get_cycles()| &amp; set a timestamp reference point \\
<span class="hl kwa">\verb</span>|delta = get_cycles() - times <span class="hl opt">[</span>tid()<span class="hl opt">]</span>| &amp; compute a timestamp delta \\
<span class="hl kwa">\end</span><span class="hl opt">{</span>tabular<span class="hl opt">}</span>

Array elements that have not been set <span class="hl opt">{</span><span class="hl kwa">\em</span> may<span class="hl opt">}</span> be fetched, and return
a dummy null value (zero or an empty string) as appropriate.  However,
assigning a null value does not delete the element: an explicit
<span class="hl kwa">\verb</span>|delete| statement is required.  <span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>null value<span class="hl opt">}{</span>A
default initialized value for globals and array elements: a zero or an
empty string, depending on type.<span class="hl opt">}</span>  Systemtap provides syntactic sugar
for these operations, in the form of explicit membership testing and
deletion.

<span class="hl kwa">\begin</span><span class="hl opt">{</span>tabular<span class="hl opt">}{</span>rl<span class="hl opt">}</span>
<span class="hl kwa">\verb</span>|if (<span class="hl opt">[</span><span class="hl num">4</span>,&quot;hello&quot;<span class="hl opt">]</span> in foo) <span class="hl opt">{ }</span>| &amp; membership test \\
<span class="hl kwa">\verb</span>|delete times<span class="hl opt">[</span>tid()<span class="hl opt">]</span>| &amp; deletion of a single element \\
<span class="hl kwa">\verb</span>|delete times| &amp; deletion of all elements \\
<span class="hl kwa">\end</span><span class="hl opt">{</span>tabular<span class="hl opt">}</span>

One final and important operation is iteration over arrays.  This uses
the keyword <span class="hl kwa">\verb</span>+foreach+.  Like <span class="hl kwa">\verb</span>+awk+, this creates a loop that
<span class="hl opt">{</span><span class="hl kwa">\em</span> iterates over key tuples<span class="hl opt">}</span> of an array, not just <span class="hl opt">{</span><span class="hl kwa">\em</span> values<span class="hl opt">}</span>.  In
addition, the iteration may be <span class="hl opt">{</span><span class="hl kwa">\em</span> sorted<span class="hl opt">}</span> by any single key or the
value by adding an extra <span class="hl kwa">\verb</span>|+| or <span class="hl kwa">\verb</span>|-| code.

The <span class="hl kwa">\verb</span>+break+ and <span class="hl kwa">\verb</span>+continue+ statements work inside
<span class="hl kwa">\verb</span>+foreach+ loops, too.  Since arrays can be large but probe
handlers must not run for long, it is a good idea to exit iteration
early if possible.  The <span class="hl kwa">\verb</span>+limit+ option in the <span class="hl kwa">\verb</span>+foreach+
expression is one way.  For simplicity, systemtap forbids any <span class="hl opt">{</span><span class="hl kwa">\em</span>
modification<span class="hl opt">}</span> of an array while it is being iterated using a
<span class="hl kwa">\verb</span>+foreach+.

<span class="hl kwa">\begin</span><span class="hl opt">{</span>tabular<span class="hl opt">}{</span>rp<span class="hl opt">{</span><span class="hl num">0.4</span><span class="hl kwa">\textwidth</span><span class="hl opt">}}</span>
<span class="hl kwa">\verb</span>|foreach (<span class="hl opt">[</span>a,b<span class="hl opt">]</span> in foo) <span class="hl opt">{</span> fuss_with(foo<span class="hl opt">[</span>a,b<span class="hl opt">]</span>) <span class="hl opt">}</span>| &amp; simple loop in arbitrary sequence \\
<span class="hl kwa">\verb</span>|foreach (<span class="hl opt">[</span>a,b<span class="hl opt">]</span> in foo+ limit <span class="hl num">5</span>) <span class="hl opt">{ }</span>| &amp; loop in increasing sequence of value, stop after <span class="hl num">5</span> \\
<span class="hl kwa">\verb</span>|foreach (<span class="hl opt">[</span>a-,b<span class="hl opt">]</span> in foo) <span class="hl opt">{ }</span>| &amp; loop in decreasing sequence of first key \\
<span class="hl kwa">\end</span><span class="hl opt">{</span>tabular<span class="hl opt">}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Aggregates<span class="hl opt">}</span>

When we said above that values can only be strings or numbers, we lied
a little.  There is a third type: statistics aggregates, or aggregates
for short.  Instances of this type are used to collect statistics on
numerical values, where it is important to accumulate new data quickly
(<span class="hl opt">{</span><span class="hl kwa">\em</span> without<span class="hl opt">}</span> exclusive locks) and in large volume (storing only
aggregated stream statistics).  This type only makes sense for global
variables, and may be stored individually or as elements of an array.
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>aggregate<span class="hl opt">}{</span>A special ``write-mostly&#39;&#39; data type used to
efficiently store aggregated statistical values of a potentially huge
data stream.<span class="hl opt">}</span>

To add a value to a statistics aggregate, systemtap uses the special
operator <span class="hl kwa">\verb</span>+&lt;&lt;&lt;+.  Think of it like C++&#39;s <span class="hl kwa">\verb</span>+&lt;&lt;+ output
streamer: the left hand side object accumulates the data sample given
on the right hand side.  This operation is efficient (taking a shared
lock) because the aggregate values are kept separately on each
processor, and are only aggregated across processors on request.

<span class="hl kwa">\begin</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
a &lt;&lt;&lt; delta_timestamp
writes<span class="hl opt">[</span>execname()<span class="hl opt">]</span> &lt;&lt;&lt; count
<span class="hl kwa">\end</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>

To read the aggregate value, special functions are available to
extract a selected statistical function. <span class="hl opt">{</span><span class="hl kwa">\em</span> The aggregate value
cannot be read by simply naming it as if it were an ordinary
variable.<span class="hl opt">}</span>  These operations take an exclusive lock on the respective
globals, and should therefore be relatively rare.  The simple ones
are: <span class="hl kwa">\verb</span>+&#64;min+, <span class="hl kwa">\verb</span>+&#64;max+, <span class="hl kwa">\verb</span>+&#64;count+, <span class="hl kwa">\verb</span>+&#64;avg+, and
<span class="hl kwa">\verb</span>+&#64;sum+, and evaluate to a single number.  In addition, histograms
of the data stream may be extracted using the <span class="hl kwa">\verb</span>+&#64;hist_log+ and
<span class="hl kwa">\verb</span>+&#64;hist_linear+.  These evaluate to a special sort of array that
may at present<span class="hl kwa">\footnote</span><span class="hl opt">{</span>We anticipate support for indexing and looping
using <span class="hl opt">{</span><span class="hl kwa">\tt</span> foreach<span class="hl opt">}</span> shortly.<span class="hl opt">}</span> only be printed.
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>extractor<span class="hl opt">}{</span>A function-like expression in a script that
computes a single statistic for a given aggregate.<span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>tabular<span class="hl opt">}{</span>rp<span class="hl opt">{</span><span class="hl num">0.5</span><span class="hl kwa">\textwidth</span><span class="hl opt">}}</span>
<span class="hl kwa">\verb</span>+&#64;avg(a)+ &amp; the average of all the values accumulated
                        into <span class="hl kwa">\verb</span>+a+ \\
<span class="hl kwa">\verb</span>+print(&#64;hist_linear(a,<span class="hl num">0</span>,<span class="hl num">100</span>,<span class="hl num">10</span>))+ &amp; print an ``ascii art&#39;&#39; linear
                  histogram of the same data stream,
     bounds $<span class="hl num">0</span> <span class="hl kwa">\ldots</span> <span class="hl num">100</span>$, bucket width is $<span class="hl num">10</span>$ \\
<span class="hl kwa">\verb</span>|&#64;count(writes<span class="hl opt">[</span>&quot;zsh&quot;<span class="hl opt">]</span>)| &amp; the number of times ``zsh&#39;&#39;
                ran the probe handler \\
<span class="hl kwa">\verb</span>+print(&#64;hist_log(writes<span class="hl opt">[</span>&quot;zsh&quot;<span class="hl opt">]</span>))+ &amp; print an ``ascii art&#39;&#39; logarithmic
                  histogram of the same data stream \\
<span class="hl kwa">\end</span><span class="hl opt">{</span>tabular<span class="hl opt">}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Safety<span class="hl opt">}</span>
<span class="hl kwa">\label</span><span class="hl opt">{</span>sec:safety<span class="hl opt">}</span>

The full expressivity of the scripting language raises good questions
of safety.  Here is a set of Q\&amp;A:

<span class="hl kwa">\begin</span><span class="hl opt">{</span>description<span class="hl opt">}</span>
<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> What about infinite loops?  recursion?<span class="hl opt">}</span> A probe handler is
bounded in time.  The C code generated by systemtap includes explicit
checks that limit the total number of statements executed to a small
number.  A similar limit is imposed on the nesting depth of function
calls.  When either limit is exceeded, that probe handler cleanly
aborts and signals an error.  The systemtap session is normally
configured to abort as a whole at that time.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> What about running out of memory?<span class="hl opt">}</span>  No dynamic memory
allocation whatsoever takes place during the execution of probe
handlers.  Arrays, function contexts, and buffers are allocated during
initialization.  These resources may run out during a session, and
generally result in errors.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> What about locking?<span class="hl opt">}</span>  If multiple probes seek conflicting
locks on the same global variables, one or more of them will time out,
and be aborted.  Such events are tallied as ``skipped&#39;&#39; probes, and a
count is displayed at session end.  A configurable number of skipped
probes can trigger an abort of the session.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> What about null pointers? division by zero?<span class="hl opt">}</span>  The C code
generated by systemtap translates potentially dangerous operations to
routines that check their arguments at run time.  These signal errors
if they are invalid.  Many arithmetic and string operations silently
overflow if the results exceed representation limits.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> What about bugs in the translator?  compiler?<span class="hl opt">}</span>  While bugs
in the translator, or the runtime layer certainly exist<span class="hl kwa">\footnote</span><span class="hl opt">{</span>See
<span class="hl kwa">\tt</span> http://sources.redhat.com/bugzilla<span class="hl opt">}</span>, our test suite gives some
assurance.  Plus, the entire generated C code may be inspected (try
the <span class="hl kwa">\verb</span>+-p3+ option).  Compiler bugs are unlikely to be of any
greater concern for systemtap than for the kernel as a whole.  In
other words, if it was reliable enough to build the kernel, it will
build the systemtap modules properly too.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> Is that the whole truth?<span class="hl opt">}</span>  In practice, there are several
weak points in systemtap and the underlying kprobes system at the time
of writing.  Putting probes indiscriminately into unusually sensitive
parts of the kernel (low level context switching, interrupt
dispatching) has reportedly caused crashes in the past.  We are
fixing these bugs as they are found, and
constructing a probe point ``blacklist&#39;&#39;, but it is not complete.
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>blacklist<span class="hl opt">}{</span>A list of probe point patterns encoded into
the translator or the kernel, where probing is prohibited for safety
reasons.<span class="hl opt">}</span>  <span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>kprobes<span class="hl opt">}{</span>A breakpoint dispatching system for
dynamic kernel probes, used by systemtap to implement some families of
probe points.<span class="hl opt">}</span>

<span class="hl kwa">\end</span><span class="hl opt">{</span>description<span class="hl opt">}</span>


<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Exercises<span class="hl opt">}</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>enumerate<span class="hl opt">}</span>
<span class="hl kwa">\item</span> Alter the last probe in <span class="hl kwa">\verb</span>+timer-jiffies.stp+ to reset the
counters and continue reporting instead of exiting.

<span class="hl kwa">\item</span> Write a script that, every ten seconds, displays the top five
most frequent users of <span class="hl kwa">\verb</span>+open+ system call during that interval.

<span class="hl kwa">\item</span> Write a script that experimentally measures the speed of the
<span class="hl kwa">\verb</span>+get_cycles()+ counter on each processor.

<span class="hl kwa">\item</span> Use any suitable probe point to get an approximate profile of
process CPU usage: which processes/users use how much of each CPU.
<span class="hl kwa">\end</span><span class="hl opt">{</span>enumerate<span class="hl opt">}</span>

<span class="hl kwa">\section</span><span class="hl opt">{</span>Tapsets<span class="hl opt">}</span>

After writing enough analysis scripts for yourself, your may become
known as an expert to your colleagues, who will want to use your
scripts.  Systemtap makes it possible to share in a controlled manner;
to build libraries of scripts that build on each other.  In fact, all
of the functions (<span class="hl kwa">\verb</span>+pid()+, etc.) used in the scripts above come
from tapset scripts like that.  A ``tapset&#39;&#39; is just a script that
designed for reuse by installation into a special directory.

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Automatic selection<span class="hl opt">}</span>

Systemtap attempts to resolve references to global symbols (probes,
functions, variables) that are not defined within the script by a
systematic search through the tapset library for scripts that define
those symbols.  Tapset scripts are installed under the default
directory named <span class="hl kwa">\verb</span>+/usr/share/systemtap/tapset+.  A user may give
additional directories with the <span class="hl kwa">\verb</span>+-I DIR+ option.  Systemtap
searches these directories for script (<span class="hl kwa">\verb</span>+.stp+) files.

The search process includes subdirectories that are specialized for a
particular kernel version and/or architecture, and ones that name only
larger kernel families.  Naturally, the search is ordered from
specific to general, as shown in Figure~<span class="hl kwa">\ref</span><span class="hl opt">{</span>fig:tapset-search<span class="hl opt">}</span>.
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>tapset search path<span class="hl opt">}{</span>A list of subdirectories searched by
systemtap for tapset scripts, allowing specialization by version
architecture.<span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>figure<span class="hl opt">}[</span>!ht<span class="hl opt">]</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}{</span><span class="hl num">6</span>in<span class="hl opt">}</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
# stap -p1 -vv -e &#39;probe begin <span class="hl opt">{ }</span>&#39; &gt; /dev/null
Created temporary directory &quot;/tmp/staplnEBh7&quot;
Searched &#39;/usr/share/systemtap/tapset/<span class="hl num">2.6.15</span>/i686/*.stp&#39;, match count <span class="hl num">0</span>
Searched &#39;/usr/share/systemtap/tapset/<span class="hl num">2.6.15</span>/*.stp&#39;, match count <span class="hl num">0</span>
Searched &#39;/usr/share/systemtap/tapset/<span class="hl num">2.6</span>/i686/*.stp&#39;, match count <span class="hl num">0</span>
Searched &#39;/usr/share/systemtap/tapset/<span class="hl num">2.6</span>/*.stp&#39;, match count <span class="hl num">0</span>
Searched &#39;/usr/share/systemtap/tapset/i686/*.stp&#39;, match count <span class="hl num">1</span>
Searched &#39;/usr/share/systemtap/tapset/*.stp&#39;, match count <span class="hl num">12</span>
Pass <span class="hl num">1</span>: parsed user script and <span class="hl num">13</span> library script(s) in <span class="hl num">350u</span>sr/<span class="hl num">10</span>sys/<span class="hl num">375</span>real ms.
Running rm -rf /tmp/staplnEBh7
<span class="hl kwa">\end</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}</span>
<span class="hl kwa">\caption</span><span class="hl opt">{</span>Listing the tapset search path.<span class="hl opt">}</span>
<span class="hl kwa">\label</span><span class="hl opt">{</span>fig:tapset-search<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>figure<span class="hl opt">}</span>

When a script file is found that <span class="hl opt">{</span><span class="hl kwa">\em</span> defines<span class="hl opt">}</span> one of the undefined
symbols, that <span class="hl opt">{</span><span class="hl kwa">\em</span> entire file<span class="hl opt">}</span> is added to the probing session being
analyzed.  This search is repeated until no more references can become
satisfied.  Systemtap signals an error if any are still unresolved.

This mechanism enables several programming idioms.  First, it allows
some global symbols to be defined only for applicable kernel
version/architecture pairs, and cause an error if their use is
attempted on an inapplicable host.  Similarly, the same symbol can be
defined differently depending on kernels, in much the same way that
different kernel <span class="hl kwa">\verb</span>+include/asm/ARCH/+ files contain macros that
provide a porting layer.

Another use is to separate the default parameters of a tapset routine
from its implementation.  For example, consider a tapset that defines
code for relating elapsed time intervals to process scheduling
activities.  The data collection code can be generic with respect to
which time unit (jiffies, wall-clock seconds, cycle counts) it can
use.  It should have a default, but should not require additional
run-time checks to let a user choose another.
Figure~<span class="hl kwa">\ref</span><span class="hl opt">{</span>fig:tapset-default<span class="hl opt">}</span> shows a way.

<span class="hl kwa">\begin</span><span class="hl opt">{</span>figure<span class="hl opt">}[</span>!ht<span class="hl opt">]</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}{</span><span class="hl num">6</span>in<span class="hl opt">}</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
# cat tapset/time-common.stp
global __time_vars
function timer_begin (name) <span class="hl opt">{</span> __time_vars<span class="hl opt">[</span>name<span class="hl opt">]</span> = __time_value () <span class="hl opt">}</span>
function timer_end (name) <span class="hl opt">{</span> return __time_value() - __time_vars<span class="hl opt">[</span>name<span class="hl opt">] }</span>

# cat tapset/time-default.stp
function __time_value () <span class="hl opt">{</span> return gettimeofday_us () <span class="hl opt">}</span>

# cat tapset-time-user.stp
probe begin
<span class="hl opt">{</span>
  timer_begin (&quot;bench&quot;)
  for (i=<span class="hl num">0</span>; i&lt;<span class="hl num">100</span>; i++) ; 
  printf (&quot;<span class="hl slc">%d cycles\n&quot;, timer_end (&quot;bench&quot;))</span>
  exit ()
<span class="hl opt">}</span>
function __time_value () <span class="hl opt">{</span> return get_ticks () <span class="hl opt">}</span> # override for greater precision

<span class="hl kwa">\end</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}</span>
<span class="hl kwa">\caption</span><span class="hl opt">{</span>Providing an overrideable default.<span class="hl opt">}</span>
<span class="hl kwa">\label</span><span class="hl opt">{</span>fig:tapset-default<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>figure<span class="hl opt">}</span>

A tapset that exports only <span class="hl opt">{</span><span class="hl kwa">\em</span> data<span class="hl opt">}</span> may be as useful as ones that
exports functions or probe point aliases (see below).  Such global
data can be computed and kept up-to-date using probes internal to the
tapset.  Any outside reference to the global variable would
incidentally activate all the required probes.

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Probe point aliases<span class="hl opt">}</span>

<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>probe point alias<span class="hl opt">}{</span>A probe point that is defined in
terms of another probe point.<span class="hl opt">}</span>  Probe point aliases allow creation of
new probe points from existing ones.  This is useful if the new probe
points are named to provide a higher level of abstraction.  For
example, the system-calls tapset defines probe point aliases of the
form <span class="hl kwa">\verb</span>+syscall.open+ etc., in terms of lower level ones like
<span class="hl kwa">\verb</span>+kernel.function(&quot;sys_open&quot;)+.  Even if some future kernel
renames <span class="hl kwa">\verb</span>+sys_open+, the aliased name can remain valid.

A probe point alias definition looks like a normal probe.  Both start
with the keyword <span class="hl kwa">\verb</span>+probe+ and have a probe handler statement block
at the end.  But where a normal probe just lists its probe points, an
alias creates a new name using the assignment (<span class="hl kwa">\verb</span>+=+) operator.
Another probe that names the new probe point will create an actual
probe, with the handler of the alias <span class="hl opt">{</span><span class="hl kwa">\em</span> prepended<span class="hl opt">}</span>.

This prepending behavior serves several purposes.  It allows the alias
definition to ``preprocess&#39;&#39; the context of the probe before passing
control to the user-specified handler.  This has several possible uses:
<span class="hl kwa">\begin</span><span class="hl opt">{</span>tabular<span class="hl opt">}{</span>rl<span class="hl opt">}</span>
<span class="hl kwa">\verb</span>+if ($flag1 != $flag2) next+ &amp; skip probe unless given condition is met \\
<span class="hl kwa">\verb</span>+name = &quot;foo&quot;+ &amp; supply probe-describing values \\
<span class="hl kwa">\verb</span>+var = $var+ &amp; extract target variable to plain local variable \\ <span class="hl slc">%$</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>tabular<span class="hl opt">}</span>

Figure~<span class="hl kwa">\ref</span><span class="hl opt">{</span>fig:probe-alias<span class="hl opt">}</span> demonstrates a probe point alias
definition as well as its use.  It demonstrates how a single probe
point alias can expand to multiple probe points, even to other
aliases.  It also includes probe point wildcarding.  These functions
are designed to compose sensibly.

<span class="hl kwa">\begin</span><span class="hl opt">{</span>figure<span class="hl opt">}[</span>!ht<span class="hl opt">]</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}{</span><span class="hl num">4.5</span>in<span class="hl opt">}</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
# cat probe-alias.stp
probe syscallgroup.io = syscall.open, syscall.close, 
                        syscall.read, syscall.write
<span class="hl opt">{</span> groupname = &quot;io&quot; <span class="hl opt">}</span>

probe syscallgroup.process = syscall.fork, syscall.execve
<span class="hl opt">{</span> groupname = &quot;process&quot; <span class="hl opt">}</span>

probe syscallgroup.* 
<span class="hl opt">{</span> groups <span class="hl opt">[</span>execname() . &quot;/&quot; . groupname<span class="hl opt">]</span> ++ <span class="hl opt">}</span>

probe end
<span class="hl opt">{</span>
  foreach (eg+ in groups)
    printf (&quot;<span class="hl slc">%s: %d\n&quot;, eg, groups[eg])</span>
<span class="hl opt">}</span>

global groups

# stap probe-alias.stp
<span class="hl num">05</span>-wait_for_sys/io: <span class="hl num">19</span>
<span class="hl num">10</span>-udev.hotplug/io: <span class="hl num">17</span>
<span class="hl num">20</span>-hal.hotplug/io: <span class="hl num">12</span>
X/io: <span class="hl num">73</span>
apcsmart/io: <span class="hl num">59</span>
<span class="hl opt">[</span>...<span class="hl opt">]</span>
make/io: <span class="hl num">515</span>
make/process: <span class="hl num">16</span>
<span class="hl opt">[</span>...<span class="hl opt">]</span>
xfce-mcs-manage/io: <span class="hl num">3</span>
xfdesktop/io: <span class="hl num">5</span>
<span class="hl opt">[</span>...<span class="hl opt">]</span>
xmms/io: <span class="hl num">7070</span>
zsh/io: <span class="hl num">78</span>
zsh/process: <span class="hl num">5</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}</span>
<span class="hl kwa">\caption</span><span class="hl opt">{</span>Classified system call activity.<span class="hl opt">}</span>
<span class="hl kwa">\label</span><span class="hl opt">{</span>fig:probe-alias<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>figure<span class="hl opt">}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Embedded C<span class="hl opt">}</span>
<span class="hl kwa">\label</span><span class="hl opt">{</span>embedded-c<span class="hl opt">}</span>

Sometimes, a tapset needs provide data values from the kernel that
cannot be extracted using ordinary target variables (<span class="hl kwa">\verb</span>+$var+).  <span class="hl slc">%$</span>
This may be because the values are in complicated data structures, may
require lock awareness, or are defined by layers of macros.  Systemtap
provides an ``escape hatch&#39;&#39; to go beyond what the language can safely
offer.  In certain contexts, you may embed plain raw C in tapsets,
exchanging power for the safety guarantees listed in
section~<span class="hl kwa">\ref</span><span class="hl opt">{</span>sec:safety<span class="hl opt">}</span>.  End-user scripts <span class="hl opt">{</span><span class="hl kwa">\em</span> may not<span class="hl opt">}</span> include
embedded C code, unless systemtap is run with the <span class="hl kwa">\verb</span>+-g+ (``guru&#39;&#39;
mode) option.  Tapset scripts get guru mode privileges automatically.
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>embedded C<span class="hl opt">}{</span>Special syntax permitting tapsets to include
literal C code.<span class="hl opt">}</span>

Embedded C can be the body of a script function.  Instead enclosing
the function body statements in <span class="hl kwa">\verb</span>+<span class="hl opt">{</span>+ and <span class="hl kwa">\verb</span>+<span class="hl opt">}</span>+, use <span class="hl kwa">\verb</span>+<span class="hl slc">%{+</span>
and <span class="hl kwa">\verb</span>+<span class="hl slc">%}+.  Any enclosed C code is literally transcribed into the</span>
kernel module: it is up to you to make it safe and correct.  In order
to take parameters and return a value, a pointer macro <span class="hl kwa">\verb</span>+THIS+ is
available.  Function parameters and a place for the return value are
available as fields of that pointer.  The familiar data-gathering
functions <span class="hl kwa">\verb</span>+pid()+, <span class="hl kwa">\verb</span>+execname()+, and their neighbours are
all embedded C functions.  Figure~<span class="hl kwa">\ref</span><span class="hl opt">{</span>fig:embedded-C<span class="hl opt">}</span> contains
another example.

Since systemtap cannot examine the C code to infer these types, an
optional<span class="hl kwa">\footnote</span><span class="hl opt">{</span>This is only necessary if the types cannot be
inferred from other sources, such as the call sites.<span class="hl opt">}</span> annotation
syntax is available to assist the type inference process.  Simply
suffix parameter names and/or the function name with <span class="hl kwa">\verb</span>+:string+ or
<span class="hl kwa">\verb</span>+:long+ to designate the string or numeric type.  In addition,
the script may include a <span class="hl kwa">\verb</span>+<span class="hl slc">%{+ \verb+%}+ block at the outermost</span>
level of the script, in order to transcribe declarative code like
<span class="hl kwa">\verb</span>+#include &lt;linux/foo.h&gt;+.  These enable the embedded C functions
to refer to general kernel types.

There are a number of safety-related constraints that should be
observed by developers of embedded C code.
<span class="hl kwa">\begin</span><span class="hl opt">{</span>enumerate<span class="hl opt">}</span>
<span class="hl kwa">\item</span> Do not dereference pointers that are not known or testable valid.
<span class="hl kwa">\item</span> Do not call any kernel routine that may cause a sleep or fault.
<span class="hl kwa">\item</span> Consider possible undesirable recursion, where your embedded C
function calls a routine that may be the subject of a probe.  If that
probe handler calls your embedded C function, you may suffer infinite
regress.  Similar problems may arise with respect to non-reentrant
locks.
<span class="hl kwa">\item</span> If locking of a data structure is necessary, use a
<span class="hl kwa">\verb</span>+trylock+ type call to attempt to take the lock.  If that fails,
give up, do not block.
<span class="hl kwa">\end</span><span class="hl opt">{</span>enumerate<span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>figure<span class="hl opt">}[</span>!ht<span class="hl opt">]</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}{</span><span class="hl num">4.5</span>in<span class="hl opt">}</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
# cat embedded-C.stp
<span class="hl slc">%{</span>
#include &lt;linux/utsname.h&gt;
<span class="hl slc">%}</span>

function utsname:string (field:long)
<span class="hl slc">%{</span>
  if (down_read_trylock (&amp; uts_sem))
    <span class="hl opt">{</span>
      const char *f =
         (THIS-&gt;field == <span class="hl num">0</span> ? system_utsname.sysname :
          THIS-&gt;field == <span class="hl num">1</span> ? system_utsname.nodename :
          THIS-&gt;field == <span class="hl num">2</span> ? system_utsname.release :
          THIS-&gt;field == <span class="hl num">3</span> ? system_utsname.version :
          THIS-&gt;field == <span class="hl num">4</span> ? system_utsname.machine :
          THIS-&gt;field == <span class="hl num">5</span> ? system_utsname.domainname : &quot;&quot;);
      strlcpy (THIS-&gt;__retvalue, f, MAXSTRINGLEN);
      up_read (&amp; uts_sem);
    <span class="hl opt">}</span>
<span class="hl slc">%}</span>

probe begin
<span class="hl opt">{</span>
  printf (&quot;<span class="hl slc">%s %s\n&quot;, utsname(0), utsname(2))</span>
  exit ()
<span class="hl opt">}</span>

# stap -g embedded-C.stp
Linux <span class="hl num">2.6.15</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>verbatim<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>boxedminipage<span class="hl opt">}</span>
<span class="hl kwa">\caption</span><span class="hl opt">{</span>Embedded C function.<span class="hl opt">}</span>
<span class="hl kwa">\label</span><span class="hl opt">{</span>fig:embedded-C<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>figure<span class="hl opt">}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Naming conventions<span class="hl opt">}</span>

Using the tapset search mechanism just described, potentially many
script files can become selected for inclusion in a single session.
This raises the problem of name collisions, where different tapsets
accidentally use the same names for functions/globals.  This can
result in errors at translate or run time.

To control this problem, systemtap tapset developers are advised to
follow naming conventions.  Here is some of the guidance.
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>naming convention<span class="hl opt">}{</span>Guidelines for naming variables and
functions to prevent unintentional duplication.<span class="hl opt">}</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>enumerate<span class="hl opt">}</span>
<span class="hl kwa">\item</span> Pick a unique name for your tapset, and substitute it for
<span class="hl opt">{</span><span class="hl kwa">\em</span> TAPSET<span class="hl opt">}</span> below.
<span class="hl kwa">\item</span> Separate identifiers meant to be used by tapset users from
those that are internal implementation artifacts.
<span class="hl kwa">\item</span> Document the first set in the appropriate <span class="hl kwa">\verb</span>+man+ pages.
<span class="hl kwa">\item</span> Prefix the names of external identifiers with <span class="hl opt">{</span><span class="hl kwa">\em</span> TAPSET<span class="hl opt">}</span><span class="hl kwa">\_</span> if
there is any likelihood of collision with other tapsets or end-user
scripts.
<span class="hl kwa">\item</span> Prefix any probe point aliases with an appropriate prefix.
<span class="hl kwa">\item</span> Prefix the names of internal identifiers with <span class="hl kwa">\_\_</span><span class="hl opt">{</span><span class="hl kwa">\em</span> TAPSET<span class="hl opt">}</span><span class="hl kwa">\_</span>.
<span class="hl kwa">\end</span><span class="hl opt">{</span>enumerate<span class="hl opt">}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Exercises<span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>enumerate<span class="hl opt">}</span>
<span class="hl kwa">\item</span> Write a tapset that implements deferred and ``cancelable&#39;&#39;
logging.  Export a function that enqueues a text string (into some
private array), returning an id token.  Include a timer-based probe
that periodically flushes the array to the standard log output.
Export another function that, if the entry was not already flushed,
allows a text string to be cancelled from the queue.

<span class="hl kwa">\item</span> Create a ``relative timestamp&#39;&#39; tapset with functions return all
the same values as the ones in the timestamp tapset, except that they
are made relative to the start time of the script.

<span class="hl kwa">\item</span> Create a tapset that exports a global array that contains a
mapping of recently seen process ID numbers to process names.
Intercept key system calls (<span class="hl kwa">\verb</span>+execve+?) to update the list
incrementally.

<span class="hl kwa">\item</span> Send your tapset ideas to the mailing list!
<span class="hl kwa">\end</span><span class="hl opt">{</span>enumerate<span class="hl opt">}</span>

<span class="hl kwa">\section</span><span class="hl opt">{</span>Further information<span class="hl opt">}</span>

For further information about systemtap, several sources are available.

There are <span class="hl kwa">\verb</span>+man+ pages:

<span class="hl kwa">\begin</span><span class="hl opt">{</span>tabular<span class="hl opt">}{</span>rl<span class="hl opt">}</span>
<span class="hl kwa">\verb</span>+stap+ &amp; systemtap program usage, language summary \\
<span class="hl kwa">\verb</span>+stapfuncs+ &amp; functions provided by tapsets \\
<span class="hl kwa">\verb</span>+stapprobes+ &amp; probes / probe aliases provided by tapsets \\
<span class="hl kwa">\verb</span>+stapex+ &amp; some example scripts \\
<span class="hl kwa">\end</span><span class="hl opt">{</span>tabular<span class="hl opt">}</span>

Then, there is the source code itself.  Since systemtap is <span class="hl opt">{</span><span class="hl kwa">\em</span> free
software<span class="hl opt">}</span>, you should have available the entire source code.  The
source files in the <span class="hl kwa">\verb</span>+tapset/+ directory are also packaged along
with the systemtap binary.  Since systemtap reads these files rather
than their documentation, they are the most reliable way to see what&#39;s
inside all the tapsets.  Use the <span class="hl kwa">\verb</span>+-v+ (verbose) command line
option, several times if you like, to show inner workings.
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>free software<span class="hl opt">}{</span>Software licensed under terms such as the
GNU GPL, which aims to enforce certain specified user freedoms such
as study, modification, and sharing.<span class="hl opt">}</span>

Finally, there is the project web site
(<span class="hl kwa">\verb</span>+http://sources.redhat.com/systemtap/+) with several articles,
an archived public mailing list for users and developers
(<span class="hl kwa">\verb</span>+systemtap&#64;sources.redhat.com+), and a live GIT source
repository.  Come join us!


<span class="hl kwa">\appendix</span>

<span class="hl kwa">\section</span><span class="hl opt">{</span>Glossary<span class="hl opt">}</span>
<span class="hl kwa">\renewcommand</span><span class="hl opt">{</span><span class="hl kwa">\nomname</span><span class="hl opt">}{}</span>
<span class="hl kwa">\printglossary</span>
<span class="hl kwa">\begin</span><span class="hl opt">{</span>htmlonly<span class="hl opt">}</span>
<span class="hl opt">{</span><span class="hl kwa">\em</span> Sorry, not available in HTML.<span class="hl opt">}</span>
<span class="hl kwa">\end</span><span class="hl opt">{</span>htmlonly<span class="hl opt">}</span>

<span class="hl kwa">\section</span><span class="hl opt">{</span>Errors<span class="hl opt">}</span>

We explain some common systemtap error messages in this section.  Most
error messages include line/character numbers with which one can
locate the precise location of error in the script code.  There is
sometimes a subsequent or prior line that elaborates.

<span class="hl opt">{</span><span class="hl kwa">\large</span> <span class="hl opt">{</span><span class="hl kwa">\em</span> error<span class="hl opt">} {</span><span class="hl kwa">\tt</span> at:<span class="hl opt">} {</span><span class="hl kwa">\em</span> filename<span class="hl opt">}</span>:<span class="hl opt">{</span><span class="hl kwa">\em</span> line<span class="hl opt">}</span>:<span class="hl opt">{</span><span class="hl kwa">\em</span> column<span class="hl opt">}</span>: <span class="hl opt">{</span><span class="hl kwa">\em</span> details<span class="hl opt">}}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Parse errors<span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>description<span class="hl opt">}</span>
<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> parse error: expected <span class="hl opt">{</span><span class="hl kwa">\em</span> foo<span class="hl opt">}</span>, saw <span class="hl opt">{</span><span class="hl kwa">\em</span> bar<span class="hl opt">}</span> $<span class="hl kwa">\ldots</span>$<span class="hl opt">}</span> \\
The script contained a grammar error.  A different type of construct
was expected in the given context.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> parse error: embedded code in unprivileged script<span class="hl opt">}</span> \\ The
script contained unsafe constructs such as embedded C (section
<span class="hl kwa">\ref</span><span class="hl opt">{</span>embedded-c<span class="hl opt">}</span>), but was run without the <span class="hl kwa">\verb</span>+-g+ (guru mode)
option.  Confirm that the constructs are used safely, then try
again with <span class="hl kwa">\verb</span>+-g+.
<span class="hl kwa">\end</span><span class="hl opt">{</span>description<span class="hl opt">}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Type errors<span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>description<span class="hl opt">}</span>
<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> semantic error: type mismatch for identifier &#39;<span class="hl opt">{</span><span class="hl kwa">\em</span> foo<span class="hl opt">}</span>&#39;
$<span class="hl kwa">\ldots</span>$ string vs. long<span class="hl opt">}</span> \\ In this case, the identifier <span class="hl opt">{</span><span class="hl kwa">\em</span> foo<span class="hl opt">}</span>
was previously inferred as a numeric type (``long&#39;&#39;), but at the given
point is being used as a string.  Similar messages appear if an array
index or function parameter slot is used with conflicting types.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> semantic error: unresolved type for identifier &#39;<span class="hl opt">{</span><span class="hl kwa">\em</span> foo<span class="hl opt">}</span>&#39;<span class="hl opt">}</span>
\\ The identifier <span class="hl opt">{</span><span class="hl kwa">\em</span> foo<span class="hl opt">}</span> was used, for example in a <span class="hl kwa">\verb</span>+print+,
but without any operations that could assign it a type.  Similar
messages may appear if a symbol is misspelled by a typo.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> semantic error: Expecting symbol or array index expression<span class="hl opt">}</span>
\\ Something other than an assignable lvalue was on the left hand sign
of an assignment.
<span class="hl kwa">\end</span><span class="hl opt">{</span>description<span class="hl opt">}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Symbol errors<span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>description<span class="hl opt">}</span>
<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> while searching for arity <span class="hl opt">{</span><span class="hl kwa">\em</span> N<span class="hl opt">}</span> function, semantic error:
unresolved function call<span class="hl opt">}</span> \\ The script calls a function with <span class="hl opt">{</span><span class="hl kwa">\em</span> N<span class="hl opt">}</span>
arguments that does not exist.  The function may exist with different
arity.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> semantic error: array locals not supported: $<span class="hl kwa">\ldots</span>$<span class="hl opt">}</span> \\ An
array operation is present for which no matching global declaration
was found.  Similar messages appear if an array is used with
inconsistent arities.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> semantic error: variable &#39;<span class="hl opt">{</span><span class="hl kwa">\em</span> foo<span class="hl opt">}</span>&#39; modified during &#39;foreach&#39;<span class="hl opt">}</span> \\
The array <span class="hl opt">{</span><span class="hl kwa">\em</span> foo<span class="hl opt">}</span> is being modified (being assigned to or deleted from)
within an active <span class="hl kwa">\verb</span>+foreach+ loop.  This invalid operation is also
detected within a function called from within the loop. 
<span class="hl kwa">\end</span><span class="hl opt">{</span>description<span class="hl opt">}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Probing errors <span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>description<span class="hl opt">}</span>
<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> semantic error: probe point mismatch at position <span class="hl opt">{</span><span class="hl kwa">\em</span> N<span class="hl opt">}</span>,
while resolving probe point <span class="hl opt">{</span><span class="hl kwa">\em</span> foo<span class="hl opt">}}</span> \\ A probe point was named that
neither directly understood by systemtap, nor defined as an alias by a
tapset script.  The divergence from the ``tree&#39;&#39; of probe point
namespace is at position <span class="hl opt">{</span><span class="hl kwa">\em</span> N<span class="hl opt">}</span> (starting with zero at left).

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> semantic error: no match for probe point, while resolving
probe point <span class="hl opt">{</span><span class="hl kwa">\em</span> foo<span class="hl opt">}}</span> \\ A probe point cannot be resolved for any of
a variety of reasons.  It may be a debuginfo-based probe point such as
<span class="hl kwa">\verb</span>+kernel.function(&quot;foobar&quot;)+ where no <span class="hl kwa">\verb</span>+foobar+ function was
found.  This can occur if the script specifies a wildcard on function
names, or an invalid file name or source line number.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> semantic error: unresolved target-symbol expression<span class="hl opt">}</span> \\ A
target variable was referred to in a probe handler that was not
resolvable.  Or, a target variable is not valid at all in a context
such as a script function.  This variable may have been elided by an
optimizing compiler, or may not have a suitable type, or there might
just be an annoying bug somewhere.  Try again with a slightly
different probe point (use <span class="hl kwa">\verb</span>+statement()+ instead of
<span class="hl kwa">\verb</span>+function()+) to search for a more cooperative neighbour in the
same area.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> semantic error: libdwfl failure $<span class="hl kwa">\ldots</span>$<span class="hl opt">}</span> \\ There was a
problem processing the debugging information.  It may simply be
missing, or may have some consistency / correctness problems.  Later
compilers tend to produce better debugging information, so if you can
upgrade and recompile your kernel/application, it may help.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> semantic error: cannot find <span class="hl opt">{</span><span class="hl kwa">\em</span> foo<span class="hl opt">}</span> debuginfo<span class="hl opt">}</span> \\ Similarly,
suitable debugging information was not found.  Check that your kernel
build/installation includes a matching version of debugging data.
<span class="hl kwa">\end</span><span class="hl opt">{</span>description<span class="hl opt">}</span>

<span class="hl kwa">\subsection</span><span class="hl opt">{</span>Runtime errors<span class="hl opt">}</span>

<span class="hl kwa">\begin</span><span class="hl opt">{</span>description<span class="hl opt">}</span>

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> WARNING: Number of errors: <span class="hl opt">{</span><span class="hl kwa">\em</span> N<span class="hl opt">}</span>, skipped probes: <span class="hl opt">{</span><span class="hl kwa">\em</span> M<span class="hl opt">}}</span> \\
Errors and/or skipped probes occurred during this run.
<span class="hl kwa">\nomenclature</span><span class="hl opt">{</span>skipped probe<span class="hl opt">}{</span>A probe handler that should have run but
couldn&#39;t, due to contention or temporary resource problems.<span class="hl opt">}</span>

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> division by <span class="hl num">0</span><span class="hl opt">}</span> \\ The script code performed an invalid
division.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> aggregate element not found<span class="hl opt">}</span> \\ An statistics extractor
function other than <span class="hl kwa">\verb</span>+&#64;count+ was invoked on an aggregate that has
not had any values accumulated yet.  This is similar to a division by
zero.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> aggregation overflow<span class="hl opt">}</span> \\ An array containing aggregate
values contains too many distinct key tuples at this time.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> MAXNESTING exceeded<span class="hl opt">}</span> \\ Too many levels of function call nesting
were attempted.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> MAXACTION exceeded<span class="hl opt">}</span> \\ The probe handler attempted to execute
too many statements.

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> kernel/user string copy fault at <span class="hl opt">{</span><span class="hl kwa">\em</span> <span class="hl num">0xadd</span>r<span class="hl opt">}}</span> \\
The probe handler attempted to copy a string from kernel or user space
at an invalid address. 

<span class="hl kwa">\item</span><span class="hl opt">{</span><span class="hl kwa">\bf</span> pointer dereference fault<span class="hl opt">}</span> \\ 
There was a fault encountered during a pointer dereference operation such
as a target variable evaluation.

<span class="hl kwa">\end</span><span class="hl opt">{</span>description<span class="hl opt">}</span>


<span class="hl kwa">\section</span><span class="hl opt">{</span>Acknowledgments<span class="hl opt">}</span>

The author thanks Martin Hunt, Will Cohen, and Jim Keniston for
improvement advice for this paper.

<span class="hl kwa">\end</span><span class="hl opt">{</span><span class="hl kwb">document</span><span class="hl opt">}</span>
</code></pre></td></tr></table>
</div> <!-- class=content -->
<div class='footer'>generated by <a href='https://git.zx2c4.com/cgit/about/'>cgit </a> (<a href='https://git-scm.com/'>git 2.34.1</a>) at 2025-08-13 11:29:04 +0000</div>
</div> <!-- id=cgit -->
</body>
</html>
