1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
|
Please follow the following steps (TODO: Automate these steps)
0. Update package version in AC_INIT (configure.in)
1. Tag the current sources in trunk to the specified version
1. Run autoconf to generate the configure file again (on changing configure.in)
2. Run configure with --prefix=<desired prefix> default is /usr/local
3. This step should generate
a. Makefile
b. libcgroup.spec
4. To build the RPM, carry out the following steps
a. Create a tag with the same version as specified in step 0
a. Run mktarball
b. copy tarball and spec file to correct place in the RPMBUILD
c. Build an rpm of the spec file
Design
========
After cgroup system has taken shape, its time to have some basic tools
in user space which can enable a user to use the resource management
functionality effictively.
One of the needed functionality is rule based placement of a task. In general,
there can be either uid or gid or exec based rules. Admin/root will
control/enforce uid/gid based rules and exec based rules can be defined by
user in a config file residing in user's home dir and fully controlled by user.
uid/gid based rules will be defined in /etc/cgrules.conf config file and
this file will be managed by root.
Basic idea is that to begin with provide facility to implement rules
based on uid and gid. So a hierarchy might look like as follows.
/mnt/cgroup
| |
gid1 gid2
| |
uid1 uid2
| |
proj1 proj2
Admin will write rules to control the resources among users. Then users
can manage their own cgroup at their own (proj1 and proj2) and control
the resources as they want.
Following are the few methods using which tasks can be placed in right
cgroups.
- Use pam_cgroup PAM plugin which will make sure users are placed in right
cgroup at login time and any tasks launch after login, will continue to run
in user's cgroup. (PAM module is still on TODO list)
- Use command line tool "cgexec" to launch the task in right cgroup.
- Modify the program and use libcgroup provided APIs for placing a task
in right cgroup before doing exec().
- Use "cgclassify" tool to classify a already running task.
- May be, a user space daemon can also be implemented which will listen to
kernel events and place the task in right group based on the rules.
This method involves few concerns.
- Reliability of netlink socket. Messages can be dropped.
- Change the netlink with a cgroup controller which exports the
events.
- Delay incurred since the event took place and task was actually placed
in right cgroup.
- daemon will interfere with container's tasks which is not desired.
HOWTO
=====
Section 1:
----------
To use "cgexec" to place the task in right cgroup.
- make cgexec
- Run a task using cgexec. Following is the cgexec syntax.
cgexec [-g <list of controllers>:<relative path to cgroup>] command [arguments]
Note: Cgroup controllers and path are optional. If user does not know the
right cgroup, cgexec will automatically place the task in right
cgroup based on /etc/cgrules.conf
Example:
cgexec -g *:test1 ls
cgexec -g cpu,memory:test1 ls -l
cgexec -g cpu,memory:test1 -g swap:test2 ls -l
Section 2
---------
To use "cgclassify" to place task in right cgroup.
- make cgclassify
- Pick a task's pid to be classified, and run
cgclassify <list of pids>
Example:
--------
cgclassify 2140 4325
Note: After classification check out whether tasks 2140 and 4325
are in the right cgroup or not (Based on rules in /etc/cgrules.conf)
Section 3:
----------
To use a pam plugin which will automatically place the task in right
cgroup upon login.
- Download pam source tree and apply the patch for pam_cgroup module.
- Build pam_cgroup.so
./configure; make
- Copy pam_cgroup.so to /lib/security/
- Edit /etc/pam.d/su to make use of pam_cgroup.so session module upon
execution of su.
example: Add following line to the end.
session optional pam_cgroup.so
- Now launch a shell for a user "xyz" using su and the resulting shell
should be running in the cgroup designated for the user as specified
by cgrules.conf
ex. "su test1"
Section 4:
----------
To use cgrulesengd which will move a task to right cgroup based on
rules in /etc/cgrules.conf do following.
- build and install latest libcgroup.so
- build cgrulesengd
make cgrulesengd
- specify some uid/gid based rules in /etc/cgrules.conf
- Mount some controllers and create an hierarchy of cgroups (matching
your rules).
- Run cgrulesengd.
- ./cgrulesengd
- Launch some task or login as a user to the sytem. Daemon should automatically
place the task in right cgroup.
FAQ
===
Q. Why admin should not control "exec" based rules.
A. Unix file system provides access control only based on uid/gid. So
even if admin starts putting tasks based on uid/gid, it can't enforce
it. For example, consider following scenario.
Lets say an admin creates following cgroup hierarchy.
/container
| |
database browser
| | | |
user1 user2 user1 user2
Now admin wants to run all the database jobs under /container/database/
and all the firefox jobs under /container/browser/. Based on which user
launched it, jobs should go in either user1 or user2 dir.
Now assume that database subdir has to more cpu resources as compared to
browser subdir. Then a user, say user2, can always move his browser job
also to /container/database/user2 dir to get more cpu resources and
admin will not be able to control that.
[Note: user2 will control what tasks can be added in /container/database/user2
and will contol what further subdirs can be created under user2 dir. Root
should not restrict the control to root only for practical purposes. Its
something like that till /container/databse, admin controls the resources
and below that how resources are further subdivided among various projects
should be controlled by respective user].
In the light of above, it seems to make more sense that admin should enforce
rules only based on uid and gid. Probably later we can have a per user exec
based rules config file (~/.cgrules.conf), which can be parsed by cgrulesd
and then jobs launched by user will be placed in right cgroup based on
combination of rules in /etc/cgrules.conf and ~/cgrules.conf.
|
