Setting Up Solaris 10 Projects to Control Resource Usage

Overview

This is one of my favorite features of the Solaris 10 operating system. Nowadays, even in relatively small machines, the resources such as CPUs, memory, etc., allow the execution of a great number of processes in a single box. The control an administration can enforce on resource utilization is fundamental for the machine power to be fully utilized without jeopardizing the responsiveness of certain processes. Moreover, the usage profile of the resource may be a complex function which depends also of parameters such as time. Solaris 10, by default, would adapt itself dynamically so that all of the running applications have equal access to resources. The default behavior can be customized so that applications can access resources on a preferential basis or even be denied access under certain conditions.

This post will give a quick and basic description of resource management on the Solaris 10 operating system. As usual, the official documentation can be consulted on Sun Microsystems' Documentation Center.

Projects and Tasks

Projects and tasks are the basic entities which are used to identify workloads in the Solaris 10 operating system. A project is associated with a set of users and a set of groups. Users and groups can run its processes in the context of a project they're member of. Both users and groups can be members of more than one project so that the relations (project, user) and (project, group) are n to n relationships. The project is the basic entity against which the usage of resources can be restricted. The task is the entity to which a process is associated. The project, indeed, is associated with a set of tasks. Tasks will be described later on.

Default Projects

Every user and every group are associated to a default project, which is the project under which their processes are run if not differently specified. The algorithm that Solaris 10 uses when determining the default project of an user or of a group is the following:

• it checks if there's an explicit association in the /etc/user_attr database by means of a project attribute. If there's one, that's the default project.
• it checks if it exists a project named user.user-id or group.group-id, in this order, in the projects database. If there's one, that's the default project.
• it checks if it exists the special default project in the projects database. If it does, that's the default project.

The Project Database

The projects database stores all the information related to existing projects in the operating system. The project database is a plain text file which can be read and modified by a set of commands such as:

• projadd, to add projects
• projmod, to modify projects
• projects, to read the project database
• projdel, to remove projects

The structure of the file is pretty simple:

project-name:project-id:comment:user-list:group-list:attributes

Field	Description
project-name	the name of the project which can contain only alphanumeric characters and the - and _ characters. The . character only can appear in default user's and default group's project.
project-id	the numerical id of the project, which can be a number between 0 and UID_MAX.
comment	a string which describes the project
user-list	the list of users which can be member of this project whose syntax is described later
group-list	the list of groups which can be member of this project whose syntax is described later
attributes	a set of attributes which apply to the project whose syntax is described later

Although the file syntax is very simple, this and many other Solaris configuration files should not be edited manually. Use the corresponding commands instead.

User and Group List

The list of users and groups in the projects database is a comma separated list of values which can be one of the following:

• an user or group name
• *, to allow all users and groups to join the project
• !*, to allow nobody to join the project
• !name, to disallow a specific user or group to join the project

Attributes

The set of attributes for a project is a semicolon (;) separated list of (key, value) pairs with the syntax:

key=value

Both the semicolon (;) and the colon (:) cannot be used in the value definition.

Task

Whenever a user log in into a project, a new task for that project is created and it contains the login process. The task is given its task id end every process launched in that login session will be associated with the task which owns the login process. The task is the basic workload entity and can be viewed as a process group, also. Indeed, many operations which are supported on process groups can also be executed on tasks. Commands which create tasks are the following:

• login
• su
  
• newtask
• cron
• setproject

Determine the Current Project and Task

If you're wondering which project you're logged in into and which task you're running your processes under, you can check it with the id command:

$ id -p
uid=101(enrico) gid=10(staff) projid=10(custom-project)

I'm currently logged in as user enrico, member of group staff and I'm currently member of project custom-project.

The ps command can also show project and task information:

$ ps -o user,uid,pid,projid
USER UID PID PROJID
enrico 101 15274 10

Other Useful Commands

The Solaris 10 operating system has got a number of commands which can be used to view and manage processes using the project or the task they're member of.

• pgrep -T | -J, to look for processes associated with the specified task, using the -T option, or the specified project, using the -J option
  
• pkill -T | -J, to send a signal to the processes associated with the specified task, using the -T option, or the specified project, using the -J option
• prstat -T | -J, to view process statistics with task statistics, using the -T option, or project statistics, using the -J option

Creating an User and a Project

Let's assume we want to cap the CPU utilization of a subset of well-known CPU-intensive processes. We do this because we're not concerned about the time these processes need to end their job but we're concerned in that at least 50% of the available CPUs on our workstation be free for other processes to run during normal system and user activity. The first thing we can do is creating a project for these processes and then run these processes in a task associated with such project. In our case, being the processes non-interactive, the easier path probably is creating a dedicated user for them and creating a default user project with the characteristics we need. This way we don't even have to bother creating a specific task manually to launch these processes: they'll be run in the default user project. Let's call the user custom-user and put it for simplicity's sake in the existing staff group:

# useradd -d /export/home/custom-user -m -k /etc/skel -P a-suitable-profile -G staff -c "custom-user" -s /bin/bash custom-user

Our user would probably need to be given a suitable profile for RBAC access control and this step will depend on your specific needs and environment. If this user needs no special RBAC configuration, we can omit the -P option. Other user related configuration activities are omitted from this example.

Once the user is created, we can create its default project using the user.user-id syntax:

# projadd -U custom-user -K "project.cpu-cap=(privileged,50,deny)" user.custom-user

Projects and Resource Caps

This command will create a project, named user.custom-user, that will be the default project for the user custom-user. This project has the following attribute:

project.cpu-cap=(privileged,50,deny)

This is one of the many resource controls that Solaris 10 puts at our disposal. For a complete list of resource control please read the official documentation. This resource control limits the amount of CPU available at the project level. It's a privileged operation and a value of 50 corresponds to the 50% of one CPU. When the limit is reached, the associated behavior is deny, which actually denies the project more CPU. The machine I'm running is two processors machine so all of the processes running under the user.custom-user project will have the 50% of one CPU, leaving free the remaining 50% of that CPU and the entire other CPU.

I use this project to execute long-running tasks which in reality aren't always expensive in terms of CPU, but with such configuration I can sit at my workstation, launch the processes I need in this project and let them run without worrying that the machine be left without CPU power to run the desktop and the application I'm using when logged in.

I configured many different projects for different classes of processes I run and their administration is pretty easy, once you're familiar with the resource controls available in Solaris 10. The mechanism of associating a default project to users and groups, moreover, is a quick way for an administrator to cap the resources used by a particular user, based on individual needs, or by a particular group. In this sense the word project really fits this kind of functionality: we have groups of people working at some projects with different resource needs each and Solaris 10 puts at our disposal a concept which really reflects the organization of our work.

Creating a Task and Moving Processes to Projects

As described in the previous section, the default project association is a quick means for administering the server resources. Our users, indeed, aren't even conscious that they're running their processes in such a resource capped environment.

Sometimes, however, you'll need to launch processes in different project or move processes from one project to another. For example, I haven't configured a project for myself as I'm not assigned a project during my working day. Sometimes, though, I need to execute processes in a particular project, as the long running processes I was describing earlier, or move an existing process from a project to another: this is the case if, for example, a process is unexpectedly consuming a larger amount of resources than I wish, and prefer to move it to a resource-capped project in order to improve the machine responsiveness.

If you need to create a task in a specific project, given that your user can join such project, can be done with the following command:

$ newtask -v -p my-project
25

This command creates a new task in the my-project project and the -v flag lets you see the new task generated for this task, which can be useful when used with other commands that accept this parameter. Launching the newtask command in your shell also has the side effect of putting your shell into the newly generated task, thus allowing you to immediately launch new process to be executed into this task.

Let's now suppose that you've detected a process which is consuming too much a resource and you prefer to move inside a properly capped project. The first thing you need is the process id of this process which can be obtained in a number of ways:

$ pgrep process-to-move
15257

Now that we have the process id of the process to be moved, we can create a new task in the desider project and move the process inside it. This can easily be done with just a one liner:

$ newtask -v -p my-project -c 15257

27

It's not necessary, but if you want to be sure that the process is now executing in the context of the desider project, you can use for example:

$ pgrep -T 27
15257

which confirms that the process number 15257 is executing inside the task 27.

Associating a SMF Managed Service with a Project

So far, we've seen the basic tools to define and manage Solaris 10 projects. The administrator is now able to:

• Define projects.
• Define resource caps for a projects.
• Move processes to tasks and assign tasks to projects.
• Assign a project to an user and/or a group of users.

So far, so good but there's still a little gotcha. How do you assign the identity (or the project) under which a SMF-managed service will run? In another installment we'll see how to do it.

Resource Controls for Network Virtualization

Solaris Network Virtualization facility takes advantage of resource controls as well. You can read this blog post for a quick walkthrough.

Taking Advance of Solaris Projects and Resource Control for Virtualization Technologies

Solaris Projects and Resource Controls are of great advantage when used in conjunction with virtualization technologies such as VirtualBox. You can read this blog post for a quick introduction and a walkthrough to get you started.

Conclusion

This is a really short introductory walk through inside Solaris 10 capabilities of configuring and monitoring resource utilization. The official documentation describes the list of resource controls that Solaris 10 puts at the administrator's disposal and the complete set of commands that can be used to monitor and even change these parameters at runtime, when they could also be life-saviours. Knowing of their existence, at least, may one day ease your life.

Next Steps

You can now:

• Read this post about how to set up Subversion as an inted service.
• Read this post about setting up SSH access to Subversion repositories.
• Read this post about how to set up Subversion post-commit hooks.
• Read this post about filtering Subversion commit using a post-commit hook.