Physical machines are home to many types of resources these days. The traditional cores, memory, disk, now share space with gpus, co-processors or even protein sequence analysis accelerators.
To facilitate use and management of these resources, a new feature is available in HTCondor for extending machine resources. Analogous to concurrency limits, which operate on a pool / global level, machine resources operate on a machine / local level.
The feature allows a machine to advertise that it has specific types of resources available. Jobs can then specify that they require those specific types of resources. And the matchmaker will take into account the new resource types.
By example, a machine may have some GPU resources, an RS232 connected to your favorite telescope, and a number of physical spinning hard disk drives. The configuration for this would be,
MACHINE_RESOURCE_NAMES = GPU, RS232, SPINDLE MACHINE_RESOURCE_GPU = 2 MACHINE_RESOURCE_RS232 = 1 MACHINE_RESOURCE_SPINDLE = 4 SLOT_TYPE_1 = cpus=100%,auto SLOT_TYPE_1_PARTITIONABLE = TRUE NUM_SLOTS_TYPE_1 = 1
Aside – cpus=100%,auto instead of just auto because of GT3327. Also, the configuration for SLOT_TYPE_1 will likely go away in the future when all slots are partitionable by default.
Once a machine with this configuration is running,
$ condor_status -long | grep -i MachineResources MachineResources = "cpus memory disk swap gpu rs232 spindle" $ condor_status -long | grep -i -e TotalCpus -e TotalMemory -e TotalGpu -e TotalRs232 -e TotalSpindle TotalCpus = 24 TotalMemory = 49152 TotalGpu = 2 TotalRs232 = 1 TotalSpindle = 4 $ condor_status -long | grep -i -e ^Cpus -e ^Memory -e ^Gpu -e ^Rs232 -e ^Spindle Cpus = 24 Memory = 49152 Gpu = 2 Rs232 = 1 Spindle = 4
As you can see, the machine is reporting the different types of resources, how many of each it has and how many are currently available.
A job can take advantage of these new types of resources using a syntax already familiar for requesting resources from partitionable slots.
To consume one of the GPUs,
cmd = luxmark.sh request_gpu = 1 queue
Or for a disk intensive workload,
cmd = hadoop_datanode.sh request_spindle = 1 queue
With these jobs submitted and running,
$ condor_status
Name OpSys Arch State Activity LoadAv Mem ActvtyTime
slot1@eeyore LINUX X86_64 Unclaimed Idle 0.400 48896 0+00:00:28
slot1_1@eeyore LINUX X86_64 Claimed Busy 0.000 128 0+00:00:04
slot1_2@eeyore LINUX X86_64 Claimed Busy 0.000 128 0+00:00:04
Machines Owner Claimed Unclaimed Matched Preempting
X86_64/LINUX 3 0 2 1 0 0
Total 3 0 2 1 0 0
$ condor_status -l slot1@eeyore | grep -i -e ^Cpus -e ^Memory -e ^Gpu -e ^Rs232 -e ^Spindle
Cpus = 22
Memory = 48896
Gpu = 1
Rs232 = 1
Spindle = 3
That’s 22 cores, 1 gpu and 3 spindles still available.
Submit four more of the spindle consuming jobs and you’ll find the fourth does not run, because the available number of spindles is 0.
$ condor_status -l slot1@eeyore | grep -i -e ^Cpus -e ^Memory -e ^Gpu -e ^Rs232 -e ^Spindle Cpus = 19 Memory = 48512 Gpu = 1 Rs232 = 1 Spindle = 0
Since these custom resources are available as attributes in various ClassAds the same way Cpu, Memory and Disk are, all the policy, management and reporting capabilities you would expect is available.
