Moe Jette | 24 Apr 01:03 2015

Slurm versions 14.11.6 is now available

Slurm version 14.11.6 is now available with quite a few bug fixes as  
listed below.

Slurm downloads are available from

* Changes in Slurm 14.11.6
  -- If SchedulerParameters value of bf_min_age_reserve is configured, then
     a newly submitted job can start immediately even if there is a higher
     priority non-runnable job which has been waiting for less time than
  -- qsub wrapper modified to export "all" with -V option
  -- RequeueExit and RequeueExitHold configuration parameters modified  
to accept
     numeric ranges. For example "RequeueExit=1,2,3,4" and  
"RequeueExit=1-4" are
  -- Correct the job array specification parser to accept brackets in job array
     expression (e.g. "123_[4,7-9]").
  -- Fix for misleading job submit failure errors sent to users. Previous error
     could indicate why specific nodes could not be used (e.g. too  
small memory)
     when other nodes could be used, but were not for another reason.
  -- Fix squeue --array to display correctly the array elements when the
     % separator is specified at the array submission time.
  -- Fix priority from not being calculated correctly due to memory issues.
  -- Fix a transient pending reason 'JobId=job_id has invalid QOS'.
  -- A non-administrator change to job priority will not be persistent except
(Continue reading)

Jacob Jenson | 1 Apr 18:12 2015

CFP: Slurm User Group Meeting 2015

You are invited to submit an abstract of a tutorial, technical presentation or site report to be given at the Slurm User Group Meeting 2015. This event is sponsored and organized by SchedMD and The George Washington University. The meeting will be held in Washington, DC on 15-16 September 2015.

This international event is opened to everyone who wants to:
Learn more about Slurm, a highly scalable Resource Manager and Job Scheduler
Share their knowledge and experience with other users and administrators
Get detailed information about the latest features and developments
Share requirements and discuss future developments

Everyone who wants to present their own usage, developments, site report, or tutorial about Slurm is invited to send an abstract to 

Important Dates:
1 June 2015: Abstracts due
15 June 2015: Notification of acceptance
15-16 September 2015: Slurm User Group Meeting 2015

Program Committee:
Yiannis Georgiou (Bull)
Brian Gilmer (Cray)
Matthieu Hautreux (CEA)
Morris Jette (SchedMD)
Bruce Pfaff (NASA Goddard Space Flight Center)
Tim Wickberg (The George Washington University)

Jacob Jenson
Slurm Support and Development
Moe Jette | 19 Mar 22:29 2015

Slurm versions 14.11.5 and 15.08.0-pre3 are now available

Version 14.11.5 contains quite a few bug fixes generated over the past  
five weeks including two high impact bugs. There is a fix for the  
slurmdbd daemon aborting if a node is set to a DOWN state and it's  
"reason" field is NULL. The other important bug fix will prevent  
someone from being able to kill a job array belonging to another user.  
Details about all of the changes are appended.

Version 15.08.0-pre3 represents the current state of Slurm development  
for the release planned in August 2015 and is intended for development  
and test purposes only. Notable enhancements include power capping  
support for Cray systems and add the ability for a compute node to be  
allocated to multiple jobs, but restricted to one user at a time.

Both versions can be downloaded from

* Changes in Slurm 14.11.5
  -- Correct the squeue command taking into account that a node can
     have NULL name if it is not in DNS but still in slurm.conf.
  -- Fix slurmdbd regression which would cause a segfault when a node is set
     down with no reason.
  -- BGQ - Fix issue with job arrays not being handled correctly
     in the runjob_mux plugin.
  -- Print FAIR_TREE, if configured, in "scontrol show config" output for
  -- Add SLURM_JOB_GPUS environment variable to those available in the Prolog.
  -- Load lua-5.2 library if using lua5.2 for lua job submit plugin.
  -- GRES logic: Prevent bad node_offset due to not preserving no_consume flag.
  -- Fix wrong variables used in the wrapper functions needed for systems that
     don't support strong_alias
  -- Fix code for apple computers SOL_TCP is not defined
  -- Cray/BASIL - Check for mysql credentials in /root/.my.cnf.
  -- Fix sprio showing wrong priority for job arrays until priority is
  -- Account to batch step all CPUs that are allocated to a job not
     just one since the batch step has access to all CPUs like other steps.
  -- Fix job getting EligibleTime set before meeting dependency requirements.
  -- Correct the initialization of QOS MinCPUs per job limit.
  -- Set the debug level of information messages in cgroup plugin to debug2.
  -- For job running under a debugger, if the exec of the task fails, then
     cancel its I/O and abort immediately rather than waiting 60 seconds for
     I/O timeout.
  -- Fix associations not getting default qos set until after a restart.
  -- Set the value of total_cpus not to be zero before invoking
  -- MySQL - When requesting cluster resources, only return resources for the
     cluster(s) requested.
  -- Add TaskPluginParam=autobind=threads option to set a default  
binding in the
     case that "auto binding" doesn't find a match.
  -- Introduce a new SchedulerParameters variable nohold_on_prolog_fail.
     If configured don't requeue jobs on hold is a Prolog fails.
  -- Make it so sched_params isn't read over and over when an epilog complete
     message comes in
  -- Fix squeue -L <licenses> not filtering out jobs with licenses.
  -- Changed the implementation of xcpuinfo_abs_to_mac() be identical
     _abs_to_mac() to fix CPUs allocation using cpuset cgroup.
  -- Improve the explanation of the unbuffered feature in the
     srun man page.
  -- Make taskplugin=cgroup work for core spec.  needed to have task/cgroup
  -- Fix reports not using the month usage table.
  -- BGQ - Sanity check given for translating small blocks into slurm  
  -- Fix bug preventing the requeue/hold or requeue/special_exit of  
job from the
     completing state.
  -- Cray - Fix for launching batch step within an existing job allocation.
  -- Cray - Add ALPS_APP_ID_ENV environment variable.
  -- Increase maximum MaxArraySize configuration parameter value from 1,000,001
     to 4,000,001.
  -- Added new SchedulerParameters value of bf_min_age_reserve. The backfill
     scheduler will not reserve resources for pending jobs until they have
     been pending for at least the specified number of seconds. This can be
     valuable if jobs lack time limits or all time limits have the same value.
  -- Fix support for --mem=0 (all memory of a node) with  
select/cons_res plugin.
  -- Fix bug that can permit someone to kill job array belonging to  
another user.
  -- Don't set the default partition on a license only reservation.
  -- Show a NodeCnt=0, instead of NO_VAL, in "scontrol show res" for a license
     only reservation.
  -- BGQ - When using static small blocks make sure when clearing the job the
     block is set up to it's original state.
  -- Start job allocation using lowest numbered sockets for block task
     distribution for consistency with cyclic distribution.

* Changes in Slurm 15.08.0pre3
-- CRAY - addition of acct_gather_energy/cray plugin.
-- Add job credential to "Run Prolog" RPC used with a configuration of
    PrologFlags=alloc. This allows the Prolog to be passed identification of
    GPUs allocated to the job.
-- Add SLURM_JOB_CONSTAINTS to environment variables available to the Prolog.
-- Added "--mail=stage_out" option to job submission commands to notify user
    when burst buffer state out is complete.
-- Require a "Reason" when using scontrol to set a node state to DOWN.
-- Mail notifications on job BEGIN, END and FAIL now apply to a job array as a
    whole rather than generating individual email messages for each task in the
    job array.
-- task/affinity - Fix memory binding to NUMA with cpusets.
-- Display job's estimated NodeCount based off of partition's configured
    resources rather than the whole system's.
-- Add AuthInfo option of "cred_expire=#" to specify the lifetime of a job
    step credential. The default value was changed from 1200 to 120 seconds.
-- Set the delay time for job requeue to the job credential lifetime (120
    seconds by default). This insures that prolog runs on every node when a
    job is requeued. (This change will slow down launch of re-queued jobs).
-- Add AuthInfo option of "cred_expire=#" to specify the lifetime of a job
    step credential.
-- Remove srun --max-launch-time option. The option has not been functional
    since Slurm version 2.0.
-- Add sockets and cores to TaskPluginParams' autobind option.
-- Added LaunchParameters configuration parameter. Have srun command test
    locally for the executable file if LaunchParameters=test_exec or the
    environment variable SLURM_TEST_EXEC is set. Without this an invalid
    command will generate one error message per task launched.
-- Fix the slurm /etc/init.d script to return 0 upon stopping the
    daemons and return 1 in case of failure.
-- Add the ability for a compute node to be allocated to multiple jobs, but
    restricted to a single user. Added "--exclusive=user" option to salloc,
    sbatch and srun commands. Added "owner" field to node record, visible using
    the scontrol and sview commands. Added new partition configuration  

Morris "Moe" Jette
Commercial Slurm Development and Support

jette | 12 Feb 22:49 2015

Slurm versions 14.11.4 and 15.08.0-pre2 are now available

Slurm versions 14.11.4 and 15.08.0-pre2 are now available from

Version 14.11.4 contains quite a few bug fixes generated over the past  
five weeks. Several of these are related to job arrays, including one  
that can cause the slurmctld daemon to abort. Version 15.08.0-pre2  
represents the current state of Slurm development for the released  
planned in August 2015 and is intended for development and test  
purposes only. It includes some development work for burst buffers,  
power management, and inter-cluster job dependencies. More details  
about the changes are shown below.

* Changes in Slurm 14.11.4
  -- Make sure assoc_mgr locks are initialized correctly.
  -- Correct check of enforcement when filling in an association.
  -- Make sacctmgr print out classification correctly for clusters.
  -- Add array_task_str to the perlapi job info.
  -- Fix for slurmctld abort with GRES types configured and no CPU binding.
  -- Fix for GRES scheduling where count > 1 per topology type (or GRES types).
  -- Make CR_ONE_TASK_PER_CORE work correctly with task/affinity.
  -- job_submit/pbs - Fix possible deadlock.
  -- job_submit/lua - Add "alloc_node" to job information available.
  -- Fix memory leak in mysql accounting when usage rollup happens.
  -- If users specify ALL together with other variables using the
     --export sbatch/srun command line option, propagate the users'
     environ to the execution side.
  -- Fix job array scheduling anomaly that can stop scheduling of valid tasks.
  -- Fix perl api tests for libslurmdb to work correctly.
  -- Remove some misleading logs related to non-consumable GRES.
  -- Allow --ignore-pbs to take effect when read as an #SBATCH argument.
  -- Fix Slurmdb::clusters_get() in perl api from not returning information.
  -- Fix TaskPluginParam=Cpusets from logging error message about not  
being able
     to remove cpuset dir which was already removed by the release_agent.
  -- Fix sorting by time left in squeue.
  -- Fix the file name substitution for job stderr when %A, %a %j and %u
     are specified.
  -- Remove minor warning when compiling slurmstepd.
  -- Fix database resources so they can add new clusters to them after  
they have
     initially been added.
  -- Use the slurm_getpwuid_r wrapper of getpwuid_r to handle possible
  -- Correct the scontrol man page and command listing which node states can
     be set by the command.
  -- Stop sacct from printing non-existent stat information for
     Front End systems.
  -- Correct srun and acct_gather.conf man pages, mention Filesystem instead
     of Lustre.
  -- When a job using multiple partition starts send to slurmdbd only
     the partition in which the job runs.
  -- ALPS - Fix depth for MemoryAllocation in BASIL with CLE 5.2.3.
  -- Fix assoc_mgr hash to deal with users that don't have a uid yet  
when making
  -- When a job uses multiple partition set the environment variable
     SLURM_JOB_PARTITION to be the one in which the job started.
  -- Print spurious message about the absence of cgroup.conf at log  
level debug2
     instead of info.
  -- Enable CUDA v7.0+ use with a Slurm configuration of TaskPlugin=task/cgroup
     ConstrainDevices=yes (in cgroup.conf). With that configuration
     CUDA_VISIBLE_DEVICES will start at 0 rather than the device number.
  -- Fix job array logic that can cause slurmctld to abort.
  -- Report job "shared" field properly in scontrol, squeue, and sview.
  -- If a job is requeued because of RequeueExit or RequeueExitHold sent event
     REQUEUED to slurmdbd.
  -- Fix build if hwloc is in non-standard location.
  -- Fix slurmctld job recovery logic which could cause the last task in a job
     array to be lost.
  -- Fix slurmctld initialization problem which could cause requeue of the last
     task in a job array to fail if executed prior to the slurmctld loading
     the maximum size of a job array into a variable in the job_mgr.c module.
  -- Fix fatal in controller when deleting a user association of a user which
     had been previously removed from the system.
  -- MySQL - If a node state and reason are the same on a node state change
     don't insert a new row in the event table.
  -- Fix issue with "sreport cluster AccountUtilizationByUser" when using
  -- Fix perlapi tests for libslurm perl module.

* Changes in Slurm 15.08.0pre2
  -- Add the environment variables SLURM_JOB_ACCOUNT, SLURM_JOB_QOS
     and SLURM_JOB_RESERVATION in the batch/srun jobs.
  -- Add sview burst buffer display.
  -- Properly enforce partition Shared=YES option. Previously oversubscribing
     resources required gang scheduling to be configured.
  -- Enable per-partition gang scheduling resource resolution (e.g.  
the partition
     can have SelectTypeParameters=CR_CORE, while the global value is  
  -- Make it so a newer version of a slurmstepd can talk to an older srun.
     allocation. Nodes could have been added while waiting for an allocation.
  -- Expanded --cpu-freq parameters to include min-max:governor specifications.
     --cpu-freq now supported on salloc and sbatch.
  -- Add support for optimized job allocations with respect to SGI Hypercube
     NOTE: Only supported with select/linear plugin.
     NOTE: The program contribs/sgi/netloc_to_topology can be used to build
     Slurm's topology.conf file.
  -- Remove 64k validation of incoming RPC nodelist size. Validated at 64MB
     when unpacking.
  -- In slurmstepd() add the user primary group if it is not part of the
     groups sent from the client.
  -- Added BurstBuffer field to advanced reservations.
  -- For advanced reservation, replace flag "License_only" with flag  
     It can be used to indicate the an advanced reservation resources (licenses
     and/or burst buffers) can be used with any compute nodes.
  -- Allow users to specify the srun --resv-ports as 0 in which case no ports
     will be reserved. The default behaviour is to allocate one port per task.
  -- Interpret a partition configuration of "Nodes=ALL" in slurm.conf as
     including all nodes defined in the cluster.
  -- Added new configuration parameters PowerParameters and PowerPlugin.
  -- Added power management plugin infrastructure.
  -- If job already exceeded one of its QOS/Accounting limits do not
     return error if user modifies QOS unrelated job settings.
  -- Added DebugFlags value of "Power".
  -- When caching user ids of AllowGroups use both getgrnam_r() and  
     then remove eventual duplicate entries.
  -- Remove rpm dependency between slurm-pam and slurm-devel.
  -- Remove support for the XCPU (cluster management) package.
  -- Add Slurmdb::jobs_get() interface to perl api.
  -- Performance improvement when sending data from srun to stepds when
     processing fencing.
  -- Add the feature to specify arbitrary field separator when running
     sacct -p or sacct -P. The command line option is --separator.
  -- Introduce slurm.conf parameter to use Proportional Set Size (PSS) instead
     of RSS to determinate the memory footprint of a job.
     Add an slurm.conf option not to kill jobs that is over memory limit.
  -- Add job submission command options: --sicp (available for inter-cluster
     dependencies) and --power (specify power management options) to salloc,
     sbatch, and srun commands.
  -- Add DebugFlags option of SICP (inter-cluster option logging).
  -- In order to support inter-cluster job dependencies, the MaxJobID
     configuration parameter default value has been reduced from 4,294,901,760
     to 2,147,418,112 and it's maximum value is now 2,147,463,647.
  -- Add QOS name to the output of a partition in squeue/scontrol/sview/smap.

Morris "Moe" Jette
Commercial Slurm Development and Support

Danny Auble | 12 Dec 20:56 2014

Slurm version 14.11.2 and 15.08.0-pre1 are now available

Slurm versions 14.11.2 and 15.08.0-pre1 are now available. Version 
14.11.2 includes quite a few relatively minor bug fixes.

Version 15.08.0 is under active development and its release is planned 
in August 2015.  While this is the first pre-release there is already 
quite a bit of new functionality.

Both versions can be downloaded from

Highlights of the 2 versions are these

* Changes in Slurm 14.11.2
  -- Fix Centos5 compile errors.
  -- Fix issue with association hash not getting the correct index which
     could result in seg fault.
  -- Fix salloc/sbatch -B segfault.
  -- Avoid huge malloc if GRES configured with "Type" and huge "Count".
  -- Fix jobs from starting in overlapping reservations that won't 
finish before
     a "maint" reservation begins.
  -- When node gets drained while in state mixed display its status as 
     in sinfo output.
  -- Allow priority/multifactor to work with sched/wiki(2) if all priorities
     have no weight.  This allows for association and QOS decay limits 
to work.
  -- Fix "squeue --start" to override SQUEUE_FORMAT env variable.
  -- Fix scancel to be able to cancel multiple jobs that are space 
  -- Log Cray MPI job calling exit() without mpi_fini(), but do not 
treat it as
     a fatal error. This partially reverts logic added in version 14.03.9.
  -- sview - Fix displaying of suspended steps elapsed times.
  -- Increase number of messages that get cached before throwing them away
     when the DBD is down.
  -- Fix jobs from starting in overlapping reservations that won't 
finish before
     a "maint" reservation begins.
  -- Restore GRES functionality with select/linear plugin. It was broken in
     version  14.03.10.
  -- Fix bug with GRES having multiple types that can cause slurmctld abort.
  -- Fix squeue issue with not recognizing "localhost" in --nodelist option.
  -- Make sure the bitstrings for a partitions Allow/DenyQOS are up to date
     when running from cache.
  -- Add smap support for job arrays and larger job ID values.
  -- Fix possible race condition when attempting to use QOS on a system 
  -- Fix issue with accounting_storage/filetxt and job arrays not being 
  -- In proctrack/linuxproc and proctrack/pgid, check the result of strtol()
     for error condition rather than errno, which might have a vestigial 
  -- Improve information recording for jobs deferred due to advanced
  -- Exports eio_new_initial_obj to the plugins and initialize kvs_seq on
     mpi/pmi2 setup to support launching.

* Changes in Slurm 15.08.0pre1
  -- Add sbcast support for file transfer to resources allocated to a 
job step
     rather than a job allocation.
  -- Change structures with association in them to assoc to save space.
  -- Add support for job dependencies jointed with OR operator (e.g.
  -- Add "--bb" (burst buffer specification) option to salloc, sbatch, 
and srun.
  -- Added configuration parameters BurstBufferParameters and 
  -- Added burst_buffer plugin infrastructure (needs many more functions).
  -- Make it so when the fanout logic comes across a node that is down 
we abandon
     the tree to avoid worst case scenarios when the entire branch is 
down and
     we have to try each serially.
  -- Add better error reporting of invalid partitions at submission time.
  -- Move will-run test for multiple clusters from the sbatch code into 
the API
     so that it can be used with DRMAA.
  -- If a non-exclusive allocation requests --hint=nomultithread on a
     CR_CORE/SOCKET system lay out tasks correctly.
  -- Avoid including unused CPUs in a job's allocation when cores or 
sockets are
  -- Added new job state of STOPPED indicating processes have been 
stopped with a
     SIGSTOP (using scancel or sview), but retain its allocated CPUs. 
Job state
     returns to RUNNING when SIGCONT is sent (also using scancel or sview).
  -- Added EioTimeout parameter to slurm.conf. It is the number of 
seconds srun
     waits for slurmstepd to close the TCP/IP connection used to relay data
     between the user application and srun when the user application 
  -- Remove slurmctld/dynalloc plugin as the work was never completed, 
so it is
     not worth the effort of continued support at this time.
  -- Remove DynAllocPort configuration parameter.
  -- Add advance reservation flag of "replace" that causes allocated 
     to be replaced with idle resources. This maintains a pool of available
     resources that maintains a constant size (to the extent possible).
  -- Added SchedulerParameters option of "bf_busy_nodes". When selecting
     resources for pending jobs to reserve for future execution (i.e. 
the job
     can not be started immediately), then preferentially select nodes 
that are
     in use. This will tend to leave currently idle resources available for
     backfilling longer running jobs, but may result in allocations 
having less
     than optimal network topology. This option is currently only 
supported by
     the select/cons_res plugin.
  -- Permit "SuspendTime=NONE" as slurm.conf value rather than only a 
     value to match "scontrol show config" output.
  -- Add the 'scontrol show cache' command which displays the associations
     in slurmctld.
  -- Test more frequently for node boot completion before starting a job.
     Provides better responsiveness.
  -- Fix PMI2 singleton initialization.
  -- Permit PreemptType=qos and PreemptMode=suspend,gang to be used 
     A high-priority QOS job will now oversubscribe resources and gang 
     but only if there are insufficient resources for the job to be started
     without preemption. NOTE: That with PreemptType=qos, the partition's
     Shared=FORCE:# configuration option will permit one job more per 
     to be run than than specified, but only if started by preemption.
  -- Remove the CR_ALLOCATE_FULL_SOCKET configuration option.  It is now the
  -- Fix a race condition in PMI2 when fencing counters can be out of sync.
  -- Increase the MAX_PACK_MEM_LEN define to avoid PMI2 failure when fencing
     with large amount of ranks.
  -- Add QOS option to a partition.  This will allow a partition to have
     all the limits a QOS has.  If a limit is set in both QOS the partition
     QOS will override the job's QOS unless the job's QOS has the
     PartitionQOS flag set.
  -- The task_dist_states variable has been split into "flags" and "base"
     components. Added SLURM_DIST_PACK_NODES and 
     to give user greater control over task distribution. The srun 
--dist options
     has been modified to accept a "Pack" and "NoPack" option. These 
options can
     be used to override the CR_PACK_NODE configuration option.

jette | 26 Nov 19:39 2014

Slurm version 14.11.1 is now available

We have just released Slurm version 14.11.1. This includes a fix for a  
race condition that can deadlock the slurmctld daemon when job_submit  
plugins are used, plus a few minor changes as identified below. You  
can download it from:

* Changes in Slurm 14.11.1
  -- Get libs correct when doing the xtree/xhash make check.
  -- Update xhash/tree make check to work correctly with current code.
  -- Remove the reference 'experimental' for the jobacct_gather/cgroup
  -- Add QOS manipulation examples to the qos.html documentation page.
  -- If 'squeue -w node_name' specifies an unknown host name print
     an error message and return 1.
  -- Fix race condition in job_submit plugin logic that could cause  
slurmctld to
  -- Job wait reason of "ReqNodeNotAvail" expanded to identify  
unavailable nodes
     (e.g. "ReqNodeNotAvail(Unavailable:tux[3-6])").

Morris "Moe" Jette

Danny Auble | 14 Nov 00:18 2014

Slurm 14.11.0 is now available

Slurm version 14.11.0 is now available. This is a major Slurm release 
with many new features. See the RELEASE_NOTES and NEWS files in the 
distribution for detailed descriptions of the changes, a few of which 
are noted below.

Upgrading from Slurm versions 2.6 or 14.03 should proceed without loss 
of jobs or other state.  Just be sure to upgrade the slurmdbd first. 
(Upgrades from pre-releases of version 14.11 may result job loss.)

Slurm downloads are available from

Thanks to all those who helped make this release!

Highlights of changes in Slurm version 14.11.0 include:
  -- Added job array data structure and removed 64k array size restriction.
  -- Added support for reserving CPUs and/or memory on a compute node 
for system
  -- Added support for allocation of generic resources by model type for
     heterogeneous systems (e.g. request a Kepler GPU, a Tesla GPU, or a 
GPU of
     any type).
  -- Added support for non-consumable generic resources that are 
limited, but
     can be shared between jobs.
  -- Added support for automatic job requeue policy based on exit value.
  -- Refactor job_submit/lua interface. LUA FUNCTIONS NEED TO CHANGE! The
     lua script no longer needs to explicitly load meta-tables, but 
     is available directly using names slurm.reservations,,
     slurm.log_info, etc. Also, the job_submit.lua script is reloaded when
     updated without restarting the slurmctld daemon.
  -- Eliminate native Cray specific port management. Native Cray systems 
     now use the MpiParams configuration parameter to specify ports to 
be used
     for communications. When upgrading Native Cray systems from version 
     all running jobs should be killed and the switch_cray_state file (in
     SaveStateLocation of the nodes where the slurmctld daemon runs) must be
     explicitly deleted.

Danny Auble | 4 Nov 02:35 2014

Slurm versions 14.03.10 and 14.11.0-rc3 are now available

Slurm version 14.03.10 includes quite a few relatively minor bug fixes, 
and will most likely be the last 14.03 release.  Thanks to all those who 
helped make this a very stable release.

We hope to officially tag 14.11.0 before SC14.  Version 14.11.0-rc3 
includes a few bug fixes discovered in recent testing but is looking 
very stable. Thanks to everyone participating in the testing!  If you 
can, please test this release so we can attempt to fix as many issues as 
we can before we tag 14.11.0.

Just a heads up, version 15.08 is already starting development we will 
most likely tag a pre1 of this later this month.

Slurm downloads are available from

Here are some snips from the NEWS file on what has changed since the 
last releases.

* Changes in Slurm 14.03.10
  -- Fix a few sacctmgr error messages.
  -- Treat non-zero SlurmSchedLogLevel without SlurmSchedLogFile as a fatal
  -- Correct sched_config.html documentation SchedulingParameters
     should be SchedulerParameters.
  -- When using gres and cgroup ConstrainDevices set correct access
     permission for the batch step.
  -- Fix minor memory leak in jobcomp/mysql on slurmctld reconfig.
  -- Fix bug that prevented preservation of a job's GRES bitmap on slurmctld
     restart or reconfigure (bug was introduced in 14.03.5 "Clear record 
of a
     job's gres when requeued" and only applies when GRES mapped to specific
  -- BGQ: Fix race condition when job fails due to hardware failure and is
     requeued. Previous code could result in slurmctld abort with NULL 
  -- Prevent negative job array index, which could cause slurmctld to crash.
  -- Fix issue with squeue/scontrol showing correct node_cnt when only tasks
     are specified.
  -- Check the status of the database connection before using it.
  -- ALPS - If an allocation requests -n set the BASIL -N option to the
     amount of tasks / number of node.
  -- ALPS - Don't set the env var APRUN_DEFAULT_MEMORY, it is not needed 
  -- Fix potential buffer overflow.
  -- Give better estimates on pending node count if no node count is 
  -- BLUEGENE - Fix issue where requeuing jobs could cause an assert.

* Changes in Slurm 14.11.0rc3
  -- Allow envs to override autotools binaries in
  -- Added system services files.
  -- If the jobs pends with DependencyNeverSatisfied keep it pending 
even after
     the job which it was depending upon was cleaned.
  -- Let operators (in addition to user root and SlurmUser) see job 
script for
     other user's jobs.
  -- Perl API modified to return node state of MIXED rather than 
     only some CPUs allocated.
  -- Double Munge connect retry timeout from 1 to 2 seconds.
  -- sview - Remove unneeded code that was resolved globally in commit
  -- Collect and report the accounting of the batch step and its children.
  -- Add configure checks for faccessat and eaccess, and make use of one of
     them if available.
  -- Make configure --enable-developer also set --enable-debug
  -- Introduce a SchedulerParameters variable kill_invalid_depend, if set
     then jobs pending with invalid dependency are going to be terminated.
  -- Move spank_user_task() call in slurmstepd after the task_g_pre_launch()
     so that the task affinity information is available to spank.
  -- Make /etc/init.d/slurm script return value 3 when the daemon is
     not running. This is required by Linux Standard Base Core
     Specification 3.1

jette | 17 Oct 23:08 2014

Slurm versions 14.03.9 and 14.11.0-rc2 are now available

Slurm versions 14.03.9 and 14.11.0-rc2 are now available.
Version 14.03.9 includes quite a few relatively minor bug fixes.
Version 14.11.0-rc2 includes a few bug fixes discovered in recent testing.
Thanks to everyone participating in the testing!
Version 14.11.0 is no longer under active development, but is undergoing
testing for a planned release in early November.

Slurm downloads are available from

* Changes in Slurm 14.03.9
  -- If slurmd fails to stat(2) the configuration print the string describing
     the error code.
  -- Fix for mixing core base reservations with whole node based reservations
     to avoid overlapping erroneously.
  -- BLUEGENE - Remove references to Base Partition.
  -- sview - If compiled on a non-bluegene system then used to view a BGQ fix
     to allow sview to display blocks correctly.
  -- Fix bug in update reservation. When modifying the reservation the end time
     was set incorrectly.
  -- The start time of a reservation that is in ACTIVE state cannot be  
  -- Update the cgroup documentation about release agent for devices.
  -- MYSQL - fix for setting up preempt list on a QOS for multiple QOS.
  -- Correct a minor error in the scancel.1 man page related to the
     --signal option.
  -- Enhance the scancel.1 man page to document the sequence of signals sent
  -- Fix slurmstepd core dump if the cgroup hierarchy is not completed
     when terminating the job.
  -- Fix hostlist_shift to be able to give correct node names on names with a
     different number of dimensions than the cluster.
  -- BLUEGENE - Fix invalid pointer in corner case in the plugin.
  -- Make sure on a reconfigure the select information for a node is preserved.
  -- Correct logic to support job GRES specification over 31 bits (problem
     in logic converting int to uint32_t).
  -- Remove logic that was creating GRES bitmap for node when not needed (only
     needed when GRES mapped to specific files).
  -- BLUEGENE - Fix sinfo -tr before it would only print idle nodes correctly.
  -- BLUEGENE - Fix for licenses_only reservation on bluegene systems.
  -- sview - Verify pointer before using strchr.
  -- -M option on tools talking to a Cray from a non-Cray fixed.
  -- CRAY - Fix rpmbuild issue for missing file slurm.conf.template.
  -- Fix race condition when dealing with removing many associations at
     different times when reservations are using the associations that are
     being deleted.
  -- When a node's state is set to power_down/power_up, then execute
     SuspendProgram/ResumeProgram even if previously executed for that node.
  -- Fix logic determining when job configuration (i.e. running node power up
     logic) is complete.
  -- Setting the state of a node in powered down state node to "resume" will
     no longer cause it to reboot, but only clear the "drain" state flag.
  -- Fix srun documentation to remove SLURM_NODELIST being equivalent as the -w
     option (since it isn't).
  -- Fix issue with --hint=nomultithread and allocations with steps running
     arbitrary layouts (test1.59).
  -- PrivateData=reservation modified to permit users to view the reservations
     which they have access to (rather then preventing them from seeing ANY
     reservation).  Backport from 14.11 commit 77c2bd25c.
  -- Fix PrivateData=reservation when using associations to give privileges to
     a reservation.
  -- Better checking to see if select plugin is linear or not.
  -- Add support for time specification of "fika" (3 PM).
  -- Standardize qstat wrapper more.
  -- Provide better estimate of minimum node count for pending jobs using more
     job parameters.
  -- ALPS - Add SubAllocate to cray.conf file for those who like the way <=2.5
     did the ALPS reservation.
  -- Safer check to avoid invalid reads when shutting down the slurmctld with
     lots of jobs.
  -- Fix minor memory leak in the backfill scheduler when shutting down.
  -- Add ArchiveResvs to the output of sacctmgr show config and init  
the variable
     on slurmdbd startup.
  -- SLURMDBD - Only set the archive flag if purging the object
     (i.e ArchiveJobs PurgeJobs).  This is only a cosmetic change.
  -- Fix for job step memory allocation logic if step requests GRES and memory
     is not allocations are not managed.
  -- Fix sinfo to display mixed nodes as allocated in '%F' output.
  -- Sview - Fix cpu and node counts for partitions.
  -- Ignore NO_VAL in SLURMDB_PURGE_* macros.
  -- ALPS - Don't drain nodes if epilog fails.  It leaves them in drain state
     with no way to get them out.
  -- Fix issue with task/affinity oversubscribing cpus erroneously when
     using --ntasks-per-node.
  -- MYSQL - Fix load of archive files.
  -- Treat Cray MPI job calling exit() without mpi_fini() as fatal error for
     that specific task and let srun handle all timeout logic.
  -- Fix small memory leak in jobcomp/mysql.
  -- Correct tracking of licenses for suspended jobs on slurmctld  
reconfigure or
  -- If failed to launch a batch job, requeue it in held state.

* Changes in Slurm 14.11.0rc2
  -- Logs for jobs which are explicitly requeued will say so rather than saying
     that a node in their allocation failed.
  -- Updated the documentation about the remote licenses served by
     the Slurm database.
  -- Insure that slurm_spank_exit() is only called once from srun.
  -- Change the signature of net_set_low_water() to use 4 bytes instead of 8.
  -- Export working_cluster_rec in as well as move some function
     definitions needed for drmaa.
  -- If using cons_res or serial cause a fatal in the plugin instead of causing
     the SelectTypeParameters to magically set to CR_CPU.
  -- Enhance task/affinity auto binding to consider tasks * cpus-per-task.
  -- Fix regression the priority/multifactor which would cause memory  
     Issue is only in rc1.
  -- Add PrivateData value of "cloud". If set, powered down nodes in the cloud
     will be visible.
  -- Sched/backfill - Eliminate clearing start_time of running jobs.
  -- Fix various backwards compatibility issues.
  -- If failed to launch a batch job, requeue it in hold.

Morris "Moe" Jette

jette | 29 Sep 18:38 2014

Slurm User Group Meeting 2014: Presentations now online

About 70 people attended the Slurm User Group Meeting last week in  
Lugano Switzerland. There were a lot of good presentations and  
discussions. Copies of the presentations are now available online at

NOTE: A few of the presentations are missing, but will be posted when  

Morris "Moe" Jette

Danny Auble | 17 Sep 22:58 2014

Slurm versions 14.03.8 and 14.11.0-pre5 are now available

Slurm versions 14.03.8 and 14.11.0-pre5 are now available. Version 
14.03.8 includes quite a few relatively minor bug fixes.

Version 14.11.0 is under active development and its release is planned 
in November 2014.  Much of its features and performance enhancements 
will be discussed next week at SLUG 2014 in Lugano Switzerland.

Note to all developers, code freeze for new features in 14.11 will be at 
the end of this month (September).

Slurm downloads are available from

Highlights of the 2 versions are these

* Changes in Slurm 14.03.8
  -- Fix minor memory leak when Job doesn't have nodes on it (Meaning 
the job
     has finished)
  -- Fix sinfo/sview to be able to query against nodes in reserved and other
  -- Make sbatch/salloc read in (SLURM|(SBATCH|SALLOC))_HINT in order to
     handle sruns in the script that will use it.
  -- srun properly interprets a leading "." in the executable name based 
     the working directory of the compute node rather than the submit host.
  -- Fix Lustre misspellings in hdf5 guide
  -- Fix wrong reference in slurm.conf man page to what --profile option 
     be used for AcctGatherFilesystemType.
  -- Update HDF5 document to point out the SlurmdUser is who creates the
     ProfileHDF5Dir directory as well as all it's sub-directories and files.
  -- CRAY NATIVE - Remove error message for srun's ran inside an salloc that
     had --network= specified.
  -- Defer job step initiation of required GRES are in use by other 
steps rather
     than immediately returning an error.
  -- Deprecate --cpu_bind from sbatch and salloc.  These never worked 
     and only caused confusion since the cpu_bind options mostly refer to a
     step we opted to only allow srun to set them in future versions.
  -- Modify sgather to work if Nodename and NodeHostname differ.
  -- Changed use of JobContainerPlugin where it should be JobContainerType.
  -- Fix for possible error if job has GRES, but the step explicitly 
requests a
     GRES count of zero.
  -- Make "srun --gres=none ..." work when executed without a job 
  -- Change the global eio_shutdown_time to a field in eio handle.
  -- Advanced reservation fixes for heterogeneous systems, especially when
     reserving cores.
  -- If --hint=nomultithread is used in a job allocation make sure any 
     ran inside the allocation can read the environment correctly.
  -- If batchdir can't be made set errno correctly so the slurmctld is 
  -- Remove repeated batch complete if batch directory isn't able to be made
     since the slurmd will send the same message.
  -- sacctmgr fix default format for list transactions.
  -- BLUEGENE - Fix backfill issue with backfilling jobs on blocks already
     reserved for higher priority jobs.
  -- When creating job arrays the job specification files for each elements
     are hard links to the first element specification files. If the 
     fails to make the links the files are copied instead.
  -- Fix error handling for job array create failure due to inability to 
     job files (script and environment).
  -- Added patch in the contribs directory for integrating make version 
4.0 with
     Slurm and renamed the previous patch "make-3.81.slurm.patch".
  -- Don't wait for an update message from the DBD to finish before 
sending rc
     message back.  In slow systems with many associations this could speed
     responsiveness in sacctmgr after adding associations.
  -- Eliminate race condition in enforcement of MaxJobCount limit for 
job arrays.
  -- Fix anomaly allocating cores for GRES with specific device/CPU mapping.
  -- cons_res - When requesting exclusive access make sure we set the number
     of cpus in the job_resources_t structure so as nodes finish the correct
     cpu count is displayed in the user tools.
  -- If the job_submit plugin calls take longer than 1 second to run, 
print a
  -- Make sure transfer_s_p_options transfers all the portions of the
     s_p_options_t struct.
  -- Correct the srun man page, the SLURM_CPU_BIND_VERBOSE, 
     SLURM_CPU_BIND_LIST environment variable are set only when 
     plugin is configured.
  -- sacct - Initialize variables correctly to avoid incorrect structure
  -- Performance adjustment to avoid calling a function multiple times 
when it
     only needs to be called once.
  -- Give more correct waiting reason if job is waiting on association/QOS
     MaxNode limit.
  -- DB - When sending lft updates to the slurmctld only send 
non-deleted lfts.
  -- BLUEGENE - Fix documentation on how to build a reservation less than
     a midplane.
  -- If Slurmctld fails to read the job environment consider it an error
     and abort the job.
  -- Add the name of the node a job is running on to the message printed by
     slurmstepd when terminating a job.
  -- Remove unsupported options from sacctmgr help and the dump function.
  -- Update sacctmgr man page removing reference to obsolete parameter
  -- Added more validity checking of incoming job submit requests.

* Changes in Slurm 14.11.0pre5
  -- Fix sbatch --export=ALL, it was treated by srun as a request to 
     export only the environment variable named "ALL".
  -- Improve scheduling of jobs in reservations that overlap other 
  -- Modify sgather to make global file systems easier to configure.
  -- Added sacctmgr reconfig to reread the slurmdbd.conf in the slurmdbd.
  -- Modify scontrol job operations to accept comma delimited list of 
job IDs.
     Applies to job update, hold, release, suspend, resume, requeue, and
     requeuehold operations.
  -- Refactor job_submit/lua interface. LUA FUNCTIONS NEED TO CHANGE! The
     lua script no longer needs to explicitly load meta-tables, but 
     is available directly using names slurm.reservations,,
     slurm.log_info, etc. Also, the job_submit.lua script is reloaded when
     updated without restarting the slurmctld daemon.
  -- Allow users to specify --resv_ports to have value 0.
  -- Cray MPMD (Multiple-Program Multiple-Data) support completed.
  -- Added ability for "scontrol update" to references jobs by JobName (and
     filtered optionally by UserID).
  -- Add support for an advanced reservation start time that remains 
     relative to the current time. This can be used to prevent the 
starting of
     longer running jobs on select nodes for maintenance purpose. See the
     reservation flag "TIME_FLOAT" for more information.
  -- Enlarge the jobid field to 18 characters in squeue output.
  -- Added "scontrol write config" option to save a copy of the current
     configuration in a file containing a time stamp.
  -- Eliminate native Cray specific port management. Native Cray systems 
     now use the MpiParams configuration parameter to specify ports to 
be used
     for commmunications. When upgrading Native Cray systems from 
version 14.03,
     all running jobs should be killed and the switch_cray_state file (in
     SaveStateLocation of the nodes where the slurmctld daemon runs) must be
     explicitly deleted.