Moe Jette | 21 Aug 01:35 2015
Gravatar

Slurm version 15.08.0-rc1 is now available


We are pleased to announce the availability of Slurm version 15.08.0-rc1
(release candidate 1). This version contains all of the features intended for
release 15.08 (with the exception of some minor burst buffer work) and we are
moving into a testing phase. You are invited to download this version  
and assist
in testing. Some highlights in this release include:
  -- Add TRES (Trackable resources) to track utilization of memory, GRES, burst
     buffer, license, and any other configurable resources in the accounting
     database.
  -- Add configurable billing weight that takes into consideration any  
TRES when
     calculating a job's resource utilization (for fair-share calculation).
  -- Add configurable prioritization factor that takes into consideration any
     TRES when calculating a job's resource utilization.
  -- Add burst buffer support infrastructure. Currently available  
plugin include
     burst_buffer/generic (uses administrator supplied programs to manage file
     staging) and burst_buffer/cray (uses Cray APIs to manage buffers).
  -- Add support for job dependencies joined with OR operator (e.g.
     "--depend=afterok:123?afternotok:124").
  -- Add advance reservation flag of "replace" that causes allocated resources
     to be replaced with idle resources. This maintains a pool of available
     resources that maintains a constant size (to the extent possible).
  -- Permit PreemptType=qos and PreemptMode=suspend,gang to be used together.
     A high-priority QOS job will now oversubscribe resources and gang  
schedule,
     but only if there are insufficient resources for the job to be started
     without preemption. NOTE: That with PreemptType=qos, the partition's
     Shared=FORCE:# configuration option will permit one job more per resource
(Continue reading)

Jacob Jenson | 8 Jul 02:02 2015

2015 Slurm User Group Meeting - Agenda and Registration

The 2015 Slurm User Group Meeting will be held on September 15 and 16 in Washington, DC. The meeting will include an assortment of tutorials, technical presentations, and site reports.

Reregistration and a preliminary agenda are available online:
http://slurm.schedmd.com/slurm_ug_agenda.html


Please register early to take advantage of the Early registration special
  • Early Registration - $250 USD per person for early registration by 31 July 2015
  • Standard Registration - $350 USD per person for standard registration by 31 August 2015
  • Late Registration - $600 USD per person for late registration starting 1 September 2015

Moe Jette | 8 Jul 01:06 2015
Gravatar

Slurm versions 14.11.8 and 15.08.0-pre6 are now available


Slurm version 14.11.8 includes about 30 relatively minor bug fixes  
developed over the past seven weeks while version 15.08.0-pre6  
contains new development scheduled for release next month. Details of  
changes are shown below. Slurm downloads are available from
http://www.schedmd.com/#repos

Also note that registration is open for the 2015 Slurm User Group  
Meeting. A preliminary agenda, registration and hotel information are  
available from
http://slurm.schedmd.com/slurm_ug_agenda.html

* Changes in Slurm 14.11.8
==========================
  -- Eliminate need for user to set user_id on job_update calls.
  -- Correct list of unavailable nodes reported in a job's "reason" field when
     that job can not start.
  -- Map job --mem-per-cpu=0 option to --mem=0.
  -- Fix squeue -o %m and %d unit conversion to Megabytes.
  -- Fix issue with incorrect time calculation in the priority plugin when
     a job runs past it's time limit.
  -- Prevent users from setting job's partition to an invalid partition.
  -- Fix sreport core dump when requesting
     'job SizesByAccount grouping=individual'.
  -- select/linear: Correct count of CPUs allocated to job on system with
     hyperthreads.
  -- Fix race condition where last array task might not get updated in the db.
  -- CRAY - Remove libpmi from rpm install
  -- Fix squeue -o %X output to correctly handle NO_VAL and suffix.
  -- When deleting a job from the system set the job_id to 0 to avoid memory
     corruption if thread uses the pointer basing validity off the id.
  -- Fix issue where sbatch would set ntasks-per-node to 0 making any srun
     afterward cause a divide by zero error.
  -- switch/cray: Refine logic to set PMI_CRAY_NO_SMP_ENV environment variable.
  -- When sacctmgr loads archives with version less than 14.11 set the array
     task id to NO_VAL, so sacct can display the job ids correctly.
  -- When using memory cgroup if a task uses more memory than requested
     the failures are logged into memory.failcnt count file by cgroup
     and the user is notified by slurmstepd about it.
  -- Fix scheduling inconsistency with GRES bound to specific CPUs.
  -- If user belongs to a group which has split entries in /etc/group
     search for its username in all groups.
  -- Do not consider nodes explicitly powered up as DOWN with reason of "Node
     unexpected rebooted".
  -- Use correct slurmd spooldir when creating cpu-frequency locks.
  -- Note that TICKET_BASED fairshare will be deprecated in the  
future. Consider
     using the FAIR_TREE algorithm instead.
  -- Set job's reason to BadConstaints when job can't run on any node.
  -- Prevent abort on update of reservation with no nodes (licenses only).
  -- Prevent slurmctld from dumping core ifjob_resrcs is missing in the
     job data structure.
  -- Fix squeue to print array task ids according to man page when
     SLURM_BITSTR_LEN is defined in the environment.
  -- In squeue sort jobs based on array job ID if available.
  -- Fix the calculation of job energy by not including the NO_VAL values.
  -- Advanced reservation fixes: enable update of bluegene reservation, avoid
     abort on multi-core reservations.
  -- Set the totalview_stepid to the value of the job step instead of NO_VAL.
  -- Fix slurmdbd core dump if the daemon does not have connection with
     the database.
  -- Display error message when attempting to modify priority of a held job.
  -- Backfill scheduler: The configured backfill_interval value (default 30
     seconds) is now interpretted as a maximum run time for the backfill
     scheduler. Once reached, the scheduler will build a new job queue and
     start over, even if not all jobs have been tested.
  -- Backfill scheduler now considers OverTimeLimit and KillWait configuration
     parameters to estimate when running jobs will exit.
  -- Correct task layout with CR_Pack_Node option and more than 1 CPU per task.
  -- Fix the scontrol man page describing the release argument.
  -- When job QOS is modified, do so before attempting to change partition in
     order to validate the partition's Allow/DenyQOS parameter.

* Changes in Slurm 15.08.0pre6
==============================
  -- Add scontrol options to view and modify layouts tables.
  -- Add MsgAggregationParams which controls a reverse tree to the slurmctld
     which can be used to aggregate messages to the slurmctld into a single
     message to reduce communication to the slurmctld.  Currently only epilog
     complete messages and node registration messages use this logic.
  -- Add sacct and squeue options to print trackable resources.
  -- Add sacctmgr option to display trackable resources.
  -- If an salloc or srun command is executed on a "front-end" configuration,
     that job will be assigned a slurmd shepherd daemon on the same  
host as used
     to execute the command when possible rather than an slurmd daemon on an
     arbitrary front-end node.
  -- Add srun --accel-bind option to control how tasks are bound to  
GPUs and NIC
     Generic RESources (GRES).
  -- gres/nic plugin modified to set OMPI_MCA_btl_openib_if_include environment
     variable based upon allocated devices (usable with OpenMPI and Melanox).
  -- Make it so info options for srun/salloc/sbatch print with just 1  
-v instead
     of 4.
  -- Add "no_backup_scheduling" SchedulerParameter to prevent jobs from being
     scheduled when the backup takes over. Jobs can be submitted, modified and
     cancelled while the backup is in control.
  -- Enable native Slurm backup controller to reside on an external Cray node
     when the "no_backup_scheduling" SchedulerParameter is used.
  -- Removed TICKET_BASED fairshare. Consider using the FAIR_TREE algorithm.
  -- Disable advanced reservation "REPLACE" option on IBM Bluegene systems.
  -- Add support for control distribution of tasks across cores (in addition
     to existing support for nodes and sockets, (e.g. "block", "cyclic" or
     "fcyclic" task distribution at 3 levels in the hardware rather than 2).
  -- Create db index on <cluster>_assoc_table.acct. Deleting accounts  
that didn't
     have jobs in the job table could take a long time.
  -- The performance of Profiling with HDF5 is improved. In addition, internal
     structures are changed to make it easier to add new profile types,
     particularly energy sensors. sh5util will continue to work with either
     format.
  -- Add partition information to sshare output if the --partition option
     is specified on the sshare command line.
  -- Add sreport -T/--tres option to identify Trackable RESources (TRES) to
     report.
  -- Display job in sacct when single step's cpus are different from the job
     allocation.
  -- Add association usage information to "scontrol show cache" command output.
  -- MPI/MVAPICH plugin now requires Munge for authentication.
  -- job_submit/lua: Add default_qos fields. Add job record qos.  Add partition
     record allow_qos and qos_char fields.
--

-- 
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support

Moe Jette | 2 Jun 00:37 2015
Gravatar

Slurm User Group Meeting, CFP extension


Due to several requests, the deadline for submitting abstracts to the  
Slurm User Group meeting has been extended to June 5. Meeting  
information is available here:
http://slurm.schedmd.com/slurm_ug_cfp.html
--

-- 
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support

Moe Jette | 25 May 17:33 2015
Gravatar

Slurm User Group Meeting 2015, Abstracts for talks due 1 June


You are invited to submit an abstract of a tutorial, technical  
presentation or site report to be given at the Slurm User Group  
Meeting 2015. This event is sponsored and organized by SchedMD and The  
George Washington University. The meeting will be held in Washington,  
DC on 15-16 September 2015.

This international event is opened to everyone who wants to:
Learn more about Slurm, a highly scalable Resource Manager and Job Scheduler
Share their knowledge and experience with other users and administrators
Get detailed information about the latest features and developments
Share requirements and discuss future developments

Everyone who wants to present their own usage, developments, site  
report, or tutorial about Slurm is invited to send an abstract to  
slugc@...

*Important Dates:*
1 June 2015: Abstracts due
15 June 2015: Notification of acceptance
15-16 September 2015: Slurm User Group Meeting 2015

*Web Site:*
http://slurm.schedmd.com/slurm_ug_cfp.html

*Program Committee:*
Yiannis Georgiou (Bull)
Brian Gilmer (Cray)
Matthieu Hautreux (CEA)
Morris Jette (SchedMD)
Bruce Pfaff (NASA Goddard Space Flight Center)
Tim Wickberg (The George Washington University)
--

-- 
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support

Danny Auble | 21 May 22:40 2015
Gravatar

Slurm versions 14.11.7 and 15.08.0-0pre5 are now available


Slurm version 14.11.7 is now available with quite a few bug fixes as
listed below.

A development tag for 15.08 (pre5) has also been made.  It represents 
the current state of Slurm development for the release planned in August 
2015 and is intended for development and test purposes only.  One 
notable enhancement included is the idea of Trackable Resources (TRES) 
for accounting for cpu, memory, energy, GRES, licenses, etc.

Both are available for download at
http://slurm.schedmd.com/download.html

Notable changes for these versions are these...

* Changes in Slurm 14.11.7
==========================
  -- Initialize some variables used with the srun --no-alloc option that may
     cause random failures.
  -- Add SchedulerParameters option of sched_min_interval that controls the
     minimum time interval between any job scheduling action. The 
default value
     is zero (disabled).
  -- Change default SchedulerParameters=max_sched_time from 4 seconds to 2.
  -- Refactor scancel so that all pending jobs are cancelled before starting
     cancellation of running jobs. Otherwise they happen in parallel and the
     pending jobs can be scheduled on resources as the running jobs are 
being
     cancelled.
  -- ALPS - Add new cray.conf variable NoAPIDSignalOnKill.  When set to 
yes this
     will make it so the slurmctld will not signal the apid's in a batch 
job.
     Instead it relies on the rpc coming from the slurmctld to kill the 
job to
     end things correctly.
  -- ALPS - Have the slurmstepd running a batch job wait for an ALPS release
     before ending the job.
  -- Initialize variables in consumable resource plugin to prevent core 
dump.
  -- Fix scancel bug which could return an error on attempt to signal a 
job step.
  -- In slurmctld communication agent, make the thread timeout be the 
configured
     value of MessageTimeout rather than 30 seconds.
  -- sshare -U/--Users only flag was used uninitialized.
  -- Cray systems, add "plugstack.conf.template" sample SPANK 
configuration file.
  -- BLUEGENE - Set DB2NOEXITLIST when starting the slurmctld daemon to 
avoid
     random crashing in db2 when the slurmctld is exiting.
  -- Make full node reservations display correctly the core count instead of
     cpu count.
  -- Preserve original errno on execve() failure in task plugin.
  -- Add SLURM_JOB_NAME env variable to an salloc's environment.
  -- Overwrite SLURM_JOB_NAME in an srun when it gets an allocation.
  -- Make sure each job has a wckey if that is something that is tracked.
  -- Make sure old step data is cleared when job is requeued.
  -- Load libtinfo as needed when building ncurses tools.
  -- Fix small memory leak in backup controller.
  -- Fix segfault when backup controller takes control for second time.
  -- Cray - Fix backup controller running native Slurm.
  -- Provide prototypes for init_setproctitle()/fini_setproctitle on NetBSD.
  -- Add configuration test to find out the full path to su command.
  -- preempt/job_prio plugin: Fix for possible infinite loop when 
identifying
     preemptable jobs.
  -- preempt/job_prio plugin: Implement the concept of Warm-up Time 
here. Use
     the QoS GraceTime as the amount of time to wait before preempting.
     Basically, skip preemption if your time is not up.
  -- Make srun wait KillWait time when a task is cancelled.
  -- switch/cray: Revert logic added to 14.11.6 that set 
"PMI_CRAY_NO_SMP_ENV=1"
     if CR_PACK_NODES is configured.
  -- Prevent users from setting job's partition to an invalid partition.

* Changes in Slurm 15.08.0pre5
==============================
  -- Add jobcomp/elasticsearch plugin. Libcurl is required for build. 
Configure
     the server as follows: 
"JobCompLoc=http://YOUR_ELASTICSEARCH_SERVER:9200".
  -- Scancel logic large re-written to better support job arrays.
  -- Added a slurm.conf parameter PrologEpilogTimeout to control how long
     prolog/epilog can run.
  -- Added TRES (Trackable resources) to track Mem, GRES, license, etc
     utilization.
  -- Add re-entrant versions of glibc time functions (e.g. localtime) to 
Slurm
     in order to eliminate rare deadlock of slurmstepd fork and exec calls.
  -- Constrain kernel memory (if available) in cgroups.
  -- Add PrologFlags option of "Contain" to create a proctrack container at
     job resource allocation time.
  -- Disable the OOM Killer in slurmd and slurmstepd's memory cgroup 
when using
     MemSpecLimit.

Moe Jette | 6 May 00:04 2015
Gravatar

Slurm User Group Meeting 2015, CFP


You are invited to submit an abstract of a tutorial, technical  
presentation or site report to be given at the Slurm User Group  
Meeting 2015. This event is sponsored and organized by SchedMD and The  
George Washington University. The meeting will be held in Washington,  
DC on 15-16 September 2015.

This international event is opened to everyone who wants to:
Learn more about Slurm, a highly scalable Resource Manager and Job Scheduler
Share their knowledge and experience with other users and administrators
Get detailed information about the latest features and developments
Share requirements and discuss future developments

Everyone who wants to present their own usage, developments, site  
report, or tutorial about Slurm is invited to send an abstract to  
slugc@...

*Important Dates:*
1 June 2015: Abstracts due
15 June 2015: Notification of acceptance
15-16 September 2015: Slurm User Group Meeting 2015

*Program Committee:*
Yiannis Georgiou (Bull)
Brian Gilmer (Cray)
Matthieu Hautreux (CEA)
Morris Jette (SchedMD)
Bruce Pfaff (NASA Goddard Space Flight Center)
Tim Wickberg (The George Washington University)
--

-- 
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support

Moe Jette | 24 Apr 01:03 2015
Gravatar

Slurm versions 14.11.6 is now available


Slurm version 14.11.6 is now available with quite a few bug fixes as  
listed below.

Slurm downloads are available from
http://slurm.schedmd.com/download.html

* Changes in Slurm 14.11.6
==========================
  -- If SchedulerParameters value of bf_min_age_reserve is configured, then
     a newly submitted job can start immediately even if there is a higher
     priority non-runnable job which has been waiting for less time than
     bf_min_age_reserve.
  -- qsub wrapper modified to export "all" with -V option
  -- RequeueExit and RequeueExitHold configuration parameters modified  
to accept
     numeric ranges. For example "RequeueExit=1,2,3,4" and  
"RequeueExit=1-4" are
     equivalent.
  -- Correct the job array specification parser to accept brackets in job array
     expression (e.g. "123_[4,7-9]").
  -- Fix for misleading job submit failure errors sent to users. Previous error
     could indicate why specific nodes could not be used (e.g. too  
small memory)
     when other nodes could be used, but were not for another reason.
  -- Fix squeue --array to display correctly the array elements when the
     % separator is specified at the array submission time.
  -- Fix priority from not being calculated correctly due to memory issues.
  -- Fix a transient pending reason 'JobId=job_id has invalid QOS'.
  -- A non-administrator change to job priority will not be persistent except
     for holding the job. User's wanting to change a job priority on a  
persistent
     basis should reset it's "nice" value.
  -- Print buffer sizes as unsigned values when failed to pack messages.
  -- Fix race condition where sprio would print factors without  
weights applied.
  -- Document the sacct option JobIDRaw which for arrays prints the  
jobid instead
     of the arrayTaskId.
  -- Allow users to modify MinCPUsNode, MinMemoryNode and MinTmpDiskNode of
     their own jobs.
  -- Increase the jobid print field in SQUEUE_FORMAT in
     opt_modulefiles_slurm.in.
  -- Enable compiling without optimizations and with debugging symbols by
     default. Disable this by configuring with --disable-debug.
  -- job_submit/lua plugin: Add mail_type and mail_user fields.
  -- Correct output message from sshare.
  -- Use standard statvfs(2) syscall if available, in preference to
     non-standard statfs.
  -- Add a new option -U/--Users to sshare to display only users
     information, parent and ancestors are not printed.
  -- Purge 50000 records at a time so that locks can released periodically.
  -- Fix potentially uninitialized variables
  -- ALPS - Fix issue where a frontend node could become unresponsive and never
     added back into the system.
  -- Gate epilog complete messages as done with other messages
  -- If we have more than a certain number of agents (50) wait longer  
when gating
     rpcs.
  -- FrontEnd - ping non-responding or down nodes.
  -- switch/cray: If CR_PACK_NODES is configured, then set the environment
     variable "PMI_CRAY_NO_SMP_ENV=1"
  -- Fix invalid memory reference in SlurmDBD when putting a node up.
  -- Allow opening of plugstack.conf even when a symlink.
  -- Fix scontrol reboot so that rebooted nodes will not be set down  
with reason
     'Node xyz unexpectedly rebooted' but will be correctly put back  
to service.
  -- CRAY - Throttle the post NHC operations as to not hog the job write lock
     if many steps/jobs finish at once.
  -- Disable changes to GRES count while jobs are running on the node.
  -- CRAY - Fix issue with scontrol reconfig.
  -- slurmd: Remove wrong reporting of "Error reading step  ... memory limit".
     The logic was treating success as an error.
  -- Eliminate "Node ping apparently hung" error messages.
  -- Fix average CPU frequency calculation.
  -- When allocating resources with resolution of sockets, charge the  
job for all
     CPUs on allocated sockets rather than just the CPUs on used cores.
  -- Prevent slurmdbd error if cluster added or removed while rollup  
in progress.
     Removing a cluster can cause slurmdbd to abort. Adding a cluster can cause
     the slurmdbd rollup to hang.
  -- sview - When right clicking on a tab make sure we don't display the page
     list, but only the column list.
  -- FRONTEND - If doing a clean start make sure the nodes are brought  
up in the
     database.
  -- MySQL - Fix issue when using the TrackSlurmctldDown and nodes are down at
     the same time, don't double bill the down time.
  -- MySQL - Various memory leak fixes.
  -- sreport - Fix Energy displays
  -- Fix node manager logic to keep unexpectedly rebooted node in state
     NODE_STATE_DOWN even if already down when rebooted.
  -- Fix for array jobs submitted to multiple partitions not starting.
  -- CRAY - Enable ALPs mpp compatibility code in sbatch for native Slurm.
  -- ALPS - Move basil_inventory to less confusing function.
  -- Add SchedulerParameters option of "sched_max_job_start="  to limit the
     number of jobs that can be started in any single execution of the main
     scheduling logic.
  -- Fixed compiler warnings generated by gcc version >= 4.6.
  -- sbatch to stop parsing script for "#SBATCH" directives after  
first command,
     which matches the documentation.
  -- Overwrite the SLURM_JOB_NAME in sbatch if already exist in the environment
     and use the one specified on the command line --job-name.
  -- Remove xmalloc_nz from unpack functions.  If the unpack ever failed the
     free afterwards would not have zeroed out memory on the variables that
     didn't get unpacked.
  -- Improve database interaction from controller.
  -- Fix for data shift when loading job archives.
  -- ALPS - Added new SchedulerParameters=inventory_interval to specify how
     often an inventory request is handled.
  -- ALPS - Don't run a release on a reservation on the slurmctld for a batch
     job.  This is already handled on the stepd when the script finishes.
--

-- 
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support

Jacob Jenson | 1 Apr 18:12 2015

CFP: Slurm User Group Meeting 2015

You are invited to submit an abstract of a tutorial, technical presentation or site report to be given at the Slurm User Group Meeting 2015. This event is sponsored and organized by SchedMD and The George Washington University. The meeting will be held in Washington, DC on 15-16 September 2015.

This international event is opened to everyone who wants to:
Learn more about Slurm, a highly scalable Resource Manager and Job Scheduler
Share their knowledge and experience with other users and administrators
Get detailed information about the latest features and developments
Share requirements and discuss future developments

Everyone who wants to present their own usage, developments, site report, or tutorial about Slurm is invited to send an abstract to slugc-1gbaCZE5fTZBDgjK7y7TUQ@public.gmane.org 

Important Dates:
1 June 2015: Abstracts due
15 June 2015: Notification of acceptance
15-16 September 2015: Slurm User Group Meeting 2015

Program Committee:
Yiannis Georgiou (Bull)
Brian Gilmer (Cray)
Matthieu Hautreux (CEA)
Morris Jette (SchedMD)
Bruce Pfaff (NASA Goddard Space Flight Center)
Tim Wickberg (The George Washington University)

--
Jacob Jenson
COO, SchedMD LLC
Slurm Support and Development
Moe Jette | 19 Mar 22:29 2015
Gravatar

Slurm versions 14.11.5 and 15.08.0-pre3 are now available


Version 14.11.5 contains quite a few bug fixes generated over the past  
five weeks including two high impact bugs. There is a fix for the  
slurmdbd daemon aborting if a node is set to a DOWN state and it's  
"reason" field is NULL. The other important bug fix will prevent  
someone from being able to kill a job array belonging to another user.  
Details about all of the changes are appended.

Version 15.08.0-pre3 represents the current state of Slurm development  
for the release planned in August 2015 and is intended for development  
and test purposes only. Notable enhancements include power capping  
support for Cray systems and add the ability for a compute node to be  
allocated to multiple jobs, but restricted to one user at a time.

Both versions can be downloaded from
http://www.schedmd.com/#repos

* Changes in Slurm 14.11.5
==========================
  -- Correct the squeue command taking into account that a node can
     have NULL name if it is not in DNS but still in slurm.conf.
  -- Fix slurmdbd regression which would cause a segfault when a node is set
     down with no reason.
  -- BGQ - Fix issue with job arrays not being handled correctly
     in the runjob_mux plugin.
  -- Print FAIR_TREE, if configured, in "scontrol show config" output for
     PriorityFlags.
  -- Add SLURM_JOB_GPUS environment variable to those available in the Prolog.
  -- Load lua-5.2 library if using lua5.2 for lua job submit plugin.
  -- GRES logic: Prevent bad node_offset due to not preserving no_consume flag.
  -- Fix wrong variables used in the wrapper functions needed for systems that
     don't support strong_alias
  -- Fix code for apple computers SOL_TCP is not defined
  -- Cray/BASIL - Check for mysql credentials in /root/.my.cnf.
  -- Fix sprio showing wrong priority for job arrays until priority is
     recalculated.
  -- Account to batch step all CPUs that are allocated to a job not
     just one since the batch step has access to all CPUs like other steps.
  -- Fix job getting EligibleTime set before meeting dependency requirements.
  -- Correct the initialization of QOS MinCPUs per job limit.
  -- Set the debug level of information messages in cgroup plugin to debug2.
  -- For job running under a debugger, if the exec of the task fails, then
     cancel its I/O and abort immediately rather than waiting 60 seconds for
     I/O timeout.
  -- Fix associations not getting default qos set until after a restart.
  -- Set the value of total_cpus not to be zero before invoking
     acct_policy_job_runnable_post_select.
  -- MySQL - When requesting cluster resources, only return resources for the
     cluster(s) requested.
  -- Add TaskPluginParam=autobind=threads option to set a default  
binding in the
     case that "auto binding" doesn't find a match.
  -- Introduce a new SchedulerParameters variable nohold_on_prolog_fail.
     If configured don't requeue jobs on hold is a Prolog fails.
  -- Make it so sched_params isn't read over and over when an epilog complete
     message comes in
  -- Fix squeue -L <licenses> not filtering out jobs with licenses.
  -- Changed the implementation of xcpuinfo_abs_to_mac() be identical
     _abs_to_mac() to fix CPUs allocation using cpuset cgroup.
  -- Improve the explanation of the unbuffered feature in the
     srun man page.
  -- Make taskplugin=cgroup work for core spec.  needed to have task/cgroup
     before.
  -- Fix reports not using the month usage table.
  -- BGQ - Sanity check given for translating small blocks into slurm  
bg_records.
  -- Fix bug preventing the requeue/hold or requeue/special_exit of  
job from the
     completing state.
  -- Cray - Fix for launching batch step within an existing job allocation.
  -- Cray - Add ALPS_APP_ID_ENV environment variable.
  -- Increase maximum MaxArraySize configuration parameter value from 1,000,001
     to 4,000,001.
  -- Added new SchedulerParameters value of bf_min_age_reserve. The backfill
     scheduler will not reserve resources for pending jobs until they have
     been pending for at least the specified number of seconds. This can be
     valuable if jobs lack time limits or all time limits have the same value.
  -- Fix support for --mem=0 (all memory of a node) with  
select/cons_res plugin.
  -- Fix bug that can permit someone to kill job array belonging to  
another user.
  -- Don't set the default partition on a license only reservation.
  -- Show a NodeCnt=0, instead of NO_VAL, in "scontrol show res" for a license
     only reservation.
  -- BGQ - When using static small blocks make sure when clearing the job the
     block is set up to it's original state.
  -- Start job allocation using lowest numbered sockets for block task
     distribution for consistency with cyclic distribution.

* Changes in Slurm 15.08.0pre3
==============================
-- CRAY - addition of acct_gather_energy/cray plugin.
-- Add job credential to "Run Prolog" RPC used with a configuration of
    PrologFlags=alloc. This allows the Prolog to be passed identification of
    GPUs allocated to the job.
-- Add SLURM_JOB_CONSTAINTS to environment variables available to the Prolog.
-- Added "--mail=stage_out" option to job submission commands to notify user
    when burst buffer state out is complete.
-- Require a "Reason" when using scontrol to set a node state to DOWN.
-- Mail notifications on job BEGIN, END and FAIL now apply to a job array as a
    whole rather than generating individual email messages for each task in the
    job array.
-- task/affinity - Fix memory binding to NUMA with cpusets.
-- Display job's estimated NodeCount based off of partition's configured
    resources rather than the whole system's.
-- Add AuthInfo option of "cred_expire=#" to specify the lifetime of a job
    step credential. The default value was changed from 1200 to 120 seconds.
-- Set the delay time for job requeue to the job credential lifetime (120
    seconds by default). This insures that prolog runs on every node when a
    job is requeued. (This change will slow down launch of re-queued jobs).
-- Add AuthInfo option of "cred_expire=#" to specify the lifetime of a job
    step credential.
-- Remove srun --max-launch-time option. The option has not been functional
    since Slurm version 2.0.
-- Add sockets and cores to TaskPluginParams' autobind option.
-- Added LaunchParameters configuration parameter. Have srun command test
    locally for the executable file if LaunchParameters=test_exec or the
    environment variable SLURM_TEST_EXEC is set. Without this an invalid
    command will generate one error message per task launched.
-- Fix the slurm /etc/init.d script to return 0 upon stopping the
    daemons and return 1 in case of failure.
-- Add the ability for a compute node to be allocated to multiple jobs, but
    restricted to a single user. Added "--exclusive=user" option to salloc,
    sbatch and srun commands. Added "owner" field to node record, visible using
    the scontrol and sview commands. Added new partition configuration  
parameter
    "ExclusiveUser=yes|no".
--

-- 
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support

jette | 12 Feb 22:49 2015
Gravatar

Slurm versions 14.11.4 and 15.08.0-pre2 are now available


Slurm versions 14.11.4 and 15.08.0-pre2 are now available from

http://www.schedmd.com/#repos

Version 14.11.4 contains quite a few bug fixes generated over the past  
five weeks. Several of these are related to job arrays, including one  
that can cause the slurmctld daemon to abort. Version 15.08.0-pre2  
represents the current state of Slurm development for the released  
planned in August 2015 and is intended for development and test  
purposes only. It includes some development work for burst buffers,  
power management, and inter-cluster job dependencies. More details  
about the changes are shown below.

* Changes in Slurm 14.11.4
==========================
  -- Make sure assoc_mgr locks are initialized correctly.
  -- Correct check of enforcement when filling in an association.
  -- Make sacctmgr print out classification correctly for clusters.
  -- Add array_task_str to the perlapi job info.
  -- Fix for slurmctld abort with GRES types configured and no CPU binding.
  -- Fix for GRES scheduling where count > 1 per topology type (or GRES types).
  -- Make CR_ONE_TASK_PER_CORE work correctly with task/affinity.
  -- job_submit/pbs - Fix possible deadlock.
  -- job_submit/lua - Add "alloc_node" to job information available.
  -- Fix memory leak in mysql accounting when usage rollup happens.
  -- If users specify ALL together with other variables using the
     --export sbatch/srun command line option, propagate the users'
     environ to the execution side.
  -- Fix job array scheduling anomaly that can stop scheduling of valid tasks.
  -- Fix perl api tests for libslurmdb to work correctly.
  -- Remove some misleading logs related to non-consumable GRES.
  -- Allow --ignore-pbs to take effect when read as an #SBATCH argument.
  -- Fix Slurmdb::clusters_get() in perl api from not returning information.
  -- Fix TaskPluginParam=Cpusets from logging error message about not  
being able
     to remove cpuset dir which was already removed by the release_agent.
  -- Fix sorting by time left in squeue.
  -- Fix the file name substitution for job stderr when %A, %a %j and %u
     are specified.
  -- Remove minor warning when compiling slurmstepd.
  -- Fix database resources so they can add new clusters to them after  
they have
     initially been added.
  -- Use the slurm_getpwuid_r wrapper of getpwuid_r to handle possible
     interrupts.
  -- Correct the scontrol man page and command listing which node states can
     be set by the command.
  -- Stop sacct from printing non-existent stat information for
     Front End systems.
  -- Correct srun and acct_gather.conf man pages, mention Filesystem instead
     of Lustre.
  -- When a job using multiple partition starts send to slurmdbd only
     the partition in which the job runs.
  -- ALPS - Fix depth for MemoryAllocation in BASIL with CLE 5.2.3.
  -- Fix assoc_mgr hash to deal with users that don't have a uid yet  
when making
     reservations.
  -- When a job uses multiple partition set the environment variable
     SLURM_JOB_PARTITION to be the one in which the job started.
  -- Print spurious message about the absence of cgroup.conf at log  
level debug2
     instead of info.
  -- Enable CUDA v7.0+ use with a Slurm configuration of TaskPlugin=task/cgroup
     ConstrainDevices=yes (in cgroup.conf). With that configuration
     CUDA_VISIBLE_DEVICES will start at 0 rather than the device number.
  -- Fix job array logic that can cause slurmctld to abort.
  -- Report job "shared" field properly in scontrol, squeue, and sview.
  -- If a job is requeued because of RequeueExit or RequeueExitHold sent event
     REQUEUED to slurmdbd.
  -- Fix build if hwloc is in non-standard location.
  -- Fix slurmctld job recovery logic which could cause the last task in a job
     array to be lost.
  -- Fix slurmctld initialization problem which could cause requeue of the last
     task in a job array to fail if executed prior to the slurmctld loading
     the maximum size of a job array into a variable in the job_mgr.c module.
  -- Fix fatal in controller when deleting a user association of a user which
     had been previously removed from the system.
  -- MySQL - If a node state and reason are the same on a node state change
     don't insert a new row in the event table.
  -- Fix issue with "sreport cluster AccountUtilizationByUser" when using
     PrivateData=users.
  -- Fix perlapi tests for libslurm perl module.

* Changes in Slurm 15.08.0pre2
==============================
  -- Add the environment variables SLURM_JOB_ACCOUNT, SLURM_JOB_QOS
     and SLURM_JOB_RESERVATION in the batch/srun jobs.
  -- Add sview burst buffer display.
  -- Properly enforce partition Shared=YES option. Previously oversubscribing
     resources required gang scheduling to be configured.
  -- Enable per-partition gang scheduling resource resolution (e.g.  
the partition
     can have SelectTypeParameters=CR_CORE, while the global value is  
CR_SOCKET).
  -- Make it so a newer version of a slurmstepd can talk to an older srun.
     allocation. Nodes could have been added while waiting for an allocation.
  -- Expanded --cpu-freq parameters to include min-max:governor specifications.
     --cpu-freq now supported on salloc and sbatch.
  -- Add support for optimized job allocations with respect to SGI Hypercube
     topology.
     NOTE: Only supported with select/linear plugin.
     NOTE: The program contribs/sgi/netloc_to_topology can be used to build
     Slurm's topology.conf file.
  -- Remove 64k validation of incoming RPC nodelist size. Validated at 64MB
     when unpacking.
  -- In slurmstepd() add the user primary group if it is not part of the
     groups sent from the client.
  -- Added BurstBuffer field to advanced reservations.
  -- For advanced reservation, replace flag "License_only" with flag  
"Any_Nodes".
     It can be used to indicate the an advanced reservation resources (licenses
     and/or burst buffers) can be used with any compute nodes.
  -- Allow users to specify the srun --resv-ports as 0 in which case no ports
     will be reserved. The default behaviour is to allocate one port per task.
  -- Interpret a partition configuration of "Nodes=ALL" in slurm.conf as
     including all nodes defined in the cluster.
  -- Added new configuration parameters PowerParameters and PowerPlugin.
  -- Added power management plugin infrastructure.
  -- If job already exceeded one of its QOS/Accounting limits do not
     return error if user modifies QOS unrelated job settings.
  -- Added DebugFlags value of "Power".
  -- When caching user ids of AllowGroups use both getgrnam_r() and  
getgrent_r()
     then remove eventual duplicate entries.
  -- Remove rpm dependency between slurm-pam and slurm-devel.
  -- Remove support for the XCPU (cluster management) package.
  -- Add Slurmdb::jobs_get() interface to perl api.
  -- Performance improvement when sending data from srun to stepds when
     processing fencing.
  -- Add the feature to specify arbitrary field separator when running
     sacct -p or sacct -P. The command line option is --separator.
  -- Introduce slurm.conf parameter to use Proportional Set Size (PSS) instead
     of RSS to determinate the memory footprint of a job.
     Add an slurm.conf option not to kill jobs that is over memory limit.
  -- Add job submission command options: --sicp (available for inter-cluster
     dependencies) and --power (specify power management options) to salloc,
     sbatch, and srun commands.
  -- Add DebugFlags option of SICP (inter-cluster option logging).
  -- In order to support inter-cluster job dependencies, the MaxJobID
     configuration parameter default value has been reduced from 4,294,901,760
     to 2,147,418,112 and it's maximum value is now 2,147,463,647.
     ANY JOBS WITH A JOB ID ABOVE 2,147,463,647 WILL BE PURGED WHEN SLURM IS
     UPGRADED FROM AN OLDER VERSION!
  -- Add QOS name to the output of a partition in squeue/scontrol/sview/smap.
--

-- 
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support


Gmane