Moe Jette | 4 Nov 23:56 2015
Gravatar

Slurm version 15.08.3 now available


We are pleased to announce the availability of Slurm version 15.08.3,  
includes about 25 bug fixes developed over the past couple of weeks as  
listed below. Slurm downloads are available from:
http://www.schedmd.com/#repos

SC15
There will be a Slurm User Group meeting on Thursday 19 November at  
12:15-13:15 in Room 16AB.
Please visit the Slurm booth (#1851) to pick up a quick reference  
guide and a limited edition tee-shirt.

* Changes in Slurm 15.08.3
==========================
  -- Correct Slurm's RPM build if Munge is not installed.
  -- Job array termination status email ExitCode based upon highest exit code
     from any task in the job array rather than the last task. Also change the
     state from "Ended" or "Failed" to "Mixed" where appropriate.
  -- Squeue recombines pending job array records only if their name  
and partition
     are identical.
  -- Fix some minor leaks in the job info and step info API.
  -- Export missing QOS id when filling in association with the association
     manager.
  -- Fix invalid reference if a lua job_submit plugin references a default qos
     when a user doesn't exist in the database.
  -- Use association enforcement in the lua plugin.
  -- Fix a few spots missing defines of accounting_enforce or acct_db_conn
     in the plugins.
  -- Show requested TRES in scontrol show jobs when job is pending.
(Continue reading)

Moe Jette | 22 Oct 21:57 2015
Gravatar

Slurm version 15.08.2 now available, SC15 News


We are pleased to announce the availability of Slurm version 15.08.2,  
which includes about 40 bug fixes developed over the past four weeks  
as listed below. Slurm downloads are available from:
http://www.schedmd.com/#repos

SC15 News:
There will be a Slurm User Group meeting on Thursday 19 November at  
12:15-13:15 in Room 16AM.
Please visit the Slurm booth (#1851) to pick up a quick reference  
guide and a limited edition tee-shirt.

* Changes in Slurm 15.08.2
==========================
  -- Fix for tracking node state when jobs that have been allocated exclusive
     access to nodes (i.e. entire nodes) and later relinquish some nodes. Nodes
     would previously appear partly allocated and prevent use by other jobs.
  -- Correct some cgroup paths ("step_batch" vs. "step_4294967294",  
"step_exter"
     vs. "step_extern", and "step_extern" vs. "step_4294967295").
  -- Fix advanced reservation core selection logic with network topology.
  -- MYSQL - Remove restriction to have to be at least an operator to  
query TRES
     values.
  -- For pending jobs have sacct print 0 for nnodes instead of the bogus 2.
  -- Fix for tracking node state when jobs that have been allocated exclusive
     access to nodes (i.e. entire nodes) and later relinquish some nodes. Nodes
     would previously appear partly allocated and prevent use by other jobs.
  -- Fix updating job in db after extending job's timelimit past partition's
     timelimit.
(Continue reading)

Moe Jette | 25 Sep 22:00 2015
Gravatar

Slurm version 15.08.1 is now availabe


We are pleased to announce the availability of Slurm version 15.08.1  
with about 40 bug fixes to 15.08.0. A list of changes is appended.

Slurm downloads are available from
http://www.schedmd.com/#repos

* Changes in Slurm 15.08.1
==========================
  -- Fix test21.30 and 21.34 to check grpwall better.
  -- Add time to the partition QOS the job is running on instead of just the
     job QOS.
  -- Print usage for GrpJobs, GrpSubmitJobs and GrpWall even if there is no
     limit.
  -- If AccountingEnforce=safe is set make sure a job can finish before going
     over the limit with grpwall on a QOS or association.
  -- burst_buffer/cray - Major updates based upon recent Cray changes.
  -- Improve clean up logic of pmi2 plugin.
  -- Improve job state reason string when required nodes not available.
  -- Fix missing else when packing an update partition message
  -- Fix srun from inheriting the SLURM_CPU_BIND and SLURM_MEM_BIND environment
     variables when running in an existing srun (e.g. an srun within  
an salloc).
  -- Fix missing else when packing an update partition message.
  -- Use more flexible mechnanism to find json installation.
  -- Make sure safe_limits was initialized before processing limits in the
     slurmctld.
  -- Fix for burst_buffer/cray to parse type option correctly.
  -- Fix memory error and version number in the nonstop plugin and reservation
     code.
(Continue reading)

Moe Jette | 21 Sep 19:01 2015
Gravatar

Slurm User Group Meeting, presentations online


Thanks to everyone who helped make the Slurm User Group Meeting last  
week a big success. Copies of the presentations are now on-line here:
http://slurm.schedmd.com/publications.html
--

-- 
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support

Danny Auble | 1 Sep 02:26 2015
Gravatar

Slurm versions 15.08.0 and 14.11.9 have been released!


Greetings everyone.

We are pleased to announce the release of 15.08.0!  It contains many new 
features and performance enhancements.  Please read the RELEASE_NOTES 
file to get an idea of the new items that have been added.  The on-line 
Slurm documentation has been updated to reflect this release.  Thanks to 
everyone that helped in this release.

Some notable changes are listed here.

  -- Added TRES (Trackable resources) to track utilization of memory, GRES,
     burst buffer, license, and any other configurable resources in the
     accounting database.
  -- Add configurable billing weight that takes into consideration any 
TRES when
     calculating a job's resource utilization.
  -- Add configurable prioritization factor that takes into 
consideration any
     TRES when calculating a job's resource utilization.
  -- Add burst buffer support infrastructure. Currently available plugin 
include
     burst_buffer/generic (uses administrator supplied programs to 
manage file
     staging) and burst_buffer/cray (uses Cray APIs to manage buffers).
  -- Add power capping support for Cray systems with automatic 
rebalancing of
     power allocation between nodes.
  -- Modify slurmctld outgoing RPC logic to support more parallel tasks 
(up to
(Continue reading)

Moe Jette | 21 Aug 01:35 2015
Gravatar

Slurm version 15.08.0-rc1 is now available


We are pleased to announce the availability of Slurm version 15.08.0-rc1
(release candidate 1). This version contains all of the features intended for
release 15.08 (with the exception of some minor burst buffer work) and we are
moving into a testing phase. You are invited to download this version  
and assist
in testing. Some highlights in this release include:
  -- Add TRES (Trackable resources) to track utilization of memory, GRES, burst
     buffer, license, and any other configurable resources in the accounting
     database.
  -- Add configurable billing weight that takes into consideration any  
TRES when
     calculating a job's resource utilization (for fair-share calculation).
  -- Add configurable prioritization factor that takes into consideration any
     TRES when calculating a job's resource utilization.
  -- Add burst buffer support infrastructure. Currently available  
plugin include
     burst_buffer/generic (uses administrator supplied programs to manage file
     staging) and burst_buffer/cray (uses Cray APIs to manage buffers).
  -- Add support for job dependencies joined with OR operator (e.g.
     "--depend=afterok:123?afternotok:124").
  -- Add advance reservation flag of "replace" that causes allocated resources
     to be replaced with idle resources. This maintains a pool of available
     resources that maintains a constant size (to the extent possible).
  -- Permit PreemptType=qos and PreemptMode=suspend,gang to be used together.
     A high-priority QOS job will now oversubscribe resources and gang  
schedule,
     but only if there are insufficient resources for the job to be started
     without preemption. NOTE: That with PreemptType=qos, the partition's
     Shared=FORCE:# configuration option will permit one job more per resource
(Continue reading)

Jacob Jenson | 8 Jul 02:02 2015

2015 Slurm User Group Meeting - Agenda and Registration

The 2015 Slurm User Group Meeting will be held on September 15 and 16 in Washington, DC. The meeting will include an assortment of tutorials, technical presentations, and site reports.

Reregistration and a preliminary agenda are available online:
http://slurm.schedmd.com/slurm_ug_agenda.html


Please register early to take advantage of the Early registration special
  • Early Registration - $250 USD per person for early registration by 31 July 2015
  • Standard Registration - $350 USD per person for standard registration by 31 August 2015
  • Late Registration - $600 USD per person for late registration starting 1 September 2015

Moe Jette | 8 Jul 01:06 2015
Gravatar

Slurm versions 14.11.8 and 15.08.0-pre6 are now available


Slurm version 14.11.8 includes about 30 relatively minor bug fixes  
developed over the past seven weeks while version 15.08.0-pre6  
contains new development scheduled for release next month. Details of  
changes are shown below. Slurm downloads are available from
http://www.schedmd.com/#repos

Also note that registration is open for the 2015 Slurm User Group  
Meeting. A preliminary agenda, registration and hotel information are  
available from
http://slurm.schedmd.com/slurm_ug_agenda.html

* Changes in Slurm 14.11.8
==========================
  -- Eliminate need for user to set user_id on job_update calls.
  -- Correct list of unavailable nodes reported in a job's "reason" field when
     that job can not start.
  -- Map job --mem-per-cpu=0 option to --mem=0.
  -- Fix squeue -o %m and %d unit conversion to Megabytes.
  -- Fix issue with incorrect time calculation in the priority plugin when
     a job runs past it's time limit.
  -- Prevent users from setting job's partition to an invalid partition.
  -- Fix sreport core dump when requesting
     'job SizesByAccount grouping=individual'.
  -- select/linear: Correct count of CPUs allocated to job on system with
     hyperthreads.
  -- Fix race condition where last array task might not get updated in the db.
  -- CRAY - Remove libpmi from rpm install
  -- Fix squeue -o %X output to correctly handle NO_VAL and suffix.
  -- When deleting a job from the system set the job_id to 0 to avoid memory
     corruption if thread uses the pointer basing validity off the id.
  -- Fix issue where sbatch would set ntasks-per-node to 0 making any srun
     afterward cause a divide by zero error.
  -- switch/cray: Refine logic to set PMI_CRAY_NO_SMP_ENV environment variable.
  -- When sacctmgr loads archives with version less than 14.11 set the array
     task id to NO_VAL, so sacct can display the job ids correctly.
  -- When using memory cgroup if a task uses more memory than requested
     the failures are logged into memory.failcnt count file by cgroup
     and the user is notified by slurmstepd about it.
  -- Fix scheduling inconsistency with GRES bound to specific CPUs.
  -- If user belongs to a group which has split entries in /etc/group
     search for its username in all groups.
  -- Do not consider nodes explicitly powered up as DOWN with reason of "Node
     unexpected rebooted".
  -- Use correct slurmd spooldir when creating cpu-frequency locks.
  -- Note that TICKET_BASED fairshare will be deprecated in the  
future. Consider
     using the FAIR_TREE algorithm instead.
  -- Set job's reason to BadConstaints when job can't run on any node.
  -- Prevent abort on update of reservation with no nodes (licenses only).
  -- Prevent slurmctld from dumping core ifjob_resrcs is missing in the
     job data structure.
  -- Fix squeue to print array task ids according to man page when
     SLURM_BITSTR_LEN is defined in the environment.
  -- In squeue sort jobs based on array job ID if available.
  -- Fix the calculation of job energy by not including the NO_VAL values.
  -- Advanced reservation fixes: enable update of bluegene reservation, avoid
     abort on multi-core reservations.
  -- Set the totalview_stepid to the value of the job step instead of NO_VAL.
  -- Fix slurmdbd core dump if the daemon does not have connection with
     the database.
  -- Display error message when attempting to modify priority of a held job.
  -- Backfill scheduler: The configured backfill_interval value (default 30
     seconds) is now interpretted as a maximum run time for the backfill
     scheduler. Once reached, the scheduler will build a new job queue and
     start over, even if not all jobs have been tested.
  -- Backfill scheduler now considers OverTimeLimit and KillWait configuration
     parameters to estimate when running jobs will exit.
  -- Correct task layout with CR_Pack_Node option and more than 1 CPU per task.
  -- Fix the scontrol man page describing the release argument.
  -- When job QOS is modified, do so before attempting to change partition in
     order to validate the partition's Allow/DenyQOS parameter.

* Changes in Slurm 15.08.0pre6
==============================
  -- Add scontrol options to view and modify layouts tables.
  -- Add MsgAggregationParams which controls a reverse tree to the slurmctld
     which can be used to aggregate messages to the slurmctld into a single
     message to reduce communication to the slurmctld.  Currently only epilog
     complete messages and node registration messages use this logic.
  -- Add sacct and squeue options to print trackable resources.
  -- Add sacctmgr option to display trackable resources.
  -- If an salloc or srun command is executed on a "front-end" configuration,
     that job will be assigned a slurmd shepherd daemon on the same  
host as used
     to execute the command when possible rather than an slurmd daemon on an
     arbitrary front-end node.
  -- Add srun --accel-bind option to control how tasks are bound to  
GPUs and NIC
     Generic RESources (GRES).
  -- gres/nic plugin modified to set OMPI_MCA_btl_openib_if_include environment
     variable based upon allocated devices (usable with OpenMPI and Melanox).
  -- Make it so info options for srun/salloc/sbatch print with just 1  
-v instead
     of 4.
  -- Add "no_backup_scheduling" SchedulerParameter to prevent jobs from being
     scheduled when the backup takes over. Jobs can be submitted, modified and
     cancelled while the backup is in control.
  -- Enable native Slurm backup controller to reside on an external Cray node
     when the "no_backup_scheduling" SchedulerParameter is used.
  -- Removed TICKET_BASED fairshare. Consider using the FAIR_TREE algorithm.
  -- Disable advanced reservation "REPLACE" option on IBM Bluegene systems.
  -- Add support for control distribution of tasks across cores (in addition
     to existing support for nodes and sockets, (e.g. "block", "cyclic" or
     "fcyclic" task distribution at 3 levels in the hardware rather than 2).
  -- Create db index on <cluster>_assoc_table.acct. Deleting accounts  
that didn't
     have jobs in the job table could take a long time.
  -- The performance of Profiling with HDF5 is improved. In addition, internal
     structures are changed to make it easier to add new profile types,
     particularly energy sensors. sh5util will continue to work with either
     format.
  -- Add partition information to sshare output if the --partition option
     is specified on the sshare command line.
  -- Add sreport -T/--tres option to identify Trackable RESources (TRES) to
     report.
  -- Display job in sacct when single step's cpus are different from the job
     allocation.
  -- Add association usage information to "scontrol show cache" command output.
  -- MPI/MVAPICH plugin now requires Munge for authentication.
  -- job_submit/lua: Add default_qos fields. Add job record qos.  Add partition
     record allow_qos and qos_char fields.
--

-- 
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support

Moe Jette | 2 Jun 00:37 2015
Gravatar

Slurm User Group Meeting, CFP extension


Due to several requests, the deadline for submitting abstracts to the  
Slurm User Group meeting has been extended to June 5. Meeting  
information is available here:
http://slurm.schedmd.com/slurm_ug_cfp.html
--

-- 
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support

Moe Jette | 25 May 17:33 2015
Gravatar

Slurm User Group Meeting 2015, Abstracts for talks due 1 June


You are invited to submit an abstract of a tutorial, technical  
presentation or site report to be given at the Slurm User Group  
Meeting 2015. This event is sponsored and organized by SchedMD and The  
George Washington University. The meeting will be held in Washington,  
DC on 15-16 September 2015.

This international event is opened to everyone who wants to:
Learn more about Slurm, a highly scalable Resource Manager and Job Scheduler
Share their knowledge and experience with other users and administrators
Get detailed information about the latest features and developments
Share requirements and discuss future developments

Everyone who wants to present their own usage, developments, site  
report, or tutorial about Slurm is invited to send an abstract to  
slugc@...

*Important Dates:*
1 June 2015: Abstracts due
15 June 2015: Notification of acceptance
15-16 September 2015: Slurm User Group Meeting 2015

*Web Site:*
http://slurm.schedmd.com/slurm_ug_cfp.html

*Program Committee:*
Yiannis Georgiou (Bull)
Brian Gilmer (Cray)
Matthieu Hautreux (CEA)
Morris Jette (SchedMD)
Bruce Pfaff (NASA Goddard Space Flight Center)
Tim Wickberg (The George Washington University)
--

-- 
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support

Danny Auble | 21 May 22:40 2015
Gravatar

Slurm versions 14.11.7 and 15.08.0-0pre5 are now available


Slurm version 14.11.7 is now available with quite a few bug fixes as
listed below.

A development tag for 15.08 (pre5) has also been made.  It represents 
the current state of Slurm development for the release planned in August 
2015 and is intended for development and test purposes only.  One 
notable enhancement included is the idea of Trackable Resources (TRES) 
for accounting for cpu, memory, energy, GRES, licenses, etc.

Both are available for download at
http://slurm.schedmd.com/download.html

Notable changes for these versions are these...

* Changes in Slurm 14.11.7
==========================
  -- Initialize some variables used with the srun --no-alloc option that may
     cause random failures.
  -- Add SchedulerParameters option of sched_min_interval that controls the
     minimum time interval between any job scheduling action. The 
default value
     is zero (disabled).
  -- Change default SchedulerParameters=max_sched_time from 4 seconds to 2.
  -- Refactor scancel so that all pending jobs are cancelled before starting
     cancellation of running jobs. Otherwise they happen in parallel and the
     pending jobs can be scheduled on resources as the running jobs are 
being
     cancelled.
  -- ALPS - Add new cray.conf variable NoAPIDSignalOnKill.  When set to 
yes this
     will make it so the slurmctld will not signal the apid's in a batch 
job.
     Instead it relies on the rpc coming from the slurmctld to kill the 
job to
     end things correctly.
  -- ALPS - Have the slurmstepd running a batch job wait for an ALPS release
     before ending the job.
  -- Initialize variables in consumable resource plugin to prevent core 
dump.
  -- Fix scancel bug which could return an error on attempt to signal a 
job step.
  -- In slurmctld communication agent, make the thread timeout be the 
configured
     value of MessageTimeout rather than 30 seconds.
  -- sshare -U/--Users only flag was used uninitialized.
  -- Cray systems, add "plugstack.conf.template" sample SPANK 
configuration file.
  -- BLUEGENE - Set DB2NOEXITLIST when starting the slurmctld daemon to 
avoid
     random crashing in db2 when the slurmctld is exiting.
  -- Make full node reservations display correctly the core count instead of
     cpu count.
  -- Preserve original errno on execve() failure in task plugin.
  -- Add SLURM_JOB_NAME env variable to an salloc's environment.
  -- Overwrite SLURM_JOB_NAME in an srun when it gets an allocation.
  -- Make sure each job has a wckey if that is something that is tracked.
  -- Make sure old step data is cleared when job is requeued.
  -- Load libtinfo as needed when building ncurses tools.
  -- Fix small memory leak in backup controller.
  -- Fix segfault when backup controller takes control for second time.
  -- Cray - Fix backup controller running native Slurm.
  -- Provide prototypes for init_setproctitle()/fini_setproctitle on NetBSD.
  -- Add configuration test to find out the full path to su command.
  -- preempt/job_prio plugin: Fix for possible infinite loop when 
identifying
     preemptable jobs.
  -- preempt/job_prio plugin: Implement the concept of Warm-up Time 
here. Use
     the QoS GraceTime as the amount of time to wait before preempting.
     Basically, skip preemption if your time is not up.
  -- Make srun wait KillWait time when a task is cancelled.
  -- switch/cray: Revert logic added to 14.11.6 that set 
"PMI_CRAY_NO_SMP_ENV=1"
     if CR_PACK_NODES is configured.
  -- Prevent users from setting job's partition to an invalid partition.

* Changes in Slurm 15.08.0pre5
==============================
  -- Add jobcomp/elasticsearch plugin. Libcurl is required for build. 
Configure
     the server as follows: 
"JobCompLoc=http://YOUR_ELASTICSEARCH_SERVER:9200".
  -- Scancel logic large re-written to better support job arrays.
  -- Added a slurm.conf parameter PrologEpilogTimeout to control how long
     prolog/epilog can run.
  -- Added TRES (Trackable resources) to track Mem, GRES, license, etc
     utilization.
  -- Add re-entrant versions of glibc time functions (e.g. localtime) to 
Slurm
     in order to eliminate rare deadlock of slurmstepd fork and exec calls.
  -- Constrain kernel memory (if available) in cgroups.
  -- Add PrologFlags option of "Contain" to create a proctrack container at
     job resource allocation time.
  -- Disable the OOM Killer in slurmd and slurmstepd's memory cgroup 
when using
     MemSpecLimit.


Gmane