Danny Auble | 18 Dec 21:01 2015
Gravatar

Slurm version 15.08.6 now available


We are pleased to announce the availability of Slurm version 15.08.6. 
This release is primarily in response to the regression in 15.08.5 with 
respects to finding the lua library.  It also contains a few other minor 
bug fixes you may find interesting.  Slurm downloads are available from: 
http://www.schedmd.com/#repos

We hope everyone has a great holiday and thanks for a great year!

* Changes in Slurm 15.08.6
==========================
  -- In slurmctld log file, log duplicate job ID found by slurmd. 
Previously was
     being logged as prolog/epilog failure.
  -- If a job is requeued while in the process of being launch, remove it's
     job ID from slurmd's record of active jobs in order to avoid 
generating a
     duplicate job ID error when launched for the second time (which would
     drain the node).
  -- Cleanup messages when handling job script and environment variables in
     older directory structure formats.
  -- Prevent triggering gang scheduling within a partition if configured 
with
     PreemptType=partition_prio and PreemptMode=suspend,gang.
  -- Decrease parallelism in job cancel request to prevent denial of service
     when canceling huge numbers of jobs.
  -- If all ephemeral ports are in use, try using other port numbers.
  -- Revert way lib lua is handled when doing a dlopen, fixing a 
regression in
     15.08.5.
(Continue reading)

jette | 11 Dec 02:35 2015
Gravatar

Slurm version 15.08.5 now available


We are pleased to announce the availability of Slurm version 15.08.5  
which includes about 30 bug fixes developed over the past few weeks as  
listed below. Slurm downloads are available from:
http://www.schedmd.com/#repos

* Changes in Slurm 15.08.5
==========================
  -- Prevent "scontrol update job" from updating jobs that have  
already finished.
  -- Show requested TRES in "squeue -O tres" when job is pending.
  -- Backfill scheduler: Test association and QOS node limits before reserving
     resources for pending job.
  -- burst_buffer/cray: If teardown operations fails, sleep and retry.
  -- Clean up the external pids when using the PrologFlags=Contain feature
     and the job finishes.
  -- burst_buffer/cray: Support file staging when job lacks job-specific buffer
     (i.e. only persistent burst buffers).
  -- Added srun option of --bcast to copy executable file to compute nodes.
  -- Fix for advanced reservation of burst buffer space.
  -- BurstBuffer/cray: Add logic to terminate dw_wlm_cli child processes at
     shutdown.
  -- If job can't be launch or requeued, then terminate it.
  -- BurstBuffer/cray: Enable clearing of burst buffer string on completed job
     as a means of recovering from a failure mode.
  -- Fix wrong memory free when parsing SrunPortRange=0-0 configuration.
  -- BurstBuffer/cray: Fix job record purging if cancelled from pending state.
  -- BGQ - Handle database throw correctly when syncing users on blocks.
  -- MySQL - Make sure we don't have a NULL string returned when not
     requesting any specific association.
(Continue reading)

Danny Auble | 13 Nov 23:58 2015
Gravatar

Slurm version 15.08.4 is now available


Slurm version 15.08.4 is now available it includes about 25 bug fixes 
developed over the past couple of weeks.

One notable fix is found in commits 8e66e2677 and d72f132d42 which will 
fix a slurmctld bug in which a pending job array could be canceled by a 
user different from the owner or the administrator. This appears to 
exist in the 15.08.* as well as the 14.11.* branches.

It is recommended you update at your earliest convenience.  If upgrading 
isn't an option generating a patch from those 2 commits is recommended.

Details about the changes are listed in the distribution's NEWS file. 
Slurm downloads are available from http://www.schedmd.com/#repos.

See you all at SC15 next week, Slurm booth #1851!

* Changes in Slurm 15.08.4
==========================
  -- Fix typo for the "devices" cgroup subsystem in pam_slurm_adopt.c
  -- Fix TRES_MAX flag to work correctly.
  -- Improve the systemd startup files.
  -- Added burst_buffer.conf flag parameter of "TeardownFailure" which will
     teardown and remove a burst buffer after failed stage-in or stage-out.
     By default, the buffer will be preserved for analysis and manual 
teardown.
  -- Prevent a core dump in srun if the signal handler runs during the job
     allocation causing the step context to be NULL.
  -- Don't fail job if multiple prolog operations in progress at slurmctld
     restart time.
(Continue reading)

Moe Jette | 4 Nov 23:56 2015
Gravatar

Slurm version 15.08.3 now available


We are pleased to announce the availability of Slurm version 15.08.3,  
includes about 25 bug fixes developed over the past couple of weeks as  
listed below. Slurm downloads are available from:
http://www.schedmd.com/#repos

SC15
There will be a Slurm User Group meeting on Thursday 19 November at  
12:15-13:15 in Room 16AB.
Please visit the Slurm booth (#1851) to pick up a quick reference  
guide and a limited edition tee-shirt.

* Changes in Slurm 15.08.3
==========================
  -- Correct Slurm's RPM build if Munge is not installed.
  -- Job array termination status email ExitCode based upon highest exit code
     from any task in the job array rather than the last task. Also change the
     state from "Ended" or "Failed" to "Mixed" where appropriate.
  -- Squeue recombines pending job array records only if their name  
and partition
     are identical.
  -- Fix some minor leaks in the job info and step info API.
  -- Export missing QOS id when filling in association with the association
     manager.
  -- Fix invalid reference if a lua job_submit plugin references a default qos
     when a user doesn't exist in the database.
  -- Use association enforcement in the lua plugin.
  -- Fix a few spots missing defines of accounting_enforce or acct_db_conn
     in the plugins.
  -- Show requested TRES in scontrol show jobs when job is pending.
(Continue reading)

Moe Jette | 22 Oct 21:57 2015
Gravatar

Slurm version 15.08.2 now available, SC15 News


We are pleased to announce the availability of Slurm version 15.08.2,  
which includes about 40 bug fixes developed over the past four weeks  
as listed below. Slurm downloads are available from:
http://www.schedmd.com/#repos

SC15 News:
There will be a Slurm User Group meeting on Thursday 19 November at  
12:15-13:15 in Room 16AM.
Please visit the Slurm booth (#1851) to pick up a quick reference  
guide and a limited edition tee-shirt.

* Changes in Slurm 15.08.2
==========================
  -- Fix for tracking node state when jobs that have been allocated exclusive
     access to nodes (i.e. entire nodes) and later relinquish some nodes. Nodes
     would previously appear partly allocated and prevent use by other jobs.
  -- Correct some cgroup paths ("step_batch" vs. "step_4294967294",  
"step_exter"
     vs. "step_extern", and "step_extern" vs. "step_4294967295").
  -- Fix advanced reservation core selection logic with network topology.
  -- MYSQL - Remove restriction to have to be at least an operator to  
query TRES
     values.
  -- For pending jobs have sacct print 0 for nnodes instead of the bogus 2.
  -- Fix for tracking node state when jobs that have been allocated exclusive
     access to nodes (i.e. entire nodes) and later relinquish some nodes. Nodes
     would previously appear partly allocated and prevent use by other jobs.
  -- Fix updating job in db after extending job's timelimit past partition's
     timelimit.
(Continue reading)

Moe Jette | 25 Sep 22:00 2015
Gravatar

Slurm version 15.08.1 is now availabe


We are pleased to announce the availability of Slurm version 15.08.1  
with about 40 bug fixes to 15.08.0. A list of changes is appended.

Slurm downloads are available from
http://www.schedmd.com/#repos

* Changes in Slurm 15.08.1
==========================
  -- Fix test21.30 and 21.34 to check grpwall better.
  -- Add time to the partition QOS the job is running on instead of just the
     job QOS.
  -- Print usage for GrpJobs, GrpSubmitJobs and GrpWall even if there is no
     limit.
  -- If AccountingEnforce=safe is set make sure a job can finish before going
     over the limit with grpwall on a QOS or association.
  -- burst_buffer/cray - Major updates based upon recent Cray changes.
  -- Improve clean up logic of pmi2 plugin.
  -- Improve job state reason string when required nodes not available.
  -- Fix missing else when packing an update partition message
  -- Fix srun from inheriting the SLURM_CPU_BIND and SLURM_MEM_BIND environment
     variables when running in an existing srun (e.g. an srun within  
an salloc).
  -- Fix missing else when packing an update partition message.
  -- Use more flexible mechnanism to find json installation.
  -- Make sure safe_limits was initialized before processing limits in the
     slurmctld.
  -- Fix for burst_buffer/cray to parse type option correctly.
  -- Fix memory error and version number in the nonstop plugin and reservation
     code.
(Continue reading)

Moe Jette | 21 Sep 19:01 2015
Gravatar

Slurm User Group Meeting, presentations online


Thanks to everyone who helped make the Slurm User Group Meeting last  
week a big success. Copies of the presentations are now on-line here:
http://slurm.schedmd.com/publications.html
--

-- 
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support

Danny Auble | 1 Sep 02:26 2015
Gravatar

Slurm versions 15.08.0 and 14.11.9 have been released!


Greetings everyone.

We are pleased to announce the release of 15.08.0!  It contains many new 
features and performance enhancements.  Please read the RELEASE_NOTES 
file to get an idea of the new items that have been added.  The on-line 
Slurm documentation has been updated to reflect this release.  Thanks to 
everyone that helped in this release.

Some notable changes are listed here.

  -- Added TRES (Trackable resources) to track utilization of memory, GRES,
     burst buffer, license, and any other configurable resources in the
     accounting database.
  -- Add configurable billing weight that takes into consideration any 
TRES when
     calculating a job's resource utilization.
  -- Add configurable prioritization factor that takes into 
consideration any
     TRES when calculating a job's resource utilization.
  -- Add burst buffer support infrastructure. Currently available plugin 
include
     burst_buffer/generic (uses administrator supplied programs to 
manage file
     staging) and burst_buffer/cray (uses Cray APIs to manage buffers).
  -- Add power capping support for Cray systems with automatic 
rebalancing of
     power allocation between nodes.
  -- Modify slurmctld outgoing RPC logic to support more parallel tasks 
(up to
(Continue reading)

Moe Jette | 21 Aug 01:35 2015
Gravatar

Slurm version 15.08.0-rc1 is now available


We are pleased to announce the availability of Slurm version 15.08.0-rc1
(release candidate 1). This version contains all of the features intended for
release 15.08 (with the exception of some minor burst buffer work) and we are
moving into a testing phase. You are invited to download this version  
and assist
in testing. Some highlights in this release include:
  -- Add TRES (Trackable resources) to track utilization of memory, GRES, burst
     buffer, license, and any other configurable resources in the accounting
     database.
  -- Add configurable billing weight that takes into consideration any  
TRES when
     calculating a job's resource utilization (for fair-share calculation).
  -- Add configurable prioritization factor that takes into consideration any
     TRES when calculating a job's resource utilization.
  -- Add burst buffer support infrastructure. Currently available  
plugin include
     burst_buffer/generic (uses administrator supplied programs to manage file
     staging) and burst_buffer/cray (uses Cray APIs to manage buffers).
  -- Add support for job dependencies joined with OR operator (e.g.
     "--depend=afterok:123?afternotok:124").
  -- Add advance reservation flag of "replace" that causes allocated resources
     to be replaced with idle resources. This maintains a pool of available
     resources that maintains a constant size (to the extent possible).
  -- Permit PreemptType=qos and PreemptMode=suspend,gang to be used together.
     A high-priority QOS job will now oversubscribe resources and gang  
schedule,
     but only if there are insufficient resources for the job to be started
     without preemption. NOTE: That with PreemptType=qos, the partition's
     Shared=FORCE:# configuration option will permit one job more per resource
(Continue reading)

Jacob Jenson | 8 Jul 02:02 2015

2015 Slurm User Group Meeting - Agenda and Registration

The 2015 Slurm User Group Meeting will be held on September 15 and 16 in Washington, DC. The meeting will include an assortment of tutorials, technical presentations, and site reports.

Reregistration and a preliminary agenda are available online:
http://slurm.schedmd.com/slurm_ug_agenda.html


Please register early to take advantage of the Early registration special
  • Early Registration - $250 USD per person for early registration by 31 July 2015
  • Standard Registration - $350 USD per person for standard registration by 31 August 2015
  • Late Registration - $600 USD per person for late registration starting 1 September 2015

Moe Jette | 8 Jul 01:06 2015
Gravatar

Slurm versions 14.11.8 and 15.08.0-pre6 are now available


Slurm version 14.11.8 includes about 30 relatively minor bug fixes  
developed over the past seven weeks while version 15.08.0-pre6  
contains new development scheduled for release next month. Details of  
changes are shown below. Slurm downloads are available from
http://www.schedmd.com/#repos

Also note that registration is open for the 2015 Slurm User Group  
Meeting. A preliminary agenda, registration and hotel information are  
available from
http://slurm.schedmd.com/slurm_ug_agenda.html

* Changes in Slurm 14.11.8
==========================
  -- Eliminate need for user to set user_id on job_update calls.
  -- Correct list of unavailable nodes reported in a job's "reason" field when
     that job can not start.
  -- Map job --mem-per-cpu=0 option to --mem=0.
  -- Fix squeue -o %m and %d unit conversion to Megabytes.
  -- Fix issue with incorrect time calculation in the priority plugin when
     a job runs past it's time limit.
  -- Prevent users from setting job's partition to an invalid partition.
  -- Fix sreport core dump when requesting
     'job SizesByAccount grouping=individual'.
  -- select/linear: Correct count of CPUs allocated to job on system with
     hyperthreads.
  -- Fix race condition where last array task might not get updated in the db.
  -- CRAY - Remove libpmi from rpm install
  -- Fix squeue -o %X output to correctly handle NO_VAL and suffix.
  -- When deleting a job from the system set the job_id to 0 to avoid memory
     corruption if thread uses the pointer basing validity off the id.
  -- Fix issue where sbatch would set ntasks-per-node to 0 making any srun
     afterward cause a divide by zero error.
  -- switch/cray: Refine logic to set PMI_CRAY_NO_SMP_ENV environment variable.
  -- When sacctmgr loads archives with version less than 14.11 set the array
     task id to NO_VAL, so sacct can display the job ids correctly.
  -- When using memory cgroup if a task uses more memory than requested
     the failures are logged into memory.failcnt count file by cgroup
     and the user is notified by slurmstepd about it.
  -- Fix scheduling inconsistency with GRES bound to specific CPUs.
  -- If user belongs to a group which has split entries in /etc/group
     search for its username in all groups.
  -- Do not consider nodes explicitly powered up as DOWN with reason of "Node
     unexpected rebooted".
  -- Use correct slurmd spooldir when creating cpu-frequency locks.
  -- Note that TICKET_BASED fairshare will be deprecated in the  
future. Consider
     using the FAIR_TREE algorithm instead.
  -- Set job's reason to BadConstaints when job can't run on any node.
  -- Prevent abort on update of reservation with no nodes (licenses only).
  -- Prevent slurmctld from dumping core ifjob_resrcs is missing in the
     job data structure.
  -- Fix squeue to print array task ids according to man page when
     SLURM_BITSTR_LEN is defined in the environment.
  -- In squeue sort jobs based on array job ID if available.
  -- Fix the calculation of job energy by not including the NO_VAL values.
  -- Advanced reservation fixes: enable update of bluegene reservation, avoid
     abort on multi-core reservations.
  -- Set the totalview_stepid to the value of the job step instead of NO_VAL.
  -- Fix slurmdbd core dump if the daemon does not have connection with
     the database.
  -- Display error message when attempting to modify priority of a held job.
  -- Backfill scheduler: The configured backfill_interval value (default 30
     seconds) is now interpretted as a maximum run time for the backfill
     scheduler. Once reached, the scheduler will build a new job queue and
     start over, even if not all jobs have been tested.
  -- Backfill scheduler now considers OverTimeLimit and KillWait configuration
     parameters to estimate when running jobs will exit.
  -- Correct task layout with CR_Pack_Node option and more than 1 CPU per task.
  -- Fix the scontrol man page describing the release argument.
  -- When job QOS is modified, do so before attempting to change partition in
     order to validate the partition's Allow/DenyQOS parameter.

* Changes in Slurm 15.08.0pre6
==============================
  -- Add scontrol options to view and modify layouts tables.
  -- Add MsgAggregationParams which controls a reverse tree to the slurmctld
     which can be used to aggregate messages to the slurmctld into a single
     message to reduce communication to the slurmctld.  Currently only epilog
     complete messages and node registration messages use this logic.
  -- Add sacct and squeue options to print trackable resources.
  -- Add sacctmgr option to display trackable resources.
  -- If an salloc or srun command is executed on a "front-end" configuration,
     that job will be assigned a slurmd shepherd daemon on the same  
host as used
     to execute the command when possible rather than an slurmd daemon on an
     arbitrary front-end node.
  -- Add srun --accel-bind option to control how tasks are bound to  
GPUs and NIC
     Generic RESources (GRES).
  -- gres/nic plugin modified to set OMPI_MCA_btl_openib_if_include environment
     variable based upon allocated devices (usable with OpenMPI and Melanox).
  -- Make it so info options for srun/salloc/sbatch print with just 1  
-v instead
     of 4.
  -- Add "no_backup_scheduling" SchedulerParameter to prevent jobs from being
     scheduled when the backup takes over. Jobs can be submitted, modified and
     cancelled while the backup is in control.
  -- Enable native Slurm backup controller to reside on an external Cray node
     when the "no_backup_scheduling" SchedulerParameter is used.
  -- Removed TICKET_BASED fairshare. Consider using the FAIR_TREE algorithm.
  -- Disable advanced reservation "REPLACE" option on IBM Bluegene systems.
  -- Add support for control distribution of tasks across cores (in addition
     to existing support for nodes and sockets, (e.g. "block", "cyclic" or
     "fcyclic" task distribution at 3 levels in the hardware rather than 2).
  -- Create db index on <cluster>_assoc_table.acct. Deleting accounts  
that didn't
     have jobs in the job table could take a long time.
  -- The performance of Profiling with HDF5 is improved. In addition, internal
     structures are changed to make it easier to add new profile types,
     particularly energy sensors. sh5util will continue to work with either
     format.
  -- Add partition information to sshare output if the --partition option
     is specified on the sshare command line.
  -- Add sreport -T/--tres option to identify Trackable RESources (TRES) to
     report.
  -- Display job in sacct when single step's cpus are different from the job
     allocation.
  -- Add association usage information to "scontrol show cache" command output.
  -- MPI/MVAPICH plugin now requires Munge for authentication.
  -- job_submit/lua: Add default_qos fields. Add job record qos.  Add partition
     record allow_qos and qos_char fields.
--

-- 
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support


Gmane