Dale Curtis | 1 Mar 02:14 2011
Picon

[PATCH] Host scheduler refactoring. Move HostScheduler out of monitor_db.

In order to facilitate site extensibility of HostScheduler we need to factor out the dependence on global variables in monitor_db. I modeled this refactoring off of monitor_db_cleanup.

The main changes I've made are as follows:
1. Move BaseHostScheduler, site import, and SchedulerError out of monitor_db. SchedulerError must be moved to prevent a cyclical dependency.
2. Convert staticmethod/classmethods in BaseHostScheduler to normal methods.
3. Fix unit tests and monitor_db to import SchedulerError from host_scheduler.


Signed-off-by: Dale Curtis <dalecurtis <at> google.com>
---
 scheduler/host_scheduler.py |  424 +++++++++++++++++++++++++++++++++++++++
 scheduler/monitor_db.py     |  457 ++-----------------------------------------
 2 files changed, 444 insertions(+), 437 deletions(-)
 create mode 100644 scheduler/host_scheduler.py

_______________________________________________
Autotest mailing list
Autotest <at> test.kernel.org
http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
Gregory P. Smith | 1 Mar 07:35 2011

Re: [PATCH] Host scheduler refactoring. Move HostScheduler out of monitor_db.

LGTM

On Mon, Feb 28, 2011 at 5:14 PM, Dale Curtis <dalecurtis <at> google.com> wrote:
In order to facilitate site extensibility of HostScheduler we need to factor out the dependence on global variables in monitor_db. I modeled this refactoring off of monitor_db_cleanup.

The main changes I've made are as follows:
1. Move BaseHostScheduler, site import, and SchedulerError out of monitor_db. SchedulerError must be moved to prevent a cyclical dependency.
2. Convert staticmethod/classmethods in BaseHostScheduler to normal methods.
3. Fix unit tests and monitor_db to import SchedulerError from host_scheduler.


Signed-off-by: Dale Curtis <dalecurtis <at> google.com>
---
 scheduler/host_scheduler.py |  424 +++++++++++++++++++++++++++++++++++++++
 scheduler/monitor_db.py     |  457 ++-----------------------------------------
 2 files changed, 444 insertions(+), 437 deletions(-)
 create mode 100644 scheduler/host_scheduler.py


_______________________________________________
Autotest mailing list
Autotest <at> test.kernel.org
http://test.kernel.org/cgi-bin/mailman/listinfo/autotest


_______________________________________________
Autotest mailing list
Autotest <at> test.kernel.org
http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
Lucas Meneghel Rodrigues | 1 Mar 22:31 2011
Picon

[PATCH] KVM test: Define OS specific info in the test variants

Some KVM autotest tests are only meant to run on Linux
platform, others are only meant to run on Windows. With
the new parser it is possible to state this information
even if the Linux and Windows blocks are defined later
inside tests_base.cfg. We advise test authors to follow
this convention from now on.

Signed-off-by: Lucas Meneghel Rodrigues <lmr <at> redhat.com>
---
 client/tests/kvm/tests_base.cfg.sample |   29 +++++++++++++++++++++++++----
 1 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/client/tests/kvm/tests_base.cfg.sample b/client/tests/kvm/tests_base.cfg.sample
index 661d6fe..eef8c97 100644
--- a/client/tests/kvm/tests_base.cfg.sample
+++ b/client/tests/kvm/tests_base.cfg.sample
 <at>  <at>  -113,15 +113,18  <at>  <at>  variants:
                 redirs += " unattended_install"
             # Install guest from http/ftp url
             - url:
+                only Linux
                 medium = url
                 url = REPLACE_THIS_WITH_TREE_URL
             # Install guest from nfs nfs_server:nfs_dir
             - nfs:
+                only Linux
                 medium = nfs
                 nfs_server = REPLACE_THIS_WITH_NFS_SERVER
                 nfs_dir = REPLACE_THIS_WITH_NFS_DIRECTORY
             # Install guest with a remote kickstart
             - remote_ks:
+                only Linux
                 medium = url
                 extra_params = " --append ks=REPLACE_THIS_WITH_URL_OF_KS"
                 url = REPLACE_THIS_WITH_TREE_URL
 <at>  <at>  -170,6 +173,7  <at>  <at>  variants:
                 iterations = 1
                 type = migration_with_file_transfer
             - with_autotest:
+                only Linux
                 type = autotest
                 migrate_background = yes
                 test_timeout = 1800
 <at>  <at>  -204,6 +208,7  <at>  <at>  variants:
         kill_vm = yes

     - autotest:     install setup unattended_install.cdrom
+        only Linux
         type = autotest
         test_timeout = 1800
         variants:
 <at>  <at>  -246,6 +251,7  <at>  <at>  variants:

 
     - linux_s3:     install setup unattended_install.cdrom
+        only Linux
         type = linux_s3

     - timedrift:    install setup unattended_install.cdrom
 <at>  <at>  -329,6 +335,7  <at>  <at>  variants:
         used_mem = 2560

     - guest_test:       install setup unattended_install.cdrom
+        only Windows
         type = guest_test
         login_timeout = 360
         test_timeout = 600
 <at>  <at>  -358,11 +365,13  <at>  <at>  variants:
                         dst_rsc_path = "C:\powershell\stub\stub.ps1"

     - iozone_windows: unattended_install.cdrom
+        only Windows
         type = iozone_windows
         iozone_cmd = "D:\IOzone\iozone.exe -a"
         iozone_timeout = 3600

     - whql:         install setup unattended_install.cdrom
+        only Windows
         nic_mode = tap
         # Replace this with the address of an installed DTM server
         server_address = 10.20.30.40
 <at>  <at>  -614,6 +623,7  <at>  <at>  variants:
         type = qmp_basic_rhel6

     - vlan:  install setup unattended_install.cdrom
+        only Linux
         type = vlan
         # subnet should not be used by host
         subnet = "192.168"
 <at>  <at>  -635,6 +645,7  <at>  <at>  variants:
         nics += ' nic2'

     - jumbo: install setup unattended_install.cdrom
+        only Linux
         type = jumbo

     - file_transfer: install setup unattended_install.cdrom
 <at>  <at>  -646,6 +657,7  <at>  <at>  variants:
                 transfer_type = remote

     - nicdriver_unload:  install setup unattended_install.cdrom
+        only Linux
         type = nicdriver_unload
         nic_mode = tap
         filesize = 100
 <at>  <at>  -654,12 +666,14  <at>  <at>  variants:
         sessions_num = 10

     - nic_promisc:  install setup unattended_install.cdrom
+        only Linux
         type = nic_promisc
         filesize = 400
         transfer_timeout = 100
         transfer_type = remote

     - multicast: install setup unattended_install.cdrom
+        only Linux
         type = multicast
         nic_mode = tap
         mcast = 225.0.0.1
 <at>  <at>  -680,10 +694,12  <at>  <at>  variants:
         pxe_timeout = 60

     - mac_change: install setup unattended_install.cdrom
+        only Linux
         type = mac_change
         kill_vm = yes

     - netperf: install setup unattended_install.cdrom
+        only Linux
         type = netperf
         nic_mode = tap
         netperf_files = netperf-2.4.5.tar.bz2 wait_before_data.patch
 <at>  <at>  -699,11 +715,13  <at>  <at>  variants:
                 protocols = "TCP_RR TCP_CRR UDP_RR"

     - ethtool: install setup unattended_install.cdrom
+        only Linux
         type = ethtool
         filesize = 512
         nic_mode = tap

     - nic_bonding:
+        only Linux
         type = nic_bonding
         nics += ' nic2 nic3 nic4'
         image_snapshot = yes
 <at>  <at>  -729,6 +747,7  <at>  <at>  variants:
         catch_uuid_cmd = dmidecode | awk -F: '/UUID/ {print $2}'

     - ksm_overcommit:
+        only Linux
         # Don't preprocess any vms as we need to change its params
         vms = ''
         image_snapshot = yes
 <at>  <at>  -751,9 +770,11  <at>  <at>  variants:
                 ksm_mode = "parallel"

     - iofuzz:
+        only Linux
         type = iofuzz

     - virtio_console:
+        only Linux
         vms = ''
         type = virtio_console
         # smoke params - $console_type:data_string
 <at>  <at>  -809,6 +830,7  <at>  <at>  variants:
         take_regular_screendumps = no

     - ioquit:
+        only Linux
         type = ioquit
         background_cmd = "for i in 1 2 3 4; do (dd if=/dev/urandom of=/tmp/file bs=102400 count=10000000 &); done"
         check_cmd = ps -a |grep dd
 <at>  <at>  -857,13 +879,16  <at>  <at>  variants:
                 image_name_snapshot2 = sn2

     - clock_getres: install setup unattended_install.cdrom
+        only Linux
         type = clock_getres

     - yum_update:    install setup unattended_install.cdrom
+        only Fedora, RHEL
         type = yum_update
         shell_prompt = "Is this ok"

     - kdump: unattended_install.cdrom
+        only Linux
         type = kdump
         # time waited for the completion of crash dump
         # crash_timeout = 360
 <at>  <at>  -945,7 +970,6  <at>  <at>  variants:
 variants:
     # Linux section
     -  <at> Linux:
-        no autoit iozone_windows whql
         shutdown_command = shutdown -h now
         reboot_command = shutdown -r now
         status_test_command = echo $?
 <at>  <at>  -1736,9 +1760,6  <at>  <at>  variants:

     # Windows section
     -  <at> Windows:
-        no autotest, linux_s3, vlan, ioquit, unattended_install.url, unattended_install.nfs, unattended_install.remote_ks
-        no jumbo, nicdriver_unload, nic_promisc, multicast, mac_change, ethtool, clock_getres
-
         shutdown_command = shutdown /s /f /t 0
         reboot_command = shutdown /r /f /t 0
         status_test_command = echo %errorlevel%
--

-- 
1.7.4
Lucas Meneghel Rodrigues | 2 Mar 01:50 2011
Picon

Re: [PATCH] Packager.py should take advantage of the parallelism from the package manager.

On Tue, 2011-02-22 at 13:01 -0800, Jean-Marc Eurin wrote:
> Packager.py should take advantage of the parallelism from the package manager.
> Create the repo directory if it doesn't exists.

Applied, thanks!

http://autotest.kernel.org/changeset/5273

> Signed-off-by: Jean-Marc Eurin <jmeurin <at> google.com>
> 
> --- autotest/client/common_lib/base_packages.py	2011-02-09 14:25:38.000000000 -0800
> +++ autotest/client/common_lib/base_packages.py	2011-02-22 12:51:10.000000000 -0800
>  <at>  <at>  -28,27 +28,36  <at>  <at> 
>              "Incorrect SSH path in global_config: %s" % repo)
>  
> 
> -def repo_run_command(repo, cmd, ignore_status=False):
> +def repo_run_command(repo, cmd, ignore_status=False, cd=True):
>      """Run a command relative to the repos path"""
>      repo = repo.strip()
>      run_cmd = None
> +    cd_str = ''
>      if repo.startswith('ssh://'):
>          username = None
>          hostline, remote_path = parse_ssh_path(repo)
> +        if cd:
> +            cd_str = 'cd %s && ' % remote_path
>          if ' <at> ' in hostline:
>              username, host = hostline.split(' <at> ')
> -            run_cmd = 'ssh %s <at> %s "cd %s && %s"' % (username, host,
> -                                                   remote_path, cmd)
> +            run_cmd = 'ssh %s <at> %s "%s%s"' % (username, host, cd_str, cmd)
>          else:
> -            run_cmd = 'ssh %s "cd %s && %s"' % (host, remote_path, cmd)
> +            run_cmd = 'ssh %s "%s%s"' % (host, cd_str, cmd)
>  
>      else:
> -        run_cmd = "cd %s && %s" % (repo, cmd)
> +        if cd:
> +            cd_str = 'cd %s && ' % repo
> +        run_cmd = "%s%s" % (cd_str, cmd)
>  
>      if run_cmd:
>          return utils.run(run_cmd, ignore_status=ignore_status)
>  
> 
> +def create_directory(repo):
> +    _, remote_path = parse_ssh_path(repo)
> +    repo_run_command(repo, 'mkdir -p %s' % remote_path, cd=False)
> +
> +
>  def check_diskspace(repo, min_free=None):
>      # Note: 1 GB = 10**9 bytes (SI unit).
>      if not min_free:
>  <at>  <at>  -272,6 +281,7  <at>  <at> 
>          if not repo.startswith('/') and not repo.startswith('ssh:'):
>              return
>          try:
> +            create_directory(repo)
>              check_diskspace(repo)
>              check_write(repo)
>          except (error.RepoWriteError, error.RepoUnknownError,
> --- autotest/utils/packager.py	2009-10-13 17:18:06.000000000 -0700
> +++ autotest/utils/packager.py	2011-02-22 12:51:10.000000000 -0800
>  <at>  <at>  -71,7 +71,7  <at>  <at> 
>  
> 
>  # Method to upload or remove package depending on the flag passed to it.
> -def process_packages(pkgmgr, pkg_type, pkg_names, src_dir, repo_url,
> +def process_packages(pkgmgr, pkg_type, pkg_names, src_dir,
>                      remove=False):
>      exclude_string = ' .'
>      names = [p.strip() for p in pkg_names.split(',')]
>  <at>  <at>  -101,12 +101,12  <at>  <at> 
>                      raise error.RepoDiskFullError(msg)
>                  tarball_path = pkgmgr.tar_package(pkg_name, pkg_dir,
>                                                    temp_dir, exclude_string)
> -                pkgmgr.upload_pkg(tarball_path, repo_url, update_checksum=True)
> +                pkgmgr.upload_pkg(tarball_path, update_checksum=True)
>              finally:
>                  # remove the temporary directory
>                  shutil.rmtree(temp_dir)
>          else:
> -            pkgmgr.remove_pkg(pkg_name, repo_url, remove_checksum=True)
> +            pkgmgr.remove_pkg(pkg_name, remove_checksum=True)
>          print "Done."
>  
> 
>  <at>  <at>  -137,7 +137,7  <at>  <at> 
>      return tarballs
>  
> 
> -def process_all_packages(pkgmgr, client_dir, upload_paths, remove=False):
> +def process_all_packages(pkgmgr, client_dir, remove=False):
>      """Process a full upload of packages as a directory upload."""
>      test_dir = os.path.join(client_dir, "tests")
>      site_test_dir = os.path.join(client_dir, "site_tests")
>  <at>  <at>  -185,22 +185,16  <at>  <at> 
>          os.chdir(temp_dir)
>          client_utils.system('md5sum * > packages.checksum')
>          os.chdir(cwd)
> -        for path in upload_paths:
> -            print "Uploading to: " + path
> -            pkgmgr.upload_pkg(temp_dir, path)
> +        pkgmgr.upload_pkg(temp_dir)
>          client_utils.run('rm -rf ' + temp_dir)
>      else:
> -        for repo_url in upload_paths:
> -            process_packages(pkgmgr, 'test', tests, client_dir, repo_url,
> -                             remove=remove)
> -            process_packages(pkgmgr, 'test', site_tests, client_dir, repo_url,
> -                             remove=remove)
> -            process_packages(pkgmgr, 'client', 'autotest', client_dir, repo_url,
> -                             remove=remove)
> -            process_packages(pkgmgr, 'dep', deps, dep_dir, repo_url,
> -                             remove=remove)
> -            process_packages(pkgmgr, 'profiler', profilers, prof_dir, repo_url,
> -                             remove=remove)
> +        process_packages(pkgmgr, 'test', tests, client_dir,remove=remove)
> +        process_packages(pkgmgr, 'test', site_tests, client_dir, remove=remove)
> +        process_packages(pkgmgr, 'client', 'autotest', client_dir,
> +                         remove=remove)
> +        process_packages(pkgmgr, 'dep', deps, dep_dir, remove=remove)
> +        process_packages(pkgmgr, 'profiler', profilers, prof_dir,
> +                         remove=remove)
>  
> 
>  # Get the list of sub directories present in a directory
>  <at>  <at>  -241,10 +235,6  <at>  <at> 
>      if len(upload_paths) == 0:
>          return
>  
> -    pkgmgr = packages.PackageManager(autotest_dir, repo_urls=repo_urls,
> -                                     upload_paths=upload_paths,
> -                                     run_function_dargs={'timeout':600})
> -
>      client_dir = os.path.join(autotest_dir, "client")
>  
>      # Bail out if the client directory does not exist
>  <at>  <at>  -267,35 +257,39  <at>  <at> 
>          # we should not be getting here
>          assert(False)
>  
> +    if options.repo:
> +        upload_path_list = [options.repo]
> +    else:
> +        upload_path_list = upload_paths
> +
> +    pkgmgr = packages.PackageManager(autotest_dir, repo_urls=repo_urls,
> +                                     upload_paths=upload_path_list,
> +                                     run_function_dargs={'timeout':600})
> +
>      if options.all:
> -        if options.repo:
> -            upload_path_list = [options.repo]
> -        else:
> -            upload_path_list = upload_paths
> -        process_all_packages(pkgmgr, client_dir, upload_path_list,
> -                             remove=remove_flag)
> +        process_all_packages(pkgmgr, client_dir, remove=remove_flag)
>  
>      if options.client:
>          process_packages(pkgmgr, 'client', 'autotest', client_dir,
> -                         options.repo, remove=remove_flag)
> +                         remove=remove_flag)
>  
>      if options.dep:
>          process_packages(pkgmgr, 'dep', options.dep, dep_dir,
> -                         options.repo, remove=remove_flag)
> +                         remove=remove_flag)
>  
>      if options.test:
>          process_packages(pkgmgr, 'test', options.test, client_dir,
> -                         options.repo, remove=remove_flag)
> +                         remove=remove_flag)
>  
>      if options.prof:
>          process_packages(pkgmgr, 'profiler', options.prof, prof_dir,
> -                         options.repo, remove=remove_flag)
> +                         remove=remove_flag)
>  
>      if options.file:
>          if remove_flag:
> -            pkgmgr.remove_pkg(options.file, options.repo, remove_checksum=True)
> +            pkgmgr.remove_pkg(options.file, remove_checksum=True)
>          else:
> -            pkgmgr.upload_pkg(options.file, options.repo, update_checksum=True)
> +            pkgmgr.upload_pkg(options.file, update_checksum=True)
>  
> 
>  if __name__ == "__main__":
> _______________________________________________
> Autotest mailing list
> Autotest <at> test.kernel.org
> http://test.kernel.org/cgi-bin/mailman/listinfo/autotest
Lucas Meneghel Rodrigues | 2 Mar 04:46 2011
Picon

[PATCH] KVM test: Create a verify_kernel_crash() VM method

A method to verify guest kernel panics can be very
useful for a number of tests. Adapted from a function
present on virtio_console test, create VM.verify_kernel_crash()
and use it on unattended_install.

Signed-off-by: Lucas Meneghel Rodrigues <lmr <at> redhat.com>
---
 client/tests/kvm/kvm_vm.py                   |   32 ++++++++++++++++++++++++++
 client/tests/kvm/tests/unattended_install.py |    1 +
 2 files changed, 33 insertions(+), 0 deletions(-)

diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
index 41f7491..ab60f71 100755
--- a/client/tests/kvm/kvm_vm.py
+++ b/client/tests/kvm/kvm_vm.py
 <at>  <at>  -105,6 +105,15  <at>  <at>  class VMDeadError(VMError):
                 (self.status, self.output))

 
+class VMDeadKernelCrashError(VMError):
+    def __init__(self, kernel_crash):
+        VMError.__init__(self, kernel_crash)
+        self.kernel_crash = kernel_crash
+
+    def __str__(self):
+        return ("VM is dead due to a kernel crash: %s" % self.kernel_crash)
+
+
 class VMAddressError(VMError):
     pass

 <at>  <at>  -1471,6 +1480,29  <at>  <at>  class VM:
         return self.serial_login(internal_timeout)

 
+    def verify_kernel_crash(self, timeout=2):
+        """
+        Find kernel crash message on serial console.
+
+         <at> param timeout: Timeout used to verify expected output.
+
+         <at> raise: VMDeadKernelCrashError, in case a kernel crash message was
+                found.
+        """
+        data = self.serial_console.read_nonblocking()
+        match = re.search("BUG:", data, re.MULTILINE)
+        if match is not None:
+            match = re.search(r"BUG:.*---\[ end trace .* \]---",
+                              data, re.DOTALL |re.MULTILINE)
+            if match is None:
+                data += self.serial_console.read_until_last_line_matches(
+                                               ["---\[ end trace .* \]---"],
+                                               timeout)
+            match = re.search(r"(BUG:.*---\[ end trace .* \]---)",
+                              data, re.DOTALL |re.MULTILINE)
+            raise VMDeadKernelCrashError(match.group(0))
+
+
      <at> error.context_aware
     def migrate(self, timeout=3600, protocol="tcp", cancel_delay=None,
                 offline=False, stable_check=False, clean=True,
diff --git a/client/tests/kvm/tests/unattended_install.py b/client/tests/kvm/tests/unattended_install.py
index 7c6d845..955f8d6 100644
--- a/client/tests/kvm/tests/unattended_install.py
+++ b/client/tests/kvm/tests/unattended_install.py
 <at>  <at>  -33,6 +33,7  <at>  <at>  def run_unattended_install(test, params, env):
     start_time = time.time()
     while (time.time() - start_time) < install_timeout:
         vm.verify_alive()
+        vm.verify_kernel_crash()
         client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
         try:
             client.connect((vm.get_address(), port))
--

-- 
1.7.4

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo <at> vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Suqin Huang | 2 Mar 05:59 2011
Picon

[PATCH] KVM-Test: Modify syntax of create_image

"()" can not connect two seperated line, 
remove it, and add "\"

Signed-off-by: Suqin Huang <shuang <at> redhat.com>
---
 client/tests/kvm/kvm_preprocessing.py |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/client/tests/kvm/kvm_preprocessing.py b/client/tests/kvm/kvm_preprocessing.py
index 2defb09..c814d3c 100644
--- a/client/tests/kvm/kvm_preprocessing.py
+++ b/client/tests/kvm/kvm_preprocessing.py
 <at>  <at>  -30,8 +30,8  <at>  <at>  def preprocess_image(test, params):
     if params.get("force_create_image") == "yes":
         logging.debug("'force_create_image' specified; creating image...")
         create_image = True
-    elif (params.get("create_image") == "yes" and not
-          os.path.exists(image_filename)):
+    elif params.get("create_image") == "yes" and not \
+          os.path.exists(image_filename):
         logging.debug("Creating image...")
         create_image = True
Jason Wang | 2 Mar 06:30 2011
Picon

[PATCH] KVM test: Create a verify_kernel_crash() VM method

Lucas Meneghel Rodrigues writes:
 > A method to verify guest kernel panics can be very
 > useful for a number of tests. Adapted from a function
 > present on virtio_console test, create VM.verify_kernel_crash()
 > and use it on unattended_install.
 > 

How about using a thread to monitor the serial console, so other test can also
benefit from this?

 > Signed-off-by: Lucas Meneghel Rodrigues <lmr <at> redhat.com>
 > ---
 >  client/tests/kvm/kvm_vm.py                   |   32 ++++++++++++++++++++++++++
 >  client/tests/kvm/tests/unattended_install.py |    1 +
 >  2 files changed, 33 insertions(+), 0 deletions(-)
 > 
 > diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
 > index 41f7491..ab60f71 100755
 > --- a/client/tests/kvm/kvm_vm.py
 > +++ b/client/tests/kvm/kvm_vm.py
 >  <at>  <at>  -105,6 +105,15  <at>  <at>  class VMDeadError(VMError):
 >                  (self.status, self.output))
 >  
 >  
 > +class VMDeadKernelCrashError(VMError):
 > +    def __init__(self, kernel_crash):
 > +        VMError.__init__(self, kernel_crash)
 > +        self.kernel_crash = kernel_crash
 > +
 > +    def __str__(self):
 > +        return ("VM is dead due to a kernel crash: %s" % self.kernel_crash)
 > +
 > +
 >  class VMAddressError(VMError):
 >      pass
 >  
 >  <at>  <at>  -1471,6 +1480,29  <at>  <at>  class VM:
 >          return self.serial_login(internal_timeout)
 >  
 >  
 > +    def verify_kernel_crash(self, timeout=2):
 > +        """
 > +        Find kernel crash message on serial console.
 > +
 > +         <at> param timeout: Timeout used to verify expected output.
 > +
 > +         <at> raise: VMDeadKernelCrashError, in case a kernel crash message was
 > +                found.
 > +        """
 > +        data = self.serial_console.read_nonblocking()
 > +        match = re.search("BUG:", data, re.MULTILINE)
 > +        if match is not None:
 > +            match = re.search(r"BUG:.*---\[ end trace .* \]---",
 > +                              data, re.DOTALL |re.MULTILINE)
 > +            if match is None:
 > +                data += self.serial_console.read_until_last_line_matches(
 > +                                               ["---\[ end trace .* \]---"],
 > +                                               timeout)
 > +            match = re.search(r"(BUG:.*---\[ end trace .* \]---)",
 > +                              data, re.DOTALL |re.MULTILINE)
 > +            raise VMDeadKernelCrashError(match.group(0))
 > +
 > +
 >       <at> error.context_aware
 >      def migrate(self, timeout=3600, protocol="tcp", cancel_delay=None,
 >                  offline=False, stable_check=False, clean=True,
 > diff --git a/client/tests/kvm/tests/unattended_install.py b/client/tests/kvm/tests/unattended_install.py
 > index 7c6d845..955f8d6 100644
 > --- a/client/tests/kvm/tests/unattended_install.py
 > +++ b/client/tests/kvm/tests/unattended_install.py
 >  <at>  <at>  -33,6 +33,7  <at>  <at>  def run_unattended_install(test, params, env):
 >      start_time = time.time()
 >      while (time.time() - start_time) < install_timeout:
 >          vm.verify_alive()
 > +        vm.verify_kernel_crash()
 >          client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 >          try:
 >              client.connect((vm.get_address(), port))
 > -- 
 > 1.7.4
 > 
 > --
 > To unsubscribe from this list: send the line "unsubscribe kvm" in
 > the body of a message to majordomo <at> vger.kernel.org
 > More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo <at> vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Goldish | 2 Mar 11:56 2011
Picon

Re: [PATCH] KVM test: Create a verify_kernel_crash() VM method

On 03/02/2011 07:30 AM, Jason Wang wrote:
> Lucas Meneghel Rodrigues writes:
>  > A method to verify guest kernel panics can be very
>  > useful for a number of tests. Adapted from a function
>  > present on virtio_console test, create VM.verify_kernel_crash()
>  > and use it on unattended_install.
>  > 
> 
> How about using a thread to monitor the serial console, so other test can also
> benefit from this?

If an exception is raised in a background thread, it won't kill the main
thread, so it's not obvious how a background thread can help here.  Or
maybe I misunderstood your suggestion?

>  > Signed-off-by: Lucas Meneghel Rodrigues <lmr <at> redhat.com>
>  > ---
>  >  client/tests/kvm/kvm_vm.py                   |   32 ++++++++++++++++++++++++++
>  >  client/tests/kvm/tests/unattended_install.py |    1 +
>  >  2 files changed, 33 insertions(+), 0 deletions(-)
>  > 
>  > diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
>  > index 41f7491..ab60f71 100755
>  > --- a/client/tests/kvm/kvm_vm.py
>  > +++ b/client/tests/kvm/kvm_vm.py
>  >  <at>  <at>  -105,6 +105,15  <at>  <at>  class VMDeadError(VMError):
>  >                  (self.status, self.output))
>  >  
>  >  
>  > +class VMDeadKernelCrashError(VMError):
>  > +    def __init__(self, kernel_crash):
>  > +        VMError.__init__(self, kernel_crash)
>  > +        self.kernel_crash = kernel_crash
>  > +
>  > +    def __str__(self):
>  > +        return ("VM is dead due to a kernel crash: %s" % self.kernel_crash)
>  > +
>  > +
>  >  class VMAddressError(VMError):
>  >      pass
>  >  
>  >  <at>  <at>  -1471,6 +1480,29  <at>  <at>  class VM:
>  >          return self.serial_login(internal_timeout)
>  >  
>  >  
>  > +    def verify_kernel_crash(self, timeout=2):
>  > +        """
>  > +        Find kernel crash message on serial console.
>  > +
>  > +         <at> param timeout: Timeout used to verify expected output.
>  > +
>  > +         <at> raise: VMDeadKernelCrashError, in case a kernel crash message was
>  > +                found.
>  > +        """
>  > +        data = self.serial_console.read_nonblocking()
>  > +        match = re.search("BUG:", data, re.MULTILINE)
>  > +        if match is not None:
>  > +            match = re.search(r"BUG:.*---\[ end trace .* \]---",
>  > +                              data, re.DOTALL |re.MULTILINE)
>  > +            if match is None:
>  > +                data += self.serial_console.read_until_last_line_matches(
>  > +                                               ["---\[ end trace .* \]---"],
>  > +                                               timeout)
>  > +            match = re.search(r"(BUG:.*---\[ end trace .* \]---)",
>  > +                              data, re.DOTALL |re.MULTILINE)
>  > +            raise VMDeadKernelCrashError(match.group(0))
>  > +
>  > +
>  >       <at> error.context_aware
>  >      def migrate(self, timeout=3600, protocol="tcp", cancel_delay=None,
>  >                  offline=False, stable_check=False, clean=True,
>  > diff --git a/client/tests/kvm/tests/unattended_install.py b/client/tests/kvm/tests/unattended_install.py
>  > index 7c6d845..955f8d6 100644
>  > --- a/client/tests/kvm/tests/unattended_install.py
>  > +++ b/client/tests/kvm/tests/unattended_install.py
>  >  <at>  <at>  -33,6 +33,7  <at>  <at>  def run_unattended_install(test, params, env):
>  >      start_time = time.time()
>  >      while (time.time() - start_time) < install_timeout:
>  >          vm.verify_alive()
>  > +        vm.verify_kernel_crash()
>  >          client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>  >          try:
>  >              client.connect((vm.get_address(), port))
>  > -- 
>  > 1.7.4
>  > 
>  > --
>  > To unsubscribe from this list: send the line "unsubscribe kvm" in
>  > the body of a message to majordomo <at> vger.kernel.org
>  > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo <at> vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo <at> vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Michael Goldish | 2 Mar 12:05 2011
Picon

Re: [PATCH] KVM test: Create a verify_kernel_crash() VM method

On 03/02/2011 05:46 AM, Lucas Meneghel Rodrigues wrote:
> A method to verify guest kernel panics can be very
> useful for a number of tests. Adapted from a function
> present on virtio_console test, create VM.verify_kernel_crash()
> and use it on unattended_install.
> 
> Signed-off-by: Lucas Meneghel Rodrigues <lmr <at> redhat.com>
> ---
>  client/tests/kvm/kvm_vm.py                   |   32 ++++++++++++++++++++++++++
>  client/tests/kvm/tests/unattended_install.py |    1 +
>  2 files changed, 33 insertions(+), 0 deletions(-)
> 
> diff --git a/client/tests/kvm/kvm_vm.py b/client/tests/kvm/kvm_vm.py
> index 41f7491..ab60f71 100755
> --- a/client/tests/kvm/kvm_vm.py
> +++ b/client/tests/kvm/kvm_vm.py
>  <at>  <at>  -105,6 +105,15  <at>  <at>  class VMDeadError(VMError):
>                  (self.status, self.output))
>  
>  
> +class VMDeadKernelCrashError(VMError):
> +    def __init__(self, kernel_crash):
> +        VMError.__init__(self, kernel_crash)
> +        self.kernel_crash = kernel_crash
> +
> +    def __str__(self):
> +        return ("VM is dead due to a kernel crash: %s" % self.kernel_crash)
> +
> +
>  class VMAddressError(VMError):
>      pass
>  
>  <at>  <at>  -1471,6 +1480,29  <at>  <at>  class VM:
>          return self.serial_login(internal_timeout)
>  
>  
> +    def verify_kernel_crash(self, timeout=2):
> +        """
> +        Find kernel crash message on serial console.
> +
> +         <at> param timeout: Timeout used to verify expected output.
> +
> +         <at> raise: VMDeadKernelCrashError, in case a kernel crash message was
> +                found.
> +        """
> +        data = self.serial_console.read_nonblocking()

I'm not sure using these methdos (read_nonblocking,
read_until_last_line_matches) is safe if verify_kernel_crash() is to be
used in tests that already use the serial console.  If a test uses the
serial console for interactive shell sessions, it might read the BUG:
message unintentionally (while waiting for a shell prompt, for example)
and then verify_kernel_crash() will not read the BUG: message because
data can't be read twice.  It's safer to use
self.serial_console.get_output(), which returns all the output seen so
far from the moment the process was started, including the output
already read by read_until_*().  However, it just returns a snapshot of
the output, so if you happen to call it while a trace is being printed,
you'll get a partial trace.  But if the intended usage for the function
is to be called periodically, that's fine, because the next time it's
called you'll get the full trace.  Does this make sense?

> +        match = re.search("BUG:", data, re.MULTILINE)
> +        if match is not None:
> +            match = re.search(r"BUG:.*---\[ end trace .* \]---",
> +                              data, re.DOTALL |re.MULTILINE)
> +            if match is None:
> +                data += self.serial_console.read_until_last_line_matches(
> +                                               ["---\[ end trace .* \]---"],
> +                                               timeout)
> +            match = re.search(r"(BUG:.*---\[ end trace .* \]---)",
> +                              data, re.DOTALL |re.MULTILINE)
> +            raise VMDeadKernelCrashError(match.group(0))
> +
> +
>       <at> error.context_aware
>      def migrate(self, timeout=3600, protocol="tcp", cancel_delay=None,
>                  offline=False, stable_check=False, clean=True,
> diff --git a/client/tests/kvm/tests/unattended_install.py b/client/tests/kvm/tests/unattended_install.py
> index 7c6d845..955f8d6 100644
> --- a/client/tests/kvm/tests/unattended_install.py
> +++ b/client/tests/kvm/tests/unattended_install.py
>  <at>  <at>  -33,6 +33,7  <at>  <at>  def run_unattended_install(test, params, env):
>      start_time = time.time()
>      while (time.time() - start_time) < install_timeout:
>          vm.verify_alive()
> +        vm.verify_kernel_crash()
>          client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>          try:
>              client.connect((vm.get_address(), port))
Lucas Meneghel Rodrigues | 2 Mar 14:07 2011
Picon

Re: [PATCH] KVM test: Create a verify_kernel_crash() VM method

On Wed, 2011-03-02 at 13:05 +0200, Michael Goldish wrote:
> On 03/02/2011 05:46 AM, Lucas Meneghel Rodrigues wrote:
> > +    def verify_kernel_crash(self, timeout=2):
> > +        """
> > +        Find kernel crash message on serial console.
> > +
> > +         <at> param timeout: Timeout used to verify expected output.
> > +
> > +         <at> raise: VMDeadKernelCrashError, in case a kernel crash message was
> > +                found.
> > +        """
> > +        data = self.serial_console.read_nonblocking()
> 
> I'm not sure using these methdos (read_nonblocking,
> read_until_last_line_matches) is safe if verify_kernel_crash() is to be
> used in tests that already use the serial console.  If a test uses the
> serial console for interactive shell sessions, it might read the BUG:
> message unintentionally (while waiting for a shell prompt, for example)
> and then verify_kernel_crash() will not read the BUG: message because
> data can't be read twice.  It's safer to use
> self.serial_console.get_output(), which returns all the output seen so
> far from the moment the process was started, including the output
> already read by read_until_*().  However, it just returns a snapshot of
> the output, so if you happen to call it while a trace is being printed,
> you'll get a partial trace.  But if the intended usage for the function
> is to be called periodically, that's fine, because the next time it's
> called you'll get the full trace.  Does this make sense?
> 

It does, certainly. It's a matter of changing the code to use
get_output() then, thanks for the suggestion, awesome!

Lucas

Gmane