Stefan Fuhrmann | 22 Aug 12:50 2014

Revprop caching plan

Situation.

In 1.8, revprop caching only works if *all* processes accessing the
same repository

1. Are located on the same machine.
2. Have revprop caching enabled.

The problem with this is that in 1.8

* we did not mention the restriction in the release notes,
* have no way to detect a violation of that rule,
* ra_local access does not enable it
  (e.g. svnsync and svn over file://; svnadmin is fine)

While many setups can be made to work within these restrictions,
the current thinking is that even if we told people what they can do,
they will still do the wrong thing.

Background.

Revprop caches are per-process as the other caches we use.
To detect changes however, we bump a "generation" number
every time we modify any revprop in the repository. That number
is part of the cache key automatically hiding outdated contents.

Access to the generation number must be quick and free of hazards.
Therefore, we keep it in a shared memory segment and use atomic
operations to modify it. This requires all processes writing revprops
to be on the same machine and update the generation info in the
shared memory, i.e. must have revprop caching enabled as soon
as any of them has.

Failure to do so can result in non-atomic revprop changes and
stale revprop contents to be delivered.

Solution.

* Revert r1619413 on /trunk.
* Disable revprop caching on /trunk (ignore the FS config option)
* Propose for 1.8 backport, inform users appropriately upon release

* Open dev branch for revised implementation
* Directly use the generation file instead of the secondary SHM
* Use hash sums stored with the generation number to detect
  generation file update hazards.
* Always bump the generation upon revprop change
  (even if not cached)
* Disable revprop caching and issue a warning when a 1.8-style
  generation file is detected.
* Before writing modified revprops, read them from disk even in
  cached mode. That guarantees change API atomicy again.

Remaining restrictions:

* All processes modifying revprops must use 1.9 or later libraries
  or revprop caching cannot be used.
* Repositories on network file systems are not strictly supported
  (independent restriction due to lack of atomicy guarantees);
  shared block devices should be fine, though.

-- Stefan^2.
Philip Martin | 22 Aug 11:45 2014

Wunused-but-set-parameter warning in svn_wc_get_diff_editor6

I see this:

../src/subversion/libsvn_wc/deprecated.c: In function 'svn_wc_get_diff_editor6':
../src/subversion/libsvn_wc/deprecated.c:1991:39: warning: parameter 'ignore_ancestry' set but
not used [-Wunused-but-set-parameter]

It was added in 1569697.  Is this bit redundant or is something missing
from the compatibility function?

--

-- 
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*

Julian Foad | 21 Aug 13:19 2014

Local diff of copied dir shows all props added

Locally copying an (unmodified) dir with properties results in a wrong local diff output. 

[[[
$ svn cp tools/hook-scripts/mailer hs
A         hs

$ svn st
A  +    hs

$ svn diff hs/
### (prints a bunch of diff headers for unchanged files
### and then...)

Index: hs
===================================================================
--- hs    (nonexistent)
+++ hs    (working copy)

Property changes on: hs
___________________________________________________________________
Added: svn:ignore
## -0,0 +1 ##
+mailer.conf
]]]

The worst part of this output is at the end where it shows an all-properties-added diff for the directory "hs".

This seems to be an error in svn_wc__diff_local_only_dir() -- it passes NULL as the copy-from source and
NULL as the copy-from propswhen it calls processor->dir_added().

I looked at the corresponding function for files, svn_wc__diff_local_only_file(), and that jumps
through some hoops to fetch the 'pristine' props to use as copy-from props, and was tempted to just copy and
paste code from there but I couldn't completely follow what it's doing.

Anyone else want to take a look at it?

(This is with current trunk; running the 'svn diff hs/' with 1.8.0 and 1.8.10 also gives a similar result.)

- Julian

Alexey Neyman | 21 Aug 02:29 2014
Picon
Picon

r1619122 (stsp) broke 'make swig-py'

The new fields added in svn_wc.h cause the following errors:

subversion/bindings/swig/python/svn_wc.c: In function 
‘_wrap_svn_wc_conflict_description2_t_prop_value_base_set’:
subversion/bindings/swig/python/svn_wc.c:8798: error: ‘_global_pool’ 
undeclared (first use in this function)
subversion/bindings/swig/python/svn_wc.c:8798: error: (Each undeclared 
identifier is reported only once
subversion/bindings/swig/python/svn_wc.c:8798: error: for each function it 
appears in.)
subversion/bindings/swig/python/svn_wc.c: In function 
‘_wrap_svn_wc_conflict_description2_t_prop_value_working_set’:
subversion/bindings/swig/python/svn_wc.c:8859: error: ‘_global_pool’ 
undeclared (first use in this function)
subversion/bindings/swig/python/svn_wc.c: In function 
‘_wrap_svn_wc_conflict_description2_t_prop_value_incoming_old_set’:
subversion/bindings/swig/python/svn_wc.c:8920: error: ‘_global_pool’ 
undeclared (first use in this function)
subversion/bindings/swig/python/svn_wc.c: In function 
‘_wrap_svn_wc_conflict_description2_t_prop_value_incoming_new_set’:
subversion/bindings/swig/python/svn_wc.c:8981: error: ‘_global_pool’ 
undeclared (first use in this function)

because the memberin typemaps for svn_string_t assume they are used in 
functions that also receive the pool argument. Trivial fix (ignoring the 
offending structure in bindings) is attached.

A better fix would probably be fix the memberin typemaps for svn_string_t, but 
I do not know the specifics of pool usage in SWIG to do that.

Regards,
Alexey.
Attachment (make-swig-py-fix.diff): text/x-patch, 546 bytes
Evgeny Kotkov | 20 Aug 14:21 2014

[RFC] Revision property caching and named atomics

We (Sergey Raevskiy <sergey.raevskiy <at> visualsvn.com> and I) had some time to
examine the revision property caching feature, and there are some conclusions
that we would like to share.  In brief, there are certain conditions when this
feature can actually *wreak havoc* in terms of the repository consistency and
possible corruptions.  This applies to both Subversion trunk <at> 1618912 and to
the released Subversion 1.8.10.

The first (and probably, the major) part of the problem is that you cannot
really have mixed configurations of this caching option.  Whenever the revprop
caching is enabled in at least one process / thread accessing the filesystem,
it implicitly requires all other processes / threads that might eventually
access the filesystem to also have it enabled.  Otherwise, you are pretty much
open to different sorts of erroneous behavior, and there is a reason for this:

 - Currently, we only bump the revision property generations in the shared
   memory if the revprop caching is enabled, see switch_to_new_revprop() and
   svn_fs_fs__set_revision_proplist() functions.  These generations are used
   as a part of the cache key; if they are out-of-date, the corresponding
   cache entries are invalid.

 - In a situation when another process does not have this caching enabled, it
   would not bump these generations when changing revision properties (and the
   changes would just go to the disk).

 - The first process currently does not have a way to see these changes in the
   terms of revision property generations, i.e. it would be completely unaware
   of the changes, and will still think that the cached entries are up-to-date.

(I have attached a patch with a failing test as an illustation)

As a consequence, things could go pretty bad, and there are certain practical
implications.  As long as there is no way to prohibit mixed configurations in
terms of the revprop caching, we basically allow different combinations of the
following parts, ...

 1. 'svn' accessing file:/// repositories, revprop caching is disabled

 2. 'svnadmin', revprop caching is enabled or disabled in compile-time based
    on the build plaform and the availability of certain intrinsics, see
    svn_named_atomic__is_efficient()

 3. Apache HTTP Server, revprop caching is controlled by the SVNCacheRevProps
    directive, disabled by default [1]

 4. 'svnserve', revprop caching is controlled by the --cache-revprops switch

...and there are some deadly combinations, which probably are not limited to
the examples below:

- Repository served by both Apache HTTP Server and 'svnserve' with different
  revprop caching options.  This makes data corruption feasible, as we could
  sometimes rewrite newer revision properties with cached (old) values, and we
  also lose our test-and-set guarantee when changing them.  This could also
  trigger a permanent state when users *cannot change* certain revision
  property until a server restart, i.e. they would be continously receiving
  a SVN_ERR_FS_PROP_BASEVALUE_MISMATCH error due to a false negative
  within the test-and-set logic.  Finally, some of the revision property
  changes performed via one protocol would remain completely invisible to
  the users of the other protocol.

- Repository served by Apache HTTP Server with SVNCacheRevProps=on and
  modified with 'svnadmin' built with "inefficient" atomics (i.e. disabled
  revprop caching), pretty much with the same consequences as above.

The second part of the problem is using the apr_mmap()'ed files for the
repositories located on a network share.  According to the documentation, it
is not possible to achieve write coherency with these files (hence, with the
named atomics) on Windows [2]:
[[[
  CreateFileMapping()
  ...

  The exception is related to remote files. Although CreateFileMapping works
  with remote files, it does not keep them coherent. For example, if two
  computers both map a file as writable, and both change the same page, each
  computer only sees its own writes to the page. When the data gets updated on
  the disk, it is not merged (!).
]]]

Furthermore, it looks like there also is a fundamental problem in the way named
atomics are implemented.  If you have two applications compiled with different
atomic "efficiencies" (see NA_SYNCHRONIZE_IS_FAST and related defines), which
is possible, for instance, if you are using Subversion executables built under
different Windows versions, you are basically left without synchronization, for
example:

 # Application 1, inefficient atomics:

 - Locks around the corresponding lock file with SVN_ERR(lock()).

 - Starts performing a non-atomic operation, expects it to be serialized
   across *all* other possible processes / threads.

   ...

 # Context switches to Application 2 with efficient atomics:

 - It executes an atomic operation, say, InterlockedExchangeAdd64().
   This application, however, does not know anything about file locks.
   So, overall, the sequence is *not* thread-safe due to the non-atomicity
   of the operation performed by Application 1.

We do not want to make any conclusions on this topic.  However, this seems
to be pretty much broken, and these problems are probably affecting existing
Subversion 1.8 users.

Any thoughts on this?

[1] http://subversion.apache.org/docs/release-notes/1.8.html#revprop-caching
[2] http://msdn.microsoft.com/en-us/library/windows/desktop/aa366537

Regards,
Evgeny Kotkov
Index: subversion/tests/libsvn_fs_fs/fs-fs-pack-test.c
===================================================================
--- subversion/tests/libsvn_fs_fs/fs-fs-pack-test.c	(revision 1619038)
+++ subversion/tests/libsvn_fs_fs/fs-fs-pack-test.c	(working copy)
 <at>  <at>  -1208,6 +1208,44  <at>  <at>  metadata_checksumming(const svn_test_opts_t *opts,
 #undef REPO_NAME

 /* ------------------------------------------------------------------------ */
+
+#define REPO_NAME "revprop_caching_on_off"
+static svn_error_t *
+revprop_caching_on_off(const svn_test_opts_t *opts,
+                       apr_pool_t *pool)
+{
+  svn_fs_t *fs1;
+  svn_fs_t *fs2;
+  apr_hash_t *fs_config;
+  svn_string_t *value;
+  const svn_string_t *new_value = svn_string_create("new", pool);
+
+  /* Open two filesystem objects, enable revision property caching
+   * in one of them. */
+  SVN_ERR(svn_test__create_fs(&fs1, REPO_NAME, opts, pool));
+
+  fs_config = apr_hash_make(pool);
+  svn_hash_sets(fs_config, SVN_FS_CONFIG_FSFS_CACHE_REVPROPS, "1");
+
+  SVN_ERR(svn_fs_open2(&fs2, svn_fs_path(fs1, pool), fs_config, pool, pool));
+
+  SVN_ERR(svn_fs_revision_prop(&value, fs2, 0, "svn:date", pool));
+  SVN_ERR(svn_fs_change_rev_prop2(fs1, 0, "svn:date", &value,
+                                  new_value, pool));
+
+  /* Expect the change to be visible through both objects.*/
+  SVN_ERR(svn_fs_revision_prop(&value, fs1, 0, "svn:date", pool));
+  SVN_TEST_STRING_ASSERT(value->data, "new");
+
+  SVN_ERR(svn_fs_revision_prop(&value, fs2, 0, "svn:date", pool));
+  SVN_TEST_STRING_ASSERT(value->data, "new");
+
+  return SVN_NO_ERROR;
+}
+
+#undef REPO_NAME
+
+/* ------------------------------------------------------------------------ */
 
 /* The test table.  */

 <at>  <at>  -1248,6 +1286,9  <at>  <at>  static struct svn_test_descriptor_t test_funcs[] =
                        "prevent recursive locking"),
     SVN_TEST_OPTS_PASS(metadata_checksumming,
                        "metadata checksums being checked"),
+    SVN_TEST_OPTS_WIMP(revprop_caching_on_off,
+                       "change revprops with enabled and disabled caching",
+                       "fails due to FSFS revprop caching implementation"),
     SVN_TEST_NULL
   };

Alexey Neyman | 20 Aug 00:08 2014
Picon
Picon

Python bindings: core.Stream missing close method?

Hi,

It looks like the core.Stream class does not close the underlying streams. It 
does save reference to a svn_stream_t wrapper in self._stream, but does not 
call svn_stream_close() on its own deletion, nor does the auto-generated 
svn_stream_t class do so.

Obviously, it can be closed by calling core.svn_stream_close(s._stream) - but 
this accesses private fields not intended to be used as external interfaces.

Am I right that it leaks open streams? If I am - perhaps, fix as attached - to 
implement the file-object-like behavior for core.Stream?

Regards,
Alexey.
Attachment (stream-close.diff): text/x-patch, 983 bytes
Lieven Govaerts | 19 Aug 15:25 2014
Picon

[PATCH] Add a configuration option to disable HTTP pipelining.

Hi,

attached patch adds a new configuration flag to disable HTTP pipelining.

There's a bug in OpenSSL's SSL renegotiation algorithm. When it's
initiated by the server  to request a client certificate, it'll fail
when on the connection pipelined requests are incoming at the server
side.

Short summary of the root cause: during renegotiation, OpenSSL reads
data from the TCP connection expecting it to be a proper client
certificate. However, if an HTTP request was still pending on the
connection or in the server's receive buffer, OpenSSL will read that
request's data, recognise its not a proper client certificate, discard
the data and report an error. Apache will then abort the connection in
response to that OpenSSL error.

Given that there's no fix planned in OpenSSL, the only available
mitigation is to disable HTTP pipelining on connections where a SSL
renegotiation can happen. Since that depends on the configuration of
the server, we can't really know or predict when such renegotiation
will happen.

Conclusion: give the user the option to disable HTTP pipelining, which
she can use in case of problems caused by renegotiation.
Attached patch implements just that.

Objections anyone? Other remarks?

Lieven

[[[
Add an option "http-pipelining" to the servers configuration, so that a user
can disable HTTP pipelining in case that causes problems, e.g. during SSL
renegotiation triggered by the server to request a client certificate.

* subversion/include/svn_config.h
  (SVN_CONFIG_OPTION_HTTP_PIPELINING): New boolean config option.

* subversion/libsvn_ra_serf/ra_serf.h
  (struct svn_ra_serf__session_t): New member variable http_pipelining.

* subversion/libsvn_ra_serf/serf.c
  (load_config): Load the value of the new option from the servers file. If not
       set, use 'HTTP pipelining is enabled' by default.
  (svn_ra_serf__open,
   ra_serf_dup_session): Set the max. nr. of outstanding requests to 1
       if HTTP pipelining is disabled.

* subversion/libsvn_ra_serf/update.c
  (open_connection_if_needed): Set the max. nr. of outstanding requests to 1
       if HTTP pipelining is disabled.

* subversion/libsvn_subr/config_file.c
  (svn_config_ensure): Add the 'http-pipelining' option in the comment section
       of the initial servers file.
]]]
Index: subversion/include/svn_config.h
===================================================================
--- subversion/include/svn_config.h	(revision 1618848)
+++ subversion/include/svn_config.h	(working copy)
 <at>  <at>  -97,6 +97,8  <at>  <at>  typedef struct svn_config_t svn_config_t;
 #define SVN_CONFIG_OPTION_HTTP_MAX_CONNECTIONS      "http-max-connections"
 /**  <at> since New in 1.9. */
 #define SVN_CONFIG_OPTION_HTTP_CHUNKED_REQUESTS     "http-chunked-requests"
+/**  <at> since New in 1.9. */
+#define SVN_CONFIG_OPTION_HTTP_PIPELINING           "http-pipelining"

 /**  <at> since New in 1.9. */
 #define SVN_CONFIG_OPTION_SERF_LOG_COMPONENTS       "serf-log-components"
Index: subversion/libsvn_ra_serf/ra_serf.h
===================================================================
--- subversion/libsvn_ra_serf/ra_serf.h	(revision 1618848)
+++ subversion/libsvn_ra_serf/ra_serf.h	(working copy)
 <at>  <at>  -144,6 +144,13  <at>  <at>  struct svn_ra_serf__session_t {
      i.e. is there a (reverse) proxy that does not support them?  */
   svn_boolean_t detect_chunking;

+  /* Can serf use HTTP pipelining, or should it send requests one by one.
+     HTTP pipelining is enabled by default. The only known case where it should
+     be disabled is when the server triggers SSL renegotiations in the middle
+     of HTTP traffic on a connection, which OpenSSL currently doesn't handle
+     well. See serf issue #135. */
+  svn_boolean_t http_pipelining;
+
   /* Our Version-Controlled-Configuration; may be NULL until we know it. */
   const char *vcc_url;

Index: subversion/libsvn_ra_serf/serf.c
===================================================================
--- subversion/libsvn_ra_serf/serf.c	(revision 1618848)
+++ subversion/libsvn_ra_serf/serf.c	(working copy)
 <at>  <at>  -244,6 +244,12  <at>  <at>  load_config(svn_ra_serf__session_t *session,
                                   SVN_CONFIG_OPTION_HTTP_CHUNKED_REQUESTS,
                                   "auto", svn_tristate_unknown));

+  /* Should we use HTTP pipelining. */
+  SVN_ERR(svn_config_get_bool(config, &session->http_pipelining,
+                              SVN_CONFIG_SECTION_GLOBAL,
+                              SVN_CONFIG_OPTION_HTTP_PIPELINING,
+                              TRUE));
+
 #if SERF_VERSION_AT_LEAST(1, 4, 0) && !defined(SVN_SERF_NO_LOGGING)
   SVN_ERR(svn_config_get_int64(config, &log_components,
                                SVN_CONFIG_SECTION_GLOBAL,
 <at>  <at>  -311,6 +317,12  <at>  <at>  load_config(svn_ra_serf__session_t *session,
                                       SVN_CONFIG_OPTION_HTTP_CHUNKED_REQUESTS,
                                       "auto", chunked_requests));

+      /* Should we use HTTP pipelining. */
+      SVN_ERR(svn_config_get_bool(config, &session->http_pipelining,
+                                  server_group,
+                                  SVN_CONFIG_OPTION_HTTP_PIPELINING,
+                                  session->http_pipelining));
+
 #if SERF_VERSION_AT_LEAST(1, 4, 0) && !defined(SVN_SERF_NO_LOGGING)
       SVN_ERR(svn_config_get_int64(config, &log_components,
                                    server_group,
 <at>  <at>  -570,6 +582,10  <at>  <at>  svn_ra_serf__open(svn_ra_session_t *session,
   if (status)
     return svn_ra_serf__wrap_err(status, NULL);

+  if (!serf_sess->http_pipelining) {
+      serf_connection_set_max_outstanding_requests(serf_sess->conns[0]->conn, 1);
+  }
+
   /* Set the progress callback. */
   serf_context_set_progress_cb(serf_sess->context, svn_ra_serf__progress,
                                serf_sess);
 <at>  <at>  -771,6 +787,10  <at>  <at>  ra_serf_dup_session(svn_ra_session_t *new_session,
   if (status)
     return svn_ra_serf__wrap_err(status, NULL);

+  if (!new_sess->http_pipelining) {
+      serf_connection_set_max_outstanding_requests(new_sess->conns[0]->conn, 1);
+  }
+
   /* Set the progress callback. */
   serf_context_set_progress_cb(new_sess->context, svn_ra_serf__progress,
                                new_sess);
Index: subversion/libsvn_ra_serf/update.c
===================================================================
--- subversion/libsvn_ra_serf/update.c	(revision 1618848)
+++ subversion/libsvn_ra_serf/update.c	(working copy)
 <at>  <at>  -723,6 +723,11  <at>  <at>  open_connection_if_needed(svn_ra_serf__session_t *
       if (status)
         return svn_ra_serf__wrap_err(status, NULL);

+      if (!sess->http_pipelining) {
+          serf_connection_set_max_outstanding_requests(sess->conns[cur]->conn,
+                                                       1);
+      }
+
       sess->num_conns++;
     }

Index: subversion/libsvn_subr/config_file.c
===================================================================
--- subversion/libsvn_subr/config_file.c	(revision 1618848)
+++ subversion/libsvn_subr/config_file.c	(working copy)
 <at>  <at>  -940,6 +940,8  <at>  <at>  svn_config_ensure(const char *config_dir, apr_pool
         "###                              HTTP operation."                   NL
         "###   http-chunked-requests      Whether to use chunked transfer"   NL
         "###                              encoding for HTTP requests body."  NL
+        "###   http-pipelining            Whether to use HTTP pipelining "   NL
+        "###                              or send requests one by one."      NL
         "###   neon-debug-mask            Debug mask for Neon HTTP library"  NL
         "###   ssl-authority-files        List of files, each of a trusted CA"
                                                                              NL
Stefan Fuhrmann | 19 Aug 13:34 2014

Moving some of our tools to "main" subversion

Hi there,

At the SHF hackathon, we talked what tools should be installed
by default and which should be part of tools.

We decided to make

* svn-bench (reported as useful in the field) and
* svnfsfs (disaster recovery tools should be available by default)

part of the standard set of binaries. They will also be moved from
./tools to the ./subversion folder.

-- Stefan^2.
Picon

[l10n] Translation status report for trunk r1618793

Translation status report for trunk <at> r1618793

  lang   trans untrans   fuzzy     obs
--------------------------------------
    de    2722      62     227     474  +++++++++++++++++++++++++++~~~oooo
    es    2229     555     790     528  ++++++++++++++++++UUUUU~~~~~~~oooo
    fr    2533     251     471     109  +++++++++++++++++++++++UU~~~~~o
    it    2095     689     921     340  ++++++++++++++++UUUUUU~~~~~~~~oo
    ja    2222     562     843     763  ++++++++++++++++++UUUUU~~~~~~~oooooo
    ko    2364     420     610     219  ++++++++++++++++++++UUUU~~~~~~o
    nb    2279     505     741     501  +++++++++++++++++++UUUU~~~~~~~oooo
    pl    2304     480     708     298  +++++++++++++++++++UUUU~~~~~~~oo
 pt_BR    2071     713     935     321  ++++++++++++++++UUUUUU~~~~~~~~oo
    sv    2693      91     258      67  ++++++++++++++++++++++++++U~~~
 zh_CN    2582     202     383      10  ++++++++++++++++++++++++UU~~~~
 zh_TW    2012     772     973     377  ++++++++++++++++UUUUUU~~~~~~~~ooo

Masaru Tsuchiyama | 18 Aug 08:59 2014
Picon

[compile for windows] gen-make.py raise exception

I tried to compile subversion trunk, and running gen-make.py,
but raise exception at build\generator\gen_win_dependencies.py.

build\generator\gen_win_dependencies.py tries to find LIBINTL_VERSION
from libintl.h, but libintl.h doesn't contain LIBINTL_VERSION.

Where can I get libintl.h which has LIBINTL_VERSION  to run gen-make.py?

C:\svnwork\subversion\trunk>gen-make.py -t vcproj --vsnet-version=2008
--with-httpd=..\common\httpd-2.2.25
--with-sqlite=..\common\sqlite-amalgamation-3071700
--with-openssl=..\common\openssl-1.0.1e
--with-swig=C:\swigwin-2.0.10
--with-zlib=..\common\zlib-1.2.8
--disable-shared
--with-serf=..\common\serf-1.3.4
--with-libintl=..\common\svn-win32-libintl
--with-berkeley-db=..\common\db4-win32-4.4.20

Generating for Visual Studio 2008

Traceback (most recent call last):
  File "C:\svnwork\subversion\trunk\gen-make.py", line 321, in <module>
    main(conf, gentype, skip_depends=skip, other_options=rest.list)
  File "C:\svnwork\subversion\trunk\gen-make.py", line 60, in main
    generator = gen_module.Generator(fname, verfname, other_options)
  File "build\generator\gen_vcnet_vcproj.py", line 36, in __init__
    'vcnet-vcproj')
  File "build\generator\gen_win.py", line 83, in __init__
    self.find_libraries(True)
  File "build\generator\gen_win_dependencies.py", line 297, in
find_libraries
    self._find_libintl(show_warnings)
  File "build\generator\gen_win_dependencies.py", line 1322, in
_find_libintl
    ver = int(match.group(1), 0)
AttributeError: 'NoneType' object has no attribute 'group'

--

-- 
Masaru Tsuchiyama <m.tmatma <at> gmail.com>

Stefan Fuhrmann | 15 Aug 18:26 2014
Picon

Sunday dinner in Sheffield

Hi there,

people interested in grabbing a bite / having a feast on Sun night,
please let me know when you'll arrive in SHF. The current idea is
to leave around 8pm but that's flexible.

-- Stefan.

My phone: +49 1522 7898103


Gmane