== PostgreSQL Weekly News - November 23 2014 ==
David Fetter <david <at> fetter.org>
2014-11-24 14:34:11 GMT
== PostgreSQL Weekly News - November 23 2014 ==
PostgreSQL 9.4RC1 released. Test!
The CfP for FOSDEM PGDay and Devroom 2015 is open.
PostgreSQL Conference 2014 Tokyo will be held December 5 in Shinagawa.
pgDaySF 2015 will be held March 10, 2015 in Burlingame, California.
== PostgreSQL Product News ==
Ora2PG 14.1, a migration system for moving from Oracle to PostgreSQL, released.
== PostgreSQL Jobs for November ==
== PostgreSQL Local ==
PG-Cuba will be November 25 and 26, 2014
The CfP for FOSDEM PGDay (January 30, 2015) and the FOSDEM PostgreSQL
Dev Room (January 31-February 1, 2015) are open until November 24th,
The CfP is open for Nordic PostgreSQL Day 2015, which will be held
March 11, 2015 in Copenhagen, Denmark.
The CfP for PGConf US 2015 is open through December 17th, 2014
Notifications will go out on January 10, 2014. The event takes place
March 25-27, 2015 in NYC.
== PostgreSQL in the News ==
Planet PostgreSQL: http://planet.postgresql.org/
PostgreSQL Weekly News is brought to you this week by David Fetter
Submit news and announcements by Sunday at 3:00pm Pacific time.
Please send English language ones to david <at> fetter.org, German language
to pwn <at> pgug.de, Italian language to pwn <at> itpug.org. Spanish language
to pwn <at> arpug.com.ar.
== Applied Patches ==
Álvaro Herrera pushed:
- Fix relpersistence setting in reindex_index. Buildfarm members with
CLOBBER_CACHE_ALWAYS advised us that commit 85b506bbfc2937 was
mistaken in setting the relpersistence value of the index directly
in the relcache entry, within reindex_index. The reason for the
failure is that an invalidation message that comes after mucking
with the relcache entry directly, but before writing it to the
catalogs, would cause the entry to become rebuilt in place from
catalogs with the old contents, losing the update. Fix by passing
the correct persistence value to RelationSetNewRelfilenode instead;
this routine also writes the updated tuple to pg_class, avoiding the
problem. Suggested by Tom Lane.
Heikki Linnakangas pushed:
- Fix WAL-logging of B-tree "unlink halfdead page" operation. There
was some confusion on how to record the case that the operation
unlinks the last non-leaf page in the branch being deleted.
_bt_unlink_halfdead_page set the "topdead" field in the WAL record
to the leaf page, but the redo routine assumed that it would be an
invalid block number in that case. This commit fixes
_bt_unlink_halfdead_page to do what the redo routine expected. This
code is new in 9.4, so backpatch there.
- Remove obsolete debugging option, RTDEBUG. The r-tree AM that used
it was removed back in 2005. Peter Geoghegan
- Add test cases for indexam operations not currently covered. That
includes VACUUM on GIN, GiST and SP-GiST indexes, and B-tree indexes
large enough to cause page deletions in B-tree. Plus some other
special cases. After this patch, the regression tests generate all
different WAL record types. Not all branches within the redo
functions are covered, but it's a step forward.
- Silence compiler warning about variable being used uninitialized.
It's a false positive - the variable is only used when 'onleft' is
true, and it is initialized in that case. But the compiler doesn't
necessarily see that.
- Revamp the WAL record format. Each WAL record now carries
information about the modified relation and block(s) in a
standardized format. That makes it easier to write tools that need
that information, like pg_rewind, prefetching the blocks to speed up
recovery, etc. There's a whole new API for building WAL records,
replacing the XLogRecData chains used previously. The new API
consists of XLogRegister* functions, which are called for each
buffer and chunk of data that is added to the record. The new API
also gives more control over when a full-page image is written, by
passing flags to the XLogRegisterBuffer function. This also
simplifies the XLogReadBufferForRedo() calls. The function can dig
the relation and block number from the WAL record, so they no longer
need to be passed as arguments. For the convenience of redo
routines, XLogReader now disects each WAL record after reading it,
copying the main data part and the per-block data into MAXALIGNed
buffers. The data chunks are not aligned within the WAL record, but
the redo routines can assume that the pointers returned by
XLogRecGet* functions are. Redo routines are now passed the
XLogReaderState, which contains the record in the already-disected
format, instead of the plain XLogRecord. The new record format also
makes the fixed size XLogRecord header smaller, by removing the
xl_len field. The length of the "main data" portion is now stored at
the end of the WAL record, and there's a separate header after
XLogRecord for it. The alignment padding at the end of XLogRecord is
also removed. This compansates for the fact that the new format
would otherwise be more bulky than the old format. Reviewed by
Andres Freund, Amit Kapila, Michael Paquier, Alvaro Herrera, Fujii
- Add pageinspect functions for inspecting GIN indexes. Patch by me,
Peter Geoghegan and Michael Paquier, reviewed by Amit Kapila.
- Fix bogus comments in XLogRecordAssemble. Pointed out by Michael
- Add a comment to regress.c explaining what it contains. Ian Barwick
- No need to call XLogEnsureRecordSpace when the relation is unlogged.
Tom Lane pushed:
- Update time zone data files to tzdata release 2014j. DST law
changes in the Turks & Caicos Islands (America/Grand_Turk) and in
Fiji. New zone Pacific/Bougainville for portions of Papua New
Guinea. Historical changes for Korea and Vietnam.
- Update 9.4 release notes for commits through today.
- Fix some bogus direct uses of realloc(). pg_dump/parallel.c was
using realloc() directly with no error check. While the odds of an
actual failure here seem pretty low, Coverity complains about it, so
fix by using pg_realloc() instead. While looking for other
instances, I noticed a couple of places in psql that hadn't gotten
the memo about the availability of pg_realloc. These aren't bugs,
since they did have error checks, but verbosely inconsistent code is
not a good thing. Back-patch as far as 9.3. 9.2 did not have
pg_dump/parallel.c, nor did it have pg_realloc available in all
- Update comments in find_typedef. These comments don't seem to have
been touched in a long time. Make them describe the current
implementation rather than what was here last century, and be a bit
more explicit about the unreferenced-typedefs issue.
- Don't require bleeding-edge timezone data in timestamptz regression
test. The regression test cases added in commits b2cbced9e et al
depended in part on the Russian timezone offset changes of Oct 2014.
While this is of no particular concern for a default Postgres build,
it was possible for a build using --with-system-tzdata to fail the
tests if the system tzdata database wasn't au courant. Bjorn Munch
and Christoph Berg both complained about this while packaging
9.4rc1, so we probably shouldn't insist on the system tzdata being
up-to-date. Instead, make an equivalent test using a zone change
that occurred in Venezuela in 2007. With this patch, the regression
tests should pass using any tzdata set from 2012 or later. (I can't
muster much sympathy for somebody using --with-system-tzdata on a
machine whose system tzdata is more than three years out-of-date.)
- Improve documentation's description of JOIN clauses. In bug #12000,
Andreas Kunert complained that the documentation was misleading in
saying "FROM T1 CROSS JOIN T2 is equivalent to FROM T1, T2". That's
correct as far as it goes, but the equivalence doesn't hold when you
consider three or more tables, since JOIN binds more tightly than
comma. I added a <note> to explain this, and ended up rearranging
some of the existing text so that the note would make sense in
context. In passing, rewrite the description of JOIN USING, which
was unnecessarily vague, and hadn't been helped any by somebody's
reliance on markup as a substitute for clear writing. (Mostly this
involved reintroducing a concrete example that was unaccountably
removed by commit 032f3b7e166cfa28.) Back-patch to all supported
- Add missing case for CustomScan. Per KaiGai Kohei. In passing
improve formatting of some code added in commit 30d7ae3c, because
otherwise pgindent will make a mess of it.
- Remove no-longer-needed phony typedefs in genbki.h. Now that we
have a policy of hiding varlena catalog fields behind "#ifdef
CATALOG_VARLEN", there is no need for their type names to be
acceptable to the C compiler. And experimentation shows that it
does not matter to pgindent either. (If it did, we'd have problems
anyway, since these typedefs are unreferenced so far as the C
compiler is concerned, and find_typedef fails to identify such
typedefs.) Hence, remove the phony typedefs that genbki.h provided
to make some varlena field definitions compilable. In passing,
rearrange #define's into what seemed a more logical order.
- Fix another oversight in CustomScan patch. disuse_physical_tlist()
must work for all plan types handled by create_scan_plan().
- Fix another oversight in CustomScan patch. execCurrent.c's
search_plan_tree() must recognize a CustomScan on the target
relation. This would only be helpful for custom providers that
support CurrentOfExpr quals, which is probably a bit far-fetched,
but it's not impossible I think. But even without assuming that, we
need to recognize a scanned-relation match so that we will properly
throw error if the desired relation is being scanned with both a
CustomScan and a regular scan (ie, self-join). Also recognize
ForeignScanState for similar reasons. Supporting WHERE CURRENT OF
on a foreign table is probably even more far-fetched than it is for
custom scans, but I think in principle you could do it with
postgres_fdw (or another FDW that supports the ctid column). This
would be a back-patchable bug fix if existing FDWs handled
CurrentOfExpr, but I doubt any do so I won't bother back-patching.
- Initial code review for CustomScan patch. Get rid of the pernicious
entanglement between planner and executor headers introduced by
commit 0b03e5951bf0a1a8868db13f02049cf686a82165. Also, rearrange
the CustomFoo struct/typedef definitions so that all the typedef
names are seen as used by the compiler. Without this pgindent will
mess things up a bit, which is not so important perhaps, but it also
removes a bizarre discrepancy between the declaration arrangement
used for CustomExecMethods and that used for CustomScanMethods and
CustomPathMethods. Clean up the commentary around
ExecSupportsMarkRestore to reflect the rather large change in its
API. Const-ify register_custom_path_provider's argument. This
necessitates casting away const in the function, but that seems
better than forcing callers of the function to do so (or else not
const-ify their method pointer structs, which was sort of the whole
point). De-export fix_expr_common. I don't like the exporting of
fix_scan_expr or replace_nestloop_params either, but this one surely
has got little excuse.
- Remove dead code supporting mark/restore in SeqScan, TidScan,
ValuesScan. There seems no prospect that any of this will ever be
useful, and indeed it's questionable whether some of it would work
if it ever got called; it's certainly not been exercised in a very
long time, if ever. So let's get rid of it, and make the comments
about mark/restore in execAmi.c less wishy-washy. The mark/restore
support for Result nodes is also currently dead code, but that's due
to planner limitations not because it's impossible that it could be
useful. So I left it in.
- Simplify API for initially hooking custom-path providers into the
planner. Instead of register_custom_path_provider and a
CreateCustomScanPath callback, let's just provide a standard
function hook in set_rel_pathlist. This is more flexible than what
was previously committed, is more like the usual conventions for
planner hooks, and requires less support code in the core. We had
discussed this design (including centralizing the set_cheapest()
calls) back in March or so, so I'm not sure why it wasn't done like
- Rearrange CustomScan API. Make it work more like FDW plans do:
instead of assuming that there are expressions in a CustomScan plan
node that the core code doesn't know about, insist that all
subexpressions that need planner attention be in a "custom_exprs"
list in the Plan representation. (Of course, the custom plugin can
break the list apart again at executor initialization.) This lets us
revert the parts of the patch that exposed setrefs.c and subselect.c
processing to the outside world. Also revert the
GetSpecialCustomVar stuff in ruleutils.c; that concept may work in
future, but it's far from fully baked right now.
- Fix mishandling of system columns in FDW queries. postgres_fdw
would send query conditions involving system columns to the remote
server, even though it makes no effort to ensure that system columns
other than CTID match what the remote side thinks. tableoid, in
particular, probably won't match and might have some use in queries.
Hence, prevent sending conditions that include non-CTID system
columns. Also, create_foreignscan_plan neglected to check local
restriction conditions while determining whether to set fsSystemCol
for a foreign scan plan node. This again would bollix the results
for queries that test a foreign table's tableoid. Back-patch the
first fix to 9.3 where postgres_fdw was introduced. Back-patch the
second to 9.2. The code is probably broken in 9.1 as well, but the
patch doesn't apply cleanly there; given the weak state of support
for FDWs in 9.1, it doesn't seem worth fixing. Etsuro Fujita,
reviewed by Ashutosh Bapat, and somewhat modified by me
- Get rid of redundant production in plpgsql grammar. There may once
have been a reason for the intermediate proc_stmts production in the
plpgsql grammar, but it isn't doing anything useful anymore, so
let's collapse it into proc_sect. Saves some code and probably a
small number of nanoseconds per statement list. In passing,
correctly alphabetize keyword lists to match pl_scanner.c; note that
for "rowtype" vs "row_count", pl_scanner.c must sort on the basis of
the lower-case spelling. Noted while fooling with a patch to
de-reserve more plpgsql keywords.
Fujii Masao pushed:
- Add --synchronous option to pg_receivexlog, for more reliable WAL
writing. Previously pg_receivexlog flushed WAL data only when WAL
file was switched. Then 3dad73e added -F option to pg_receivexlog
so that users could control how frequently sync commands were issued
to WAL files. It also allowed users to make pg_receivexlog flush WAL
data immediately after writing by specifying 0 in -F option. However
feedback messages were not sent back immediately even after a flush
location was updated. So even if WAL data was flushed in real time,
the server could not see that for a while. This commit removes -F
option from and adds --synchronous to pg_receivexlog. If
--synchronous is specified, like the standby's wal receiver,
pg_receivexlog flushes WAL data as soon as there is WAL data which
has not been flushed yet. Then it sends back the feedback message
identifying the latest flush location to the server. This option is
useful to make pg_receivexlog behave as sync standby by using
replication slot, for example. Original patch by Furuya Osamu,
heavily rewritten by me. Reviewed by Heikki Linnakangas, Alvaro
Herrera and Sawada Masahiko.
- Fix pg_receivexlog --slot so that it doesn't prevent the server
shutdown. When pg_receivexlog --slot is connecting to the server,
at the shutdown of the server, walsender keeps waiting for the last
WAL record to be replicated and flushed in pg_receivexlog. But
previously pg_receivexlog issued sync command only when WAL file was
switched. So there was the case where the last WAL was never flushed
and walsender had to keep waiting infinitely. This caused the server
shutdown to get stuck. pg_recvlogical handles this problem by
calling fsync() when it receives the request of immediate reply from
the server. That is, at shutdown, walsender sends the request,
pg_recvlogical receives it, flushes the last WAL record, and sends
the flush location back to the server. Since walsender can see that
the last WAL record is successfully flushed, it can exit cleanly.
This commit introduces the same logic as pg_recvlogical has, to
pg_receivexlog. Back-patch to 9.4 where pg_receivexlog was changed
so that it can use the replication slot. Original patch by Michael
Paquier, rewritten by me. Bug report by Furuya Osamu.
- Fix bug in the test of file descriptor of current WAL file in
pg_receivexlog. In pg_receivexlog, in order to check whether the
current WAL file is being opened or not, its file descriptor has to
be checked against -1 as an invalid value. But, oops, 7900e94 added
the incorrect test checking the descriptor against 1. This commit
fixes that bug. Back-patch to 9.4 where the bug was added. Spotted
by Magnus Hagander
Simon Riggs pushed:
- Add pg_dump --snapshot option. Allows pg_dump to use a snapshot
previously defined by a concurrent session that has either used
pg_export_snapshot() or obtained a snapshot when creating a logical
slot. When this option is used with parallel pg_dump, the snapshot
defined by this option is used and no new snapshot is taken. Simon
Riggs and Michael Paquier
- Reduce btree scan overhead for < and > strategies. For <, <=, > and >=
strategies, mark the first scan key as already matched if scanning
in an appropriate direction. If index tuple contains no nulls we
can skip the first Author: Rajeev Rastogi. Reviewer: Haribabu
Kommi. Rework of the code and comments by Simon Riggs
Robert Haas pushed:
- Avoid file descriptor leak in pg_test_fsync. This can cause
problems on Windows, where files that are still open can't be
unlinked. Jeff Janes
Peter Eisentraut pushed:
- Fix suggested layout for PGXS makefile. Custom rules must come
after pgxs inclusion, not before, because any rule added before pgxs
will break the default 'all' target. Author: Cédric Villemain
<cedric <at> 2ndquadrant.fr>
Andrew Dunstan pushed:
- Make psql's \sf and \ef honor ECHO_HIDDEN. These commands were
calling the database direct rather than calling PSQLexec like other
slash commands that needed database data. The code is also changed
not to pass the connection as a parameter to the helper functions.
It's available in a global variable, and that's what PSQLexec uses.
- Fix an error in psql that overcounted output lines. This error
counted the first line of a cell as "extra". The effect was to cause
far too frequent invocation of the pager. In most cases this can be
worked around (for example, by using the "less" pager with the -F
flag), so don't backpatch.
- Rework echo_hidden for \sf and \ef from commit e4d2817. PSQLexec's
error reporting turns out to be too verbose for this case, so revert
to using PQexec instead with minimal error reporting. Prior to
calling PQexec, we call a function that mimics just the echo_hidden
piece of PSQLexec.
- Fix memory leaks introduced by commit eca2b9b
Noah Misch pushed:
- Detect PG_PRINTF_ATTRIBUTE automatically. This eliminates gobs of
"unrecognized format function type" warnings under MinGW compilers
predating GCC 4.4.
== Rejected Patches (for now) ==
No one was disappointed this week
== Pending Patches ==
Etsuro Fujita sent in another revision of a patch to allow foreign
tables to participate in table inheritance hierarchies.
Jeff Davis and Tomas Vondra traded patches to do better memory
accounting, in part as infrastructure for a memory-bounded HashAgg.
Michael Paquier sent in a patch to fix CLOBBER_CACHE_ALWAYS.
Michael Paquier sent in another revision of a patch to track number of
files marked as ready for archiving in pg_stat_archiver.
SAWADA Masahiko sent in another revision of a patch to implement
REINDEX SCHEMA, in this case as an option to reindexdb.
Heikki Linnakangas sent in two PoC patches to deal with a performance
problem in PageRepairFragmentation.
Petr (PJMODOS) Jelinek and Simon Riggs traded patches to add a
shutdown_at_recovery_target option to recovery.conf.
Dimitri Fontaine sent in three more revisions of a patch to add "table
rewrite" as an event which can fire an event trigger.
Adam Brightwell sent in a patch to add int64-to-int8 mapping in
Magnus Hagander sent in another revision of a patch to add a
Aaron W. Swenson sent in a patch to replace configure.in with
configure dot ac.
Heikki Linnakangas sent in a patch to add some more tests for WALs.
Peter Geoghegan sent in two revisions of a patch to create an amcheck
utility, a generalization of his previous btreecheck.
Michael Paquier sent in a patch to uppercase some SQL keywords in the
Amit Kapila sent in another revision of a patch to fix some
infelicities between pg_basebackup and Windows.
Peter Eisentraut sent in a series of patches which: Fix SHLIB_PREREQS
use in contrib, allowing PGXS builds, allow the user to set VPATH
directly, and clean up some makefile changes that weren't needed.
Abhijit Menon-Sen sent in a patch to use slice-by-8 for CRC, which
increased the performance of that operation by a factor of 4+.
Adam Brightwell sent in another revision of a patch to fix Catalog.pm
for role attributes.
Laurenz Albe sent in a patch to disallow changing functions which are
part of an index definition.
Michael Paquier sent in a patch to enable GIN code managing entry
insertion not to distinguish fresh from old indexes.
Robert Haas sent in another revision of a patch to add infrastructure
to save and restore GUC values.
Michael Paquier sent in a patch to add fillfactor as storage parameter
for GIN indexes.
Petr (PJMODOS) Jelinek sent in two more revisions of a patch to track
Adam Brightwell sent in another revision of a patch to clean up
Alexander Shulgin sent in another revision of a patch to turn
recovery.conf into GUCs.
Álvaro Herrera added some opfamilies for BRIN indexes which should
help make them more useful.
Marko (johto) Tiikkaja sent in two revisions of a patch to make the
HINTs more useful for certain types common mistakes in UPDATE.
Amit Kapila sent in another revision of a patch to parallelize
David Rowley sent in another revision of a patch to support SEMI and
ANTI join removal in certain cases.
Tom Lane sent in a patch to remove 17 of PL/pgsql's reserved words.
Sent via pgsql-announce mailing list (pgsql-announce <at> postgresql.org)
To make changes to your subscription: