pgBadger 10.x Release Notes

Complete upstream release notes for the pgBadger 10.x series

Categories:

Release

11 minute read

These entries preserve the complete upstream change record for pgBadger 10.x, newest first.

Source: upstream ChangeLog at commit a1ad95a.

v10.3 · 2019-02-14

This release of pgBadger is a maintenance release that fixes some log format autodetection issues another pgBouncer log parsing issue reported by users. There is also a new feature:

The -o | --outfile option can now be used multiple time to dump
output in several format in a single command. For example:
    pgbadger -o out.html -o out.json /log/pgsql-11.log
will create two reports in html and json format saved in the
two corresponding files.

There’s also some bugs fixes and features enhancements.

Fix statistics reports when there a filter on database, user, client or application is requested. Some queries was not reported.
Fix autodetection of pg>=10 defauilt log line prefix.
Fix autodetection of log file with “non standard” log line prefix. If –prefix specify %t, %m, %n and %p or %c, set format to stderr. Thanks to Alex Danvy for the report.
Remove extra space at end of line.
Add minimal test to syslog parser.
Fix a call to autodetect_format().
Truncate statement when maxlength is used. Thanks to Thibaud Madelaine for the patch.
Add test for multiple output format.
The -o | –outfile option can now be used multiple time to dump output in several format in a single command. For example: pgbadger -o out.txt -o out.html -o - -x json /log/pgsql-11.log Here pgbadger will create two reports in text and html format saved in the two corresponding file. It will also output a json report on standard output. Thanks to Nikolay for the feature request.
Move detection of output format and setting of out filename into a dedicated function set_output_extension().
Fix another pgBouncer log parsing issue. Thanks to Douglas J. Hunley for the report.

v10.2 · 2018-12-27

This release of pgBadger is a maintenance release that fixes issues reported by users during last three months. There is also some new features:

Add support to pgbouncer 1.8 Stats log format.
Auto adjust javascript graph timezone.

There is a new command line option:

Add –exclude-db option to compute report about everything except the specified database.
Add support to http or ftp remote PostgreSQL log file download. The log file is parsed during the download using curl command and never saved to disk. With ssh remote log parsing you can use uri as command line argument to specify the PostgreSQL log file.
```
    ssh://localhost/postgresql-10-main.log
    http://localhost/postgresql-10-main.log.gz
    ftp://localhost/postgresql-10-main.log
```
with http and ftp protocol you need to specify the log file format at end of the uri:
```
    http://localhost/postgresql-10-main.log:stderr
```
You can specify multiple uri for log files to be parsed. This is useful when you have pgbouncer log file on a remote host and PostgreSQL logs in the local host.

With ssh protocol you can use wild card too like with remote mode, ex: ssh://localhost/postgresql-10-main.log*

Old syntax to parse remote log file using -r option is still working but is obsolete and might be removed in future versions.

There’s also some bugs fixes and features enhancements.

Adjust end of progress bar with files with estimate size (bz2 compressed files and remote compressed files.
Update year in copyright.
Add information about URI notation to parse remote log files.
Force progress to reach 100% at end of parsing of compressed remote file.
Extract information about PL/pgSQL function call in queries of temporary file reports. The information is append to the details display block.
Fix progress bar with csv files.
Fix reading binary file as input file instead of log file.
Encode html output of queries into UTF8 to avoid message “Wide character in print”. Thanks to Colin ’t Hart for the report.
Add Checkpoints distance key/value for distance peak.
Fix pgbouncer parsing and request throughput reports. Thanks to Levente Birta for the report.
Fix use of csvlog instead of csv for input format.
Add support to pgbouncer 1.8 Stats log format. Thanks to Levente Birta for the report.
Add warning about parallel processing disabled with csvlog. Thanks to cstdenis for the report.
Add information in usage output about single process forcing with csvlog format in -j and -J options. Thanks to cstdenis for the report.
Fix unknown line format error for multi line log while incremental analysis over ssh. Thanks to Wooyoung Cho for the report.
Add -k (–insecure) option to curl command to be able to download logs from server using a self signed certificate.
Auto adjust javascript graph timezone. Thanks to Massimino Sala for the feature request.
Add support to HTTP logfile download by pgBadger, for example: /usr/bin/pgbadger http://www.mydom.com/postgresql-10.log
Will parse the file during download using curl command.
Fix documentation. Thanks to 0xflotus for the patch.
Reapply fix on missing replacement of bind parameters after some extra code cleaning. Thanks to Bernhard J. M. Grun for the report.
Add –exclude-db option to compute report about everything except the specified database. The option can be used multiple time.

v10.1 · 2018-09-12

This release of pgBadger is a maintenance release that fixes reports in incremental mode and multiprocess with -j option. Log parsing from standard input was also broken. If you are using v10.0 please upgrade now.

Add test on pgbouncer log parser.
Some little performances improvment.
Fix not a valid file descriptor at pgbadger line 12314.
Fix unwanted newline in progressbar at startup.
Remove circleci files from the project.
Remove dependency of bats and jq for the test suite, they are replaced with Test::Simple and JSON::XS.
Add more tests especially for incremental mode and input from stdin that was broken in release 10.0.
Sync pgbadger, pod, and README, and fix some syntax errors. Thanks to Christoph Berg for the patch.
Add documentation on how to install Perl module JSON::XS from apt and yum repositories.
Fix URI for CSS in incremental mode. Thanks to Floris van Nee for the report.
Fix fatal error when looking for log from STDIN. Thanks to Jacek Szpot for the report.
Fixes SED use for OSX builds. Thanks to Steve Newson for the patch.
Fix illegal division by zero in incrental mode. Thanks to aleszeleny for the report.
Replace SQL::Beautify with v3.1 of pgFormatter::Beautify.

v10.0 · 2018-09-09

This release of pgBadger is a major release that adds some new features and fix all issues reported by users since last release.

Add support of pgbouncer syslog log file format.
Add support to all auto_explain format (text, xml, json and yaml).
Add support to %q placeholder in log_line_prefix.
Add jsonlog format of Michael Paquier extension, with -f jsonlog pgbadger will be able to parse the log.
Replace the SQL formatter/beautify with v3.0 of pgFormatter.

There is some new command line option:

Add –prettify-json command line option to prettify JSON output.
Add –log-timezone +/-XX command line option to set the number of hours from GMT of the timezone that must be used to adjust date/time read from log file before beeing parsed. Note that you might still need to adjust the graph timezone using -Z when the client has not the same timezone.
Add –include-time option to add the ability to choose times that you want to see, instead of excluding all the times you do not want to see (–exclude-time).

The pgBadger project and copyrights has been transfered from Dalibo to the author and official maintainer of the project. Please update your links:

Web site: http://pgbadger.darold.net/
Source code: https://github.com/darold/pgbadger

I want to thanks the great guys at Dalibo for all their investments into pgBadger during these years and especially Damien Clochard and Jean-paul argudo for their help to promote pgBadger.

Fix checkpoint distance and estimate not reported in incremental mode. Thanks to aleszeleny for the report.
Fix title of pgbouncer simultaneous session report. Thansks to Jehan Guillaume De Rorthais for the report.
Add support of pgbouncer syslog log file format. Thanks to djester for the feature request.
Fix error when a remote log is empty. Thanks to Parasit Hendersson for the report.
Fix test with binary format. Binary file must be generated as it is dependent of the plateform. Thanks to Michal Nowak for the report.
Fix case where an empty explain plan is generated.
Fix parsing of autodetected default format with a prefix in command line.
Remove dependency of git command in Makefile.PL.
Update documentation about options changes and remove of the [%l-1] part of the mandatory prefix.
Fix parsing of vacuum / analyze system usage for PostgreSQL 10. Thanks to Achilleas Mantzios for the patch.
Fix Temporary File Activity table.
Remove dependency to git during install.
Add –log-timezone +/-XX command line option to set the number of hours from GMT of the timezone that must be used to adjust date/time read from log file before beeing parsed. Using this option make more difficult log search with a date/time because the time will not be the same in the log. Note that you might still need to adjust the graph timezone using -Z when the client has not the same timezone. Thanks to xdexter for the feature request and Julien Tachoire for the patch.
Add support to auto_explain json output format. Thanks to dmius for the report.
Fix auto_explain parser and queries that was counted twice. Thanks to zam6ak for the report.
Fix checkpoint regex to match PostgreSQL 10 log messages. Thanks to Edmund Horner for the patch.
Update description of -f | –format option by adding information about jsonlog format.
Fix query normalisation to not duplicate with bind queries. Normalisation of values are now tranformed into a single ? and no more 0 for numbers, two single quote for string. Thanks to vadv for the report.
Fix log level count. Thanks to Jean-Christophe Arnu for the report
Make pgbadger more compliant with B::Lint bare sub name.
Made perlcritic happy.
Add –prettify-json command line option to prettify JSON output. Default output is all in single line.
Fix Events distribution report.
Fix bug with –prefix when log_line_prefix contain multiple %%. Thanks to svb007 for the report.
Add –log-timezone +/-XX command line option to set the number of hours from GMT of the timezone that must be used to adjust date/time read from log file before beeing parsed. Using this option make more difficult log search with a date/time because the time will not be the same in the log. Note that you might still need to adjust the graph timezone using -Z when the client has not the same timezone. Thanks to xdexter for the feature request.
Remove INDEXES from the keyword list and add BUFFERS to this list.
Fix normalization of query using cursors.
Remove Dockerfile and documentation about docker run. pgBadger comes as a single Perl script without any dependence and it can be used on any plateform. It is a non sens to use docker to run pgbadger, if you don’t want to install anything, just copy the file pgbadger where you want and execute it.
Fix broken grid when no temp files activity. Thanks to Pierre Giraud for the patch
Add doc warning about log_in_duration_statement vs log_duration + log_statement. Thanks to Julien Tachoire for the patch.
Apply timezone offset to bar charts. Thanks to Julien Tachoire for the patch.
Delete current temp file info if we meet an error for the same PID Thanks to Julien Tachoire for the patch.
Consistently use app= in examples, and support appname= Some of the usage examples used appname= in the prefix, but the code didn’t recognize that token. Use app= in all examples, and add appname= to the prefix parser. Thanks to Christoph Berg for the patch
Fix wrong long name for option -J that should be –Jobs intead of –job_per_file. Thanks to Chad Trabant for the report and Etienne Bersac for the patch.
Ignore blib files. Thanks to Etienne Bersac for the patch.
Add consistency tests. Thanks to damien clochard for the patch.
doc update : stderr is not a default for -f. Thanks to Christophe Courtois for the patch.
Always update pod and README. Thanks to Etienne Bersac for the patch.
Add some regression tests. Thanks to Etienne Bersac for the patch.
Add editorconfig configuration. Thanks to Etienne Bersac for the patch.
Drop vi temp files from gitignore. Thanks to Etienne Bersac for the patch.
Add –include-time option to add the ability to choose times that you want to see, instead of excluding all the times you do not want to see. This is handy when wanting to view only one or two days from a week’s worth of logs (simplifies down from multiple –exlucde-time options to one –include-time). Thanks to Wesley Bowman for the patch.
Check pod syntax. Thanks to Etienne Bersac for the patch.
Add HACKING to document tests. Thanks to Etienne Bersac for the patch.
Drop obsolete –bar-graph option. Thanks to Etienne Bersac for the patch.
Drop misleading .perltidyrc. This file date from 2012 and pgbadger code is far from compliant. perltidy unified diff is 10k lines. Let’s drop this. Thanks to Etienne Bersac for the patch.
Fix use of uninitialized value in SQL formatting. Thanks to John Krugger for the report and Jean-paul Argudo for the report.