MANATEE-371: Want ISM for SysV Shared Memory in Postgres >= 9.3

Details

Issue Type:Bug
Priority:2 - Critical
Status:Resolved
Created at:2017-09-29T22:05:36.000Z
Updated at:2020-08-24T13:33:38.370Z

People

Created by:Former user
Reported by:Former user
Assigned to:Former user

Resolution

Fixed: A fix for this issue is checked into the tree and tested.
(Resolution Date: 2017-10-06T23:48:49.000Z)

Related Issues

Description

After upgrading from Postgres 9.2.4 to 9.6.3 we noticed a large increase in memory usage by postgres on the order of one hundred gigabytes. The discrepancy in memory usage has been attributed to the lack of ISM (intimate shared memory) support in versions of Postgres >= 9.3.

A bug in the Postgres Bug Mailing List corroborated our observation.
BUG #13416: Postgres >= 9.3 doesn't use optimized shared memory on Solaris anymore. The bug report details how to reproduce this bug and it has been reproduced on SmartOS in a base-multiarch zone version 17.2.0.

Comments

Comment by Former user
Created at 2017-10-02T23:58:32.000Z

Testing Notes

Upgraded an existing Manatee sync peer running PostgreSQL 9.6.3, and checked that replication was still working. By inspecting ipcs -mA output in the zone, we see a large increase in SHM segment size indicating SYSV/ISM usage. In pmap -sx output, we also see an additional ISM segment of much larger size after the change.

The same upgrade was also performed on the primary peer in the same cluster.

Finally, a new Manatee cluster was created to check that initdb still works.


Comment by Former user
Created at 2017-10-03T01:37:23.000Z

Note that during testing in the Manatee cluster we found what appears to be another POSIX shared memory segment,
with a name of the form .SHMDPostgreSQL.NNNNNNN. This was in addition to the larger ISM segment that appeared to be for shared buffer usage.

Manatee does not currently specify a value in the PostgreSQL configuration file for dynamic_shared_memory_type, but if we explicitly set it to sysv this segment does not appear to be created. The change to manta-manatee.git to incorporate the patched PostgreSQL version will update the configuration templates to include this value.


Comment by Former user
Created at 2017-10-04T21:59:40.000Z
Updated at 2017-12-14T17:35:44.581Z

After submitting this patch upstream we received a reply from Andrews Freund that he had written a similar patch that added a tunable (GUC in postgres jargon) in postgresql.conf called shared_memory_type which has possible values mmap, sysv and windows. I have updated that patch to 9.6.4 for our use.

If we'd like to upstream this patch we will probably need to re-work this patch and turn shared_memory_type into a compile-time option. After reading the mailing list thread that resulted in the patch above, it's clear that the postgres community feels that postgresql.conf should only consist of options that adapt postgres to a particular workload, not operating system. However, our patch will make use of a runtime tunable because we would like to be able to turn off this behavior if it causes problems.

Testing results to follow.


Comment by Former user
Created at 2017-10-04T22:07:48.000Z
Updated at 2017-12-14T17:35:44.639Z

Testing Summary

Here are the different postgresql.conf configurations we could use with the first being the configuration we plan to use in production.

shared_memory_type = sysv
dynamic_shared_memory_type = sysv
FFFFFD7FF5A00000     145008     145008          -     145008 rwxsR    [ ism shmid=0x47000010 ]
FFFFFD7FFE9D0000         64         36          -          - rwx--    [ anon ]
FFFFFD7FFE9F0000          4          4          -          4 rwxsR    [ ism shmid=0x43000011 ]

----

shared_memory_type = sysv
dynamic_shared_memory_type = posix
FFFFFD7FF5A00000     145008     145008          -     145008 rwxsR    [ ism shmid=0x4500000c ]
FFFFFD7FFE9D0000         64         36          -          - rwx--    [ anon ]
FFFFFD7FFE9F0000          4          4          -          - rw-s-  .SHMDPostgreSQL.1665330762

----

shared_memory_type = mmap
dynamic_shared_memory_type = posix 
FFFFFD7FF5C00000     145008      10516          -          - rw-s-    [ anon ]
FFFFFD7FFE9C0000         64         36          -          - rwx--    [ anon ]
FFFFFD7FFE9E0000          4          4          -          - rw-s-  .SHMDPostgreSQL.434476131
FFFFFD7FFE9F0000          4          4          -          4 rwxsR    [ ism shmid=0x4700000d ]

----

shared_memory_type = mmap
dynamic_shared_memory_type = sysv
FFFFFD7FF5C00000     145008      10516          -          - rw-s-    [ anon ]
FFFFFD7FFE9C0000         64         36          -          - rwx--    [ anon ]
FFFFFD7FFE9E0000          4          4          -          4 rwxsR    [ ism shmid=0x4300000f ]
FFFFFD7FFE9F0000          4          4          -          4 rwxsR    [ ism shmid=0x4800000e ]

Comment by Former user
Created at 2017-10-06T22:35:29.000Z
Updated at 2017-12-14T17:35:44.245Z

Image bae84d22-aadb-11e7-8c06-171a6ad6b797 looks good .

[root@e79fce53 (postgres) ~]$ ipcs -mA
IPC status from <running system> as of Fri Oct  6 22:33:27 UTC 2017
T         ID      KEY        MODE        OWNER    GROUP  CREATOR   CGROUP NATTCH      SEGSZ  CPID  LPID   ATIME    DTIME    CTIME  ISMATTCH         PROJECT
Shared Memory:
m         53   0x3182344b --rw------- postgres     root postgres     root     14       2316 75561 75904 22:31:33 22:33:07 22:31:33       14          system
m         52   0x52e2c1   --rw------- postgres     root postgres     root     14   43016192 75561 75904 22:31:33 22:33:07 22:31:33       14          system
[root@e79fce53 (postgres) ~]$ pmap -sx 75561|grep ism
FFFFFD7FFB400000      18432      18432          -      18432   2M rwxsR    [ ism shmid=0x34 ]
FFFFFD7FFC600000      23576      23576          -      23576   4K rwxsR    [ ism shmid=0x34 ]
FFFFFD7FFDFB0000          4          4          -          4   4K rwxsR    [ ism shmid=0x35 ]
[root@e79fce53 (postgres) ~]$ cat /manatee/pg/data/postgresql.conf | grep sysv
shared_memory_type = sysv               # the default is the first option
dynamic_shared_memory_type = sysv       # the default is the first option

Comment by Former user
Created at 2017-10-06T22:41:46.000Z

manta-manatee commit 50dc048 (branch master, by Sam Gwydir)

MANATEE-371 Want ISM for SysV Shared Memory in Postgres >= 9.3
Reviewed by: David Pacheco <dap@joyent.com>
Approved by: David Pacheco <dap@joyent.com>


Comment by Former user
Created at 2017-10-10T17:37:17.000Z

engadm commit c45a6d4 (branch master, by Trent Mick)

TOOLS-1884 'make-check' jenkins job broken MANATEE-371: need to 'git submodule sync'