There’s a very important linux setting to change when you set up a redis server on an instance: vm.overcommit = 1.
If you haven’t set it up yet, you should do it now. Without it you’re essentially limited to half the RAM available on your machine, after that redis will be unable to create snapshots.
The high level steps of redis snapshots is this:
- Redis forks its process
- The parent continues on to accept requests
- The child converts the working memory into the rdb format and saves it to disk
- The old rdb file is removed and replaced by the new one
Forking is a UNIX system call that makes a replica of the process that calls it. This means that the child will have a copy of all of the memory that the parent had.
This means that when your 10GB redis instance forks, you will now have 2 processes with each 10GB of data.
That sounds ridiculous right?
On a 20GB RAM machine, you would need to keep 10GB free just for the snapshot process. It’s incredibly wasteful.
Fortunately, there’s something brilliant called Copy on write (CoW) which solves this issue.
This allows processes created by forking to use the parent process’ memory until a process updates that memory page. If the parent (or child) writes to its memory, it will create a new memory page and write there instead, but the child process will still reference the old memory page.
That’s cool and all, but why do you need vm.overcommit = 1?
Overcomitting is allowing a process to allocate more memory than is physically available on the system. You’re betting on never really having to use it.
0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this mode. This is the default. 1 - Always overcommit. Appropriate for some scientific applications. Classic example is code using sparse arrays and just relying on the virtual memory consisting almost entirely of zero pages. 2 - Don't overcommit. The total address space commit for the system is not permitted to exceed swap + a configurable amount (default is 50%) of physical RAM. Depending on the amount you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate. Useful for applications that want to guarantee their memory allocations will be available in the future without having to initialize every page.
You typically want to go for 0 or 2. Allowing a program to allocate more memory than is available will result in a sudden crash if it tries to actually use memory that’s not available. In most cases, this is a very bad thing. Especially when you’re dealing with a database.
Why 1 then?
Since the redis snapshot process is only used to dump memory to disk, it will not change any of its parent memory and it won’t last long enough for the parent to change a significant amount of values. Unless you’re very close to your physical limit, the case where there are enough changes done by the parent to go above the physical memory limit is very unlikely. This makes choosing always overcommit a very reasonable option for the snapshot fork.