Redis Sentinel Reset
I want to expand on the answer above by providing status meanings in a distributed setup # get sentinel / redis instance role of remote host: › redis-cli -h redis-02.aslave -p 6379 info grep role role:slave # get sentinel status of remote host: › redis-cli -h redis-01.asentinel -p 26379 info grep status master0:name=mymaster,status=ok,address=172.xx.xxx.xx:6379,slaves=2,sentinels=3. Sentinel role Sentinel is a distributed system, which guarantees the high availability of programs. It is used to monitor any number of master servers and all slave servers under these master servers. In case of failure, it selects a new master through voting mechanism and connects all slaves to the new master. Monitor Continuously check. 目前 Sentinel 系统是 Redis 的 unstable 分支的一部分, 你必须到 Redis 项目的 Github 页面 克隆一份 unstable 分值, 然后通过编译来获得 Sentinel 系统。. Sentinel 程序可以在编译后的 src 文档中发现, 它是一个命名为 redis-sentinel 的程序。. 你也可以通过下一节介绍的方法, 让 redis-server 程序.
- Troubleshooting
Using an alternate local Redis instance
Omnibus GitLab includes Redis by default. To direct the GitLabapplication to your own locally running Redis instance:
Edit
/etc/gitlab/gitlab.rb
:Reconfigure GitLab for the changes to take effect:
Making the bundled Redis reachable via TCP
Use the following settings if you want to make the Redis instancemanaged by Omnibus GitLab reachable via TCP:
Edit
/etc/gitlab/gitlab.rb
:Save the file and reconfigure GitLab for the changes to take effect:
Setting up a Redis-only server using Omnibus GitLab
If you’d like to set up Redis in a separate server than the GitLab application,you can use thebundled Redis in Omnibus GitLab.
Running with multiple Redis instances
See https://docs.gitlab.com/ee/administration/redis/replication_and_failover.html#running-multiple-redis-clusters.
Redis Sentinel
See https://docs.gitlab.com/ee/administration/redis/replication_and_failover.html.
Using Redis in a failover setup
See https://docs.gitlab.com/ee/administration/redis/replication_and_failover.html.
Using Google Cloud Memorystore
Google Cloud Memorystore does not support the Redis CLIENT
command.By default Sidekiq will attempt to set the CLIENT
for debuggingpurposes. This can be disabled via this configuration setting:
Increasing the number of Redis connections beyond the default
By default Redis will only accept 10,000 client connections. If you needmore that 10,000 connections set the maxclients
attribute to suit your needs.Be advised that adjusting the maxclients
attribute means that you will also needto take into account your systems settings for fs.file-max
(for example sysctl -w fs.file-max=20000
)
Tuning the TCP stack for Redis
The following settings are to enable a more performant Redis server instance. tcp_timeout
isa value set in seconds that the Redis server waits before terminating an idle TCP connection.The tcp_keepalive
is a tunable setting in seconds to TCP ACKs to clients in absence ofcommunication.
Setting the Redis Cache instance as an LRU
Using multiple Redis instances allows you to configure Redis as a LeastRecently Used cache. Note you should onlydo this for the Redis cache instance; the Redis queues and shared state instancesshould never be configured as an LRU, since they contain data (e.g. Sidekiqjobs) that is expected to be persistent.
To cap memory usage at 32GB, you can use:
Using Secure Sockets Layer (SSL)
Redis 5.x does NOT support SSL out of the box. However, you can encrypt aRedis connection using stunnel.AWS ElasticCache also supports Redis over SSL.
Redis Sentinel Config
Support for SSL has the following limitations:
- Omnibus GitLab doesn’t include
stunnel
or other tools to provide encryptionfor the Redis server. However, GitLab does provide client support by usingtherediss://
(as opposed toredis://
) URL scheme. - Omnibus GitLab bundles Redis Sentinel 5.0.x which does NOT support SSL.If you use Redis Sentinel, do not activate client support for SSL.Redis 6 supports SSL, and you canconfigure it to work with GitLab only as anexternal service.
To activate GitLab client support for SSL:
Add the following line to
/etc/gitlab/gitlab.rb
:Reconfigure GitLab for the changes to take effect:
SSL certificates
If you’re using custom SSL certificates for Redis, be sure to add themto the trusted certificates.
Renamed commands
By default, the KEYS
command is disabled as a security measure.
If you’d like to obfuscate or disable this command, or other commands, edit the redis['rename_commands']
setting in /etc/gitlab/gitlab.rb
to look like:
OTHER_COMMAND
is the command you want to modifyVALUE
should be one of:- A new command name.
'
, which completely disables the command.
To disable this functionality:
- Set
redis['rename_commands'] = {}
in your/etc/gitlab/gitlab.rb
file - Run
sudo gitlab-ctl reconfigure
Lazy freeing
Redis 4 introduced lazy freeing. This can improve performance when freeing large values.
This setting defaults to false
. To enable it, you can use:
Troubleshooting
x509: certificate signed by unknown authority
This error message suggests that the SSL certificates have not beenproperly added to the list of trusted certificates for the server. Tocheck whether this is an issue:
Check Workhorse logs in
/var/log/gitlab/gitlab-workhorse/current
.If you see messages that look like:
The first line should show
rediss
as the scheme with the addressof the Redis server. The second line indicates the certificate isnot properly trusted on this server. See the previous section.Verify that the SSL certificate is working via these troubleshootingsteps.
NOAUTH Authentication required
Redis Sentinel Commands
A Redis server may require a password sent via an AUTH
message beforecommands are accepted. A NOAUTH Authentication required
error messagesuggests the client is not sending a password. GitLab logs may helptroubleshoot this error:
Python Redis Sentinel
Check Workhorse logs in
/var/log/gitlab/gitlab-workhorse/current
.If you see messages that look like:
Check that the Redis client password specified in
/etc/gitlab/gitlab.rb
is correct:If you are using the Omnibus-provided Redis server, check that the server has the same password:
Redis connection reset (ECONNRESET)
If you see Redis::ConnectionError: Connection lost (ECONNRESET)
in theGitLab Rails logs (/var/log/gitlab-rails/production.log
), this mightindicate that the server is expecting SSL but the client is notconfigured to use it.
Check that the server is actually listening to the port via SSL.For example:
Check
/var/opt/gitlab/gitlab-rails/etc/resque.yml
. Youshould see something like:If
redis://
is present instead ofrediss://
, theredis_ssl
parameter may not have been configured properly, or the reconfigurestep may not have been run.
Connecting to Redis via the CLI
When connecting to Redis for troubleshooting you can use:
Redis via Unix domain sockets:
Redis via TCP:
Password to authenticate to Redis if required:
Help & feedback
Docs
Edit this page to fix an error or add an improvement in a merge requestCreate an issue to suggest an improvement to this doc
Product
Create an issue if there's something you don't like about this featurePropose functionality by submitting a feature request
Join beta testing to help shape new features
Feature availability and product trials
View pricing to see all GitLab tiers and features, or to upgradeGitLab Gold Free-trial to try all features in GitLab.com
GitLab Ultimate Free-trial to try all features in GitLab self-managed
Get Help
If you didn't find what you were looking for, search the docs.
If you want help with something specific, and could use community support, post on the GitLab forum.
For problems setting up or using this feature (depending on your GitLab subscription).