Thanks for providing your views on this. In general, I would agree. But in
our usage, we do have a couple unique requirements. One, as I mentioned,
is overall resource management for the services we are deploying. Simply
something we want to allow. We'll be specifying memory usage based on
expected profiles. The second constraint is the data model we're working
with. Right or wrong, there are dependencies between some of the various
in Redis. Allowing some of those dependent entries to be removed by Redis
would cause data inconsistencies. Yes, the application retrieving the data
additional complexities and have performance implications. I suppose we're
using Redis more of a shadow DB, rather than a pure cache. We may revisit
this at a later time, but that's the current implementation.
Post by h***@gmail.com@Chris,
I don't understand why you'd need [server or application logs] I was
comparing your explanation of Redis' behavior with the documentation
description (from the stock redis.conf file). Your description did not
seem to align with the redis documentation.
The initial post asking about the memory consumption didn't mention that
the policy was 'noeviction' (which is, IMO, an uncommon choice when
after the memory limit was exceeded. In fact, my reading of your reply: "and
it should have been returning errors" was that you were declaring that
errors were *not* being returned. (This wasn't actually what you were
saying, but it's the implication I got from the wording)
This triggered my investigation and troubleshooting instincts, and the way
to find out whether Redis was rejecting writes is to check the logfiles.
That's the reason I brought them up.
I think there may also be confusion about my viewpoint on this particular
combination of config options, so I'll explain it. The following is my own
personal viewpoint. I'm not an employee of Redis Labs, so my opinion
doesn't have anything to do with the folks who write the Redis server code
or support it. I'm just another Redis user.
When it comes to managing the memory consumption of a Redis server, a lot
of people have their focus drawn into the nuts-and-bolts details of memory
management, and some overlook the questions of practicality and prudence.
They become involved in How, but forget Who. Who should manage the Redis
server's memory consumption? I see three options for the answer to that
1. *Nobody* - the Redis server's memory grows and shrinks without
on-the-spot management. If it grows large enough for the monitoring system
to alert, a human investigates and decides what action should be taken.
2. *Client/Clients* - an external maintenance script/client watches
the Redis server's memory consumption and, when necessary, takes action to
reduce it, according to business rules. This can be a special client, or
even the regular clients that read/write data.
3. *Redis* - the Redis server watches its own memory consumption
and, when necessary, takes action to reduce it, according to the
generalized policies implemented in the Redis server code.
As I see it, the setting the config parameter 'maxmemory' is telling
Redis, "You are responsible for managing your memory consumption." Setting
the eviction policy to 'noeviction' is telling Redis, "You are responsible
for managing your memory consumption, but I give you *no* power to manage
it with."
Consider a cache. By far the most common use case for cache behavior is
to hold data that has been used recently, because it's expected the data
will be wanted again, soon. And it's faster to get it from cache than from
the original source (often a DB). Managing the memory consumption of a
cache is perfectly normal, but what data should be in the cache? As I just
mentioned, most caches are intended to hold data that was recently used.
In other words, the cache's most valuable data, the data that should have
the best guarantee of benefiting from the cache's speed, is the most recent
data. There is no data that's more recent than the data that's being used
right now. But when Redis is configured with maxmemory, eviction disabled,
Redis rejects the data written now (the most valuable data) and keeps older
data. There can certainly be caches that are intended to hold only
particular data and not the most recently used, but this is a tiny minority
of use cases.
So setting 'maxmemory' and disabling eviction represent conflicting
configurations. If you want Redis to manage its memory, give it the power
to do so. If Redis won't have the power to manage its own memory, don't
burden it with the responsibility. Especially in a typical cache.
That's why this combination of configuration options doesn't make a lot of
sense to me.
@hvarzan,
I may have caused confusion when I jumped into this thread. You're
talking to two different people here.
Post by h***@gmail.comI don't have access to your Redis server logs nor your client
application's logs.
I don't understand why you'd need that information. I was comparing your
explanation of Redis' behavior with the documentation description (from the
stock redis.conf file). Your description did not seem to align with the
redis documentation.
Post by h***@gmail.comWhat is the goal of setting maxmemory without an expiration policy?
Do you have some other client that is supposed to watch the Redis server's
memory consumption and issue delete commands to reduce it?
environment along side many other services on dedicated servers. Due to
everything that we're deploying, we have to manage our resource usage for
every service (memory, disk, cpu). We will be imposing limits on every
service (and of course, base those on actual usage expectations). Similar
will be using redis-exporter with prometheus to monitor memory usage and
alert at certain thresholds. While we'll have very generous amount of ram
available on the servers, it isn't infinite and it isn't dedicated to
redis, so specifying a maxmemory does make sense to be able to
appropriately manage resource consumption by redis along with everything
else running on the servers.
Chris
Post by h***@gmail.comYes, the client gets either a OOM with command cannot be executed/
socket error upon maximum memory is reached.
I have a requirement not to evict anything from the redis memory, since
its a primary cache with persistence enabled.
We have appropriate alerts enabled to watch and report if atleast 75% of
maxmemory is reached.
I am experimenting to understand the redis behaviour if the alerts are
attended late due to one or the other natural hurdles.
Post by h***@gmail.comI don't have access to your Redis server logs nor your client
application's logs. Do your client programs log the error when Redis
rejects a commane they submit?
I'm having trouble understand this configuration. What is the goal of
setting maxmemory without an expiration policy? Do you have some other
client that is supposed to watch the Redis server's memory consumption and
issue delete commands to reduce it?
Post by Chris Stanaway# Set a memory usage limit to the specified amount of bytes.
# When the memory limit is reached Redis will try to remove keys
# according to the eviction policy selected (see maxmemory-policy).
#
# If Redis can't remove keys according to the policy, or if the policy is
# set to 'noeviction', Redis will start to reply with errors to commands
# that would use more memory, like SET, LPUSH, and so on, and will continue
# to reply to read-only commands like GET.
#
# This option is usually useful when using Redis as an LRU or LFU
cache, or to
# set a hard memory limit for an instance (using the 'noeviction' policy).
With maxmemory_policy:noeviction, the documentation describes
maxmemory as a hard limit and it should have been returning errors.
@hvarzan, could you clarify this apparent inconsistency?
Thanks,
Chris
Post by bennet samuelThanks that answers it.
But bit nervous if it expires some old keys to compensate some fast
writes....Since policy is set as no_eviction.
Post by h***@gmail.comThe value of maxmemory_human is just an imprecise (but easier for a
human to understand) rendering of the very precise value in maxmemory.
maxmemory is a goal, not a brick-wall limit. After the memory
consumption exceeds maxmemory, Redis will take actions to reduce the
consumption until it's back below maxmemory again.
used_memory_peak_human simply shows that one of the times that
memory consumption exceeded maxmemory, it reached 2.5GB before Redis could
take action to reduce it. Why would it reach 2.5GB? Perhaps because there
were a fast burst of writes that totalled 500MB (or more), and it took
Redis some time to expire enough older keys to compensate.
Post by bennet samuelPlease find my info memory data below
# Memory
used_memory:2119143648
used_memory_human:1.97G
used_memory_rss:2181586944
used_memory_rss_human:2.03G
used_memory_peak:2778222312
used_memory_peak_human:2.59G
total_system_memory:3974836224
total_system_memory_human:3.70G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:2147483648
maxmemory_human:2.00G
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.03
mem_allocator:jemalloc-4.0.3
From my above configuration, can somebody please help me to
understand, how can used_memory_peak_human be more than the maxmemory_human
set?
My thought was that maxmemory_human is the maximum that the redis
can use. Kindly help me to undertsand this.
Thanks,
Bennet Samuel.
--
You received this message because you are subscribed to the Google
Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at https://groups.google.com/group/redis-db
<https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_group_redis-2Ddb&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=N1Msa0ZexCqrdtWLgqlbMZlNrFKF4YmyP5otZCNJ7mmERCt9HPx5t646YC5_jvy8&m=oIcisDRhOq8O3VJm7ci1UcaiiWs_gUCcsxZfGYg29S0&s=dnRcznYJDEbNFExKuEYgTEjHICAhBie2GY6N1MbdYz0&e=>
.
For more options, visit https://groups.google.com/d/optout
<https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_optout&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=N1Msa0ZexCqrdtWLgqlbMZlNrFKF4YmyP5otZCNJ7mmERCt9HPx5t646YC5_jvy8&m=oIcisDRhOq8O3VJm7ci1UcaiiWs_gUCcsxZfGYg29S0&s=MVnh5DKNgxZbY2jjLkBQkBMOOz7ZNhXi3rl8cB1gde4&e=>
.
--
You received this message because you are subscribed to the Google
Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/redis-db
<https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_group_redis-2Ddb&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=N1Msa0ZexCqrdtWLgqlbMZlNrFKF4YmyP5otZCNJ7mmERCt9HPx5t646YC5_jvy8&m=xEzzV5_TguT_QksQg6G6aGNUiotKGqYb-6eWRUPxAL4&s=LPd_3Ar2eAJEZrHrB812me_rGRvbWEZl4SMTdxkeL8w&e=>
.
For more options, visit https://groups.google.com/d/optout
<https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_optout&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=N1Msa0ZexCqrdtWLgqlbMZlNrFKF4YmyP5otZCNJ7mmERCt9HPx5t646YC5_jvy8&m=xEzzV5_TguT_QksQg6G6aGNUiotKGqYb-6eWRUPxAL4&s=rA-TYtkJrS4qtYnUsbx9Jv96Gtc-AJlZItPdjffiH94&e=>
.
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/redis-db
<https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_group_redis-2Ddb&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=N1Msa0ZexCqrdtWLgqlbMZlNrFKF4YmyP5otZCNJ7mmERCt9HPx5t646YC5_jvy8&m=09bZjmPh2VUGjkXRGBkCI3Augu7SQ803LpopdlnL_a4&s=2MjMrM7Qqpe57f_gXyi7nmTo2HImxMA4A-Nwr8_MBLY&e=>
.
For more options, visit https://groups.google.com/d/optout
<https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_optout&d=DwMFaQ&c=q3cDpHe1hF8lXU5EFjNM_A&r=N1Msa0ZexCqrdtWLgqlbMZlNrFKF4YmyP5otZCNJ7mmERCt9HPx5t646YC5_jvy8&m=09bZjmPh2VUGjkXRGBkCI3Augu7SQ803LpopdlnL_a4&s=-9weKYjAONM33tA6yR8uJMMFvTWyxSxL3Ps2QIR2aec&e=>
.