[redis-db] Streams questions

[Andrew Eisenberg]

Hi all,

Streams in redis 5.0 look fantastic. They seem to be solving many of the
problems that we are having. I've gone through Introduction to Streams
<https://getpocket.com/redirect?url=https%3A%2F%2Fredis.io%2Ftopics%2Fstreams-intro&formCheck=9373f720f6fbc17121f2fd829537f941>
and I still have a few questions:


1. It seems that streams are the only data type that is not removed when
it has no more elements (because of the state related to data groups). Is
there any way of forcing the deletion of a stream?
2. The way we are planning on using streams, we will have groups of
values that will need to be evicted at once. They won't necessarily be
sequential, but we will know which group each value belongs to when we
insert it and we are planning on storing the ids for these values in sets
so we can keep track of them. My question is: will this destroy the memory
footprint of the stream where we periodically delete groups of maybe 1000
keys spread throughout the stream? And is there something we can do to
mitigate the issue?
3. Most of our values are currently stored as JSON strings (even though
they are mostly key-value pairs). Since streams allow key-value pairs
natively, would we be better off avoiding JSON and using native key-value
pairs? Each of our values will have roughly 20 keys.

thanks for your help,
Andrew

[Salvatore Sanfilippo]

Hello Andrew,

1. We may add an option to XDEL to remove the stream if after the
deletion: a) The stream is empty. b) It has no consumer groups
associated.
2. Yes deleting does not use more memory but does not reclaim memory
unless the whole macro-node becomes empty, so if you end having a very
large percentage of the past entires deleted, your memory usage is not
going to be brilliant. If it's like 20% of deleted items you may not
care, if it's 60% or 80% you have to account for an higher memory
usage for sure. In the future it is likely I'll implement garbage
collection of nodes, it was too much complexity to add right now in
V1.
3. You'll save definitely memory using the native stream
representation, because if the fields are always the same the stream
will compress the next items using just a single bit to say "same
fields". Moreover you get the reply already parsed for you as an hash,
instead of having to decode the JSON received.

Cheers,
Salvatore
On Sun, Oct 21, 2018 at 2:33 AM Andrew Eisenberg

Post by Andrew Eisenberg
Hi all,
It seems that streams are the only data type that is not removed when it has no more elements (because of the state related to data groups). Is there any way of forcing the deletion of a stream?
The way we are planning on using streams, we will have groups of values that will need to be evicted at once. They won't necessarily be sequential, but we will know which group each value belongs to when we insert it and we are planning on storing the ids for these values in sets so we can keep track of them. My question is: will this destroy the memory footprint of the stream where we periodically delete groups of maybe 1000 keys spread throughout the stream? And is there something we can do to mitigate the issue?
Most of our values are currently stored as JSON strings (even though they are mostly key-value pairs). Since streams allow key-value pairs natively, would we be better off avoiding JSON and using native key-value pairs? Each of our values will have roughly 20 keys.
thanks for your help,
Andrew
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

--
Salvatore 'antirez' Sanfilippo
open source developer - Redis Labs https://redislabs.com

"If a system is to have conceptual integrity, someone must control the
concepts."
— Fred Brooks, "The Mythical Man-Month", 1975.

[Andrew Eisenberg]

Thanks. That really clears things up.

[Nick Farrell]

If the plan is to delete subsets of the data at a time, perhaps streams is not the right choice. At least, not on its own.

Streams are ideal for a resilient, resumable pub/sub pattern. If instead you want to partition incoming data in some way, then atomically process it and remove it, there are more appropriate data types. Hard to give a more specific suggestion without knowing your specific use case.