Seandon Mooy
2018-06-26 22:30:11 UTC
Hello!
I'm working with the new XADD/XREAD commands, and am trying to devise a
client library for working with the streams feature. XREAD is a blocking
call, but accepts multiple streams to read per XREAD call. I certainly
don't want to design a system where, for example, an API to make use of
redis streams to provide real-time streaming logs, per-user, I would need
one connection to redis per user (ie: one connection for one stream). I
would like to "pool" the subscriptions and read from multiple streams - but
this implies that I must wait the full BLOCK time to add a subscription.
Put another way, if I limit myself to one-redis-connection-per-application,
how do I effectively read from more than one stream, on demand.
I have built a very crude pool system, where I allow up to N number of
redis connections, adding new XREAD calls as requests for subscriptions
come in. When an XREAD returns, I check to see if another XREAD is handling
this connection's streams, and if not I issue a new one for all desired
streams - ie: trying hard to eventually wind up with 1 XREAD. With a
"BLOCK" of one second, this effectively limits response time after _N
requests per second_! This is a scaling issue for me and feels like a poor
design on my part.
Does anyone have any ideas on this? I've been looking for a signal I can
send down the connection that will "cancel" an XREAD call, but all I can
manage to do is quit the connection entirely :P
Thanks for reading!
P.S. huge redis fan / user - I gave a talk at RedisConf17 and would love to
do another some day!
- Seandon
I'm working with the new XADD/XREAD commands, and am trying to devise a
client library for working with the streams feature. XREAD is a blocking
call, but accepts multiple streams to read per XREAD call. I certainly
don't want to design a system where, for example, an API to make use of
redis streams to provide real-time streaming logs, per-user, I would need
one connection to redis per user (ie: one connection for one stream). I
would like to "pool" the subscriptions and read from multiple streams - but
this implies that I must wait the full BLOCK time to add a subscription.
Put another way, if I limit myself to one-redis-connection-per-application,
how do I effectively read from more than one stream, on demand.
I have built a very crude pool system, where I allow up to N number of
redis connections, adding new XREAD calls as requests for subscriptions
come in. When an XREAD returns, I check to see if another XREAD is handling
this connection's streams, and if not I issue a new one for all desired
streams - ie: trying hard to eventually wind up with 1 XREAD. With a
"BLOCK" of one second, this effectively limits response time after _N
requests per second_! This is a scaling issue for me and feels like a poor
design on my part.
Does anyone have any ideas on this? I've been looking for a signal I can
send down the connection that will "cancel" an XREAD call, but all I can
manage to do is quit the connection entirely :P
Thanks for reading!
P.S. huge redis fan / user - I gave a talk at RedisConf17 and would love to
do another some day!
- Seandon
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+***@googlegroups.com.
To post to this group, send email to redis-***@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+***@googlegroups.com.
To post to this group, send email to redis-***@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.