Discussion:
[redis-db] AOF behavior on NFS
rpbear
2018-11-14 13:39:38 UTC
Permalink
hi,
We configured our aof file to persisted on our NFS service which is
developed by ourselves(and this NFS has some file semantics problem).
After used for a very short period,AOF can't write.And this is very easy to
reproduce by these steps(redis version v3.2.11):
[***@10-9-182-230 ~]# redis-cli -h 127.0.0.1
127.0.0.1:6379> SET 1 1
OK
127.0.0.1:6379> BGREWRITEAOF
Background append only file rewriting started
127.0.0.1:6379> SET 2 2
OK
127.0.0.1:6379> BGREWRITEAOF
Background append only file rewriting started
127.0.0.1:6379> SET 3 3
OK
127.0.0.1:6379> BGREWRITEAOF
Background append only file rewriting started
127.0.0.1:6379> SET 4 4
OK
127.0.0.1:6379> BGREWRITEAOF
Background append only file rewriting started
127.0.0.1:6379> SET 5 5
OK
127.0.0.1:6379> BGREWRITEAOF
Background append only file rewriting started
127.0.0.1:6379> SET 6 6
OK
127.0.0.1:6379> BGREWRITEAOF
Background append only file rewriting started
127.0.0.1:6379>
127.0.0.1:6379> BGREWRITEAOF
Background append only file rewriting started
127.0.0.1:6379> SET 7 7
OK
127.0.0.1:6379> SET 8 8
(error) MISCONF Errors writing to the AOF file: Stale file handle
127.0.0.1:6379> SET 9 9
(error) MISCONF Errors writing to the AOF file: Stale file handle
127.0.0.1:6379>

I've read the redis code and the rewrite aof logic has these steps(if I'm
not wrong):

- open a "temp" file to write the aof data,rename this "temp" file to
"bg-temp" after finished
- open the "bg-temp" file and rename the "bg-temp" to
"appendonly.aof",assign the fd of "bg-temp" to server.aof_fd
- subsequent aof data will be write into server.aof_fd

The problem is that,in our NFS service implementation,the filehandle of one
NFS file will changed after a "NON-FIRST TIME" rename,so the server.aof_fd
now refer to a dangling file.
And it should fail when write data into it.However,sometime the aof write
will success,and I checked the server log and found it send the filehandle
of "temp",not filehandle of "bg-temp",
this is a "FIRST TIME RENAME",so this filehandle can been used.Regardless
the semantics problem here,I don't understand why it sends the wrong fd,and
the redis code seems ok.
Anyone can give me some suggestion if this is a nfs client bug or what?
Thanks for advance.
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+***@googlegroups.com.
To post to this group, send email to redis-***@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.
Loading...