Orange Pi5 kernel

^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   1) // SPDX-License-Identifier: GPL-2.0
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   2) /*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   3)  * Functions to sequence PREFLUSH and FUA writes.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   4)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   5)  * Copyright (C) 2011		Max Planck Institute for Gravitational Physics
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   6)  * Copyright (C) 2011		Tejun Heo <tj@kernel.org>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   7)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   8)  * REQ_{PREFLUSH|FUA} requests are decomposed to sequences consisted of three
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300   9)  * optional steps - PREFLUSH, DATA and POSTFLUSH - according to the request
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  10)  * properties and hardware capability.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  11)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  12)  * If a request doesn't have data, only REQ_PREFLUSH makes sense, which
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  13)  * indicates a simple flush request.  If there is data, REQ_PREFLUSH indicates
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  14)  * that the device cache should be flushed before the data is executed, and
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  15)  * REQ_FUA means that the data must be on non-volatile media on request
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  16)  * completion.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  17)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  18)  * If the device doesn't have writeback cache, PREFLUSH and FUA don't make any
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  19)  * difference.  The requests are either completed immediately if there's no data
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  20)  * or executed as normal requests otherwise.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  21)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  22)  * If the device has writeback cache and supports FUA, REQ_PREFLUSH is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  23)  * translated to PREFLUSH but REQ_FUA is passed down directly with DATA.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  24)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  25)  * If the device has writeback cache and doesn't support FUA, REQ_PREFLUSH
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  26)  * is translated to PREFLUSH and REQ_FUA to POSTFLUSH.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  27)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  28)  * The actual execution of flush is double buffered.  Whenever a request
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  29)  * needs to execute PRE or POSTFLUSH, it queues at
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  30)  * fq->flush_queue[fq->flush_pending_idx].  Once certain criteria are met, a
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  31)  * REQ_OP_FLUSH is issued and the pending_idx is toggled.  When the flush
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  32)  * completes, all the requests which were pending are proceeded to the next
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  33)  * step.  This allows arbitrary merging of different types of PREFLUSH/FUA
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  34)  * requests.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  35)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  36)  * Currently, the following conditions are used to determine when to issue
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  37)  * flush.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  38)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  39)  * C1. At any given time, only one flush shall be in progress.  This makes
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  40)  *     double buffering sufficient.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  41)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  42)  * C2. Flush is deferred if any request is executing DATA of its sequence.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  43)  *     This avoids issuing separate POSTFLUSHes for requests which shared
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  44)  *     PREFLUSH.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  45)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  46)  * C3. The second condition is ignored if there is a request which has
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  47)  *     waited longer than FLUSH_PENDING_TIMEOUT.  This is to avoid
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  48)  *     starvation in the unlikely case where there are continuous stream of
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  49)  *     FUA (without PREFLUSH) requests.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  50)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  51)  * For devices which support FUA, it isn't clear whether C2 (and thus C3)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  52)  * is beneficial.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  53)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  54)  * Note that a sequenced PREFLUSH/FUA request with DATA is completed twice.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  55)  * Once while executing DATA and again after the whole sequence is
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  56)  * complete.  The first completion updates the contained bio but doesn't
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  57)  * finish it so that the bio submitter is notified only after the whole
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  58)  * sequence is complete.  This is implemented by testing RQF_FLUSH_SEQ in
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  59)  * req_bio_endio().
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  60)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  61)  * The above peculiarity requires that each PREFLUSH/FUA request has only one
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  62)  * bio attached to it, which is guaranteed as they aren't allowed to be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  63)  * merged in the usual way.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  64)  */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  65) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  66) #include <linux/kernel.h>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  67) #include <linux/module.h>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  68) #include <linux/bio.h>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  69) #include <linux/blkdev.h>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  70) #include <linux/gfp.h>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  71) #include <linux/blk-mq.h>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  72) #include <linux/lockdep.h>
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  73) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  74) #include "blk.h"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  75) #include "blk-mq.h"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  76) #include "blk-mq-tag.h"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  77) #include "blk-mq-sched.h"
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  78) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  79) /* PREFLUSH/FUA sequences */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  80) enum {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  81) 	REQ_FSEQ_PREFLUSH	= (1 << 0), /* pre-flushing in progress */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  82) 	REQ_FSEQ_DATA		= (1 << 1), /* data write in progress */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  83) 	REQ_FSEQ_POSTFLUSH	= (1 << 2), /* post-flushing in progress */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  84) 	REQ_FSEQ_DONE		= (1 << 3),
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  85) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  86) 	REQ_FSEQ_ACTIONS	= REQ_FSEQ_PREFLUSH | REQ_FSEQ_DATA |
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  87) 				  REQ_FSEQ_POSTFLUSH,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  88) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  89) 	/*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  90) 	 * If flush has been pending longer than the following timeout,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  91) 	 * it's issued even if flush_data requests are still in flight.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  92) 	 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  93) 	FLUSH_PENDING_TIMEOUT	= 5 * HZ,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  94) };
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  95) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  96) static void blk_kick_flush(struct request_queue *q,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  97) 			   struct blk_flush_queue *fq, unsigned int flags);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  98) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300  99) static unsigned int blk_flush_policy(unsigned long fflags, struct request *rq)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 100) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 101) 	unsigned int policy = 0;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 102) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 103) 	if (blk_rq_sectors(rq))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 104) 		policy |= REQ_FSEQ_DATA;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 105) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 106) 	if (fflags & (1UL << QUEUE_FLAG_WC)) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 107) 		if (rq->cmd_flags & REQ_PREFLUSH)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 108) 			policy |= REQ_FSEQ_PREFLUSH;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 109) 		if (!(fflags & (1UL << QUEUE_FLAG_FUA)) &&
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 110) 		    (rq->cmd_flags & REQ_FUA))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 111) 			policy |= REQ_FSEQ_POSTFLUSH;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 112) 	}
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 113) 	return policy;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 114) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 115) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 116) static unsigned int blk_flush_cur_seq(struct request *rq)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 117) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 118) 	return 1 << ffz(rq->flush.seq);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 119) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 120) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 121) static void blk_flush_restore_request(struct request *rq)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 122) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 123) 	/*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 124) 	 * After flush data completion, @rq->bio is %NULL but we need to
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 125) 	 * complete the bio again.  @rq->biotail is guaranteed to equal the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 126) 	 * original @rq->bio.  Restore it.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 127) 	 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 128) 	rq->bio = rq->biotail;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 129) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 130) 	/* make @rq a normal request */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 131) 	rq->rq_flags &= ~RQF_FLUSH_SEQ;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 132) 	rq->end_io = rq->flush.saved_end_io;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 133) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 134) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 135) static void blk_flush_queue_rq(struct request *rq, bool add_front)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 136) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 137) 	blk_mq_add_to_requeue_list(rq, add_front, true);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 138) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 139) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 140) static void blk_account_io_flush(struct request *rq)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 141) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 142) 	struct hd_struct *part = &rq->rq_disk->part0;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 143) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 144) 	part_stat_lock();
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 145) 	part_stat_inc(part, ios[STAT_FLUSH]);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 146) 	part_stat_add(part, nsecs[STAT_FLUSH],
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 147) 		      ktime_get_ns() - rq->start_time_ns);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 148) 	part_stat_unlock();
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 149) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 150) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 151) /**
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 152)  * blk_flush_complete_seq - complete flush sequence
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 153)  * @rq: PREFLUSH/FUA request being sequenced
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 154)  * @fq: flush queue
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 155)  * @seq: sequences to complete (mask of %REQ_FSEQ_*, can be zero)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 156)  * @error: whether an error occurred
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 157)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 158)  * @rq just completed @seq part of its flush sequence, record the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 159)  * completion and trigger the next step.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 160)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 161)  * CONTEXT:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 162)  * spin_lock_irq(fq->mq_flush_lock)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 163)  */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 164) static void blk_flush_complete_seq(struct request *rq,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 165) 				   struct blk_flush_queue *fq,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 166) 				   unsigned int seq, blk_status_t error)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 167) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 168) 	struct request_queue *q = rq->q;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 169) 	struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx];
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 170) 	unsigned int cmd_flags;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 171) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 172) 	BUG_ON(rq->flush.seq & seq);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 173) 	rq->flush.seq |= seq;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 174) 	cmd_flags = rq->cmd_flags;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 175) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 176) 	if (likely(!error))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 177) 		seq = blk_flush_cur_seq(rq);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 178) 	else
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 179) 		seq = REQ_FSEQ_DONE;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 180) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 181) 	switch (seq) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 182) 	case REQ_FSEQ_PREFLUSH:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 183) 	case REQ_FSEQ_POSTFLUSH:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 184) 		/* queue for flush */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 185) 		if (list_empty(pending))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 186) 			fq->flush_pending_since = jiffies;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 187) 		list_move_tail(&rq->flush.list, pending);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 188) 		break;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 189) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 190) 	case REQ_FSEQ_DATA:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 191) 		list_move_tail(&rq->flush.list, &fq->flush_data_in_flight);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 192) 		blk_flush_queue_rq(rq, true);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 193) 		break;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 194) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 195) 	case REQ_FSEQ_DONE:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 196) 		/*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 197) 		 * @rq was previously adjusted by blk_insert_flush() for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 198) 		 * flush sequencing and may already have gone through the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 199) 		 * flush data request completion path.  Restore @rq for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 200) 		 * normal completion and end it.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 201) 		 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 202) 		BUG_ON(!list_empty(&rq->queuelist));
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 203) 		list_del_init(&rq->flush.list);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 204) 		blk_flush_restore_request(rq);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 205) 		blk_mq_end_request(rq, error);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 206) 		break;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 207) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 208) 	default:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 209) 		BUG();
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 210) 	}
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 211) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 212) 	blk_kick_flush(q, fq, cmd_flags);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 213) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 214) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 215) static void flush_end_io(struct request *flush_rq, blk_status_t error)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 216) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 217) 	struct request_queue *q = flush_rq->q;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 218) 	struct list_head *running;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 219) 	struct request *rq, *n;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 220) 	unsigned long flags = 0;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 221) 	struct blk_flush_queue *fq = blk_get_flush_queue(q, flush_rq->mq_ctx);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 222) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 223) 	/* release the tag's ownership to the req cloned from */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 224) 	spin_lock_irqsave(&fq->mq_flush_lock, flags);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 225) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 226) 	if (!refcount_dec_and_test(&flush_rq->ref)) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 227) 		fq->rq_status = error;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 228) 		spin_unlock_irqrestore(&fq->mq_flush_lock, flags);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 229) 		return;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 230) 	}
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 231) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 232) 	blk_account_io_flush(flush_rq);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 233) 	/*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 234) 	 * Flush request has to be marked as IDLE when it is really ended
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 235) 	 * because its .end_io() is called from timeout code path too for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 236) 	 * avoiding use-after-free.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 237) 	 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 238) 	WRITE_ONCE(flush_rq->state, MQ_RQ_IDLE);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 239) 	if (fq->rq_status != BLK_STS_OK) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 240) 		error = fq->rq_status;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 241) 		fq->rq_status = BLK_STS_OK;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 242) 	}
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 243) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 244) 	if (!q->elevator) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 245) 		flush_rq->tag = BLK_MQ_NO_TAG;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 246) 	} else {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 247) 		blk_mq_put_driver_tag(flush_rq);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 248) 		flush_rq->internal_tag = BLK_MQ_NO_TAG;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 249) 	}
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 250) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 251) 	running = &fq->flush_queue[fq->flush_running_idx];
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 252) 	BUG_ON(fq->flush_pending_idx == fq->flush_running_idx);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 253) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 254) 	/* account completion of the flush request */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 255) 	fq->flush_running_idx ^= 1;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 256) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 257) 	/* and push the waiting requests to the next stage */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 258) 	list_for_each_entry_safe(rq, n, running, flush.list) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 259) 		unsigned int seq = blk_flush_cur_seq(rq);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 260) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 261) 		BUG_ON(seq != REQ_FSEQ_PREFLUSH && seq != REQ_FSEQ_POSTFLUSH);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 262) 		blk_flush_complete_seq(rq, fq, seq, error);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 263) 	}
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 264) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 265) 	spin_unlock_irqrestore(&fq->mq_flush_lock, flags);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 266) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 267) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 268) bool is_flush_rq(struct request *rq)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 269) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 270) 	return rq->end_io == flush_end_io;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 271) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 272) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 273) /**
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 274)  * blk_kick_flush - consider issuing flush request
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 275)  * @q: request_queue being kicked
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 276)  * @fq: flush queue
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 277)  * @flags: cmd_flags of the original request
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 278)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 279)  * Flush related states of @q have changed, consider issuing flush request.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 280)  * Please read the comment at the top of this file for more info.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 281)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 282)  * CONTEXT:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 283)  * spin_lock_irq(fq->mq_flush_lock)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 284)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 285)  */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 286) static void blk_kick_flush(struct request_queue *q, struct blk_flush_queue *fq,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 287) 			   unsigned int flags)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 288) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 289) 	struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx];
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 290) 	struct request *first_rq =
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 291) 		list_first_entry(pending, struct request, flush.list);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 292) 	struct request *flush_rq = fq->flush_rq;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 293) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 294) 	/* C1 described at the top of this file */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 295) 	if (fq->flush_pending_idx != fq->flush_running_idx || list_empty(pending))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 296) 		return;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 297) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 298) 	/* C2 and C3 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 299) 	if (!list_empty(&fq->flush_data_in_flight) &&
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 300) 	    time_before(jiffies,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 301) 			fq->flush_pending_since + FLUSH_PENDING_TIMEOUT))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 302) 		return;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 303) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 304) 	/*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 305) 	 * Issue flush and toggle pending_idx.  This makes pending_idx
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 306) 	 * different from running_idx, which means flush is in flight.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 307) 	 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 308) 	fq->flush_pending_idx ^= 1;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 309) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 310) 	blk_rq_init(q, flush_rq);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 311) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 312) 	/*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 313) 	 * In case of none scheduler, borrow tag from the first request
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 314) 	 * since they can't be in flight at the same time. And acquire
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 315) 	 * the tag's ownership for flush req.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 316) 	 *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 317) 	 * In case of IO scheduler, flush rq need to borrow scheduler tag
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 318) 	 * just for cheating put/get driver tag.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 319) 	 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 320) 	flush_rq->mq_ctx = first_rq->mq_ctx;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 321) 	flush_rq->mq_hctx = first_rq->mq_hctx;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 322) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 323) 	if (!q->elevator) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 324) 		flush_rq->tag = first_rq->tag;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 325) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 326) 		/*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 327) 		 * We borrow data request's driver tag, so have to mark
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 328) 		 * this flush request as INFLIGHT for avoiding double
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 329) 		 * account of this driver tag
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 330) 		 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 331) 		flush_rq->rq_flags |= RQF_MQ_INFLIGHT;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 332) 	} else
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 333) 		flush_rq->internal_tag = first_rq->internal_tag;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 334) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 335) 	flush_rq->cmd_flags = REQ_OP_FLUSH | REQ_PREFLUSH;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 336) 	flush_rq->cmd_flags |= (flags & REQ_DRV) | (flags & REQ_FAILFAST_MASK);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 337) 	flush_rq->rq_flags |= RQF_FLUSH_SEQ;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 338) 	flush_rq->rq_disk = first_rq->rq_disk;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 339) 	flush_rq->end_io = flush_end_io;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 340) 	/*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 341) 	 * Order WRITE ->end_io and WRITE rq->ref, and its pair is the one
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 342) 	 * implied in refcount_inc_not_zero() called from
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 343) 	 * blk_mq_find_and_get_req(), which orders WRITE/READ flush_rq->ref
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 344) 	 * and READ flush_rq->end_io
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 345) 	 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 346) 	smp_wmb();
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 347) 	refcount_set(&flush_rq->ref, 1);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 348) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 349) 	blk_flush_queue_rq(flush_rq, false);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 350) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 351) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 352) static void mq_flush_data_end_io(struct request *rq, blk_status_t error)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 353) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 354) 	struct request_queue *q = rq->q;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 355) 	struct blk_mq_hw_ctx *hctx = rq->mq_hctx;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 356) 	struct blk_mq_ctx *ctx = rq->mq_ctx;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 357) 	unsigned long flags;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 358) 	struct blk_flush_queue *fq = blk_get_flush_queue(q, ctx);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 359) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 360) 	if (q->elevator) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 361) 		WARN_ON(rq->tag < 0);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 362) 		blk_mq_put_driver_tag(rq);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 363) 	}
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 364) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 365) 	/*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 366) 	 * After populating an empty queue, kick it to avoid stall.  Read
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 367) 	 * the comment in flush_end_io().
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 368) 	 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 369) 	spin_lock_irqsave(&fq->mq_flush_lock, flags);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 370) 	blk_flush_complete_seq(rq, fq, REQ_FSEQ_DATA, error);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 371) 	spin_unlock_irqrestore(&fq->mq_flush_lock, flags);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 372) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 373) 	blk_mq_sched_restart(hctx);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 374) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 375) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 376) /**
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 377)  * blk_insert_flush - insert a new PREFLUSH/FUA request
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 378)  * @rq: request to insert
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 379)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 380)  * To be called from __elv_add_request() for %ELEVATOR_INSERT_FLUSH insertions.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 381)  * or __blk_mq_run_hw_queue() to dispatch request.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 382)  * @rq is being submitted.  Analyze what needs to be done and put it on the
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 383)  * right queue.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 384)  */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 385) void blk_insert_flush(struct request *rq)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 386) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 387) 	struct request_queue *q = rq->q;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 388) 	unsigned long fflags = q->queue_flags;	/* may change, cache */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 389) 	unsigned int policy = blk_flush_policy(fflags, rq);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 390) 	struct blk_flush_queue *fq = blk_get_flush_queue(q, rq->mq_ctx);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 391) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 392) 	/*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 393) 	 * @policy now records what operations need to be done.  Adjust
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 394) 	 * REQ_PREFLUSH and FUA for the driver.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 395) 	 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 396) 	rq->cmd_flags &= ~REQ_PREFLUSH;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 397) 	if (!(fflags & (1UL << QUEUE_FLAG_FUA)))
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 398) 		rq->cmd_flags &= ~REQ_FUA;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 399) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 400) 	/*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 401) 	 * REQ_PREFLUSH|REQ_FUA implies REQ_SYNC, so if we clear any
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 402) 	 * of those flags, we have to set REQ_SYNC to avoid skewing
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 403) 	 * the request accounting.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 404) 	 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 405) 	rq->cmd_flags |= REQ_SYNC;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 406) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 407) 	/*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 408) 	 * An empty flush handed down from a stacking driver may
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 409) 	 * translate into nothing if the underlying device does not
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 410) 	 * advertise a write-back cache.  In this case, simply
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 411) 	 * complete the request.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 412) 	 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 413) 	if (!policy) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 414) 		blk_mq_end_request(rq, 0);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 415) 		return;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 416) 	}
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 417) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 418) 	BUG_ON(rq->bio != rq->biotail); /*assumes zero or single bio rq */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 419) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 420) 	/*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 421) 	 * If there's data but flush is not necessary, the request can be
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 422) 	 * processed directly without going through flush machinery.  Queue
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 423) 	 * for normal execution.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 424) 	 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 425) 	if ((policy & REQ_FSEQ_DATA) &&
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 426) 	    !(policy & (REQ_FSEQ_PREFLUSH | REQ_FSEQ_POSTFLUSH))) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 427) 		blk_mq_request_bypass_insert(rq, false, false);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 428) 		return;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 429) 	}
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 430) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 431) 	/*
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 432) 	 * @rq should go through flush machinery.  Mark it part of flush
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 433) 	 * sequence and submit for further processing.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 434) 	 */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 435) 	memset(&rq->flush, 0, sizeof(rq->flush));
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 436) 	INIT_LIST_HEAD(&rq->flush.list);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 437) 	rq->rq_flags |= RQF_FLUSH_SEQ;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 438) 	rq->flush.saved_end_io = rq->end_io; /* Usually NULL */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 439) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 440) 	rq->end_io = mq_flush_data_end_io;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 441) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 442) 	spin_lock_irq(&fq->mq_flush_lock);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 443) 	blk_flush_complete_seq(rq, fq, REQ_FSEQ_ACTIONS & ~policy, 0);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 444) 	spin_unlock_irq(&fq->mq_flush_lock);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 445) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 446) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 447) /**
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 448)  * blkdev_issue_flush - queue a flush
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 449)  * @bdev:	blockdev to issue flush for
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 450)  * @gfp_mask:	memory allocation flags (for bio_alloc)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 451)  *
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 452)  * Description:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 453)  *    Issue a flush for the block device in question.
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 454)  */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 455) int blkdev_issue_flush(struct block_device *bdev, gfp_t gfp_mask)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 456) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 457) 	struct bio *bio;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 458) 	int ret = 0;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 459) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 460) 	bio = bio_alloc(gfp_mask, 0);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 461) 	bio_set_dev(bio, bdev);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 462) 	bio->bi_opf = REQ_OP_WRITE | REQ_PREFLUSH;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 463) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 464) 	ret = submit_bio_wait(bio);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 465) 	bio_put(bio);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 466) 	return ret;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 467) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 468) EXPORT_SYMBOL(blkdev_issue_flush);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 469) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 470) struct blk_flush_queue *blk_alloc_flush_queue(int node, int cmd_size,
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 471) 					      gfp_t flags)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 472) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 473) 	struct blk_flush_queue *fq;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 474) 	int rq_sz = sizeof(struct request);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 475) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 476) 	fq = kzalloc_node(sizeof(*fq), flags, node);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 477) 	if (!fq)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 478) 		goto fail;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 479) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 480) 	spin_lock_init(&fq->mq_flush_lock);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 481) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 482) 	rq_sz = round_up(rq_sz + cmd_size, cache_line_size());
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 483) 	fq->flush_rq = kzalloc_node(rq_sz, flags, node);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 484) 	if (!fq->flush_rq)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 485) 		goto fail_rq;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 486) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 487) 	INIT_LIST_HEAD(&fq->flush_queue[0]);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 488) 	INIT_LIST_HEAD(&fq->flush_queue[1]);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 489) 	INIT_LIST_HEAD(&fq->flush_data_in_flight);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 490) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 491) 	lockdep_register_key(&fq->key);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 492) 	lockdep_set_class(&fq->mq_flush_lock, &fq->key);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 493) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 494) 	return fq;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 495) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 496)  fail_rq:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 497) 	kfree(fq);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 498)  fail:
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 499) 	return NULL;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 500) }
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 501) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 502) void blk_free_flush_queue(struct blk_flush_queue *fq)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 503) {
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 504) 	/* bio based request queue hasn't flush queue */
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 505) 	if (!fq)
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 506) 		return;
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 507) 
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 508) 	lockdep_unregister_key(&fq->key);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 509) 	kfree(fq->flush_rq);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 510) 	kfree(fq);
^8f3ce5b39 (kx 2023-10-28 12:00:06 +0300 511) }