1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
|
From patchwork Mon Feb 10 09:52:20 2020
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
X-Patchwork-Submitter: Lukas Wunner <lukas@wunner.de>
X-Patchwork-Id: 11372935
Return-Path:
<SRS0=uiPw=36=lists.infradead.org=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@kernel.org>
Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org
[172.30.200.123])
by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6E6F4109A
for <patchwork-linux-arm@patchwork.kernel.org>;
Mon, 10 Feb 2020 09:52:52 +0000 (UTC)
Received: from bombadil.infradead.org (bombadil.infradead.org
[198.137.202.133])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(No client certificate requested)
by mail.kernel.org (Postfix) with ESMTPS id 3ABD621835
for <patchwork-linux-arm@patchwork.kernel.org>;
Mon, 10 Feb 2020 09:52:52 +0000 (UTC)
Authentication-Results: mail.kernel.org;
dkim=pass (2048-bit key) header.d=lists.infradead.org
header.i=@lists.infradead.org header.b="sLV1I6RP"
DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3ABD621835
Authentication-Results: mail.kernel.org;
dmarc=none (p=none dis=none) header.from=wunner.de
Authentication-Results: mail.kernel.org;
spf=none
smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=lists.infradead.org; s=bombadil.20170209; h=Sender:
Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe:
List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:To:Subject:Date:
From:References:In-Reply-To:Message-Id:Reply-To:Content-ID:
Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc
:Resent-Message-ID:List-Owner;
bh=52m4zqqDQ15r/4EHqLM3tCdaghuCQFRSRI636FPzyTg=; b=sLV1I6RPSfIAq0DmE3ADnTyGjh
NnOUSjVDCUDlMsaRe7dhAfC7D3enDiNCrnGXNT3kb6+57+bXrj31EU+9NhIevN+Rr+lrXe+8PaWKw
Q2LbzZDJ3owcXTu2EZAYrQPHg/BScQT5NiMMHrLmISp9B3erAO45imXsyolyfBhUFbvxPY/ijVw2C
rPXYS/w47IaC6T79wZVTZ1Iuqn0jR/qvksA7yRUBMTsnHcc3fv6yuD+cD0BFmmao7LAxKlA3uu8Ba
MXB0/7aQIlb5yOssEfnwThbbnbrpxmnIS0CT00K1yAYIkzDAv7lS7imZ5qTy2uahyJFWjQF8T++wX
TgZL/6mQ==;
Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org)
by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux))
id 1j15kG-0008ES-UU; Mon, 10 Feb 2020 09:52:48 +0000
Received: from mailout1.hostsharing.net ([83.223.95.204])
by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux))
id 1j15kC-0008Cu-Hl; Mon, 10 Feb 2020 09:52:46 +0000
Received: from h08.hostsharing.net (h08.hostsharing.net
[IPv6:2a01:37:1000::53df:5f1c:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "*.hostsharing.net",
Issuer "COMODO RSA Domain Validation Secure Server CA" (not verified))
by mailout1.hostsharing.net (Postfix) with ESMTPS id 32EE510192623;
Mon, 10 Feb 2020 10:52:31 +0100 (CET)
Received: from localhost (unknown [87.130.102.138])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits))
(No client certificate requested)
by h08.hostsharing.net (Postfix) with ESMTPSA id E85DF60AD5E1;
Mon, 10 Feb 2020 10:52:30 +0100 (CET)
X-Mailbox-Line: From 8be2f3e95fb29abdf80240f2b8a38621c42eb2a9 Mon Sep 17
00:00:00 2001
Message-Id:
<8be2f3e95fb29abdf80240f2b8a38621c42eb2a9.1581327911.git.lukas@wunner.de>
In-Reply-To: <713627a200d9c8fd7cac424d69e98166@kernel.org>
References: <713627a200d9c8fd7cac424d69e98166@kernel.org>
From: Lukas Wunner <lukas@wunner.de>
Date: Mon, 10 Feb 2020 10:52:20 +0100
Subject: [PATCH v2] irqchip/bcm2835: Quiesce IRQs left enabled by bootloader
To: Thomas Gleixner <tglx@linutronix.de>, Jason Cooper <jason@lakedaemon.net>,
Marc Zyngier <maz@kernel.org>,
"Nicolas Saenz Julienne" <nsaenzjulienne@suse.de>
X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3
X-CRM114-CacheID: sfid-20200210_015244_903439_EE700514
X-CRM114-Status: GOOD ( 18.64 )
X-Spam-Score: -0.7 (/)
X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary:
Content analysis details: (-0.7 points)
pts rule name description
---- ----------------------
--------------------------------------------------
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/,
low trust [83.223.95.204 listed in list.dnswl.org]
0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3)
[83.223.95.204 listed in wl.mailspike.net]
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
0.0 SPF_NONE SPF: sender does not publish an SPF Record
0.0 RCVD_IN_MSPIKE_WL Mailspike good senders
X-BeenThere: linux-arm-kernel@lists.infradead.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <linux-arm-kernel.lists.infradead.org>
List-Unsubscribe:
<http://lists.infradead.org/mailman/options/linux-arm-kernel>,
<mailto:linux-arm-kernel-request@lists.infradead.org?subject=unsubscribe>
List-Archive: <http://lists.infradead.org/pipermail/linux-arm-kernel/>
List-Post: <mailto:linux-arm-kernel@lists.infradead.org>
List-Help: <mailto:linux-arm-kernel-request@lists.infradead.org?subject=help>
List-Subscribe:
<http://lists.infradead.org/mailman/listinfo/linux-arm-kernel>,
<mailto:linux-arm-kernel-request@lists.infradead.org?subject=subscribe>
Cc: Florian Fainelli <f.fainelli@gmail.com>,
Kristina Brooks <notstina@gmail.com>, Scott Branden <sbranden@broadcom.com>,
Ray Jui <rjui@broadcom.com>, Serge Schneider <serge@raspberrypi.org>,
linux-kernel@vger.kernel.org, Phil Elwell <phil@raspberrypi.org>,
Matthias Brugger <mbrugger@suse.com>, bcm-kernel-feedback-list@broadcom.com,
linux-rpi-kernel@lists.infradead.org, Martin Sperl <kernel@martin.sperl.org>,
linux-arm-kernel@lists.infradead.org, Stefan Wahren <wahrenst@gmx.net>
MIME-Version: 1.0
Sender: "linux-arm-kernel" <linux-arm-kernel-bounces@lists.infradead.org>
Errors-To:
linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org
Customers of our "Revolution Pi" open source PLCs (which are based on
the Raspberry Pi) have reported random lockups as well as jittery eMMC,
UART and SPI latency. We were able to reproduce the lockups in our lab
and hooked up a JTAG debugger:
It turns out that the USB controller's interrupt is already enabled when
the kernel boots. All interrupts are disabled when the chip comes out
of power-on reset, according to the spec. So apparently the bootloader
enables the interrupt but neglects to disable it before handing over
control to the kernel.
The bootloader is a closed source blob provided by the Raspberry Pi
Foundation. Development of an alternative open source bootloader was
begun by Kristina Brooks but it's not fully functional yet. Usage of
the blob is thus without alternative for the time being.
The Raspberry Pi Foundation's downstream kernel has a performance-
optimized USB driver (which we use on our Revolution Pi products).
The driver takes advantage of the FIQ fast interrupt. Because the
regular USB interrupt was left enabled by the bootloader, both the
FIQ and the normal interrupt is enabled once the USB driver probes.
The spec has the following to say on simultaneously enabling the FIQ
and the normal interrupt of a peripheral:
"One interrupt source can be selected to be connected to the ARM FIQ
input. An interrupt which is selected as FIQ should have its normal
interrupt enable bit cleared. Otherwise a normal and an FIQ interrupt
will be fired at the same time. Not a good idea!"
^^^^^^^^^^^^^^^
https://www.raspberrypi.org/app/uploads/2012/02/BCM2835-ARM-Peripherals.pdf
page 110
On a multicore Raspberry Pi, the Foundation's kernel routes all normal
interrupts to CPU 0 and the FIQ to CPU 1. Because both the FIQ and the
normal interrupt is enabled, a USB interrupt causes CPU 0 to spin in
bcm2836_chained_handle_irq() until the FIQ on CPU 1 has cleared it.
Interrupts with a lower priority than USB are starved as long.
That explains the jittery eMMC, UART and SPI latency: On one occasion
I've seen CPU 0 blocked for no less than 2.9 msec. Basically,
everything not USB takes a performance hit: Whereas eMMC throughput
on a Compute Module 3 remains relatively constant at 23.5 MB/s with
this commit, it irregularly dips to 23.0 MB/s without this commit.
The lockups occur when CPU 0 receives a USB interrupt while holding a
lock which CPU 1 is trying to acquire while the FIQ is temporarily
disabled on CPU 1.
I've tested old releases of the Foundation's bootloader as far back as
1.20160202-1 and they all leave the USB interrupt enabled. Still older
releases fail to boot a contemporary kernel on a Compute Module 1 or 3,
which are the only Raspberry Pi variants I have at my disposal for
testing.
Fix by disabling IRQs left enabled by the bootloader. Although the
impact is most pronounced on the Foundation's downstream kernel,
it seems prudent to apply the fix to the upstream kernel to guard
against such mistakes in any present and future bootloader.
Signed-off-by: Lukas Wunner <lukas@wunner.de>
Cc: Serge Schneider <serge@raspberrypi.org>
Cc: Kristina Brooks <notstina@gmail.com>
Cc: stable@vger.kernel.org
Reviewed-by: Florian Fainelli <f.fainelli@gmail.com>
---
Changes since v1:
* Use "relaxed" MMIO accessors to avoid memory barriers (Marc)
* Use u32 instead of int for register access (Marc)
* Quiesce FIQ as well (Marc)
* Quiesce IRQs after mapping them for better readability
* Drop alternative approach from commit message (Marc)
Link to v1:
https://lore.kernel.org/lkml/988737dbbc4e499c2faaaa4e567ba3ed8deb9a89.1581089797.git.lukas@wunner.de/
drivers/irqchip/irq-bcm2835.c | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/drivers/irqchip/irq-bcm2835.c b/drivers/irqchip/irq-bcm2835.c
index 418245d31921..63539c88ac3a 100644
--- a/drivers/irqchip/irq-bcm2835.c
+++ b/drivers/irqchip/irq-bcm2835.c
@@ -61,6 +61,7 @@
| SHORTCUT1_MASK | SHORTCUT2_MASK)
#define REG_FIQ_CONTROL 0x0c
+#define REG_FIQ_ENABLE 0x80
#define NR_BANKS 3
#define IRQS_PER_BANK 32
@@ -135,6 +136,7 @@ static int __init armctrl_of_init(struct device_node *node,
{
void __iomem *base;
int irq, b, i;
+ u32 reg;
base = of_iomap(node, 0);
if (!base)
@@ -157,6 +159,19 @@ static int __init armctrl_of_init(struct device_node *node,
handle_level_irq);
irq_set_probe(irq);
}
+
+ reg = readl_relaxed(intc.enable[b]);
+ if (reg) {
+ writel_relaxed(reg, intc.disable[b]);
+ pr_err(FW_BUG "Bootloader left irq enabled: "
+ "bank %d irq %*pbl\n", b, IRQS_PER_BANK, ®);
+ }
+ }
+
+ reg = readl_relaxed(base + REG_FIQ_CONTROL);
+ if (reg & REG_FIQ_ENABLE) {
+ writel_relaxed(0, base + REG_FIQ_CONTROL);
+ pr_err(FW_BUG "Bootloader left fiq enabled\n");
}
if (is_2836) {
|
