tag:api.aurora.dev,2005:/historyAurora Status - Incident History2024-03-28T18:39:40ZAuroratag:api.aurora.dev,2005:Incident/178435102023-07-13T13:23:46Z2023-07-13T13:23:46ZTestnet RPC Writepath<p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>13:23</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>13</var>, <var data-var='time'>11:56</var> UTC</small><br><strong>Identified</strong> - We are performing a major systems update. During the update the writepath (sendrawTransactions, get_gasPrice) will not be available.</p>tag:api.aurora.dev,2005:Incident/172658332023-05-16T15:32:54Z2023-05-16T15:32:54ZEth_call and related RPC calls return error after on-chain update<p><small>May <var data-var='date'>16</var>, <var data-var='time'>15:32</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>16</var>, <var data-var='time'>13:18</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'>16</var>, <var data-var='time'>12:02</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:api.aurora.dev,2005:Incident/172317402023-05-12T23:00:55Z2023-05-12T23:14:16ZEthereum->NEAR/Aurora transfers are delayed because of Ethereum Mainnet attestation degradation<p><small>May <var data-var='date'>12</var>, <var data-var='time'>23:00</var> UTC</small><br><strong>Resolved</strong> - The bridge is fully synced an operational.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>19:58</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>18:45</var> UTC</small><br><strong>Identified</strong> - Ethereum Mainnet is currently experiencing issues with the attestation and finalization of the Beacon Chain. Because of that, Rainbow Bridge transfers for Ethereum->NEAR/Aurora directions are delayed.</p>tag:api.aurora.dev,2005:Incident/160950072023-02-12T18:52:52Z2023-02-12T18:52:52ZRainbow Bridge is temporarily paused<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>18:52</var> UTC</small><br><strong>Resolved</strong> - The Rainbow Bridge operations has been fully restored. One can execute new transfers as well as finalise the ones that has been started before the pause.<br />In-depth review of the issue to follow.</p><p><small>Feb <var data-var='date'>10</var>, <var data-var='time'>23:30</var> UTC</small><br><strong>Investigating</strong> - The Rainbow Bridge was paused as a precautionary measure. No funds are lost. We are working on getting the bridge back up quickly. Thank you for your patience.</p>tag:api.aurora.dev,2005:Incident/119454942022-10-14T14:53:50Z2022-10-14T14:53:50ZAurora Testnet block progression halted. No transactions possible<p><small>Oct <var data-var='date'>14</var>, <var data-var='time'>14:53</var> UTC</small><br><strong>Resolved</strong> - The Near testnet has stabilized and our nodes are in sync again.</p><p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>09:16</var> UTC</small><br><strong>Monitoring</strong> - The Aurora Testnet currently does not reliably produce blocks nor does it reliably accept transactions. The underlying issue is that the NEAR testnet - on which Aurora relies - is currently experiencing instabilities. There is no ETA for resolution.<br />Aurora is monitoring the situation. As soon as the NEAR testnet stabilizes, the Aurora testnet should become operational at once.</p>tag:api.aurora.dev,2005:Incident/111013202022-09-27T01:52:28Z2022-09-27T01:52:28ZRainbow Bridge transfers to/from Ethereum are paused due to the upcoming The Merge upgrade<p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>01:52</var> UTC</small><br><strong>Resolved</strong> - Rainbow Bridge transfers to/from Ehtereum are unpaused. See: https://twitter.com/auroraisnear/status/1574573522453639168</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>15:30</var> UTC</small><br><strong>Update</strong> - -</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>15:28</var> UTC</small><br><strong>Monitoring</strong> - -</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>15:28</var> UTC</small><br><strong>Investigating</strong> - The Rainbow Bridge and transfer finalization to Ethereum was paused on September 11 at 12:00 UTC. We will resume operations after The Merge, as soon as we feel certain about the safety and success of the upgrade. Users will be able to finalize any pending transfers after the Rainbow Bridge is unpaused. Read this blog article to learn what you need to know about this important event: https://aurora.dev/blog/the-ethereum-merge-and-what-to-expect-on-aurora</p>tag:api.aurora.dev,2005:Incident/110578412022-09-16T01:05:12Z2022-09-16T01:05:12ZWrong transaction count/nonce returned from RPC.<p><small>Sep <var data-var='date'>16</var>, <var data-var='time'>01:05</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>16</var>, <var data-var='time'>01:04</var> UTC</small><br><strong>Update</strong> - Data re-indexing has been completed.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>13:39</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>13:38</var> UTC</small><br><strong>Monitoring</strong> - Some small amount of EOAs will receive wrong transaction count from the RPC and not be able to make transactions using the derived nonce. <br />The root cause of this is an indexing error that has been identified and fixed, but re-indexing of the chain will take until Sept 9th.<br />Until then, users can manually set the nonce to a correct value for one transaction, after which the RPC will return the correct value going forward.</p>tag:api.aurora.dev,2005:Incident/95663562022-03-17T16:49:33Z2022-03-17T16:49:33ZAurora Mainnet and Bridge partial outage<p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>16:49</var> UTC</small><br><strong>Resolved</strong> - The NEAR Mainnet incident has been resolved so Aurora Mainnet and Rainbow Bridge are fully operational again.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>Identified</strong> - The issue was identified: there was a high load on NEAR Mainnet and its RPCs. Also, some NEAR validators were in the process of upgrading to nearcore 1.25.0. Due to these things NEAR Mainnet (and thus Aurora) was quite congested so some users might have experienced degraded performance.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>14:56</var> UTC</small><br><strong>Investigating</strong> - Due to NEAR Mainnet RPC partial outage, Aurora Mainnet RPC and Rainbow Bridge are having issues and could be temporarily unavailable. We are working hard with our colleagues at NEAR to restore service soonest.</p>tag:api.aurora.dev,2005:Incident/89992512022-01-04T12:58:53Z2022-01-04T12:58:53ZEthereum->NEAR/Aurora transactions are delayed<p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>12:58</var> UTC</small><br><strong>Resolved</strong> - The relayer is synced, all delayed Ethereum->NEAR/Aurora transactions are finalized.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>10:30</var> UTC</small><br><strong>Monitoring</strong> - We identified and resolved the issue, the relayer is catching up.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>Investigating</strong> - Ethereum->NEAR/Aurora transactions made after 08:30 UTC are delayed. We are investigating the issue.</p>tag:api.aurora.dev,2005:Incident/89225192021-12-23T19:15:51Z2021-12-23T19:55:02ZSome transactions are delayed<p><small>Dec <var data-var='date'>23</var>, <var data-var='time'>19:15</var> UTC</small><br><strong>Resolved</strong> - NEAR RPC issue was fixed by NEAR team. The bridge is back to full performance.</p><p><small>Dec <var data-var='date'>23</var>, <var data-var='time'>17:30</var> UTC</small><br><strong>Monitoring</strong> - The NEAR RPC is currently experiencing intermittent issues that are causing delays on transfers to NEAR and Aurora, and may affect connection to NEAR wallets. We do not have an ETA on when the NEAR team will have these issues resolved, but they are actively working on it.<br /><br />Transferred funds are safe, and transfers will be completed once these RPC issues are resolved. It is also safe to initiate transfers currently; just be aware their finalization may be delayed.</p>tag:api.aurora.dev,2005:Incident/88824312021-12-19T21:15:28Z2021-12-20T01:19:58ZRainbow Bridge upgrade to nearcore 1.23.0<p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>21:15</var> UTC</small><br><strong>Resolved</strong> - The bridge has been successfully upgraded to nearcore v1.23.0</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>14:51</var> UTC</small><br><strong>Identified</strong> - We are currently upgrading the bridge to nearcore 1.23.0. Until the upgrade is finished, all new transactions from NEAR/Aurora to Ethereum will be delayed.</p>tag:api.aurora.dev,2005:Incident/87947602021-12-17T02:07:16Z2021-12-17T02:24:46ZNEAR/Aurora->Ethereum: finalization of a set of transfers is delayed<p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>02:07</var> UTC</small><br><strong>Resolved</strong> - The transactions that were stuck are finalized on behalf of Aurora team.<br />If you think that your transaction is not finalized, please reach to our support team at support@aurora.dev</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>08:00</var> UTC</small><br><strong>Update</strong> - The fix is in the review process but a custom solution is being prepared to speed up the finalization.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>07:00</var> UTC</small><br><strong>Monitoring</strong> - The fix is in the review process.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>06:15</var> UTC</small><br><strong>Update</strong> - The fix is prepared by NEAR core team (https://github.com/near/nearcore/pull/5761) and is being reviewed.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>02:00</var> UTC</small><br><strong>Identified</strong> - The issue is identified with the collaboration with the NEAR Core team. There is an edge case that produced incorrect proofs for some specific epoch by NEAR RPC. The fix is being prepared.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>20:00</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate the issue.</p><p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>20:00</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate the issue.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>21:15</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>19:03</var> UTC</small><br><strong>Investigating</strong> - Finalization of transfers to Ethereum made between Tuesday, Dec 7, 10 am UTC and Dec 7, 9 pm UTC is delayed. We are investingating.</p>tag:api.aurora.dev,2005:Incident/88559762021-12-15T19:46:21Z2021-12-15T19:46:21ZMainnet degraded performance<p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>19:46</var> UTC</small><br><strong>Resolved</strong> - The incident has been resolved.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>16:53</var> UTC</small><br><strong>Identified</strong> - There is some performance degradation on the NEAR Protocol mainnet due to certain validators being offline. The network is still operational but transactions may take longer to finish as block speed has decreased.<br /><br />https://twitter.com/NEARChainStatus/status/1471160003780399109</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>16:06</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:api.aurora.dev,2005:Incident/85589192021-11-18T00:15:06Z2021-11-18T00:15:06ZDegraded performance on Mainnet<p><small>Nov <var data-var='date'>18</var>, <var data-var='time'>00:15</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>15:10</var> UTC</small><br><strong>Monitoring</strong> - NEAR identified that this incident was due to the protocol upgrade for sharding, whereby NEAR nodes were preoccupied doing state splits and therefore experienced sluggish RPC responses during a time with very high RPC load due to other ongoing events. The state-splitting has largely completed on all nodes now and RPC performance is back to nominal.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>13:49</var> UTC</small><br><strong>Investigating</strong> - Due to an ongoing incident with the NEAR Protocol, certain RPC operations on the Aurora Mainnet are currently experiencing significantly longer response latencies. We are investigating in collaboration with our colleagues at NEAR and will update here soon.</p>tag:api.aurora.dev,2005:Incident/83127122021-10-29T07:17:10Z2021-10-29T07:17:10ZMainnet: DDoS attack<p><small>Oct <var data-var='date'>29</var>, <var data-var='time'>07:17</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>03:42</var> UTC</small><br><strong>Monitoring</strong> - We've deployed more anti-abuse measures and are monitoring the results.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>19:10</var> UTC</small><br><strong>Investigating</strong> - We've identified large-scale abuse of the Mainnet RPC endpoints and are working to address the issue.</p>tag:api.aurora.dev,2005:Incident/80091692021-09-16T18:11:37Z2021-09-16T18:44:48ZTestnet outage<p><small>Sep <var data-var='date'>16</var>, <var data-var='time'>18:11</var> UTC</small><br><strong>Resolved</strong> - The NEAR Protocol incident has been resolved and the Aurora Testnet is now back online.<br /><br />More details at: https://twitter.com/NEARChainStatus/status/1438566159629144071</p><p><small>Sep <var data-var='date'>16</var>, <var data-var='time'>10:26</var> UTC</small><br><strong>Identified</strong> - Due to an outage of the NEAR Protocol Testnet, the Aurora Testnet is temporarily unavailable. We are working hard with our colleagues at NEAR to restore service soonest.</p>tag:api.aurora.dev,2005:Incident/72801122021-06-18T22:57:31Z2021-06-18T22:57:31ZTestnet: Intermittent transaction submission errors<p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>22:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>19:23</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>13:29</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>13:25</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>12:52</var> UTC</small><br><strong>Investigating</strong> - We are aware of and investigating intermittent error responses when submitting transactions to the Testnet RPC endpoint.</p>