Comments on reddit

On 28 August 2023 NATS, the UK’s air traffic control operator, suffered a
major technical incident. The BBC reports that more than 2000 flights were
cancelled and the cost has been estimated
at over £100 million GBP. The incident probably affected hundreds of thousands
of people.

The press initially reported the cause was a faulty flight plan: UK air traffic
control: inquiry into whether French error caused failure (The Times) and in
typical Mail Online reporting style: “Did blunder by French airline spark air
traffic control issues? Officials probe if a single badly filed travel plan
caused UK’s entire flight-control system to collapse in worst outage for a
decade – with 1,000 flights cancelled and chaos set to last DAYS”.

So what happened? These are notes on my reading of the incident report:

NATS Major Incident Preliminary Report
Flight Plan Reception Suite Automated (FPRSA-R) Sub-system Incident 28th August 2023
pdf.

NATS is a “public-private” company in the UK that is responsible for all of
the UK’s air traffic control:

Air Traffic Control (ATC) is the provision and operation of a safe system for
controlling and monitoring aircraft.
[..]
aircraft [..] are required to file a flight plan.
[..]
ATC ensures that aircraft are safely separated laterally and vertically.

What went wrong

The start of the sequence of events leading to the incident can be tracked
back to the point at which a flight plan was entered into the flight planning
system.

[Airlines] submit the plan into Eurocontrol’s Integrated Initial Flight Plan
Processing System (IFPS).
[..]

If the submitted flight plan is accepted by IFPS, i.e. it is compliant with
IFPS defined parameters […] this is sufficient for a flight to depart with
local ATC approval. The flight plan will be sent from IFPS to all relevant
ANSPs who need to manage the flight.
[..]

Within the NATS En-route operations at Swanwick Centre, the data is passed to
FPRSA-R. The FPRSA-R sub-system exists to convert the data received from IFPS
(in a format known as ATS Data Exchange Presentation, ADEXP) into a format
that is compatible with the UK National Airspace System (NAS). NAS is the
flight data processing system which contains all of the relevant airspace and
routings.
[..]

FPRSA-R has a primary and backup system monitored both by dedicated Control
and Monitoring (C&M) systems and also an aggregated central C&M system.
Further resilience is provided by NAS storing 4 hours of previously filed
flight data to allow the operation to continue in the event of the loss of
automatic processing of flight data.
[..]

In addition to the technical resilience provided by backup systems, and the 4
hours of stored flight data, there is operational contingency available to
allow safe service to continue. This is provided through the ability to input
flight data manually, directly into NAS using a manual input system.

To summarise:

• Flight plans are first submitted to a European-wide authority IFPS.
• If a plan is accepted, the flight is cleared for takeoff.
• NATS requires the flight plan be transferred to them at least 4 hours before
  the aircraft is due to enter UK airspace. This is supposed to give NATS a
  4-hour window to be able to fix any problems in processing flight plans.
• It seems that there is also probably some process which delays flight plans
  until close to the deadline (see below). This might be to avoid congesting
  the system with flight plans too early, or lots of plans that may later
  change. Still, this results in flight plans being received by NATS sometimes
  hours after the flight has taken off.

The NATS ATC System was operating normally.
[..]

[On] 28 August the airline submitted an ICAO4444 compliant flight plan into
Eurocontrol’s flight planning distribution system, IFPS.

ICAO stands for International Civil Aviation
Organization, a United Nations agency.
An ICAO4444 flight plan looks like this:

(FPL-TTT123-IS
-C550/L-SDE1E2GHIJ3J5RWZ/SB1D1
-KPWM1225
-N0440F310 SSOXS5 SSOXS DCT BUZRD
DCT SEY DCT HTO J174 ORF J121
CHS EESNT LUNNI1
-KJAX0214 KMCO
-PBN/A1L1B1C1D1O1T1 NAV/Z1 GBAS
DAT/1FANS2PDC SUR/260B RSP180
DOF/220501 REG/N123A SEL/BPAM
CODE/A05ED7)

Such messages are in a format that is meant to be read by machines, but also by
humans if necessary. The format is spec’d over many many pages of PDF, but is
roughly:

( FPL-ACID-Flt Rules Flight Type
- AC Type/Wake Cat-
Equip.&Capability
- Departure EOBT
- Speed Altitude [sp] Route
- Destination ETE [sp]
Alternate(s)
- Other Information )

The route part (in this example: N0440F310 SSOXS5 SSOXS DCT BUZRD DCT SEY DCT HTO J174 ORF J121 CHS EESNT LUNNI1) encodes an overall speed (here N0440
meaning 440 knots), an overall altitude (here F310 which means “Flight
Level 310” which means 310 × 100 ft (can also be in km)), and a sequence of
waypoints (referenced by name) separated by a description of how to get from the
previous waypoint to the next one, usually by referencing a “known route” by
name.

The flight plan was accepted by IFPS
[..]
the aircraft was cleared to depart at 04:00.
[..]

At 08:32 the flight plan was received by NATS’ FPRSA-R sub-system from
Eurocontrol’s IFPS system. This is consistent with the 4 hour rule mentioned
above. The purpose of the FPRSA-R software is to extract the UK portion of the
flight plan [..]

The flight plans delivered to FPRSA-R by IFPS are converted from [..] ICAO4444
to [..] ADEXP. ADEXP is a European-wide flight plan specification that
includes, amongst other data, additional geographical waypoints within the
European region specific to the route of a flight. For flights transiting
through UK airspace, rather than landing in the UK, this will include
additional waypoints outside of UK airspace required for its onward journey.
Following this conversion the ADEXP version of a flight plan includes, amongst
other aspects, the original ICAO4444 flight plan plus an additional list of
waypoints and other data.

ADEXP looks like this:

-TITLE IFPL
-BEGIN ADDR
  -FAC LIIRZEZX
  [...]
  -FAC LYZZEBXX
-END ADDR
-ADEP EDDF
-ADES LGTS
-ARCID KIM1
-ARCTYP B738
-CEQPT SDGRWY
-EOBD 170729
-EOBT 0715
-FILTIM 280832
-IFPLID AT00441635
-ORIGIN -NETWORKTYPE SITA -FAC FRAOXLH
-SEQPT C
-WKTRC M
-PBN B2
-REG DABHM
-SEL KMGJ
-SRC FPL
-TTLEET 0210
-RFL F330
-SPEED N0417
-FLTRUL I
-FLTTYP S
-ROUTE N0417F330 ANEKI8L ANEKI Y163 NATOR UN850 TRA UP131 RESIA Q333
BABAG UN606 PEVAL DCT PETAK UL607 PINDO UM603 EDASI
-ALTRNT1 LBSF
-BEGIN RTEPTS
  -PT -PTID EDDF -FL F004 -ETO 170729073000
  -PT -PTID RID -FL F100 -ETO 170729073404
  -PT -PTID ANEKI -FL F210 -ETO 170729073856
  -PT -PTID NEKLO -FL F214 -ETO 170729073911
  -PT -PTID BADLI -FL F248 -ETO 170729074118
  -PT -PTID PABLA -FL F279 -ETO 170729074348
  -PT -PTID HERBI -FL F308 -ETO 170729074624
  -PT -PTID NATOR -FL F330 -ETO 170729074911
  -PT -PTID TITIX -FL F330 -ETO 170729075154
  -PT -PTID TRA -FL F330 -ETO 170729075323
  -PT -PTID ARGAX -FL F330 -ETO 170729080055
  -PT -PTID RESIA -FL F330 -ETO 170729080731
  -PT -PTID UNTAD -FL F330 -ETO 170729081243
  -PT -PTID DIKEM -FL F330 -ETO 170729081627
  -PT -PTID ROKIB -FL F330 -ETO 170729081824
  -PT -PTID BABAG -FL F330 -ETO 170729082816
  -PT -PTID PEVAL -FL F330 -ETO 170729082916
  -PT -PTID PETAK -FL F330 -ETO 170729091754
  -PT -PTID PINDO -FL F330 -ETO 170729093322
  -PT -PTID EDASI -FL F165 -ETO 170729094347
  -PT -PTID LGTS -FL F000 -ETO 170729095713
-END RTEPTS
-SID ANEKI8L
-ATSRT Y163 ANEKI NATOR
-ATSRT UN850 NATOR TRA
-ATSRT UP131 TRA RESIA
-ATSRT Q333 RESIA BABAG
-ATSRT UN606 BABAG PEVAL
-DCT PEVAL PETAK
-ATSRT UL607 PETAK PINDO
n -ATSRT UM603 PINDO EDASI

You can read about ADEXP in the official spec.
Some notable fields (page 48):

Adexp Primary Field	Kind	Syntax	Semantic
route	b	'-' "ROUTE" {LIM_CHAR}	Complete ICAO Field 15 information containing speed, RFL and route (conforming to the syntax given in Ref. [3]).
rtepts	c	'-' "BEGIN" "RTEPTS" { pt I ad / vec} '-' "END" "RTEPTS"	List of route points. May also contain an aerodrome identifier.

In the example, we have the ICAO route:

-ROUTE N0417F330 ANEKI8L ANEKI Y163 NATOR UN850 TRA UP131 RESIA Q333 BABAG UN606 PEVAL DCT PETAK UL607 PINDO UM603 EDASI

(9 waypoints, 11 if you add the start and end waypoints)

Visually, routes look like:

(You can play around with flight plans at
flightplandatabase.com, a website for people
who like playing with flight simulators)

We can indent the “route” parts between the waypoints in the ICAO plan to make
things clearer:

N0417F330
  ANEKI8L 
  ANEKI 
    Y163
  NATOR
    UN850
  TRA
    UP131
  RESIA
    Q333
  BABAG
    UN606
  PEVAL
    DCT
  PETAK
    UL607
  PINDO
    UM603
  EDASI

E.g. ANEKI Y163 NATOR means “go from waypoint ANEKI to waypoint NATOR via
the route Y163“. DCT means “direct”.

The ADEXP format has more waypoints, along with more precision about altitude and estimated time at each waypoint:

-BEGIN RTEPTS
-PT -PTID EDDF  -FL F004 -ETO 170729073000
-PT -PTID RID   -FL F100 -ETO 170729073404
-PT -PTID ANEKI -FL F210 -ETO 170729073856
-PT -PTID NEKLO -FL F214 -ETO 170729073911
-PT -PTID BADLI -FL F248 -ETO 170729074118
-PT -PTID PABLA -FL F279 -ETO 170729074348
-PT -PTID HERBI -FL F308 -ETO 170729074624
-PT -PTID NATOR -FL F330 -ETO 170729074911
-PT -PTID TITIX -FL F330 -ETO 170729075154
-PT -PTID TRA   -FL F330 -ETO 170729075323
-PT -PTID ARGAX -FL F330 -ETO 170729080055
-PT -PTID RESIA -FL F330 -ETO 170729080731
-PT -PTID UNTAD -FL F330 -ETO 170729081243
-PT -PTID DIKEM -FL F330 -ETO 170729081627
-PT -PTID ROKIB -FL F330 -ETO 170729081824
-PT -PTID BABAG -FL F330 -ETO 170729082816
-PT -PTID PEVAL -FL F330 -ETO 170729082916
-PT -PTID PETAK -FL F330 -ETO 170729091754
-PT -PTID PINDO -FL F330 -ETO 170729093322
-PT -PTID EDASI -FL F165 -ETO 170729094347
-PT -PTID LGTS  -FL F000 -ETO 170729095713
-END RTEPTS

(21 waypoints)

We can mark which of the ADEXP waypoints have a corresponding waypoint in the ICAO plan (with a +) and which are implicit (with a |):

EDDF   |
RID    |
ANEKI  + 
NEKLO  |
BADLI  |
PABLA  |
HERBI  |
NATOR  +
TITIX  |
TRA    +
ARGAX  |
RESIA  +
UNTAD  |
DIKEM  |
ROKIB  |
BABAG  +
PEVAL  |
PETAK  +
PINDO  +
EDASI  +
LGTS   |

Note that the ICAO waypoints do not contain the start and end, since in the
original ICAO format these are specified in other fields (so it would waste
space to list them again in this list).

The ADEXP waypoints plan included two waypoints along its route that were
geographically distinct but which have the same designator.

This means there were two lines like:

-PT -PTID RESIA -FL F330 -ETO 170729080731

that had the same PTID string like "RESIA".

Although there has been work by ICAO and other bodies to eradicate non-unique
waypoint names there are duplicates around the world. In order to avoid
confusion latest standards state that such identical designators should be
geographically widely spaced. In this specific event, both of the waypoints
were located outside of the UK, one towards the beginning of the route and one
towards the end; approximately 4000 nautical miles apart.

4000 nautical miles is 7408km. Here is an arc of that length on the globe:

Once the ADEXP file had been received, the FPRSA-R software commenced
searching for the UK airspace entry point in the waypoint information per the
ADEXP flight plan, commencing at the first line of that waypoint data. FPRSA-R
was able to specifically identify the character string as it appeared in the
ADEXP flight plan text.

The programming style is very imperative. Furthermore, the description sounds
like the procedure is working directly on the textual representation of the
flight plan, rather than a data structure parsed from the text file. This would
be quite worrying, but it might also just be how it is explained.

Having correctly identified the entry point, the software moved on to search
for the exit point from UK airspace in the waypoint data.

Having completed those steps,

This part of the code identified entry and exit waypoints to UK airspace in
the list of ADEXP waypoints.

FPRSA-R then searches the ICAO4444 section of
the ADEXP file.

It seems at this point, having identified the entry and exit points from the
list of ADEXP waypoints, it will try to extract the UK portion of the flight plan from the ICAO route.

It initially searches from the beginning of that data, to find
the identified UK airspace entry point. This was successfully found. Next, it
searches backwards, from the end of that section, to find the UK airspace exit
point. This did not appear in that section of t